[illumos-Developer] "preallocation" style taskqs

[Garrett D'Amore]

So I've been wanting to do this for a long time.... and I think that it
relates to certain bugs.

In illumos, taskqs are used *heavily* for scheduling, because they are
extremely convenient to work with.

However, one thing that often gets overlooked is that the implementation
in illumos requires hitting up a shared list, or possibly even doing a
kmem_cache_alloc() to get a task entry.

Many of the uses of taskqs could be made to eliminate this code path,
reducing contention on hot paths (e.g. taskq dispatch can be called up
to once per packet for networking!) by having support for preallocated
taskq_ent_t's that were stored within data structures.

For example, zio_t could use an inline taskq_ent_t, to handle
dispatching of zio_execute or zio_reexecute (although note that this
only makes sense where a zio can only be dispatched onto a single such
taskq.)

This could also offer a "guaranteed" form of taskq dispatch, since if
the data structure is already preallocated there is no chance of an
allocation failure, even in situations which might be challenging, like
interrupt context.

For folks trying to achieve extreme performance, I think this could win.

To this end, I'm proposing the following:

sizeof (taskq_ent_t) become a fixed part of the DDI.  (The actual
structure contents would be undocumented/opaque.)

creation of a new interface:

void ddi_taskq_dispatch_prealloc(ddi_taskq_t *, void (*func)(void *),
void *arg, uint_t flags, taskq_ent_t *);

Note that from a simple matter of ease of implementation, this interface
would *only* be available to non-DYNAMIC taskqs.  (With the potential to
allocate new threads, etc. dynamic taskqs don't seem well suited to
preallocation.)

Note that IIRC NetBSD taskqs work much like I'm suggesting here, except
that they are *always* prealloc style.  (That may have changed since I
last looked though.)

Thoughts?

	- Garrett