[illumos-Developer] "preallocation" style taskqs

[David Gwynne]

this makes perfect sense to me. i did this exact same thing in openbsd 18 months ago for workqs.

dlg

On 12/02/2011, at 7:14 AM, Garrett D'Amore wrote:

> So I've been wanting to do this for a long time.... and I think that it
> relates to certain bugs.
> 
> In illumos, taskqs are used *heavily* for scheduling, because they are
> extremely convenient to work with.
> 
> However, one thing that often gets overlooked is that the implementation
> in illumos requires hitting up a shared list, or possibly even doing a
> kmem_cache_alloc() to get a task entry.
> 
> Many of the uses of taskqs could be made to eliminate this code path,
> reducing contention on hot paths (e.g. taskq dispatch can be called up
> to once per packet for networking!) by having support for preallocated
> taskq_ent_t's that were stored within data structures.
> 
> For example, zio_t could use an inline taskq_ent_t, to handle
> dispatching of zio_execute or zio_reexecute (although note that this
> only makes sense where a zio can only be dispatched onto a single such
> taskq.)
> 
> This could also offer a "guaranteed" form of taskq dispatch, since if
> the data structure is already preallocated there is no chance of an
> allocation failure, even in situations which might be challenging, like
> interrupt context.
> 
> For folks trying to achieve extreme performance, I think this could win.
> 
> To this end, I'm proposing the following:
> 
> sizeof (taskq_ent_t) become a fixed part of the DDI.  (The actual
> structure contents would be undocumented/opaque.)
> 
> creation of a new interface:
> 
> void ddi_taskq_dispatch_prealloc(ddi_taskq_t *, void (*func)(void *),
> void *arg, uint_t flags, taskq_ent_t *);
> 
> Note that from a simple matter of ease of implementation, this interface
> would *only* be available to non-DYNAMIC taskqs.  (With the potential to
> allocate new threads, etc. dynamic taskqs don't seem well suited to
> preallocation.)
> 
> Note that IIRC NetBSD taskqs work much like I'm suggesting here, except
> that they are *always* prealloc style.  (That may have changed since I
> last looked though.)
> 
> Thoughts?
> 
> 	- Garrett
> 
> 
> _______________________________________________
> Developer mailing list
> Developer at lists.illumos.org
> http://lists.illumos.org/m/listinfo/developer