[illumos-Developer] "preallocation" style taskqs

[Garrett D'Amore]

On Fri, 2011-02-11 at 17:34 -0500, Eric Schrock wrote:
> Seems reasonable to me.  Out of curiosity, do you have any perf
> numbers?

No numbers... its just something that has bugged me many many times.
There are cases where I know of bugs that result in far too many
allocations due to dispatching to the same queue millions of jobs, and
other cases where I know we are hitting the backing kmem store pretty
hard.

Collecting numbers would be done on a case by case basis as we convert
individual callers to use the new interface.

> 
> Do we need it to be part of the DDI, or can we just make it
> consolidation private?  My concern is that if there is going to be any
> significant effort around taskq scalability (which there probably
> should be), we may want to change this structure.  And what does it
> even mean to be in the DDI in Illumos?  Do we have a 9F manpage
> repository that would be modified to include this?

I'd be happy to have a non-DDI version to start with.  The question
about what a DDI means are interesting... that presumes a functioning
ARC (there is developer-council, but that's mostly quiescent at
present), and real doc efforts, etc.


> On a side note, if you're looking at general taskq scalability, one
> major problem is the lack of CPU affinity for non-dynamic taskqs.

Yes, although its not entirely clear how to handle that generically --
but perhaps tagging a cpuid and dispatching to a per-cpu list first
would be a start.

	- Garrett
> 
> 
> Thanks,
> 
> 
> - Eric
> 
> On Fri, Feb 11, 2011 at 4:14 PM, Garrett D'Amore <garrett at damore.org>
> wrote:
>         So I've been wanting to do this for a long time.... and I
>         think that it
>         relates to certain bugs.
>         
>         In illumos, taskqs are used *heavily* for scheduling, because
>         they are
>         extremely convenient to work with.
>         
>         However, one thing that often gets overlooked is that the
>         implementation
>         in illumos requires hitting up a shared list, or possibly even
>         doing a
>         kmem_cache_alloc() to get a task entry.
>         
>         Many of the uses of taskqs could be made to eliminate this
>         code path,
>         reducing contention on hot paths (e.g. taskq dispatch can be
>         called up
>         to once per packet for networking!) by having support for
>         preallocated
>         taskq_ent_t's that were stored within data structures.
>         
>         For example, zio_t could use an inline taskq_ent_t, to handle
>         dispatching of zio_execute or zio_reexecute (although note
>         that this
>         only makes sense where a zio can only be dispatched onto a
>         single such
>         taskq.)
>         
>         This could also offer a "guaranteed" form of taskq dispatch,
>         since if
>         the data structure is already preallocated there is no chance
>         of an
>         allocation failure, even in situations which might be
>         challenging, like
>         interrupt context.
>         
>         For folks trying to achieve extreme performance, I think this
>         could win.
>         
>         To this end, I'm proposing the following:
>         
>         sizeof (taskq_ent_t) become a fixed part of the DDI.  (The
>         actual
>         structure contents would be undocumented/opaque.)
>         
>         creation of a new interface:
>         
>         void ddi_taskq_dispatch_prealloc(ddi_taskq_t *, void
>         (*func)(void *),
>         void *arg, uint_t flags, taskq_ent_t *);
>         
>         Note that from a simple matter of ease of implementation, this
>         interface
>         would *only* be available to non-DYNAMIC taskqs.  (With the
>         potential to
>         allocate new threads, etc. dynamic taskqs don't seem well
>         suited to
>         preallocation.)
>         
>         Note that IIRC NetBSD taskqs work much like I'm suggesting
>         here, except
>         that they are *always* prealloc style.  (That may have changed
>         since I
>         last looked though.)
>         
>         Thoughts?
>         
>                - Garrett
>         
>         
>         _______________________________________________
>         Developer mailing list
>         Developer at lists.illumos.org
>         http://lists.illumos.org/m/listinfo/developer
> 
> 
> 
> -- 
> Eric Schrock
> Delphix
> 
> 
> 275 Middlefield Road, Suite 50
> Menlo Park, CA 94025
> http://www.delphix.com
> 
>