[illumos-Developer] "preallocation" style taskqs

[Garrett D'Amore]

On Sat, 2011-02-12 at 12:06 +1000, David Gwynne wrote:
> To my mind the performance is less interesting than making code more
> robust by avoiding the possibility that the taskq ent allocation can
> fail.

Me too... dealing with scheduling failure in interrupt context can be a
significant PITA.  That's the first time this came up in my brain, a few
years ago... I've just never been bothered enough to fix it.  (Then
there was that whole PSARC thing... :-p)

	- Garrett
> 
> Sent from my phone
> 
> ----- Reply message -----
> From: "Garrett D'Amore" <garrett at damore.org>
> Date: Sat, Feb 12, 2011 08:49
> Subject: [illumos-Developer] "preallocation" style taskqs
> To: "Eric Schrock" <eric.schrock at delphix.com>
> Cc: <developer at lists.illumos.org>
> 
> 
> On Fri, 2011-02-11 at 17:34 -0500, Eric Schrock wrote:
> > Seems reasonable to me.  Out of curiosity, do you have any perf
> > numbers?
> 
> No numbers... its just something that has bugged me many many times.
> There are cases where I know of bugs that result in far too many
> allocations due to dispatching to the same queue millions of jobs, and
> other cases where I know we are hitting the backing kmem store pretty
> hard.
> 
> Collecting numbers would be done on a case by case basis as we convert
> individual callers to use the new interface.
> 
> > 
> > Do we need it to be part of the DDI, or can we just make it
> > consolidation private?  My concern is that if there is going to be
> any
> > significant effort around taskq scalability (which there probably
> > should be), we may want to change this structure.  And what does it
> > even mean to be in the DDI in Illumos?  Do we have a 9F manpage
> > repository that would be modified to include this?
> 
> I'd be happy to have a non-DDI version to start with.  The question
> about what a DDI means are interesting... that presumes a functioning
> ARC (there is developer-council, but that's mostly quiescent at
> present), and real doc efforts, etc.
> 
> 
> > On a side note, if you're looking at general taskq scalability, one
> > major problem is the lack of CPU affinity for non-dynamic taskqs.
> 
> Yes, although its not entirely clear how to handle that generically --
> but perhaps tagging a cpuid and dispatching to a per-cpu list first
> would be a start.
> 
> - Garrett
> > 
> > 
> > Thanks,
> > 
> > 
> > - Eric
> > 
> > On Fri, Feb 11, 2011 at 4:14 PM, Garrett D'Amore
> <garrett at damore.org>
> > wrote:
> >         So I've been wanting to do this for a long time.... and I
> >         think that it
> >         relates to certain bugs.
> >         
> >         In illumos, taskqs are used *heavily* for scheduling,
> because
> >         they are
> >         extremely convenient to work with.
> >         
> >         However, one thing that often gets overlooked is that the
> >         implementation
> >         in illumos requires hitting up a shared list, or possibly
> even
> >         doing a
> >         kmem_cache_alloc() to get a task entry.
> >         
> >         Many of the uses of taskqs could be made to eliminate this
> >         code path,
> >         reducing contention on hot paths (e.g. taskq dispatch can be
> >         called up
> >         to once per packet for networking!) by having support for
> >         preallocated
> >         taskq_ent_t's that were stored within data structures.
> >         
> >         For example, zio_t could use an inline taskq_ent_t, to
> handle
> >         dispatching of zio_execute or zio_reexecute (although note
> >         that this
> >         only makes sense where a zio can only be dispatched onto a
> >         single such
> >         taskq.)
> >         
> >         This could also offer a "guaranteed" form of taskq dispatch,
> >         since if
> >         the data structure is already preallocated there is no
> chance
> >         of an
> >         allocation failure, even in situations which might be
> >         challenging, like
> >         interrupt context.
> >         
> >         For folks trying to achieve extreme performance, I think
> this
> >         could win.
> >         
> >         To this end, I'm proposing the following:
> >         
> >         sizeof (taskq_ent_t) become a fixed part of the DDI.  (The
> >         actual
> >         structure contents would be undocumented/opaque.)
> >         
> >         creation of a new interface:
> >         
> >         void ddi_taskq_dispatch_prealloc(ddi_taskq_t *, void
> >         (*func)(void *),
> >         void *arg, uint_t flags, taskq_ent_t *);
> >         
> >         Note that from a simple matter of ease of implementation,
> this
> >         interface
> >         would *only* be available to non-DYNAMIC taskqs.  (With the
> >         potential to
> >         allocate new threads, etc. dynamic taskqs don't seem well
> >         suited to
> >         preallocation.)
> >         
> >         Note that IIRC NetBSD taskqs work much like I'm suggesting
> >         here, except
> >         that they are *always* prealloc style.  (That may have
> changed
> >         since I
> >         last looked though.)
> >         
> >         Thoughts?
> >         
> >                - Garrett
> >         
> >         
> >         _______________________________________________
> >         Developer mailing list
> >         Developer at lists.illumos.org
> >         http://lists.illumos.org/m/listinfo/developer
> > 
> > 
> > 
> > -- 
> > Eric Schrock
> > Delphix
> > 
> > 
> > 275 Middlefield Road, Suite 50
> > Menlo Park, CA 94025
> > http://www.delphix.com
> > 
> > 
> 
> 
> 
> _______________________________________________
> Developer mailing list
> Developer at lists.illumos.org
> http://lists.illumos.org/m/listinfo/developer
> 
>