[illumos-Developer] Available MSI(-x) interrupt limit.

[Dmitry Yusupov]

Great news!

> -----Original Message-----
> From: Garrett D'Amore
> Sent: Wednesday, June 08, 2011 9:03 AM
> To: Richard Elling
> Cc: Dmitry Yusupov; illumos-dev
> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
> 
> illumos / OpenIndiana and future Nexenta 4 should be able to
dynamically
> adjust these resources on demand, and also enable many more interrupts
on
> systems with a newer APIX.  So this problem should be temporary.
> 
>   -- Garrett D'Amore
> 
> On Jun 8, 2011, at 7:59 PM, "Richard Elling"
> <richard.elling at richardelling.com> wrote:
> 
> > On Jun 7, 2011, at 11:30 AM, Dmitry Yusupov wrote:
> >
> >> Who came up with 2? Why not 3 or 8? Why not 1 ? :-)
> >
> > For NexentaOS and OpenSolaris, the default MSI-X
> > (ddi_msix_alloc_limit) has been 8 for many years. This causes
another
> > problem in that 2x dual-port ixgbe uses 2x2x8 MSI-X interrupts and
> > leaves none for other devices, such as igb, e1000g, etc. Newer igb
can
> > work around this by using MSI when MSI-X are not available. This
causes
> still more problems for some hardware configurations.
> >
> > For non-SPARC hardware, I see good results limiting
ddi_msix_alloc_limit
> to 6.
> > -- richard
> >
> >>
> >> This is obviously wrong way to limit system and lock user into
> >> pre-MSI days... But do I understand you correctly that driver can
> >> allocate more than 2 if needed? Can you please post an example of
> >> snippet code in here?
> >>
> >>> -----Original Message-----
> >>> From: Garrett D'Amore
> >>> Sent: Tuesday, June 07, 2011 10:53 AM
> >>> To: Dmitry Yusupov
> >>> Cc: Alexey Zaytsev; illumos-dev
> >>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt
limit.
> >>>
> >>> You misunderstand.  The default is 2 now.  If you want more, it
take
> >>> extraordinary measures.
> >>>
> >>> On illumos there is a more modern API that allows "cooperative"
> >>> device drivers to get more, with a promise that they will yield
the
> >>> resources
> >> if other
> >>> devices need them later.
> >>>
> >>> -- Garrett D'Amore
> >>>
> >>> On Jun 7, 2011, at 8:59 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
> >>> wrote:
> >>>
> >>>> This seems like not logical way to fix ill driver... :-) Instead
of
> >>>> harming overall system performance we should fix the driver(s).
And
> >>>> those vendors who ships Illumos can apply limiting settings if
they
> >>>> care. I would guess that in most cases they will not use Illumos
> >> with
> >>>> devices/drivers which would consume all the interrupts and would
> >>>> rather prefer to focus the driver...
> >>>>
> >>>> Why don't we reverse this default setting from max to min ?
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Garrett D'Amore [mailto:garrett at nexenta.com]
> >>>>> Sent: Tuesday, June 07, 2011 7:51 AM
> >>>>> To: Alexey Zaytsev
> >>>>> Cc: illumos-dev
> >>>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt
limit.
> >>>>>
> >>>>> Yes.    The limit comes from the fact that ill behaved
> >> devices/drivers
> >>>> could
> >>>>> easily consume all the interrupts on a system, preventing other
> >>>> devices from
> >>>>> attaching.  There was a recent rearchitecture of the interrupt
> >>>>> code specifically for MSI-X that offers far more interrupt
vectors
> >>>>> to
> >>>> devices, in
> >>>>> exchange for the device registering a callback indicating a
> >>>> willingness to
> >>>>> return interrupts back to the system when they become scarce.
> >>>>>
> >>>>> There's also a legacy override somewhere that you can use for
> >>>>> specific devices, but it isn't documented.  I don't remember the
> >> name
> >>>>> of the
> >>>> override,
> >>>>> but it can grow the set up to 8.
> >>>>>
> >>>>> -- Garrett D'Amore
> >>>>>
> >>>>> On Jun 7, 2011, at 5:49 PM, "Alexey Zaytsev"
> >>>> <alexey.zaytsev at gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hey.
> >>>>>>
> >>>>>> Any ideas, what's the reason to have the 2 msi per device
limit?
> >>>>>> It's been there from the start of the history.
> >>>>>>
> >>>>>> https://github.com/illumos/illumos-
> >>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L296
> >>>>>> https://github.com/illumos/illumos-
> >>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L40
> >>>>>> https://github.com/illumos/illumos-
> >>>>> gate/blob/master/usr/src/uts/common/sys/ddi_intr_impl.h#L146
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Developer mailing list
> >>>>>> Developer at lists.illumos.org
> >>>>>> http://lists.illumos.org/m/listinfo/developer
> >>>>>
> >>>>> _______________________________________________
> >>>>> Developer mailing list
> >>>>> Developer at lists.illumos.org
> >>>>> http://lists.illumos.org/m/listinfo/developer
> >>
> >> _______________________________________________
> >> Developer mailing list
> >> Developer at lists.illumos.org
> >> http://lists.illumos.org/m/listinfo/developer
> >