[illumos-Developer] Available MSI(-x) interrupt limit.

[Dmitry Yusupov]

What about NICs with 2 or more HW rings? What about HBAs with more than
2 channels?
And as we know PIO are very expensive in virtualization environments...
hence we punishing everyone who is running Illumos in VMware/Xen/KVM
environments.

> -----Original Message-----
> From: Garrett D'Amore
> Sent: Tuesday, June 07, 2011 11:36 AM
> To: Dmitry Yusupov
> Cc: Alexey Zaytsev; illumos-dev
> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
> 
> Two is very sensible.  It comes from NICs.  One for transmit, one for
receive.
> :-)   It also is unlikely to be large enough to lead to interrupt
resource
> exhaustion.
> 
> In illumos, the default is higher.  Allocating more in older revs (say
> NexentaStor 3.x) requires a property change.
> 
> The newer callback interfaces are part of a PSARC case and I don't
remember
> the number.  I think the ixgbe driver uses these interfaces though.
Again,
> these interfaces require a recent OpenSolaris or illumos build.
> 
>   -- Garrett D'Amore
> 
> On Jun 7, 2011, at 10:30 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
> wrote:
> 
> > Who came up with 2? Why not 3 or 8? Why not 1 ? :-)
> >
> > This is obviously wrong way to limit system and lock user into
pre-MSI
> > days... But do I understand you correctly that driver can allocate
> > more than 2 if needed? Can you please post an example of snippet
code
> > in here?
> >
> >> -----Original Message-----
> >> From: Garrett D'Amore
> >> Sent: Tuesday, June 07, 2011 10:53 AM
> >> To: Dmitry Yusupov
> >> Cc: Alexey Zaytsev; illumos-dev
> >> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
> >>
> >> You misunderstand.  The default is 2 now.  If you want more, it
take
> >> extraordinary measures.
> >>
> >> On illumos there is a more modern API that allows "cooperative"
> >> device drivers to get more, with a promise that they will yield the
> >> resources
> > if other
> >> devices need them later.
> >>
> >>  -- Garrett D'Amore
> >>
> >> On Jun 7, 2011, at 8:59 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
> >> wrote:
> >>
> >>> This seems like not logical way to fix ill driver... :-) Instead
of
> >>> harming overall system performance we should fix the driver(s).
And
> >>> those vendors who ships Illumos can apply limiting settings if
they
> >>> care. I would guess that in most cases they will not use Illumos
> > with
> >>> devices/drivers which would consume all the interrupts and would
> >>> rather prefer to focus the driver...
> >>>
> >>> Why don't we reverse this default setting from max to min ?
> >>>
> >>>> -----Original Message-----
> >>>> From: Garrett D'Amore [mailto:garrett at nexenta.com]
> >>>> Sent: Tuesday, June 07, 2011 7:51 AM
> >>>> To: Alexey Zaytsev
> >>>> Cc: illumos-dev
> >>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt
limit.
> >>>>
> >>>> Yes.    The limit comes from the fact that ill behaved
> > devices/drivers
> >>> could
> >>>> easily consume all the interrupts on a system, preventing other
> >>> devices from
> >>>> attaching.  There was a recent rearchitecture of the interrupt
code
> >>>> specifically for MSI-X that offers far more interrupt vectors to
> >>> devices, in
> >>>> exchange for the device registering a callback indicating a
> >>> willingness to
> >>>> return interrupts back to the system when they become scarce.
> >>>>
> >>>> There's also a legacy override somewhere that you can use for
> >>>> specific devices, but it isn't documented.  I don't remember the
> > name
> >>>> of the
> >>> override,
> >>>> but it can grow the set up to 8.
> >>>>
> >>>> -- Garrett D'Amore
> >>>>
> >>>> On Jun 7, 2011, at 5:49 PM, "Alexey Zaytsev"
> >>> <alexey.zaytsev at gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hey.
> >>>>>
> >>>>> Any ideas, what's the reason to have the 2 msi per device limit?
> >>>>> It's been there from the start of the history.
> >>>>>
> >>>>> https://github.com/illumos/illumos-
> >>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L296
> >>>>> https://github.com/illumos/illumos-
> >>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L40
> >>>>> https://github.com/illumos/illumos-
> >>>> gate/blob/master/usr/src/uts/common/sys/ddi_intr_impl.h#L146
> >>>>>
> >>>>> _______________________________________________
> >>>>> Developer mailing list
> >>>>> Developer at lists.illumos.org
> >>>>> http://lists.illumos.org/m/listinfo/developer
> >>>>
> >>>> _______________________________________________
> >>>> Developer mailing list
> >>>> Developer at lists.illumos.org
> >>>> http://lists.illumos.org/m/listinfo/developer