[illumos-Developer] Available MSI(-x) interrupt limit.

[Garrett D'Amore]

Of course hardware with more rings or channels would want more.  The point is that nearly *all* hardware can use at least 2.  And, there is much less gain unless you have more cores than you have interrupt channels.  (With MSI it doesn't matter much anyway... the system generally puts all MSI interrupts for a device on the same core.)

Please be aware that these defaults are diffferent for MSI-X.  MSI-X and the new code base is a totally different beast.

  -- Garrett D'Amore

On Jun 7, 2011, at 10:44 PM, "Dmitry Yusupov" <dmitry at nexenta.com> wrote:

> What about NICs with 2 or more HW rings? What about HBAs with more than
> 2 channels?
> And as we know PIO are very expensive in virtualization environments...
> hence we punishing everyone who is running Illumos in VMware/Xen/KVM
> environments.
> 
>> -----Original Message-----
>> From: Garrett D'Amore
>> Sent: Tuesday, June 07, 2011 11:36 AM
>> To: Dmitry Yusupov
>> Cc: Alexey Zaytsev; illumos-dev
>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>> 
>> Two is very sensible.  It comes from NICs.  One for transmit, one for
> receive.
>> :-)   It also is unlikely to be large enough to lead to interrupt
> resource
>> exhaustion.
>> 
>> In illumos, the default is higher.  Allocating more in older revs (say
>> NexentaStor 3.x) requires a property change.
>> 
>> The newer callback interfaces are part of a PSARC case and I don't
> remember
>> the number.  I think the ixgbe driver uses these interfaces though.
> Again,
>> these interfaces require a recent OpenSolaris or illumos build.
>> 
>>  -- Garrett D'Amore
>> 
>> On Jun 7, 2011, at 10:30 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
>> wrote:
>> 
>>> Who came up with 2? Why not 3 or 8? Why not 1 ? :-)
>>> 
>>> This is obviously wrong way to limit system and lock user into
> pre-MSI
>>> days... But do I understand you correctly that driver can allocate
>>> more than 2 if needed? Can you please post an example of snippet
> code
>>> in here?
>>> 
>>>> -----Original Message-----
>>>> From: Garrett D'Amore
>>>> Sent: Tuesday, June 07, 2011 10:53 AM
>>>> To: Dmitry Yusupov
>>>> Cc: Alexey Zaytsev; illumos-dev
>>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>>>> 
>>>> You misunderstand.  The default is 2 now.  If you want more, it
> take
>>>> extraordinary measures.
>>>> 
>>>> On illumos there is a more modern API that allows "cooperative"
>>>> device drivers to get more, with a promise that they will yield the
>>>> resources
>>> if other
>>>> devices need them later.
>>>> 
>>>> -- Garrett D'Amore
>>>> 
>>>> On Jun 7, 2011, at 8:59 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
>>>> wrote:
>>>> 
>>>>> This seems like not logical way to fix ill driver... :-) Instead
> of
>>>>> harming overall system performance we should fix the driver(s).
> And
>>>>> those vendors who ships Illumos can apply limiting settings if
> they
>>>>> care. I would guess that in most cases they will not use Illumos
>>> with
>>>>> devices/drivers which would consume all the interrupts and would
>>>>> rather prefer to focus the driver...
>>>>> 
>>>>> Why don't we reverse this default setting from max to min ?
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Garrett D'Amore [mailto:garrett at nexenta.com]
>>>>>> Sent: Tuesday, June 07, 2011 7:51 AM
>>>>>> To: Alexey Zaytsev
>>>>>> Cc: illumos-dev
>>>>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt
> limit.
>>>>>> 
>>>>>> Yes.    The limit comes from the fact that ill behaved
>>> devices/drivers
>>>>> could
>>>>>> easily consume all the interrupts on a system, preventing other
>>>>> devices from
>>>>>> attaching.  There was a recent rearchitecture of the interrupt
> code
>>>>>> specifically for MSI-X that offers far more interrupt vectors to
>>>>> devices, in
>>>>>> exchange for the device registering a callback indicating a
>>>>> willingness to
>>>>>> return interrupts back to the system when they become scarce.
>>>>>> 
>>>>>> There's also a legacy override somewhere that you can use for
>>>>>> specific devices, but it isn't documented.  I don't remember the
>>> name
>>>>>> of the
>>>>> override,
>>>>>> but it can grow the set up to 8.
>>>>>> 
>>>>>> -- Garrett D'Amore
>>>>>> 
>>>>>> On Jun 7, 2011, at 5:49 PM, "Alexey Zaytsev"
>>>>> <alexey.zaytsev at gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey.
>>>>>>> 
>>>>>>> Any ideas, what's the reason to have the 2 msi per device limit?
>>>>>>> It's been there from the start of the history.
>>>>>>> 
>>>>>>> https://github.com/illumos/illumos-
>>>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L296
>>>>>>> https://github.com/illumos/illumos-
>>>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L40
>>>>>>> https://github.com/illumos/illumos-
>>>>>> gate/blob/master/usr/src/uts/common/sys/ddi_intr_impl.h#L146
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Developer mailing list
>>>>>>> Developer at lists.illumos.org
>>>>>>> http://lists.illumos.org/m/listinfo/developer
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Developer mailing list
>>>>>> Developer at lists.illumos.org
>>>>>> http://lists.illumos.org/m/listinfo/developer
> 
> _______________________________________________
> Developer mailing list
> Developer at lists.illumos.org
> http://lists.illumos.org/m/listinfo/developer