[illumos-Developer] Available MSI(-x) interrupt limit.

[Richard Elling]

On Jun 7, 2011, at 11:30 AM, Dmitry Yusupov wrote:

> Who came up with 2? Why not 3 or 8? Why not 1 ? :-)

For NexentaOS and OpenSolaris, the default MSI-X (ddi_msix_alloc_limit)
has been 8 for many years. This causes another problem in that 2x dual-port
ixgbe uses 2x2x8 MSI-X interrupts and leaves none for other devices, such as
igb, e1000g, etc. Newer igb can work around this by using MSI when MSI-X are
not available. This causes still more problems for some hardware configurations.

For non-SPARC hardware, I see good results limiting ddi_msix_alloc_limit to 6.
 -- richard

> 
> This is obviously wrong way to limit system and lock user into pre-MSI
> days... But do I understand you correctly that driver can allocate more
> than 2 if needed? Can you please post an example of snippet code in
> here?
> 
>> -----Original Message-----
>> From: Garrett D'Amore
>> Sent: Tuesday, June 07, 2011 10:53 AM
>> To: Dmitry Yusupov
>> Cc: Alexey Zaytsev; illumos-dev
>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>> 
>> You misunderstand.  The default is 2 now.  If you want more, it take
>> extraordinary measures.
>> 
>> On illumos there is a more modern API that allows "cooperative" device
>> drivers to get more, with a promise that they will yield the resources
> if other
>> devices need them later.
>> 
>>  -- Garrett D'Amore
>> 
>> On Jun 7, 2011, at 8:59 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
>> wrote:
>> 
>>> This seems like not logical way to fix ill driver... :-) Instead of
>>> harming overall system performance we should fix the driver(s). And
>>> those vendors who ships Illumos can apply limiting settings if they
>>> care. I would guess that in most cases they will not use Illumos
> with
>>> devices/drivers which would consume all the interrupts and would
>>> rather prefer to focus the driver...
>>> 
>>> Why don't we reverse this default setting from max to min ?
>>> 
>>>> -----Original Message-----
>>>> From: Garrett D'Amore [mailto:garrett at nexenta.com]
>>>> Sent: Tuesday, June 07, 2011 7:51 AM
>>>> To: Alexey Zaytsev
>>>> Cc: illumos-dev
>>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>>>> 
>>>> Yes.    The limit comes from the fact that ill behaved
> devices/drivers
>>> could
>>>> easily consume all the interrupts on a system, preventing other
>>> devices from
>>>> attaching.  There was a recent rearchitecture of the interrupt code
>>>> specifically for MSI-X that offers far more interrupt vectors to
>>> devices, in
>>>> exchange for the device registering a callback indicating a
>>> willingness to
>>>> return interrupts back to the system when they become scarce.
>>>> 
>>>> There's also a legacy override somewhere that you can use for
>>>> specific devices, but it isn't documented.  I don't remember the
> name
>>>> of the
>>> override,
>>>> but it can grow the set up to 8.
>>>> 
>>>> -- Garrett D'Amore
>>>> 
>>>> On Jun 7, 2011, at 5:49 PM, "Alexey Zaytsev"
>>> <alexey.zaytsev at gmail.com>
>>>> wrote:
>>>> 
>>>>> Hey.
>>>>> 
>>>>> Any ideas, what's the reason to have the 2 msi per device limit?
>>>>> It's been there from the start of the history.
>>>>> 
>>>>> https://github.com/illumos/illumos-
>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L296
>>>>> https://github.com/illumos/illumos-
>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L40
>>>>> https://github.com/illumos/illumos-
>>>> gate/blob/master/usr/src/uts/common/sys/ddi_intr_impl.h#L146
>>>>> 
>>>>> _______________________________________________
>>>>> Developer mailing list
>>>>> Developer at lists.illumos.org
>>>>> http://lists.illumos.org/m/listinfo/developer
>>>> 
>>>> _______________________________________________
>>>> Developer mailing list
>>>> Developer at lists.illumos.org
>>>> http://lists.illumos.org/m/listinfo/developer
> 
> _______________________________________________
> Developer mailing list
> Developer at lists.illumos.org
> http://lists.illumos.org/m/listinfo/developer