[illumos-Developer] Available MSI(-x) interrupt limit.

[Garrett D'Amore]

illumos / OpenIndiana and future Nexenta 4 should be able to dynamically adjust these resources on demand, and also enable many more interrupts on systems with a newer APIX.  So this problem should be temporary.

  -- Garrett D'Amore

On Jun 8, 2011, at 7:59 PM, "Richard Elling" <richard.elling at richardelling.com> wrote:

> On Jun 7, 2011, at 11:30 AM, Dmitry Yusupov wrote:
> 
>> Who came up with 2? Why not 3 or 8? Why not 1 ? :-)
> 
> For NexentaOS and OpenSolaris, the default MSI-X (ddi_msix_alloc_limit)
> has been 8 for many years. This causes another problem in that 2x dual-port
> ixgbe uses 2x2x8 MSI-X interrupts and leaves none for other devices, such as
> igb, e1000g, etc. Newer igb can work around this by using MSI when MSI-X are
> not available. This causes still more problems for some hardware configurations.
> 
> For non-SPARC hardware, I see good results limiting ddi_msix_alloc_limit to 6.
> -- richard
> 
>> 
>> This is obviously wrong way to limit system and lock user into pre-MSI
>> days... But do I understand you correctly that driver can allocate more
>> than 2 if needed? Can you please post an example of snippet code in
>> here?
>> 
>>> -----Original Message-----
>>> From: Garrett D'Amore
>>> Sent: Tuesday, June 07, 2011 10:53 AM
>>> To: Dmitry Yusupov
>>> Cc: Alexey Zaytsev; illumos-dev
>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>>> 
>>> You misunderstand.  The default is 2 now.  If you want more, it take
>>> extraordinary measures.
>>> 
>>> On illumos there is a more modern API that allows "cooperative" device
>>> drivers to get more, with a promise that they will yield the resources
>> if other
>>> devices need them later.
>>> 
>>> -- Garrett D'Amore
>>> 
>>> On Jun 7, 2011, at 8:59 PM, "Dmitry Yusupov" <dmitry at nexenta.com>
>>> wrote:
>>> 
>>>> This seems like not logical way to fix ill driver... :-) Instead of
>>>> harming overall system performance we should fix the driver(s). And
>>>> those vendors who ships Illumos can apply limiting settings if they
>>>> care. I would guess that in most cases they will not use Illumos
>> with
>>>> devices/drivers which would consume all the interrupts and would
>>>> rather prefer to focus the driver...
>>>> 
>>>> Why don't we reverse this default setting from max to min ?
>>>> 
>>>>> -----Original Message-----
>>>>> From: Garrett D'Amore [mailto:garrett at nexenta.com]
>>>>> Sent: Tuesday, June 07, 2011 7:51 AM
>>>>> To: Alexey Zaytsev
>>>>> Cc: illumos-dev
>>>>> Subject: Re: [illumos-Developer] Available MSI(-x) interrupt limit.
>>>>> 
>>>>> Yes.    The limit comes from the fact that ill behaved
>> devices/drivers
>>>> could
>>>>> easily consume all the interrupts on a system, preventing other
>>>> devices from
>>>>> attaching.  There was a recent rearchitecture of the interrupt code
>>>>> specifically for MSI-X that offers far more interrupt vectors to
>>>> devices, in
>>>>> exchange for the device registering a callback indicating a
>>>> willingness to
>>>>> return interrupts back to the system when they become scarce.
>>>>> 
>>>>> There's also a legacy override somewhere that you can use for
>>>>> specific devices, but it isn't documented.  I don't remember the
>> name
>>>>> of the
>>>> override,
>>>>> but it can grow the set up to 8.
>>>>> 
>>>>> -- Garrett D'Amore
>>>>> 
>>>>> On Jun 7, 2011, at 5:49 PM, "Alexey Zaytsev"
>>>> <alexey.zaytsev at gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey.
>>>>>> 
>>>>>> Any ideas, what's the reason to have the 2 msi per device limit?
>>>>>> It's been there from the start of the history.
>>>>>> 
>>>>>> https://github.com/illumos/illumos-
>>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L296
>>>>>> https://github.com/illumos/illumos-
>>>>> gate/blob/master/usr/src/uts/common/os/ddi_intr_impl.c#L40
>>>>>> https://github.com/illumos/illumos-
>>>>> gate/blob/master/usr/src/uts/common/sys/ddi_intr_impl.h#L146
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Developer mailing list
>>>>>> Developer at lists.illumos.org
>>>>>> http://lists.illumos.org/m/listinfo/developer
>>>>> 
>>>>> _______________________________________________
>>>>> Developer mailing list
>>>>> Developer at lists.illumos.org
>>>>> http://lists.illumos.org/m/listinfo/developer
>> 
>> _______________________________________________
>> Developer mailing list
>> Developer at lists.illumos.org
>> http://lists.illumos.org/m/listinfo/developer
>