Low latency and interrupt mitigation

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Low latency and interrupt mitigation

Dan Shechter-2
Hi All,

I have a very latency sensitive application. I need to move packets
from one interface to another

I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB

My latency sensitive application reads packets from one em interface
using libpcap and sends packet to another em interface using udp
socket.

Does em has interrupt mitigation enabled?
If interrupt mitigation is enabled would disabling it will reduce latency?
How can I disable the interrupt mitigation for em NICs?
What would change in 5.3?

Best regards,
Dan

Reply | Threaded
Open this post in threaded view
|

Re: Low latency and interrupt mitigation

Mike Belopuhov-5
On 11 February 2013 12:19, Dan Shechter <[hidden email]> wrote:

> Hi All,
>
> I have a very latency sensitive application. I need to move packets
> from one interface to another
>
> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>
> My latency sensitive application reads packets from one em interface
> using libpcap and sends packet to another em interface using udp
> socket.
>
> Does em has interrupt mitigation enabled?

It does.

> If interrupt mitigation is enabled would disabling it will reduce latency?

It would.

> How can I disable the interrupt mitigation for em NICs?

I suggest you try increasing MAX_INTS_PER_SEC define by the factor
or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
can try commenting out this line:

E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);

> What would change in 5.3?
>

Nothing regarding interrupt moderation.

> Best regards,
> Dan
>

Cheers,
Mike

Reply | Threaded
Open this post in threaded view
|

Re: Low latency and interrupt mitigation

Dan Shechter-2
Thanks,

Are there any other tips to reduce latency?


Best regards,
Dan


On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov <[hidden email]> wrote:

> On 11 February 2013 12:19, Dan Shechter <[hidden email]> wrote:
>> Hi All,
>>
>> I have a very latency sensitive application. I need to move packets
>> from one interface to another
>>
>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>
>> My latency sensitive application reads packets from one em interface
>> using libpcap and sends packet to another em interface using udp
>> socket.
>>
>> Does em has interrupt mitigation enabled?
>
> It does.
>
>> If interrupt mitigation is enabled would disabling it will reduce latency?
>
> It would.
>
>> How can I disable the interrupt mitigation for em NICs?
>
> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
> can try commenting out this line:
>
> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>
>> What would change in 5.3?
>>
>
> Nothing regarding interrupt moderation.
>
>> Best regards,
>> Dan
>>
>
> Cheers,
> Mike

Reply | Threaded
Open this post in threaded view
|

Re: Low latency and interrupt mitigation

Mike Belopuhov-5
On 11 February 2013 12:53, Dan Shechter <[hidden email]> wrote:
> Thanks,
>
> Are there any other tips to reduce latency?
>

Using pcap means copying packets, so I'd say you want to
put your code into the kernel to avoid copying and maybe
queueing as well, but this is not something that can be
trivially explained and is certainly not something that
OpenBSD supports or would want to support out of the box.

FreeBSD netmap interface would be helpful in your situation
but is not currently supported by OpenBSD.

Cheers,
Mike

>
> Best regards,
> Dan
>
>
> On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov <[hidden email]> wrote:
>> On 11 February 2013 12:19, Dan Shechter <[hidden email]> wrote:
>>> Hi All,
>>>
>>> I have a very latency sensitive application. I need to move packets
>>> from one interface to another
>>>
>>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>>
>>> My latency sensitive application reads packets from one em interface
>>> using libpcap and sends packet to another em interface using udp
>>> socket.
>>>
>>> Does em has interrupt mitigation enabled?
>>
>> It does.
>>
>>> If interrupt mitigation is enabled would disabling it will reduce latency?
>>
>> It would.
>>
>>> How can I disable the interrupt mitigation for em NICs?
>>
>> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
>> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
>> can try commenting out this line:
>>
>> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>>
>>> What would change in 5.3?
>>>
>>
>> Nothing regarding interrupt moderation.
>>
>>> Best regards,
>>> Dan
>>>
>>
>> Cheers,
>> Mike

Reply | Threaded
Open this post in threaded view
|

Re: Low latency and interrupt mitigation

Dan Shechter-2
Thanks.

If interrupt mitigation disabling won't be enough I will have do
decide: freebs with netmap or openbsd with if_ethersubr.c
modification.

netmap looks much easier to code. No need to do the mbuf dance (or I
hope that with netmap there would be no such need).

BTW, would HW TX IP/UDP checksum reduce the latency?
Best regards,
Dan


On Mon, Feb 11, 2013 at 2:11 PM, Mike Belopuhov <[hidden email]> wrote:

> On 11 February 2013 12:53, Dan Shechter <[hidden email]> wrote:
>> Thanks,
>>
>> Are there any other tips to reduce latency?
>>
>
> Using pcap means copying packets, so I'd say you want to
> put your code into the kernel to avoid copying and maybe
> queueing as well, but this is not something that can be
> trivially explained and is certainly not something that
> OpenBSD supports or would want to support out of the box.
>
> FreeBSD netmap interface would be helpful in your situation
> but is not currently supported by OpenBSD.
>
> Cheers,
> Mike
>
>>
>> Best regards,
>> Dan
>>
>>
>> On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov <[hidden email]> wrote:
>>> On 11 February 2013 12:19, Dan Shechter <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I have a very latency sensitive application. I need to move packets
>>>> from one interface to another
>>>>
>>>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>>>
>>>> My latency sensitive application reads packets from one em interface
>>>> using libpcap and sends packet to another em interface using udp
>>>> socket.
>>>>
>>>> Does em has interrupt mitigation enabled?
>>>
>>> It does.
>>>
>>>> If interrupt mitigation is enabled would disabling it will reduce latency?
>>>
>>> It would.
>>>
>>>> How can I disable the interrupt mitigation for em NICs?
>>>
>>> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
>>> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
>>> can try commenting out this line:
>>>
>>> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>>>
>>>> What would change in 5.3?
>>>>
>>>
>>> Nothing regarding interrupt moderation.
>>>
>>>> Best regards,
>>>> Dan
>>>>
>>>
>>> Cheers,
>>> Mike

Reply | Threaded
Open this post in threaded view
|

Re: Low latency and interrupt mitigation

Mike Belopuhov-5
On 11 February 2013 13:22, Dan Shechter <[hidden email]> wrote:

> Thanks.
>
> If interrupt mitigation disabling won't be enough I will have do
> decide: freebs with netmap or openbsd with if_ethersubr.c
> modification.
>
> netmap looks much easier to code. No need to do the mbuf dance (or I
> hope that with netmap there would be no such need).
>
> BTW, would HW TX IP/UDP checksum reduce the latency?

I don't know exactly, but compared to the amount of work
done by bpf/pcap and syscalls (at least write), I'd say
it's negligible.

> Best regards,
> Dan
>
>
> On Mon, Feb 11, 2013 at 2:11 PM, Mike Belopuhov <[hidden email]> wrote:
>> On 11 February 2013 12:53, Dan Shechter <[hidden email]> wrote:
>>> Thanks,
>>>
>>> Are there any other tips to reduce latency?
>>>
>>
>> Using pcap means copying packets, so I'd say you want to
>> put your code into the kernel to avoid copying and maybe
>> queueing as well, but this is not something that can be
>> trivially explained and is certainly not something that
>> OpenBSD supports or would want to support out of the box.
>>
>> FreeBSD netmap interface would be helpful in your situation
>> but is not currently supported by OpenBSD.
>>
>> Cheers,
>> Mike
>>
>>>
>>> Best regards,
>>> Dan
>>>
>>>
>>> On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov <[hidden email]> wrote:
>>>> On 11 February 2013 12:19, Dan Shechter <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> I have a very latency sensitive application. I need to move packets
>>>>> from one interface to another
>>>>>
>>>>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>>>>
>>>>> My latency sensitive application reads packets from one em interface
>>>>> using libpcap and sends packet to another em interface using udp
>>>>> socket.
>>>>>
>>>>> Does em has interrupt mitigation enabled?
>>>>
>>>> It does.
>>>>
>>>>> If interrupt mitigation is enabled would disabling it will reduce latency?
>>>>
>>>> It would.
>>>>
>>>>> How can I disable the interrupt mitigation for em NICs?
>>>>
>>>> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
>>>> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
>>>> can try commenting out this line:
>>>>
>>>> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>>>>
>>>>> What would change in 5.3?
>>>>>
>>>>
>>>> Nothing regarding interrupt moderation.
>>>>
>>>>> Best regards,
>>>>> Dan
>>>>>
>>>>
>>>> Cheers,
>>>> Mike