I built a C++ library for linux kernel bypass networking such as Solarflare ef_vi, DPDK, Verbs(mlx5) and also AF_XDP. Using this library I measured the RTT latency using a loopback on Solarflare X2522-Plus(ef_vi + DPDK), ConnectX-4 Lx(DPDK + Verbs + AF_XDP) and Intel XXV710-DA2(DPDK + AF_XDP). The benchmarks were run for 24 hours with median all the way upto P99.999 and max.
Solarflare X2522-Plus produced a RTT median of 1.866us and a max of 3.307us with 45B frames over 24 hours. This was the best result. Comprehensive results are all documented in the repository.
Over this project I did come across numerous driver(and NIC) specific quirks and bugs. These are all documented too.
Solarflare X2522-Plus is quite fast due to a Tx trick known as cut-through programmed IO(CTPIO). But I found this to be fragile, it has three modes: fast, default and in-order. Fast mode produces the lowest median but has a measurable percentage of failed CTPIO known as CTPIO “fallbacks” or “poison”. The CPTIO fallbacks significantly hurt the tail latency(P99 - P99.999). The CPTIO poison frames can be measured via “ethtool”. I used the in-order CPTIO mode which gives marginally higher medium but a tight tail latency.
Another strange issue I found was the PTP subsystem in the Solarflare X2522-Plus. So the default sfc driver runs a in-kernel IEEE 1588 subsystem with a PTP hardware clock, a dedicated PTP channel and management controller firmware activity. I found this to be a problem as a noticeable number of interrupts were mounting in “/proc/interrupts” and the stock driver with ef_vi produced a max of 7.679us which seemed very high.
Then I ran the same benchmark with DPDK which produced a max of 5.628us. DPDK unbinds the driver so the PTP subsystem does not work. Then I compiled out the PTP subsystem from the sfc driver and ran the 24 hour benchmark again and the max was reduced to only 3.307us.
There is a lot of more interesting stuff but I think the post is already getting long so I won’t mention any more. Everything is documented in the repository. It is also 100% open source with an Apache-2.0 license.
I built a C++ library for linux kernel bypass networking such as Solarflare ef_vi, DPDK, Verbs(mlx5) and also AF_XDP. Using this library I measured the RTT latency using a loopback on Solarflare X2522-Plus(ef_vi + DPDK), ConnectX-4 Lx(DPDK + Verbs + AF_XDP) and Intel XXV710-DA2(DPDK + AF_XDP). The benchmarks were run for 24 hours with median all the way upto P99.999 and max.
Solarflare X2522-Plus produced a RTT median of 1.866us and a max of 3.307us with 45B frames over 24 hours. This was the best result. Comprehensive results are all documented in the repository.
Over this project I did come across numerous driver(and NIC) specific quirks and bugs. These are all documented too.
Solarflare X2522-Plus is quite fast due to a Tx trick known as cut-through programmed IO(CTPIO). But I found this to be fragile, it has three modes: fast, default and in-order. Fast mode produces the lowest median but has a measurable percentage of failed CTPIO known as CTPIO “fallbacks” or “poison”. The CPTIO fallbacks significantly hurt the tail latency(P99 - P99.999). The CPTIO poison frames can be measured via “ethtool”. I used the in-order CPTIO mode which gives marginally higher medium but a tight tail latency.
Another strange issue I found was the PTP subsystem in the Solarflare X2522-Plus. So the default sfc driver runs a in-kernel IEEE 1588 subsystem with a PTP hardware clock, a dedicated PTP channel and management controller firmware activity. I found this to be a problem as a noticeable number of interrupts were mounting in “/proc/interrupts” and the stock driver with ef_vi produced a max of 7.679us which seemed very high. Then I ran the same benchmark with DPDK which produced a max of 5.628us. DPDK unbinds the driver so the PTP subsystem does not work. Then I compiled out the PTP subsystem from the sfc driver and ran the 24 hour benchmark again and the max was reduced to only 3.307us.
There is a lot of more interesting stuff but I think the post is already getting long so I won’t mention any more. Everything is documented in the repository. It is also 100% open source with an Apache-2.0 license.