【容器平台】Kubernetes网络解决方案的比较

Kubernetes要求集群中的每个容器都具有唯一的可路由的IP。 Kubernetes本身不分配IP，将任务交给第三方解决方案。

在这项研究中，我们的目标是找到具有最低延迟，最高吞吐量和最低安装成本的解决方案。 由于我们的负载对延迟敏感，因此我们的目的是在相对高的网络利用率下测量高百分比的延迟。 我们特别关注性能低于最大负载的30-50％，因为我们认为这最好代表了非超载系统的最常见用例。

竞争对手

Docker与--net =主机

这是我们的参考设置。 所有其他的竞争对手都与这种设置进行比

-net = host选项意味着容器继承其主机的IP，即不涉及网络容器化。

先验，没有网络集装箱比任何网络容器化表现更好; 这就是为什么我们使用这个设置作为参考。

Flannel

Flannel是由CoreOS项目维护的虚拟网络解决方案。 这是一个经过充分测试，生产就绪的解决方案，所以它的安装成本最低。

当您将一台带Flannel的机器添加到集群时，Flannel做三件事情：

使用etcd为新机器分配一个子网

在机器上创建虚拟桥接器（称为docker0桥）

设置数据包转发后端：

AWS-VPC

在Amazon AWS实例表中注册计算机子网。 此表中的记录数限制为50，即如果使用aws-vpc Flannel，则集群中的计算机数不能超过50台。 而且，这个后端只适用于Amazon的AWS。

主机-GW

通过远程计算机IP创建到子网的IP路由。 运行Flannel的主机之间需要直接的二层连接。

VXLAN

创建一个虚拟的VXLAN接口。

由于flannel使用网桥接口来转发数据包，因此每个数据包在从一个容器移动到另一个容器时会通过两个网络堆栈。

IPvlan

在Linux内核中，IPvlan是驱动程序，使您可以创建具有唯一IP的虚拟接口，而无需使用网桥接口。

要将IP分配给具有IPvlan的容器，您必须：

创建一个没有网络接口的容器

在默认网络命名空间中创建一个ipvlan接口

将接口移动到容器的网络名称空间

IPvLAN是一个相对较新的解决方案，所以没有现成的工具来自动化这个过程。 这使得用许多机器和容器来部署IPvlan很困难，即安装成本高。

但是，IPvLAN不需要网桥接口，直接将数据包从网卡转发到虚拟接口，所以我们期望它的性能比Flannel更好。

负载测试方案

对于每个竞争对手，我们运行这些步骤：

在两台物理机上建立网络

在一台机器上的一个容器中运行tcpkali，让我们以恒定的速率发送请求

在另一台机器的容器中运行Nginx，让它用固定大小的文件进行响应

捕获系统度量和tcpkali结果

我们以每秒50,000到450,000个请求（RPS）的请求速率运行基准测试。

在每个请求中，Nginx都以一个固定大小的静态文件进行响应：350B（100B内容，250B头）或4KB。

结果

IPvlan显示最低延迟和最高吞吐量。带有host-gw和aws-vpc的法兰绒紧随其后，但是host-gw在最大负荷下显示效果更好。

与vxlan法兰绒显示在所有测试中最差的结果。但是，我们怀疑其99.999％的分数是由于一个错误。

4 KB响应的结果与350 B响应的结果类似，有两个显着的差异：

最大RPS点要低得多，因为4 KB响应只需要约270k RPS即可完全加载10 Gbps NIC

在接近吞吐量限制的情况下，IPvlan更接近--net = host

我们目前的选择是与主机gw法兰绒。它没有太多的依赖关系（例如，不需要AWS或新的Linux版本），与IPvlan相比很容易设置，并且具有足够的性能特征。 IPvlan是我们的备份解决方案。如果在某些时候flannel增加了对IPvLAN的支持，我们将切换到它。

尽管aws-vpc比host-gw表现得稍微好一些，但它的50台机器限制和它连接亚马逊AWS的事实对我们来说是一个破坏者。

50,000 RPS, 350 B

每秒5万个请求，所有候选人表现出可接受的性能。 您已经可以看到主要趋势：IPVlan表现最好，host-gw和aws-vpc紧随其后，vxlan是最差的。

150,000 RPS, 350 B

IPvLAN稍好于host-gw和aws-vpc，但是它具有最差的99.99个百分点。 host-gw比aws-vpc执行略好。

250,000 RPS, 350 B

这种负荷预计在生产中也是常见的，所以这些结果是特别重要的。

IPvLAN再次显示最佳性能，但aws-vpc具有最好的99.99和99.999百分点。 host-gw在95和99百分位上胜过aws-vpc。

350,000 RPS, 350 B¶

在大多数情况下，潜伏期接近250,000 RPS，350 B的情况，但在99.5百分位之后迅速增长，这意味着我们已经接近最大RPS。

450,000 RPS, 350 B

This is the maximum RPS that produced sensible results.

IPvlan leads again with latency ≈30% worse than that of --net-host:

Interestingly, host-gw performs much better than aws-vpc:

500,000 RPS, 350 B

Under 500,000 RPS, only IPvlan still works and even outperforms --net=host, but the latency is so high that we think it would be of no use to latency-sensitive applications.

50k RPS, 4 KB

Bigger response results in higher network usage, but the leaderboard looks pretty much the same as with the smaller response:

Latency percentiles at 50k RPS (≈20% of maximum RPS), ms

Setup	95 %ile	99 %ile	99.5 %ile	99.99 %ile	99.999 %ile	Max Latency
IPvlan	0.6	0.8	0.9	5.7	9.6	15.8
aws-vpc	0.7	0.9	1	5.6	9.8	403.1
host-gw	0.7	0.9	1	7.4	12	202.5
vxlan	0.8	1.1	1.2	5.7	201.5	402.5
--net=host	0.5	0.7	0.7	6.4	9.9	14.8

150k RPS, 4 KB

Host-gw has a surprisingly poor 99.999 percentile, but it still shows good results for lower percentiles.

Latency percentiles at 150k RPS (≈60% of maximum RPS), ms

Setup	95 %ile	99 %ile	99.5 %ile	99.99 %ile	99.999 %ile	Max Latency
IPvlan	1	1.3	1.5	5.3	201.3	405.7
aws-vpc	1.2	1.5	1.7	6	11.1	405.1
host-gw	1.2	1.5	1.7	7	211	405.3
vxlan	1.4	1.7	1.9	6	202.51	406
--net=host	0.9	1.2	1.3	4.2	9.5	404.7

250k RPS, 4 KB

This is the maximum RPS with big response. aws-vpc performs much better than host-gw, unlike the small response case.

Vxlan was excluded from the graph once again.

Test Environment

Background

To understand this article and reproduce our test environment, you should be familiar with the basics of high performance.

These articles provide useful insights on the topic:

• How to receive a million packets per second by CloudFlare

• How to achieve low latency with 10Gbps Ethernet by CloudFlare

• Scaling in the Linux Networking Stack from the Linux kernel documentation

Machines

• We use two c4.8xlarge instances by Amazon AWS EC2 with CentOS 7.

• Both machines have enhanced networking enabled.

• Each machine is NUMA with 2 processors; each processor has 9 cores, each core has 2 hyperthreads, which effectively allows to run 36 threads on each machine.

• Each machine has a 10Gbps network interface card (NIC) and 60 GB memory.

• To support enhanced networking and IPvlan, we’ve installed Linux kernel 4.3.0 with Intel’s ixgbevf driver.

Setup

Modern NICs offer Receive Side Scaling (RSS) via multiple interrupt request (IRQ) lines. EC2 provides only two interrupt lines in a virtualized environment, so we tested several RSS and Receive Packet Steering (RPS) Receive Packet Steering (RPS) configurations and ended up with following configuration, partly suggested by the Linux kernel documentation:

• IRQ

• The first core on each of the two NUMA nodes is configured to receive interrupts from NIC.

  To match a CPU to a NUMA node, use lscpu:

  $ lscpu | grep NUMA
  NUMA node(s): 2
  NUMA node0 CPU(s): 0-8,18-26
  NUMA node1 CPU(s): 9-17,27-35

  This is done by writing 0 and 9 to /proc/irq/<num>/smp_affinity_list, where IRQ numbers are obtained with grep eth0 /proc/interrupts:

  $ echo 0 > /proc/irq/265/smp_affinity_list
  $ echo 9 > /proc/irq/266/smp_affinity_list

• RPS

• Several combinations for RPS have been tested. To improve latency, we offloaded the IRQ handling processors by using only CPUs 1–8 and 10–17. Unlike IRQ’s smp_affinity, the rps_cpus sysfs file entry doesn’t have a _list counterpart, so we use bitmasks to list the CPUs to which RPS can forward traffic1:

  $ echo "00000000,0003fdfe" > /sys/class/net/eth0/queues/rx-0/rps_cpus
  $ echo "00000000,0003fdfe" > /sys/class/net/eth0/queues/rx-1/rps_cpus

• Transmit Packet Steering (XPS)

• All NUMA 0 processors (including HyperThreading, i.e. CPUs 0-8, 18-26) were set to tx-0 and NUMA 1 (CPUs 9-17, 27-37) to tx-12:

  $ echo "00000000,07fc01ff" > /sys/class/net/eth0/queues/tx-0/xps_cpus
  $ echo "0000000f,f803fe00" > /sys/class/net/eth0/queues/tx-1/xps_cpus

• Receive Flow Steering (RFS)

• We’re planning to use 60k permanent connections, official documentation suggests to round up it to the nearest power of two:

  $ echo 65536 > /proc/sys/net/core/rps_sock_flow_entries
  $ echo 32768 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
  $ echo 32768 > /sys/class/net/eth0/queues/rx-1/rps_flow_cnt

• Nginx

• Nginx uses 18 workers, each worker has it’s own CPU (0-17). This is set by the worker_cpu_affinity option:

  workers 18;worker_cpu_affinity 1 10 100 1000 10000 ...;

• Tcpkali

• Tcpkali doesn’t have built-in CPU affinity support. In order to make use of RFS, we run tcpkali in a taskset and tune the scheduler to make thread migrations happen rarely:

  $ echo 10000000 > /proc/sys/kernel/sched_migration_cost_ns
  $ taskset -ac 0-17 tcpkali --threads 18 ...

This setup allows us to spread interrupt load across the CPU cores more uniformly and achieve better throughput with the same latency compared to the other setups we have tried.

Cores 0 and 9 deal exclusively with NIC interrupts and don’t serve packets, but they still are the most busy ones:

RedHat’s tuned was also used with the network-latency profile on.

To minimize the influence of nf_conntrack, NOTRACK rules were added.

Sysctls was tuned to support large number of tcp connections:

fs.file-max = 1024000net.ipv4.ip_local_port_range = "2000 65535"net.ipv4.tcp_max_tw_buckets = 2000000net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_fin_timeout = 10net.ipv4.tcp_slow_start_after_idle = 0net.ipv4.tcp_low_latency = 1

Footnotes

[1]	Linux kernel documentation: RPS Configuration

[2]	Linux kernel documentation: XPS Configuration