# Firecracker network performance numbers

This document provides details about Firecracker network performance. The
numbers presented are dependent on the hardware (CPU, networking card, etc.), OS
version and settings. Scope of the measurements is to illustrate the limits for
the emulation thread.

## TCP Throughput

| Segment size/ Direction | 1460bytes | 256bytes | 128bytes | 96bytes |
| ----------------------- | --------- | -------- | -------- | ------- |
| Ingress                 | 25Gbps    | 23Gbps   | 20Gbps   | 18Gbps  |
| Egress                  | 25Gbps    | 23Gbps   | 20Gbps   | 18Gbps  |
| Bidirectional           | 18Gbps    | 18Gbps   | 18Gbps   | 18Gbps  |

**Setup and test description** Throughput measurements were done using
[iperf3](https://iperf.fr/). The target is to fully saturate the emulation
thread and keep it at 100% utilization. No adjustments were done to socket
buffer, or any other network related kernel parameters.

To identify the limit of emulation thread, TCP throughput was measured between
host and guest. An EC2
[M5d.metal](https://aws.amazon.com/ec2/instance-types/m5/) instance, running
[Amazon Linux 2](https://aws.amazon.com/amazon-linux-ami/), was used as a host.

For ingress or egress throughput measurements, a Firecracker microVM running
Kernel 4.14 with 4GB of Ram, 8 vCPUs and one network interface was used. The
measurements were taken using 6 iperf3 clients running on host and 6 iperf3
serves running on guest and vice versa.

For bidirectional throughput measurements, a Firecracker microVM running Amazon
Linux 2, Kernel 4.14 with 4GB of Ram, 12 vCPUs and one network interface was
used. The measurements were taken using 4 iperf3 clients and 4 iperf3 servers
running on both host and guest.

## Latency

The virtualization layer, Firecracker emulation thread plus host kernel stack,
is responsible for adding on average 0.06ms of network latency.

**Setup and test description** Latency measurements were done using ping round
trip times. 2 x EC2 M5d.metal instances running Amazon Linux 2 within the
same[VPC](https://aws.amazon.com/vpc/) were used, with a security group
configured so that it would allow traffic from instances using private IPs. A
10Mbps background traffic was running between instances.

Round trip time between instances was measured.

`rtt min/avg/max/mdev = 0.101/0.198/0.237/0.044 ms`

On one of the instances, a Firecracker microVM running Kernel 4.14, with 1 GB of
RAM, 2 vCPUs, one network interface running was used. Round trip between the
microVM and the other instance was measured, while a 10Mbps background traffic
was running.

`rtt min/avg/max/mdev = 0.191/0.321/0.519/0.058  ms`

From the difference between those we can conclude that ~0.06ms are the
virtualization overhead.
