A Discussion On Storage Overhead

Let’s talk about transmission overhead.

For various types of communications protocols, ranging from Ethernet to Fibre Channel to SATA to PCIe, there’s typically additional bits that are transmitted to help with error correction, error detection, and/or clock sync. These additional bits eat up some of the bandwidth, and is referred to generally as just “the overhead”.

For 1 Gigabit Ethernet and 8 Gigabit Fibre Channel as well as SATA I, II, and III, they use 8/10 overhead. Which means for every eight bits of data, an additional two bits are sent.

The difference is who pays for those extra bits. With Ethernet, Ethernet pays. With Fibre Channel and SATA, the user pays.

1 Gigabit Ethernet has a raw transmit rate of 1 gigabit per second. However, the actual transmission rate (baud, the rate at which raw 1s and 0s are transmitted) for Gigabit Ethernet is 1.25 gigabaud. This is to make up for the 8/10 overhead.

SATA and Fibre Channel, however, do not up the baud rate to accommodate for the 8/10 overhead. As such, even though 1,000 Gigabit / 8 bits per byte = 125 MB/s, Gigabit Fibre Channel only provides 100 MB/s. 25 MB/s is eaten up by the extra 2 bits in the encoding. The same is true for SATA. SATA 3 is capable of transmitting at 6 Gigabits per second, which is 750 MB/s. However, 150 MB/s of that is eaten up by the extra 2 bits, so SATA III can transmit 600 MB/s instead.

PAM 4

There’s a new type of raw data transmission hitting the networking world called PAM 4. Right now it’s used in 400 Gigabit Ethernet. 400 Gigabit Ethernet is 4 channels of 50 Gigabit links. You’ll probably notice the math on that doesn’t check out: 4 x 50 = 200, not 400. That’s where PAM 4 comes in: The single rate change is still 50 gigabaud, but instead of the signal switching between two possible values (0, 1), it switches between 4 possible values (0, 1, 2, 3). Thus, each clock cycle can represent 2 bits of data in stead of 1 bit of data, doubling the transmission rate.

Higher Level Protocol Overhead

For networking storage on Ethernet, there’s also additional overhead for IP, TCP/UDP, and possibly others (VXLAN for example). In my next article, I’ll talk about why they don’t really matter that much.

One Response to A Discussion On Storage Overhead

  1. Wes Felter says:

    A note on PAM4: Cheaper 400G-CR8/SR8 is eight lanes of 50G and each lane runs at 25 (technically 26.5625) Gbaud with PAM4. More expensive 400G-DR4/FR4/LR4 has four lanes of 100G using PAM4 at 50 (really 53.125) Gbaud. These flavors aren’t interoperable so you have to plan carefully.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: