Recurse SP2'23 #9: Domain Fronting and Packet Gymnastics

I was reading up on domain fronting recently. In short, network censors can’t see the Host header when your traffic is encrypted with TLS, but they can see the TLS SNI (Server Name Indication) extension, so they just drop any TLS Client Hello packets containing a censored domain in the SNI. Domain fronting is a censorship circumvention strategy that relies on content delivery networks accepting an SNI for an unblocked domain, while allowing a different Host header to be provided in the HTTP request.

Domain fronting is an old idea, but I thought it would still be a nice exercise to try and sniff SNI’s with tcpdump, which brought me to some parts of the Berkeley Packet Filter (BPF) language I hadn’t previously encountered.

First, I’ll give some extra context about domain fronting, including a quick look at the role SNI plays in content delivery networks (CDNs) like Cloudflare, Amazon Cloudfront, Fastly, etc. Then, I’ll pick apart this little packet filter:

tcpdump '(tcp[((tcp[12:1] & 0xf0) >> 2)+5:1] = 0x01) and (tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16)'

(My hope is that this article is approachable to anyone familiar with what an IP address is, what SSL/TLS does at a high level, is vaguely aware of how a content delivery service works, and has probably tried using tcpdump to monitor packets on a network interface before.)

SNI by example

Suppose we want to read a comic with our morning beverage, so we pop on over to https://xkcd.com in our browser.

What’s happening when we do that?

b@tincan:~$ dig +short xkcd.com
151.101.0.67
151.101.192.67
151.101.64.67
151.101.128.67
b@tincan:~$ whois 151.101.128.67 | grep OrgName
OrgName:        Fastly, Inc.

To use Fastly as a CDN, xkcd.com has pointed its DNS to Fastly’s four anycast IP addresses. An anycast address is an IP address used by different devices at different locations. When you try to reach it from your own network, you get routed to just one of these devices. This is usually, but not always, the closest one to you. This is one of the strategies used by maintainers of “edge” networks like Fastly to quickly serve content to users - reducing latency by being close to them!

After we connect to https://xkcd.com, Fastly sees that our browser set the header Host: xkcd.com, so it routes the request to the service run by the author, Randall Munroe. At that point, it probably finds the page’s contents in its cache, and it reaches out to xkcd’s backend host if not.

But before we ever send that Host header, we require that Fastly present a certificate for xkcd.com. Since Fastly services many, many domains on these IP addresses, our browser has to communicate what certificate it expects prior to connecting. This is exactly what the SNI extension of TLS does: it lets you list one or more hostnames you’ll accept a certificate for.

SNI and network censorship

As SSL and then TLS became more popular, it became increasingly difficult for network censors to inspect packet contents. And as large-scale edge hosting services (Cloudflare, Amazon Cloudfront, Fastly, etc) grew, they relied increasingly on anycast IP addresses to serve client content. As we saw above, this required relying on the TLS SNI extension header to identify which certificate to serve the client before receiving an HTTP request with a Host header.

Prior to the mass adoption of SSL/TLS, a network censor could sniff the Host header from unencrypted HTTP traffic. But as encryption became ubiquitous, censors were left with the tough decision to either block all traffic to the handful of anycast IP addresses used by a large CDN, or block none of it to avoid too much collateral blocking of content they deemed beneficial.

But in TLS, SNI is sent in the clear, so a censor could block a domain served by a CDN simply by dropping any Client Hello packets with the corresponding SNI.

(There’s actually an exception here: TLS 1.3 added the Encrypted Client Hello (ECH) extension, which encrypts the entire message (Client Hello) that would contain the SNI. Unfortunately, some of the more active censors decided they would just block ECH traffic outright, since the adoption of ECH isn’t yet high enough to yield enough collateral damage from blocking it.)

So what is domain fronting?

Domain fronting is a relatively old (like a decade) strategy in censorship circumvention that relies on CDNs not confirming that the TLS SNI header and the HTTP Host header match.

Suppose that your network really doesn’t want you to read silly comics about space or whatever, and they drop all Client Hellos with xkcd.com among the SNI’s. We can get around this by taking advantage of the fact that Fastly isn’t (currently) requiring these fields to match. We make a request to one domain, https://fastly.com, to set the SNI, but override the Host header with the target domain:

b@tincan:~$ curl -s https://fastly.com | grep '<title>'
b@tincan:~$ curl -s -H "Host: xkcd.com" https://fastly.com | grep '<title>'
<title>xkcd: College Knowledge</title>

Well, that was easy! Both curl requests went to https://fastly.com, but the second one retrieved the “censored” page.

We can add some extra parameters to our request to confirm what’s going on.

-v to see the TLS connection information, prefixed with an asterisk *.
-I to make a HEAD request and show response headers, prefixed with greater than >. This way we confirm the Host header.

b@tincan:~$ curl -v4sI -H "Connection: close" -H "Host: xkcd.com" https://fastly.com 2>&1 | grep 'fastly\|xkcd'
* Connected to fastly.com (151.101.129.57) port 443 (#0)
*  subject: CN=www.fastly.com
*  subjectAltName: host "fastly.com" matched cert's "fastly.com"
> Host: xkcd.com
* Connection #0 to host fastly.com left intact

We see that we connected to fastly.com, then received and accepted a cert for the same domain - which means our SNI was for fastly.com. However, the Host header of the HTTP request was xkcd.com, as we specified via curl.

Building on domain fronting

You may be wondering: does it matter that we chose fastly.com? The answer is no, we could have chosen any uncensored domain whose DNS resolved to at least one of those anycast IP addresses. Likewise, if our front domain was eventually blocked, we could keep choosing other domains on the service.

The cool thing is that this strategy doesn’t have to be restricted to accessing blocked sites on the CDN! The blocked host can provide access to the broader internet in some fashoin. In the simplest case, it could be a web proxy to specific blocked sites (or to arbitrary sites,) but it’s generally tricky to host a proxy behind a CDN.

Proxying to HTTPS sites is done using the CONNECT method, which CDN’s tend to disallow.
Proxying over HTTP is generally tricky, since you would normally set the Host header for the destination web server, but you need to instead set the Host header for the proxy service behind the CDN.
I actually played around with this, and I’m planning to write a follow-up on this specific work.

More realistically, domain-fronting is used for pluggable transports for Tor like meek. The writing on the above post is rather dated, so it’d be interesting to read further on where meek lives in the present. (For example, contrasting it with obfs4 and Snowflake. There’s a short note on this in the Tor Browser manual, but it’d be great to read further.)

As I mentioned, domain fronting is old news. Censors caught on and pressured CDNs to do something about it, and eventually some of them acquiesced. Plenty of words have already been spilled on this by more qualified folks, so I won’t get into that further.

Let’s get to the bit-fiddling!

Packet gymnastics: snooping on SNIs with tcpdump

I haven’t had an excuse to reach for tcpdump in a while, and I figured sniffing SNIs was as good of an excuse as any to dust it off. So, I did a quick search for “tcpdump detect sni” and found this answer on Stack Exchange which suggested the following:

tcpdump '(tcp[((tcp[12:1] & 0xf0) >> 2)+5:1] = 0x01) and (tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16)'

Like many tools, tcpdump uses the Berkeley Packet Filter (BPF) language to select which packets to match for collection or further processing. I’ve used tcpdump a good bit in the past, but I never ran into this more complex BPF syntax. However, I did recently encounter a very similar language for processing the u32 iptables match extension (described in man iptables-extensions 8,) so I wanted to know more about this element of BPF.

A source of truth

I found it surprisingly difficult to find complete and definitive documentation for the language here, as the manual pages for tcpdump and bpf didn’t cover it in full. The Wikipedia page for BPF referenced an early paper, The BSD Packet Filter: A New Architecture for User-level Packet Capture, which noted that tcpdump was actually implementing its own high-level language on top of the actual BPF machine language. This was a surprise for me - what I thought of as BPF wasn’t BPF at all! Poking around the tcpdump homepage, I found that man pcap-filter 7 is the documentation for this language. You can read it online here.

Most of the above seemed intuitive, but I did have to read around a bit to confirm the meaning of the colon (:) operator. This syntax is meant to take the form proto [ expr : size ], where:

proto selects the protocol layer of interest, like ether, arp, ip, tcp, etc.
expr is a sub-expression which is ultimately evaluated to a value representing a byte offset from the beginning of that protocol’s layer.
size is how much to select.
In other words, tcp[A:B] just means “read B octets starting at octet A, counting from 0.”

Dissecting the filter

Before we dive in, here are a couple of visual references that might be helpful:

For navigating your way through a TCP packet, have a look at the chart of the TCP segment structure on Wikipedia.
For the contents of the TLS payload of the TCP packet, see xargs.org, which provides a great illustrated guide to TLS 1.3. In particular, see Client Hello.

Okay, let’s go. We know from earlier that tcp[12:1] will select the contents of the 13th octet of the TCP header.

This octet has two halves - the data offset field, and then four reserved bits.
The data offset field gives the length of the TCP header.
- It’s important to note that the data offset isn’t the length in bytes, but in 32-bit words.
(tcp[12:1] & 0xf0) extracts the first half of the octet, trimming off the four reserved bits.

Next we have the bit shift: >> 2. What’s that about?

Since our data is in the left half of the octet, we want to create a byte of it instead.
The obvious way to do that would normally be to shift by 4 using >> 4. For example, if we had B=00110000 for 3 words (3*4=12 bytes), then B >> 4 would be 00000011.

But we’re counting 32-bit words here, not bytes! Since there are four bytes to each word, we can multiply by four to get 00001100… but that’s just the same as doing >> 2! Mystery solved: the bitshift is just turning the data offset into a byte count.

So:

(tcp[12:1] & 0xf0) >> 2)+5 gets the data offset, converts it to bytes, and adds 5: this is number of bytes from the beginning of the TCP header and the 6th byte (we count from 0) of the TCP payload.
tcp[(tcp[12:1] & 0xf0) >> 2)+5:1] means “the contents of exactly one byte at the address above.” Here the previously computed value is passed as a byte offset to find a different part of the packet.
tcp[(tcp[12:1] & 0xf0) >> 2)+5:1] == 0x01 completes the filter expression - this just requires that the value stored at that octet is 0x01.

This is testing that the TLS Handshake Header is set correctly for Client Hello, in which case this byte is 1. Again, see xargs.org for a deep dive into an example packet.

(tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16) is a similar incantation that instead tests that this is a handshake packet: it checks that the first byte of the payload (which is the first byte of the Record Header) is 0x16.

Testing this out

First, we’ll want to update our curl command a bit:

b@tincan:~$ curl -4s -H "Connection: close" -H "Host: xkcd.com" https://fastly.com | grep '<title>'                       
<title>xkcd: College Knowledge</title>

-4 ensures we use IPv4, since we’re trying to sniff traffic to Fastly’s anycast IPv4 addresses.
-H "Connection: close" ensures that we don’t reuse our previous connection, in which case we wouldn’t need to send another Client Hello.

If we run the above command in one terminal after starting the following tcpdump in another terminal, here’s what we’ll see:

b@tincan:~$ sudo tcpdump -nX '(tcp[((tcp[12:1] & 0xf0) >> 2)+5:1] = 0x01) and (tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16) and (host 151.101.65.57 or host 151.101.1.57 or host 151.101.65.57 or host 151.101.193.57)'                      
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wlp61s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:42:01.789055 IP 192.168.1.239.43732 > 151.101.65.57.443: Flags [P.], seq 1770067559:1770068076, ack 1642030537, win 502, options [nop,nop,TS val 3434428401 ecr 3549486851], length 517
        0x0000:  4500 0239 bb43 4000 4006 e245 c0a8 01ef  E..9.C@.@..E....
        0x0010:  9765 4139 aad4 01bb 6981 1667 61df 65c9  .eA9....i..ga.e.
        0x0020:  8018 01f6 5536 0000 0101 080a ccb5 37f1  ....U6........7.
        0x0030:  d390 df03 1603 0102 0001 0001 fc03 03a0  ................
        0x0040:  c2d1 36cb deb1 640f 4cee 6c56 b742 da03  ..6...d.L.lV.B..
        0x0050:  e6a5 b264 e5c5 d660 a5c3 b6d6 dbc1 1520  ...d...`........
        0x0060:  5223 8092 6408 0887 d41e 33eb adba 078b  R#..d.....3.....
        0x0070:  fa11 17d6 86f7 34ee ad05 1376 d5d3 7bc8  ......4....v..{.
        0x0080:  003e 1302 1303 1301 c02c c030 009f cca9  .>.......,.0....
        0x0090:  cca8 ccaa c02b c02f 009e c024 c028 006b  .....+./...$.(.k
        0x00a0:  c023 c027 0067 c00a c014 0039 c009 c013  .#.'.g.....9....
        0x00b0:  0033 009d 009c 003d 003c 0035 002f 00ff  .3.....=.<.5./..
        0x00c0:  0100 0175 0000 000f 000d 0000 0a66 6173  ...u.........fas <-- Confirmed!
        0x00d0:  746c 792e 636f 6d00 0b00 0403 0001 0200  tly.com.........
        0x00e0:  0a00 1600 1400 1d00 1700 1e00 1900 1801  ................
        0x00f0:  0001 0101 0201 0301 0433 7400 0000 1000  .........3t.....
        0x0100:  0e00 0c02 6832 0868 7474 702f 312e 3100  ....h2.http/1.1.
        0x0110:  1600 0000 1700 0000 3100 0000 0d00 2a00  ........1.....*.
        0x0120:  2804 0305 0306 0308 0708 0808 0908 0a08  (...............
-----8<-----8<-----8<----8<-----8<-----snipped-----8<-----8<----8<-----8<-----8<-----

And we see fastly.com in roughly where we’d expect it in the packet. Nice!

Let’s take a quick peek at that tcpdump command:

-X prints the packet payload in hex and ASCII (use -XX if you want the link-level header as well)
The filter starts with what we already worked through above:
- '(tcp[((tcp[12:1] & 0xf0) >> 2)+5:1] = 0x01) and (tcp[((tcp[12:1] & 0xf0) >> 2):1] = 0x16) and ...
The filter ends with ensuring we match only the Fastly anycast addresses we found at the beginning:
- ...and (host 151.101.65.57 or host 151.101.1.57 or host 151.101.65.57 or host 151.101.193.57)'

Getting just the SNI?

This is great and all, but it’s a lot of output! Can we extract just the SNI?

It turns out that this is a bit tricky without a proper programming language. For simplicity, let’s only worry about TLS1.3 for now. Recall that we can explore the Client Hello message at xargs.org.

After the Compression Methods header comes a two octet Extensions Length header, which gives the total length in bytes of the extension fields that follow. Each extension field in turn begins with two octets that identify it - for SNI, this is 00 00. This field then has several variable-length fields nested within each other, to allow for a list of multiple hostnames of different types.

And of course, before all of this, both the Session ID and Cipher Suites fields have variable length. So that’s several jumps we’d have to do in order to get to the SNI values, of which there could be several to parse out!

In other words, we don’t have a reliable way to grab extension fields like SNI without a TLS parser. So maybe that’s a fun exercise for another time. :)

Thank you to the kind Recurse folks who took the time to provide feedback on this post!