During a recent penetration test on a customer application, I noticed weird interactions between the web front-end and back-end. This would eventually turn out to be a vulnerability called HTTP request smuggling, enabled by the fact that the front-end was configured to downgrade HTTP/2 requests to HTTP/1.1.

With the help from my colleague Thomas Stacey, we were able to construct an exploit chain with response queue desynchronization along with traditional HTTP/1.1 request smuggling techniques. This resulted in us being able to capture requests from legitimate users, which would enable attackers to take over legitimate user accounts, and obtain sensitive information.

I recommend reading Thomas’s blog post,“Using HTTP request smuggling to hijack a user’s session – exploit walkthrough”, which covers the technical details of the exploit. This blog will instead focus on how HTTP/2 request smuggling works, and how we were able to exploit it.

What is HTTP request smuggling?

HTTP request smuggling is a security vulnerability that arises from a disparity between the front-end and back-end systems in how they handle the size of a message’s body. This issue is commonly observed in HTTP/1.1 when multiple message length headers (Content-Length and Transfer-Encoding), are added to the same request. Malicious actors can exploit this vulnerability to “smuggle” their own requests into the back-end by embedding them within the message’s body. As a result, these illicit requests have the potential to interfere with subsequent requests from legitimate users on the website, posing a significant threat to its security.

HTTP/2 introduced a secure and improved approach for calculating the length of messages, effectively mitigating the risk of request smuggling (read more about that here). However, potential dangers can still arise if HTTP/2 is not implemented end-to-end.

To gain a better understanding of this vulnerability, let’s start by looking at the distinctions and similarities between HTTP/2 and HTTP/1.1, specifically how they handle the length of a request and its body.

HTTP/1.1

Since its inception in 1997, HTTP/1.1 has been the staple for handling the fetching of web resources. In this version, all requests and responses are sent and handled in the plain text message format, consisting of simple line-oriented sequence of characters. The two main data transfer mechanisms that are available are Content-Length and Transfer-Encoding: chunked. Both require a predetermined value for the request being sent, character length of the data for Content-Length and the number of bytes for Transfer-Encoding: chunked.

HTTP/2

HTTP/2 was first introduced in 2015 and quickly adopted by most major web browsers. Some of the goals with HTTP/2 were reducing latency by using request and response multiplexing, and minimizing protocol overhead via HTTP header compression, and request prioritization. A TCP connection is still established, same as for HTTP/1.1, but because of multiplexing, the number of active connections needed is greatly reduced.

HTTP/2 request are not sent as plain text messages, but instead converted to binary format. The request is represented as “frames”, where each frame has an explicit length field that tells the server how many bytes to read. This makes the total length of the request the sum of all the frames length field. Therefore, eliminating the need for using a predetermined header such as Content-Length and Transfer-Encoding: chunked. This eliminates the possibility of user supplied data interfering with the request length, thereby solving request smuggling issues.

Reason for downgrading to HTTP/2

While developing HTTP/2, one of the major goals was to keep a high-level compatibility with HTTP/1.1 by maintaining status codes, methods, and most header fields. In the end, this means that both versions of the protocol are presenting the same information in different ways, see image below:

Since HTTP/2 is still new compared to its predecessor, there is a high probability that web servers that use it still communicate with back-end servers that only use HTTP/1.1. This is when HTTP/2 downgrading comes into play. As mentioned earlier, both versions approximately represent the same information in different ways, so it is straightforward to convert these requests for the server. This works the other way around also, when the back-end server sends the HTTP/1.1 response, the front-end server will then take it and “upgrade” it to HTTP/2.

Now that we are a bit familiar with how HTTP/1.1 and HTTP/2 work, and the fundamentals of downgrading, we can start discussing where the problems arise.

Although HTTP/2 eliminates the need for message length headers, the process of HTTP downgrading reintroduces this requirement due to the back-end HTTP/1.1 server’s reliance on them. However, it is important to note that the front-end HTTP/2 server is likely to disregard or can be manipulated to disregard these message length headers. Consequently, HTTP downgrading creates the potential for a desynchronization between the front-end and back-end servers.

Detecting HTTP/2 Request Smuggling

In this specific customer’s case, we had a H2:TE variant and so we will use that as an example. The other variant, H2.CL can be read more about here.

To begin with, the HTTP/2 specification recommends any request containing the Transfer-Encoding header should either have it stripped or blocked by the front-end server. But if the front-end server ignores the specification and lets the request pass with the header intact, it subsequently downgrades the request to HTTP/1.1 and forwards it to a back-end server that supports chunked encoding. Now a request smuggling vulnerability is likely to be introduced.

Just like with its HTTP/1.1 counterpart, the detection is performed by sending a malformed chunked body. If the request has a much longer response time than a normal request, you can most likely deduce that the back-end server is waiting for the remaining bytes of the request. 

Once we have detected the vulnerability, we can attempt to smuggle a request prefix as shown in the image below.

In this case, our smuggled prefix may allow us to bypass the front-end protections for accessing the admin panel by smuggling the request for the admin panel past the front-end.

Using CRLF injection to bypass defenses

As HTTP/2 request smuggling is a known attack vector, most environments block potentially dangerous downgrade requests by stripping and blocking the Transfer-Encoding header for H2.TE attacks and validating the content-length. However, because of HTTP/2’s binary nature there are multiple ways to attempt to bypass this. We will focus on a CRLF injection that lets us smuggle headers past the HTTP/2 front-end.

CRLF, is a combination of the control characters CR (Carriage Return, \r) and LF (Line Feed, \n). CR moves the cursor to the beginning of the line without advancing to the next line and LF moves the cursor down to the next line without returning to the beginning of the line. In combination with each other, they move the cursor down to the next line and then to the beginning of the line. CRLF sequences interacts differently in HTTP/1.1 and HTTP/2 which opens more opportunities for request smuggling attacks when HTTP downgrading is in use.

In HTTP/1.1, a full CRLF sequence in a header’s value indicates that the current header is terminated and a new one will start on the next line. However, since HTTP/2 is not plain text based and is instead binary based, CRLF sequences have no specific meaning. This means that you can include the binary representation of CRLF sequences inside the request header, without it breaking the specification. In a case where HTTP downgrading is in play, the front-end may split the header containing the CRLF sequence into two separate headers for the HTTP/1.1 request.

The example above shows the HTTP/2 version of the header and how it will end up in the HTTP/1.1 request after downgrading. Since the front-end uses HTTP/2, it will only see that arbitrary “foo” header and its value. Even if the Transfer-Encoding and the CRLF sequence are in the value, the front-end will not see this as a new header and forward it to the back-end HTTP/1.1 server. The back-end server will instead read the value of the “foo” header, see the CRLF sequence and terminate the header. The server will then read the remaining part of the value as if it were the next header, successfully injecting the Transfer-Encoding header.

War story

During a regularly scheduled customer penetration test, I noticed weird responses when sending message length headers to certain endpoints. Therefore, I triggered a HTTP/2 probe, a scanning option that comes with Portswigger’s “HTTP request smuggling” tool. After a while, I saw that it had detected an issue related to request smuggling. With HTTP request smuggling being rather uncommon these days, I was quite skeptical. I ended up running the probe against several endpoints before truly believing that the tool had found something worth investigating. All the endpoints returned the following issue:

This response from the probe specifies that a HTTP downgrading might be in use, and provides the method the probe used to trigger the desynchronization. After some careful consideration, the plan after was to first confirm that downgrading was in use and that the back-end server accepted the chunked Transfer Encoding header.

Sending a request with a proper chunked body to the endpoint worked, and the application responded normally, while sending a malformed request caused a delay. This confirmed that the front-end was downgrading my requests and that the back-end server supported chunked encoding. See the different response time in the requests below.

The next step was to confirm that we could affect the next request in the request queue. First, I tried a simple H2.TE smuggling request, that would smuggle a request to “robots.txt” that I knew existed on the site (note that the request is HTTP/2 but will be represented in HTTP/1.1 syntax):

I tried to get the above request to work but realized after a while that the site was blocking any post request to the vulnerable endpoints. At the same time, it appeared that the Transfer-Encoding header was being sanitized by the back-end server when suspicious requests were sent.

I continued with attempts by attempting to obfuscate the Transfer-Encoding header, such as adding an additional space, an additional tab space, and a duplicate header. These attempts ended the same way that the previous did with no results at all. After asking my colleagues for help with building a working payload, we figured out that the following criteria had to be met for a request to be smuggled:

• GET request must be used, not a POST request.
• The Transfer-Encoding needs to be injected into the value of an arbitrary header with a CRLF sequence, for instance Foo: bar\r\nTransfer-Encoding: chunked.
• A duplicate of the arbitrary header needs to be present in the request.
• The value of the Content-Length header needs to match the length of the entire body for the request to be properly smuggled. 

In the end, the final proof of concept request looked like this (note that this request is represented in HTTP/1.1 syntax):

Since the application used multiple back-end servers, we needed the smuggled request to be on the same server. By changing the endpoint to one that matched robots.txt server, we successfully smuggled robots.txt and got 200 OK status code response that reflected in the body of the vulnerable endpoint.

Now that we have a proof of concept, we could start construct an attack that would have a higher impact. While request smuggling itself greatly affects the availability of the website, serving random users robots.txt is not the optimal showcase of this rare vulnerability. Therefore, the way we wanted to showcase it was through a response queue poisoning, just as in Thomas’ blog post, we realized that this was the way to construct a higher impact vulnerability.

A response queue poisoning works by smuggling an extra request after the first one. The first request will trigger the response queue desynchronization, causing users to receive responses belonging to other users. In figure 11, we only smuggled a prefix after the first request, causing the next request in queue to be appended to our prefix. When we smuggle an entire request to the back-end server, the front-end server will be confused since it has an additional response to deal with. This enables an attacker to receive a victim’s response rather than their own.

First, we needed a page with a POST body parameter which was reflected in the response, and preferably did not limit the number of characters to act as our “reflection gadget”. The reflection gadget is a page on the application that reflects the request body in the response. We had some good candidates, but none that could fit and reflect the entire user’s request into it. That is when I remembered that earlier I had found that the website still had a bunch of default Apache Tomcat pages on it, including the example pages. One of the examples pages is “RequestParamExample”, which lets you input two parameters, that can contain a large amount of data. With this, we had all the pieces in place to build a functional exploit.

The following payload contains two smuggled requests, that when combined, will cause us to receive other uses requests. The first request will cause a response queue desynchronization and only smuggles the prefix before the next request starts. The next request being our reflection gadget. Since the “Content-Length” header is properly set the backend server will read everything up to and including the prefix as one request and the following reflection gadget as a separate request. The victim’s request will then likely be appended to the body of our gadget and the response served to us.

Since this exploit can be a bit confusing, the steps are summarized below:  

• The crafted request will be sent to the front-end server, which process the “Content-Length” header and interprets the entire request as normal, even if there is a malformed chunked encoding body. The front-end then downgrades the request to HTTP/1.1 and forwards it to the back-end server with the CLRF injected Transfer-Encoding header. Making the injected Transfer-Encoding header get converted to a legitimate header once the downgrading from HTTP/2 to HTTP/1.1 is complete.
• When the request reaches the back-end server, the server will instead prioritize the “Transfer-

Encoding” header and read the request using the chunked method and skipping over the Content-Length headers value all together. It will see the “0” character, which indicates the end of the body when using chunked encoding. Everything in the request after the 0 will be smuggled while the server returns the response for the original endpoint.

• The request towards the root page (the “/” path) causes the response queue desynchronization and has a “Content-Length” header with length 1, which will leave the follow-up RequestParamExample request in the back-end’s queue.
• The victim’s request will, depending on the request timings, arrive to the back-end server and be appended to the body of the reflection gadget. The victim will instead be served the response for the request to the root page (due to the response queue poisoning).
• Another request is then sent by the attacker, with the hope that the application will serve them the response from the reflection gadget, containing the victims request.

By sending this request in quick succession, there is a possibility we might be able to capture another users’ request:

Request smuggling mitigation

Make sure to use HTTP/2 end-to-end connection and do not downgrade it. This will prevent all that was brought up here. If you are setting up or managing an HTTP/2 server, especially one that supports downgrading, make sure to reject requests containing over ambitious headers specifying the request body’s size. Another mitigation strategy is also to block any headers containing specific sequences, such as the CRLF sequence mentioned in this blog.

How pen testing as a service (PTaaS) can help

Discover the power of Outpost24’s PTaaS to uncover complex application vulnerabilities. Unlike automated testing, our advanced pen testing services provide manual and in-depth analysis that can detect elusive vulnerabilities like HTTP request smuggling.

Our skilled testers go above and beyond to push vulnerabilities to their limits, ensuring that even the most intricate flaws are identified. Don’t leave your applications vulnerable to potential threats. Discover the power of our PTaaS solution by requesting a live demo today.

Ghost Labs  

Ghost Labs is the specialist security unit within Outpost24, offering enhanced security services such as advanced network penetration testing, web application testing, Red Teaming assessments and complex exploitation. In addition, the Ghost Labs team is an active contributor to the security community with vulnerability research and coordinated responsible disclosure programs.  

Ghost Labs performs hundreds of successful penetration tests for its customers ranging from global enterprises to SMEs. Our team consists of highly skilled ethical hackers, covering a wide range of advanced testing services to help companies keep up with evolving threats and new technologies.