Thursday, April 30, 2015

Broken, Abandoned, and Forgotten Code, Part 2

In the part 1, I showed how the Netgear R6200's upnpd binary contains what appears to be a hidden SOAP action related to the string "SetFirmware". I also showed how we can get into the upnp_receive_firmware_packets() function if we play timing games and send our request in multiple parts.

In this part I'll describe additional timing considerations needed to avoid hanging the server. I'll also discuss sloppy parsing of the SOAP request, and I'll make some guesses as to how that request should be formed.

If you're following along, the first proof-of-concept code is available. Clone my git repo from:

Each installment in this series that has new or updated code will have a separate directory in the repository. This week's code is under part_2.

Receiving Firmware Bytes

The conditions I described previously are:
  • The request should be broken up into two or more parts, with the first being no larger than 8,190 bytes.
  • "Content-length:" should be somewhere in the data, presumably in the HTTP headers (because this would make sense), but not necessarily.
  • The content length should be greater than 102,401 bytes.
  • The string "SetFirmware" should be somewhere in the data.
If those conditions are satisfied, then upnp_receive_firmware_packets() gets called from upnp_main() at 0x4144E4. In this function, a select(), recv(), and memcpy() loop receives the remainder of the request. This proceeds fairly sanely, with one problem.

upnp receive firmware select loop
The select() and recv() loop doesn't check for closed connections

If the client closes the connection immediately after sending the request, this function gets caught in an infinite loop. The cause for this is a little tricky to explain.

From the select(2) Linux man page:
A file descriptor is considered ready if it is possible to perform the corresponding I/O operation (e.g., read(2)) without blocking.

If the peer has closed its end of the connection, then select() indicates the socket is ready because a recv() would not block. The way Unix TCP sockets work, when the remote end of a connection closes, a recv() on that socket returns zero. In the loop, the return value from recv() is checked for errors (negative values), but if there are no errors, it is assumed that data was received, and the loop returns to select(). This results in the function looping indefinitely if the client shuts down the connection too soon.

The only two ways this loop ever terminates are (a) if select() or recv() return an error, or (b) if select() returns zero, indicating a timeout with no file descriptors ready for I/O. This means the requesting client must not close the connection immediately after it has sent the request. It should send the request, and then pause before closing the connection. Sleeping a few seconds should suffice.

However, there's an additional implication. Recall from before that we had to sleep 1-2 seconds in upnp_main() in order to get into this function. It turns out that if we slept longer, then the select() would time out, returning zero, and the loop would end before we had sent the rest of the request. So, while it's critical to sleep a second or two, it's also critical to sleep no more than that.

In review, the steps should be:

  • Send 8,190 bytes or fewer, but hold the connection open
  • Sleep 1-2 seconds, but no more
  • Send the rest of the request, but hold the connection open
  • Sleep a few more seconds
  • Close the connection

The following code fragment sends chunks with appropriate sizes and sleep periods to get us into upnp_receive_firmware_packets() and to avoid getting into an infinite loop with select():

def special_upnp_send(addr,port,data):
    #only send first 8190 bytes of request
    #sleep to ensure first recv()
    #only gets this first chunk.
    #Hopefully in upnp_receiv_firmware_packets()
    #by now, so we can send the rest.
    #Sleep a bit more so server doesn't end up
    #in an infinite select() loop.
    #Select's timeout is set to 1 sec,
    #so we need to give enough time
    #for the loop to go back to select,
    #and for the timeout to happen,
    #returning an error.

More Broken and Lazy Parsing

Once the entire request has been received, it is parsed, or "parsed" as it were, piecemeal, across several functions. The upnp_receive_firmware_packets() function calls sub_4134A8(). This function inspects the beginning of the received request (the first 1023 bytes, to be precise) for for the HTTP method. If the request is a POST, the soap_method_check() function is called at 0x413774.

Check for POST HTTP method
Checking for the POST HTTP method

Call to soap_method_check
Calling soap_method_check()

In soap_method_check() several naive stristr() calls search for a series of strings across the entire request buffer. Based on several of the more recognizable strings, such as "Public_UPNP_C1", these strings are UPnP control URLs that might be requested by the POST. Although these strings may be placed literally anywhere (starting to sound familiar?) in the request and still trigger their respective code paths, presumably a typical request would be structured like so:

POST /Public_UPNP_C1 HTTP/1.1

One of the control URLs that is checked is "soap/server_sa". If that URL is found in the request, the function sa_method_check() is called. Note that we still don't know for certain where the UPnP daemon actually expects the "SetFirmware" string to be located. However, based on other, similar string references, it seems likely that this string should be part of the UPnP control URL: "soap/server_sa/SetFirmware".

call to sa_method_check
A call to sa_method_check if "soap/server_sa" is found
The sa_method_check() function loops over a list of valid strings corresponding to the "SOAPAction:" header, and for each string in the list performs a naive stristr() across the entire request buffer. The string "DeviceConfig", if found anywhere in the request, results in a call to sub_43292C(). This enormous function repeatedly calls sa_findKeyword(), passing it the request buffer as well as various keys to be looked up in the "s_Event" dictionary.

graph view of sub_43292C
The enormous graph of sub_43292c(). This function looks for keywords in the SOAP request.

The sa_findKeyword() function searches the request buffer for the corresponding string from the "s_Event" dictionary. The original "SetFirmware" string is referenced by the key 49. If it is found, again, anywhere in the request, the function sa_parseRcvCmd() is called.

search for SetFirmware string
Repeated calls of sa_findKeyword(). Index 49 corresponds to "SetFirmware."

The following HTTP request headers should, based on what we have observed so far, get the request into the sa_parseRcvCmd() function.

request="".join["POST /soap/server_sa/SetFirmware HTTP/1.1\r\n",
                 "Accept-Encoding: identity\r\n",
                 "Content-Length: 102401\r\n",
                 "Soapaction: \"urn:DeviceConfig\"\r\n",
                 "Connection: close\r\n",
                 "Content-Type: text/xml ;charset=\"utf-8\"\r\n\r\n"]

Forming an HTTP request that would exercise the proper code path was an exercise in guesswork due to the many naive string searches littered along the way and an absence of anything resembling structured parsing.

It is in the sa_parseRcvCmd() function that an encoded firmware image is extracted and decoded from the request body, and assuming the right conditions are met, written to the router's flash storage, replacing the existing firmware.

Up until now, it has remained at least possible, however improbable, that the vendor may have designed a client to send the magic SOAP requests and to play the timing games necessary to exercise the firmware updating functionality. In the next part I'll start discussing sa_parseRcvCmd(),  a complicated function with lots of code paths and lots of bugs. It is also this function where it becomes even clearer that the firmware updating capability of this UPnP server is not completely implemented and cannot actually work under normal conditions.

Thursday, April 23, 2015

Broken, Abandoned, and Forgotten Code, Part 1


This series of posts describes how abandoned, partially implemented functionality can be exploited to gain complete, persistent control of Netgear wireless routers. I'll describe a hidden SOAP method in the UPnP stack that, at first glance, appeared to allow unauthenticated remote firmware upload to the router. After some reverse engineering, it became apparent this functionality was never fully implemented, and could never work properly given a well formed SOAP request and firmware image. If it could work at all, it would be with only the most contrived of inputs.

Someone may have thought shipping dead code was okay because an exploit scenario would be so contrived. Someone may not have considered that contrived inputs are the stock-in-trade of vulnerability researchers.

In this series, I'll describe the process of specially crafting a malicious firmware image and a SOAP request in order to route around the many artifacts of incomplete implementation in order to gain persistent control of the router. I'll discuss reverse engineering the proper firmware header format, as well as the the improper one that will work with the broken code. Together, we'll go from discovery to complete, persistent compromise.

Rules of Engagement

In order to make the challenge more interesting and to more clearly demonstrate the thesis, I decided to not take advantage of any shortcuts by exploiting vulnerabilities in the broken code path. I treated all bugs I encountered along the way as hurdles to overcome. For example, there is a buffer overflow that I will describe in a future post. I could exploit this buffer overflow to subvert the flow of execution and execute shellcode that would write my firmware, but that would be cheating. The point of this project is to show that dead code can represent a powerful attack vector, even when it is non-functional.

Target Device

The device I'll be describing in this series is the Netgear R6200 802.11ac router. Here are some specifics about the router:
  • Linux based
  • Little endian MIPS
  • Firmware version
  • Originally released in 2012
  • US$200 retail price when released
I only worked with the firmware, which I believe is the original released version. I haven't looked into later versions. That will remain an exercise for the reader. I will add that as recently as January 2015, I ordered an R6200 from Amazon, and it came with firmware installed.

I <3 UPnP

Universal Plug and Play services on SOHO routers make for a nice attack surface for vulnerability research. UPnP services are often capable of system-level modifications that are protected only by a thin veil of obscurity. When I found strings referencing "firmware" in the Netgear R6200[1] 801.11ac router's UPnP binary I knew this daemon was going to be an interesting target. Most SOHO router exploits do not offer persistence[2], owing to their read-only storage. An unauthenticated firmware upload is an opportunity to persist undetected on the gateway device for months or even years.

Firmware Unpacking and Strings Analysis

Upon unpacking the R6200's firmware, you can easily identify the UPnP daemon as /usr/sbin/upnpd. Source code is not available for this application, so research is an exercise in binary analysis.

Initial strings analysis of the binary reveals a "SetFirmware" string:

terminal - strings upnpd
Strings analysis on upnpd binary, showing "SetFirmware"

Hopefully this string is somehow related to modifying the device's firmware. Static analysis reveals how the "SetFirmware" string is referenced in the binary:

SetFirmware reference in upnpd
Reference to "SetFirmware" from upnp_main()

As shown in the above screenshot, "SetFirmware" is referenced exactly once, from upnp_main() at offset 0x4142C4.

Lazy Parsing

When upnpd receives a SOAP request, the upnp_main() function does the following:
  • recv() from a TCP socket
  • check that it received (seemingly arbitrarily) 8,190 bytes or fewer.
  • perform a lazy parse of incoming requests by performing stristr() string searches on the received data.
The upnp_main() function searches for the string "Content-length:" literally anywhere (wtf?) in the received data. If the value following "Content-length:" is greater than or equal to (again, seemingly arbitrary) 102401, as checked by atoi() another stristr() is performed, searching for the "SetFirmware" string. Again, this string may be anywhere in the received data. If the string is found, upnp_receive_firmware_packets() is called at 0x4144E4.

call to upnp_receive_firmware_packets()
A call to upnp_receive_firmware_packets()

It's worth noting the implication of these two size checks, the first for 8,190 or less and the second for a content length greater that 102,401. The request must either have a forged content-length header, or the requesting client must avoid sending the entire request in one operation. In the latter case, the request should send no more than 8,190 bytes, pause, then send the rest.

It is also worth noting that at this stage it is unclear how the "SetFirmware" request should be structured. It also is unclear if it should even be a SOAP request (we will proceed on the assumption that it is), or some other protocol. The only things that are known about the request are:
  • The request should be broken up into two or more parts, with the first being no larger than 8,190 bytes.
  • "Content-length:" should be somewhere in the data, presumably in the HTTP headers (because this would make sense), but not necessarily.
  • The content length should be greater than 102,401 bytes.
  • The string "SetFirmware" should be somewhere in the data.

This is the first bug that suggests this code doesn't actually work, at least not naturally. When you send() a request, you should be able to send the entire request in one operation. Your operating system's TCP/IP stack (usually in the kernel) will handle chunking the data as necessary. Further, the remote host's TCP/IP stack will handle unchunking the data as necessary. These details are abstracted from userspace code, and the receiving program should be able continue receiving until the remote end has closed the connection or until some maximum allowable size has been received. We're able to work around this anomalous behavior by sending only a small chunk of the data, then sleeping before sending the rest.

In the next part, I'll describe another bug, this time a misuse of select(), that also suggests this code never actually worked in the wild. I'll go on to describe how to make it work anyway. I'll also discuss how the broken, lazy parsing makes it difficult to know how the SOAP request should be formed such that execution follows a desirable code path.

[1] Although the R6200 was the primary device researched, preliminary analysis of other devices, including the R6300 v1, indicates presence of the same vulnerabilities described on this blog.
[2] It should be noted that non-persistent exploits are attractive in their own right, as the attacker may remove all traces of the compromise from the device by merely rebooting it.

Wednesday, April 22, 2015

Broken, Abandoned, and Forgotten Code: Prologue

A Secret Passage to Persistant SOHO Router Pwnage

Almost two years ago plus a house selling, a cross-country move, a house buying, a job change, and a wedding, I downloaded and unpacked the firmware for Netgear's then-new R6200 wireless router. This was one of Netgear's first entries into the nascent 802.11ac market. At around US$200 at the time, this device was at the high end of the Netgear lineup. Finding some cool vulnerabilities in some of the newest, swankiest, consumer WiFi gear would make for a neat paper, or at least a good blog post or two.

In June 2013, I started investigating the R6200. Right away, there were suspicious strings and code paths in the UPnP daemon that were too interesting ignore. If I was right, I would be able to flash a malicious firmware to the device from the local network without authentication. Answering the sirens' call, I spent a few weeks trying to unravel this shit-show of a daemon. I finally gave up, deciding the code I was investigating was too broken to ever actually work, and was therefore not exploitable.

Fast forward six months to December. Having worked through my anger from wasted weeks of work over the summer, the project was back on my mind. I decided to revisit it, this time with a new approach. My original approach was to reverse engineer what appeared to be a backdoor update capability. I gave up when I realized the backdoor was likely never completely implemented and could never actually work as intended. My new approach was to see if I could specially craft an exploit that would route around all the broken networking code and broken parsing code in order to get the router to accept my firmware without crashing.

Spoiler: In the end I was successful. The project had become interesting enough that I planned to write it up and submit it to a conference. But, well, life happened, and here we are nearly a year and a half later.

What comes next amounts, I think, to the equivalent of a small book describing this project. Over the next 14 or so posts, I'll cover all of the various challenges involved and how I solved them, including the following:
  • Reverse engineering the upnpd binary
  • Broken networking code and how to deal with it
  • Using Bowcaster to reverse engineer an undocumented firmware header
  • Unpacking, modifying, and repacking the firmware
... and many others. I plan to post about one article a week. I'll include complete, working exploit code as well as code to generate proper headers and to repack the firmware.

My hope is that, with the necessary tools and a little prerequisite reversing experience, you can follow along and reproduce this project.

In the mean time, here's a video to give you a tease. The left window is a minicom serial connection showing you what's going on under the hood. The right window is where actual exploitation is happening.

R6200 Firmware Upload from Zach on Vimeo.

Stay tuned. Hopefully it will be fun.

Update: Part 1 is up! Hope you enjoy it!