( Original text by Eclypsium )

Supermicro BMC Case Study

Baseboard Management Controllers (BMCs) are high-value targets to attackers, who can use the out-of-band management features of the BMC to compromise servers even below the level of the host OS and system firmware. BMCs are currently an area of active research for both attackers and defenders. A compromised BMC can be used by malware to provide persistence that lasts beyond a complete wipe and reinstall of the operating system as well as the ability to cross security domains by moving laterally from host-side networks to management networks which are intended to be isolated. In addition to our research, multiple teams have discovered vulnerabilities in other BMCs, such as the recent HPE iLO4 Authentication Bypass and RCEpublication and the The Unbearable Lightness of BMC’s talk at Black Hat. Even back in 2013, Dan Farmer and HD Moore (penetration tester’s guide, Supermicro IPMI) published about serious BMC vulnerabilities. Given the inherent privilege of the BMC in server architecture, a compromised BMC amounts to a significant compromise of the system.

Our research has uncovered vulnerabilities in the way that multiple vendors update their BMC firmware. These vendors typically leverage standard, off-the-shelf IPMI management tools instead of developing customized in-house management capabilities. In this case, we will go deep into the BMC update process on Supermicro systems, we found that the BMC code responsible for processing and applying firmware updates does not perform cryptographic signature verification on the provided firmware image before accepting the update and committing it to non-volatile storage. This effectively allows the attacker to load modified code onto the BMC.

After informing Supermicro of these issues, they have been working with us to improve the security of BMC firmware updates using cryptographic signatures. They have informed us that all new Supermicro products will be released with this feature, and customers who wish to upgrade existing X10 or X11 systems can contact Supermicro (details on Supermicro’s cryptographic signature page). Unfortunately, at the time of publication, updates that address this critical vulnerability are not publicly available for direct download. Also, since Supermicro’s tools do allow administrators to retrieve the firmware version and a dump of the image, Supermicro customers can check that the latest updates are installed and verify the integrity of BMC firmware from critical systems.

Impact

Using the vulnerabilities we discovered, it is possible to make arbitrary modifications to the BMC code and data. Using these modifications, an attacker can run malicious software within these highly privileged management controllers. This could be useful, for example, to survive operating system reinstallation or communicate covertly with the attacker’s infrastructure, similar to the PLATINUM malware that used manageability features to bypass detection. Alternatively, this vulnerability could be used to “brick” (permanently disable) the BMC or the entire system, creating an impact even more severe than the BlackEnergy KillDisk component. Malicious software could prevent any further firmware updates to the BMC or simply corrupt the host firmware. In both of these scenarios, the only option to recover the system is to physically attach reprogramming hardware to the SPI chips on the motherboard and restore the original image.

Because IPMI communications can be performed over the BMC LAN interface, this update mechanism could also be exploited remotely if the attacker has been able to capture the ADMIN password for the BMC. This requires access to the systems management network, which should be isolated and protected from the production network. However, the implicit trust of management networks and interfaces may generate a false sense of security, leading to otherwise-diligent administrators practicing password reuse for convenience. In such cases, an attacker who has compromised one system could capture this password and use it to spread remotely to other systems on the management network.

Many server platforms support a dedicated IPMI LAN interface with a separate physical port rather than sharing a single physical port with the host operating system through the use of the Network Controller-Sideband Interface (NC-SI) mechanism. These systems allow the use of a physically separate “management” network such that IPMI LAN traffic is isolated from normal host network traffic. When this type of network architecture is being used, we can encounter situations where hosts are isolated from talking to each other through network firewalling to reduce their attack surface, but they’re all on the same BMC management network. In this case, once we have compromised the BMC, we can use this management network to attack and compromise other BMCs when the host-side network interfaces can’t directly communicate with each other.

About the BMC

Modern server platforms include a Baseboard Management Controller (BMC) for out-of-band management via Intelligent Platform Management Interface (IPMI). the BMC contains a separate CPU with its own firmware, which can remain accessible even when the host is powered off. This system component is designed to provide administrative access to the platform and is highly privileged, allowing administrators to remotely manage the firmware and OS of the host in the case of a failure or as part of regular maintenance. As a result, compromise of the BMC can undermine trust in the system regardless of any operating system protections installed on the host processor.

Some of the features Supermicro advertises for their BMC firmware are:

• IPMI 2.0 based management
• Hardware health monitor
• Remote power control
• Keyboard, Video & Mouse (KVM) Console Redirection with multi language support
• HTML5 web Console Redirection
• Media Redirection
• Web based configuration

The AST2400 and AST2500 are common BMC components used in many server platforms. They contain an ARM processor, graphics processor, and network hardware. In addition, they have their own SPI Flash and DRAM. This allows the BMC subsystem to operate independently from the main CPU(s) in the server. Most BMCs are then connected to a wide variety of system busses, including PCIe, USB, LPC, SMBUS and possibly a shared network adapter.

Vulnerabilities

We have discovered that the X8 through X11 generation Supermicro servers use insecure firmware update mechanisms for their BMC components. Using the existing update interface to the BMC, it is possible for host software to modify BMC firmware images and run arbitrary malicious code inside the BMC processor.

X8 and X9 BMC FW Update Process

The BMC firmware in both X8 and X9 generation systems uses a WPCM450 Boot Loader (WBL) from Winbond Limited, which takes just under 64kB at the beginning of the SPI flash image. The SPI flash also contains an “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

After the bootloader and this nvram section, each region in the firmware image is stored as a series of 32kB chunks where the end of the last chunk contains metadata about the region including load address and the name of the region, such as “1stFS”, “kernel”, and “2ndFS”. These correspond to two cramfs (rootfs and webfs) filesystems and a Linux kernel image which is zipped.

In the following screenshot, the 32kB chunk of this firmware image ending at offset 0x920000 contains a metadata footer with the magic value of 0xA0FFFF9F which indicates that this is a WBL-formatted image.

Further, the value of 0x40180000 indicates the “burn address” of this section of the file. For these types of images, the base of the flash region starts at 0x40000000 and the update tool uses this “burn address” minus the flash base address to determine how far into the file this region begins. The end of the region is calculated by simply taking the end of the chunk that contained the metadata footer. In this example, the start of this region is at offset 0x180000 and the end of the region is at 0x920000. The name for this region is “1stFS” and it contains the root filesystem for the Linux distribution running inside the BMC.

Using this information, we can carve the region out of the containing firmware image and extract the contents of the filesystem:

We can use the same process to extract the kernel region which contains a zipped Linux kernel:

Inside the kernel image, we found the default boot command line containing the mtdparts= argument which specifies how the flash regions are configured.

This flash layout declaration matches up with the 1stFS, kernel, and 2ndFS regions we discovered by scanning for the WBL (Winbond Boot Loader) magic value of 0xA0FFFF9F. In addition, it also shows us that the bootloader region starts at offset 0 with size 256kB and there’s an additional “nvram” region starting at offset 256kB with size 1280kB. This “nvram” region is used by the BMC operating system to store configuration settings, event logs, and other pieces of data that should persist across BMC resets.

The X9 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed. Note that the IPMI specification does not have a requirement for update authentication.

Additionally, no cryptographic signature verification is performed on the BMC firmware images before they are written to the SPI flash chip.

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, and replace the original filesystem sections in the firmware update image and flash these modified images to the X9 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X8DT3-LN4F using the X8DT3303.zip firmware image (latest available), X9DRi-LN4F+ using the SMT_X9_348.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X9-generation IPMI/BMC firmware images.

X10 BMC FW Update Process

In X10 generation systems, the BMC processor has been replaced with AST2x00 SoCs from ASPEED. The BMC firmware in these systems uses a U-Boot bootloader which takes up approximately 128kB at the beginning of the SPI flash image. The SPI flash also contains a “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

Like the X9 generation systems, the X10 BMC firmware images also contain three additional regions; two cramfs filesystems and a Linux kernel image. The BMC firmware image contains a u-boot header for the kernel image with a crc32 checksum as well as a text description of the flash regions containing start offset, size, crc32, and name for each region.

This example is from the REDFISH_X10_327.bin firmware image:

With a little formatting to make it easier to read:

[img]: 0 20fec cf74e74e u-boot.bin

[img]: 400000 d28000 ec25e35d out_rootfs_img.bin

[img]: 1400000 177620 64d09306 out_kernel.bin

[img]: 1700000 55f00a ed586d8c out_webfs_img.bin

[end]

The columns within each [img] tag are start offset, end offset, crc32, and section name.

The X10 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed.

CRC32 checksums are calculated and checked for each region in the flash image, but no cryptographic signature verification is performed before they are written to the SPI flash chip. The update process also checks two values near an “ATENs_FW” string, but this check does not provide any security protection. An attacker can replace sections of the firmware with malicious code because these checksums are not a security mechanism and only protect against accidental corruption.

Here’s what the ATENs_FW signature looks like in this firmware update image:

One thing that initially caused some confusion when analyzing this image was that the ec25d280qed5855f0 string looked very similar to the crc32 values from the [img] sections and we took a closer look to see how these sections are related. “ec25” are ascii hex values for the upper half of the crc32 from the out_rootfs_img.bin section, but “d280” is the the upper half of the *length* of the section rather than the lower 16-bits of the crc32. Likewise, ed58 and 55f0 are the upper halves of the crc32 and length from the out_rootfs_img.bin section.

When we unpacked the out_rootfs_img.bin cramsfs, one of the executable files we found was /bin/img_crc_check which checks these specific fields in the firmware image. However, the way that this check is performed can’t possibly work with this image.

Here’s the code from find_atens_signature() that is used to find the signature and extract the values:

And here’s the code from main() that uses those values:

However, there’s no hexdecimal conversion anywhere in this code so the length and checksum fields will be incorrect. As an example, the length for the rootfs will be 0x64323830 instead of 0xd28000. It appears that we’ve found a bug in the checksum generation for these images, but it turns out that this doesn’t matter.

When the BMC receives a new firmware image over the IPMI interface, the code that checks its signature doesn’t actually check the values of the checksum fields in this section before writing the received firmware image to the SPI flash, as shown in the following code:

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, replace the original filesystem sections in the firmware update image, recalculate the necessary CRC32 values in the [img] tags, and flash these modified images to the X10 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X10SLM-F system, the REDFISH_X10_327.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X10-generation IPMI/BMC firmware images.

X11 BMC FW Update Process

X11 generation systems continue the use of ASPEED AST2x00 BMC chips. However, with this generation, Supermicro made changes to the BMC firmware update file format. This appears to be an attempt to make it more difficult to extract and replace the BMC firmware for these systems.

When we first examined the X11 BMC firmware update images, we didn’t find the standard headers for embedded filesystems or the Linux kernel we expected, but we found many byte sequences that signified LZMA compressed data along with strings that looked like file and directory names:

This looked like a cramfs filesystem, except that we’d expect to see the standard signature bytes and other header fields at the beginning. Instead, we just had garbage. This seemed suspicious, so we bought an X11 motherboard so we could look at a real system.

Immediately after receiving the X11 platform, we hooked up a SPI programmer to physically read a copy of the SPI flash chip containing the BMC firmware before we even installed the CPU:

When we took a look at the dumped SPI flash image, we discovered that the live system contains the normal cramfs and Linux kernel headers precisely where we expected to find them.

After extracting the cramfs filesystem and examining libipmi.so, we found debugging messages and function symbols that made it clear what was going on:

j_console_log("[%s] img-size: %#x\n", "flash_decrypt_check", a2);
j_console_log("[%s] Flash encryption check ...\n", "flash_decrypt_check");
v3 = j_crypto_task1(v2, 2, 0x400000);
if ( v3 >= 0 )
{
v3 = j_crypto_task1(v2, 2, 0x1700000);
if ( v3 >= 0 )
{
v5 = j_crypto_task2(v2, 2);

A little more digging and we discovered hardcoded AES keys and IVs that were used to encrypt the first 96 or 192 bytes of each of the regions of the BMC firmware image we were interested in.

Because AES is a symmetric cipher, these keys can be used to decrypt an existing firmware update, modify the image, and then re-encrypt the image such that the X11 update mechanism will accept the modified image and apply it to the system.

These keys can easily be recovered by someone with physical access to an X11 system. We were able to extract the shipped firmware image from the system, find the keys, and successfully create a new firmware update image to connect back to our command and control server with a root shell running in the BMC in less than two hours of work. And this was all done before we installed the CPU in the new motherboard.

Because the keys are only used to obfuscate the header of the cramfs filesystem and not the contents of the compressed files, it’s also possible to find the compressed libipmi.so which contains the hardcoded keys and manually extract that from the firmware update image even without physical access to a system which contains the firmware image flashed in the clear.

Mitigation

The BMC firmware update mechanism must protect itself against potentially malicious updates including performing cryptographically secure signature verification before allowing new updates to be committed to the SPI flash. NIST has published Special Publication 800-193: Platform Firmware Resiliency Guidelines which describes important attributes for each component of a computer system. An Authenticated Update Mechanism is described in section 4.2.1.1 as follows:

One or more authenticated update mechanisms anchored in the RTU shall be the exclusive means for updating device firmware, absent unambiguous physical presence through a secure local update, as defined in Section 3.5.3. Authenticated update mechanisms shall meet the following authentication guidelines:

1. Firmware update images shall be signed using an approved digital signature algorithm as specified in FIPS 186-4 [7], Digital Signature Standard, with security strength of at least 112 bits in compliance with SP 800-57, Recommendation for Key Management – Part 1: General [8].
2. Each firmware update image shall be signed by an authorized entity – usually the device manufacturer, the platform manufacturer or a trusted third party – in conformance with SP 800-89, Recommendation for Obtaining Assurances for Digital Signature Applications [9].
3. The digital signature of a new or recovery firmware update image shall be verified by an RTU or a CTU prior to the non-volatile storage completion of the update process. For example, this might be accomplished by verifying the contents of the update in RAM and then performing an update to the active flash. In another example, it could also be accomplished by loading the update into a region of flash, verifying it, and then selecting that region of flash as the active region.

While this requirement does not seem to appear in the IPMI specification, more attention on the area should drive wider adoption. After we reported these issues to Supermicro, they have implemented cryptographic signature verification for the BMC firmware update process. This appears to create a Root of Trust for Update (RTU) as described above, which is an important security feature for devices, including the BMC. Supermicro recommends reaching out to their support contacts for more information about the authentication feature on current systems.

In addition, Supermicro’s update tools provide the ability to check the firmware version which is installed on a system as well as dump a copy of the current firmware which can be used to check for known vulnerable versions of the firmware or check the integrity of the installed firmware.

Conclusion

BMCs provide highly privileged management access to servers, but vulnerabilities in these management components can undermine all of the existing protections in the host operating system and applications which are running on the server. As with any security issue, continued attention is needed for improvement. Given the recent BMC issues discovered by other researchers as well as the vulnerabilities discussed in this post, it is clear that more attention is needed. We believe that better visibility into the integrity of critical components like the BMC is an important improvement for many organizations, and we’re working to make that happen.