Debug UEFI code by single-stepping your Coffee Lake-S hardware CPU

Original text by Teddy Reed V

In the post I will cover:

  • Configuring an ASRock H370M-ITX/ac to allow DCI DbC debugging
  • Using Intel System Studio and System Debugger to single-step a Coffee Lake-S i7-8700 CPU
  • Debugging an example exploitable UEFI application on hardware

USB DCI DbC Debugging (JTAG over USB3)

TL;DR, if you have a newer CPU & chipset you can purchase a $15 off-the-shelf cable and single-step your hardware threads. The cable is a USB 3.0 debugging cable; and is similar to an ethernet crossover cable in the sense that the internal wiring is crossed. Be careful with this cable as unsupported machines will have undefined behavior due to the electronics of USB.

Newer Intel CPUs support debugging over USB3 via a proprietary Direct Connection Interface (DCI) with the use of off-the-shelf hardware. This applies to some 6th-generation CPU and chipset combinations, and most 7th-generation and newer setups. I have not found the specific CPU/chipsec combinations but my educated guess from the Core series is as follows:

  • Kaby Lake / Intel 100 or 200 series SunrisePoint
  • Coffee Lake-S / Intel Z370, H370, H310, or B360
  • Kaby Lake R / 6th-gen Intel Core
  • Whiskey Lake-U (8565U, 8265U, 8145U)
  • Coffee Lake-S / H370, H310, B360

These combinations should support «DCI USB 3.x Debug Class» debugging. This means you only need the inexpensive debug cable linked above. Note that if debug-cable debugging is not support then a proprietary interposing device is required via a purchase from Intel.

From the documentation I’ve read, the USB3 hardware on a supported machine decodes DCI commands, forwards them to an appropriate hardware module on the target CPU that translates them to JTAG sequences. Intel provides a free-to-use, renewably-licenced, Intel System Studio and System Debugger software along with a DCI implementation called OpenDCI. This debugging environment is built with Eclipse and supported on macOS, Linux, and Windows. I’ve only found OpenDCI support for DbC-compatible targets on the Windows version.

You will need a Windows 10 install and Intel System Studio if you are following along.

Enable DCI on the ASRock H370M-ITX/ac

TL;DR you will need to enable and disable undocumented settings within UEFI by flipping several bits in a UEFI variable.

If you are doing casual research on DCI you will find several references to using a BIOS version with DCI enabled or using a UEFI debug build. I am sure they will be very helpful but it is not possible to acquire this in a general sense. However, we can still follow guidance on «modding» our UEFI to enable DCI. I found eiselekd’s DCI-enable guidance extremely helpful.

  1. Use chipsec to dump your SPI contents to disk. e.g., chipsec_util spi dump rom.bin
  2. Open rom.bin with UEFITool and extract GUID 899407D7-99FE-43D8-9A21-79EC328CAC21 (the Setup UEFI variable).
  3. Use IFRExtractor to print a textual representation of the variable options.

The variables settings required for the H370M-ITX/ac are as follows, tested on version 3.10 and 4.00 UEFI releases:

  • Enable/Disable IED (Intel Enhanced Debug): offset 0x960, set to enabled 0x1
  • CPU Run Control: offset 0x663, set to enabled 0x1
  • CPU Run Control Lock: offset 0x664, set to disabled 0x0
  • Platform Debug COnnect: offset 0x114F, set to 0x03 to enable DCI DbC
  • xDCI Support: offset 0xABD, set to enabled 0x1

To modify and save these offsets follow the guidance above to use the UEFI Shell and RU.efi application by James Wang.

You can confirm that DCI is enabled by reading the USB3 device class label when you connect the debug cable into your host and target machines. The host should have Intel System Studio installed and the target is the H370M-ITC/ac. The host USB driver will read «Intel USB Native Debug Class Devices» if DCI is enabled. If there is an error you will see «Port Reset Failed«. An easy way to view the detailed USB device information is with USB Tree View. Chipsec will also report if DCI is enabled but I found that DbC-specific availability is not reported; so use the USB device driver selection in Windows to confirm the UEFI options are set correctly.

Single-stepping the i7-8700

To recap the requirements and setup:

  • You have a host machine running Windows 10 with Intel System Studio installed
  • The host machine and target i7-8700/H370M-ITX/ac are connected via a USB3 DbC cabled
  • The host machine shows a connected «Intel USB Native Debug Class Device» USB device

Interrupt the target machine’s boot such that you enter UEFI Setup (press F2). This is not required but it will help while following along with the address space and other layout details. I have not figured out how to halt the CPU on reset with DCI and DbC.

In Intel System Studio you should open System Debugger and configure your target connection to use «8th Gen Intel Core Processors (Coffee Lake-S) _ Intel H370 Chipset Intel H310 Chipset Intel B360 Chipset for Consumer (Cannon Lake PCH)» using the connection method: «Intel(R) DCI USB 3.x Debug Class«

Upon success you will see status output similar to the following:

22:02:20 [INFO ] TCA - IPConnection: Open Connection, configuration: CFL_CNP_OpenDCI_DBC_Only_ReferenceSettings.
22:02:57 [INFO ] Starting DAL ...
22:02:57 [DAL  ] The system cannot find the batch label specified - SetScriptPath
22:02:58 [DAL  ] Registering MasterFrame...
22:03:00 [DAL  ] Using Intel DAL 1.1905.602.100 
22:03:00 [DAL  ] Using python.exe 2.7.15 (64bit), .NET 2.0.50727.8940, Python.NET 2.0.19, pyreadline 2.1.1
22:03:02 [DAL  ]     Note:    The 'coregroupsactive' control variable has been set to 'GPC'
22:03:10 [DAL  ] Using CFL_CNP_OpenDCI_DBC_Only_ReferenceSettings
22:03:10 [DAL  ] >>? DAL startup completed
22:03:10 [INFO ] Connection Manager: Status change: CONNECTED
    Connection: 8th Gen Intel Core Processors (Coffee Lake-S) _ Intel H370 Chipset Intel H310 Chipset Intel B360 Chipset for Consumer (Cannon Lake PCH)
    Target: 8th Gen Intel Core Processors (Coffee Lake-S) / Intel H370 Chipset, Intel H310 Chipset, Intel B360 Chipset for Consumer (Cannon Lake PCH)
    Connection Method: Intel(R) DCI USB 3.x Debug Class

And output similar to the following screen captures:

The connection will also pause the CPU threads and show you the nearby disassembly. If the CPU is not paused and clicking the «pause» button fails you have not enabled DCI completely. For example, if you encounter either, ExecutionControlUnableToHaltAllException, or operation not allowed while the processor is in state 'running' then double-check the UEFI Setup variable options.

A successful connection will show a UI similar to the following:

And you can now View and inspect memory as well as other common JTAG-debugging features.

Debugging an example exploitable UEFI application on hardware

TL;DR this is extremely simple and thus a great toy example, due to the lack of platform runtime security in UEFI and lack of build and compile security in the UEFI development kit (EDK/UDK).

The goal is to build a «toy» vulnerable UEFI application, trigger the exploitation, and observe the behavior within the System Debugger on the connected host. The first step is to configure the edk2 build environment. This is well-documented in several places.

I will modify the HelloWorld application and replace the MdeModulePkg/Application/HelloWorld/HelloWorld.c with the following content.

#include <Uefi.h>
#include <Library/UefiLib.h>
#include <Library/UefiApplicationEntryPoint.h>

#include <Protocol/LoadedImage.h>
#include <Library/UefiBootServicesTableLib.h>
#include <Library/MemoryAllocationLib.h>

VOID RunAsm();

CHAR16* GetArgv(IN EFI_HANDLE ImageHandle)
{
  EFI_LOADED_IMAGE* li;
  EFI_GUID loaded_image_protocol = LOADED_IMAGE_PROTOCOL;
  gBS->HandleProtocol(ImageHandle, &loaded_image_protocol, (void**) &li);

  CHAR16* wargv = (CHAR16 *)li->LoadOptions;
  return wargv;
}

VOID RunMe()
{
  Print(L"You win\n");
  RunAsm();
}

UINT32 StrLenChar(CHAR8* src) {
  UINT32 ret = 0;
  while (src[ret++] != 0) {}
  return ret - 1;
}

VOID StrCpy(CHAR8* dst, CHAR16* src, UINT32 length) {
  CHAR8 *src8 = (CHAR8*)src;
  for (UINT32 i = 0; i < length; i++) {
    dst[i] = src8[(i*2)];
  }

  UINT64 loc = (UINT64)&RunMe;
  dst[length - 1] = 0;
  dst[length - 2] = 0;
  dst[length - 3] = 0;
  dst[length - 4] = 0;
  dst[length - 5] = ((loc >> (8 * 3)) & 0xFF);
  dst[length - 6] = ((loc >> (8 * 2)) & 0xFF);
  dst[length - 7] = ((loc >> (8 * 1)) & 0xFF);
  dst[length - 8] = ((loc >> (8 * 0)) & 0xFF);
}

 __attribute__((noinline)) VOID
 TestBufferOverflow(CHAR16* input)
 {
  /* Test stack buffer overflow */

  // Compiled with EDKII that auto-adds (-fno-stack-protector)
  CHAR8 buffer[32];
  StrCpy((CHAR8*)buffer, input, StrLen(input));
  buffer[StrLen(input)] = 0;
}

EFI_STATUS EFIAPI UefiMain (
  IN EFI_HANDLE        ImageHandle,
  IN EFI_SYSTEM_TABLE  *SystemTable
) {
  // Run with: fs0:X64\HelloWorld.efi A*222

  Print(L"UefiMain=0x%p\n", &UefiMain);
  CHAR16* wargv = GetArgv(ImageHandle);
  UINT32 wargv_len = StrLen(wargv);
  TestBufferOverflow(wargv);

  return EFI_SUCCESS;
}

The specific build command is

$ . ./edksetup.sh BaseTools
$ build -m MdeModulePkg/Application/HelloWorld/HelloWorld.inf -p MdeModulePkg/MdeModulePkg.dsc

And if you would like to test that this runs follow the QEMU debugging guide and use:

$ qemu-system-x86_64 -bios /usr/share/OVMF/OVMF_PURE_EFI.fd -display none -nodefaults -serial stdio -hda fat:Build/MdeModule/DEBUG_GCC5

The code above is a sythethetic stack-based buffer overflow example. It will auto-fill in the overwritten ret address for you. If you want to learn what is happening here please read Dhaval’s articles on Buffer Overflows. As a note, we could choose to make this more realistic (e.g., remove the auto-filled ret) by reading a file into the vulnerable stack variable.

The default edk2 build configuration will compile the overflow into the following flow, where the StrCpy logic is inlined:

Our goal is to copy 0x30 characters into the buffer, overflowing the expected 0x20, the 8 for the saved RBX, and 16 for RSP and RIP; at which point the final 8 will be filled in with the address of RunMe.

For some fast feedback we’ll print to ConsoleOut then reset the CPU using:

ASM_GLOBAL ASM_PFX(RunAsm)
ASM_PFX(RunAsm):
    mov $254, %al
    out %al, $100
    ret

If a console is not available then this functions well for blind-testing control of rip.

Because we are printing the location of UefiMain we can both confirm that each time the application is executed the address is constant and know what location to set a hardware breakpoint in System Debugger so we can single-step and watch the overflow.

For my UEFI build this location was 0x600BC69C, which means the .text is loaded to an offset of 0x600BB000 as this subroutine is 0x169C. From here we can add more breakpoints in System Debugger.

Реклама

Debugging UCSI firmware failures

( Original text by Rajib Dutta )

Background

The UCSI driver in Windows communicates with the firmware UCSI component (called PPM or Platform Policy Manager) through the spec-defined command notification interfaces. While driver failures can be tracked using Windows Error Reporting or WER and driver traces, the firmware may run into failures as well which might go undetected at first but result into loss of functionality later in time. UCSI compliant machines in the wild do run into such failures every day in the tens of thousands.
This document intends to describe a way in which an OEM or a firmware developer may avoid these PPM failures pro-actively running some stress tests.

  1. The most common PPM failure is UCSI command timeout.
  2. UcmUcsiCx.sys or UcmUcsi.sys is the inbox component in Windows that communicates with PPM. Here is a typical message flow between the driver and the firmware.
  3. UcmUcsiCx.sys/UcmUcsi.sys driver sends a UCSI command to PPM
    PPM asynchronously sends command complete notification (CCI:: Command Complete Indicator*).
  4. The driver receives the command complete notification and then send Ack to PPM (ACK_CC_CCI)
    PPM asynchronously send ack complete notification (CCI::Acknowledge Command Indicator).
    The driver waits for 10 seconds for an asynchronous notification (steps (2) and (4)) before giving up and generating failure report or a livedump.

*Note that the above steps are performed for all UCSI commands except PPM_RESET since after PPM_RESET the PPM Is expected to be in reset state where no notifications are enabled. The driver then polls for CCI::Reset Complete Indicator to be true.

 

Converting firmware failures to bugchecks

One way in which an OEM or a firmware developer can catch these failures in the lab or development environment is the configure the machine to bugcheck generating a kernel crashdump rather than  livedump which may go undetected. In addition to repro the failure, we advise an OEM/firmware developer to use Connection Exerciser Stress test: CxStress. CxStress is a part of MUTT software pachage found in this location.

  • Setup Connection exerciser: An example connection is shown in the following picture where the Type-C connector of the SUT is connected to the connection exerciser. The 4 Type-C ports of the connection exerciser is connected to 4 different kinds of the partners.ucsi setup.png
  • Configure UcmUcsiCx.sys to elevate livedump to crashdump
    • Locate the device node in Device Manager (devmgmt.msc) named UCM-UCSI ACPI Device. The node is under the USB Connector Manager
    • Right-click on the device, and select Properties and open the Details
    • Select Device Instance Path from the drop-down and note the property value.
    • On an elevated command prompt
      • reg add «HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\<device-instance-path>\Device Parameters» /v ForceCommandFailureCrash /t REG_DWORD /d 0x01
    • Restart the device by selecting the Disable option on the device node in Device Manager, and then selecting Enable. Alternatively, you can simply restart the PC.
  • Run CxStess.cmd (typically found in “C:\Program Files (x86)\USBTest\x64”). For completeness, we recommend you to run this test overnight (~8 hours).
  • Keep an eye on bugchecks that might happen due to the above test. The next section talks about how to go about finding their root cause.

Analyzing Firmware Failures

This section goes through analysis of an example crash that occurred when UcmUcsiCx driver was configured to generate a crashdump instead of a livedump.

Run !analyze to check the failure type. Bugcheck code: 0x1D4 is the UCSI failure code.

 

1: kd> !analyze -v

UCMUCSI_FAILURE (1d4)
The UcmUcsi driver has encountered an error.
Arguments:
Arg1: 0000000000000000, A UCSI command has timed out because the firmware did not respond to the command in time.
Arg2: 0000000000000005, The UCSI command value.
Arg3: ffff988646b2f8d0, If non-zero, the pointer to additional information (dt UcmUcsiCx!UCMUCSICX_TRIAGE).
Arg4: 0000000000000000

Debugging Details:
..

 

 

Get UcmUcsiCx driver traces. The following text contains the last few lines of the trace which we are interested in. The last few entries in the traces are related to the driver trying to stop the state machine and generating a dump. Look for the state machine event called CommandTimeOut and retrace the state machine transitions back a few steps to find out which command has timed out. In this specific example, it happens to be SetNotificationEnable UCSI command. Refer line 1076, 1083 and 1085 in the following WPP trace dump from Windbg.

1: kd> !rcdrkd.rcdrlogdump ucmucsicx
Trace searchpath is: 

Trace format prefix is: %7!u!: %!FUNC! - 
Trying to extract TMF information from - C:\ProgramData\Dbg\sym\UcmUcsiCx.pdb\DA9957412F663B283F7EF8BE9406D0551\UcmUcsiCx.pdb
--- start of log ---
…
…
…
1073: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: External][Source: WaitingForCommand][Event : CommandAvailable][Target: ProcessNextCommand]
1074: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: Call][Source: ProcessNextCommand][Event : _noevent_][Target: RetreiveCommandType]
1075: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: External][Source: RetreiveCommandType][Event : AsyncCommand][Target: SendNewCommandToPPM]
1076: UcmUcsiCx::CommandHandler::SendUCSICommandToPpm - [UCMUCSIPPM: 0x0000507062114698] Sending Command UcsiCommandSetNotificationEnable to client


1077: UcmUcsiCx::CommandHandler::EvtLogEventEnqueue - [COMMAND HANDLER] SmFx event: RequestCompletedSuccessfully
1078: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: Call][Source: SendNewCommandToPPM][Event : _noevent_][Target: WaitingForRequestCompletion]
1079: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: ExplicitPop][Source: WaitingForRequestCompletion][Event : RequestCompletedSuccessfully][Target: SendNewCommandToPPM]
1080: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: External][Source: SendNewCommandToPPM][Event : Succeeded][Target: StartingCommandCompleteTimer]
1081: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: External][Source: StartingCommandCompleteTimer][Event : Done][Target: WaitingForCommandComplete]
1082: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: Call][Source: EnableCmdCompleteNotificationOnPowerUp][Event : _noevent_][Target: WaitingForCommandCompletion]
1083: UcmUcsiCx::CommandHandler::PrettyPrintCCI - [CCI] ConnChange:0x0 | DataLength:0x0 | NotSupported:0 | CancelCompleted:0 | ResetCompleted:1 | Busy:0 | AckCommand:0 | Error:0 | CommandComplete:0 |
1084: UcmUcsiCx::CxDevice::EvtDeviceWdmPreprocessSetPowerIrp - [WDFDEVICE: 0x000050706D0133C8] System Power Down to SYSTEM_POWER_STATE:0x5
1085: UcmUcsiCx::CommandHandler::EvtLogEventEnqueue - [COMMAND HANDLER] SmFx event: CommandTimedOut
1086: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: ExplicitPop][Source: WaitingForCommandComplete][Event : CommandTimedOut][Target: ProcessNextCommand]
1087: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: ImplicitPop][Source: ProcessNextCommand][Event : UnrecoverableFailure][Target: Started]
1088: UcmUcsiCx::CommandHandler::EvtLogTransition - [COMMAND HANDLER] SmFx transition [Transition: External][Source: Started][Event : UnrecoverableFailure][Target: Failed]
1089: UcmUcsiCx::Opm::EvtCommandCompletionReceived - [UCMUCSIPPM: 0x0000507062114698] Response timed out for command UcsiCommandSetNotificationEnable, 0x00000102(STATUS_TIMEOUT)
1090: UcmUcsiCx::Opm::EvtLogEventEnqueue - [OPM] SmFx event: Stop
1091: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: External][Source: WaitingForCommandCompletion][Event : Stop][Target: WaitingForCommandCompletionAfterStop]
1092: UcmUcsiCx::Opm::EvtLogEventEnqueue - [OPM] SmFx event: CommandComplete
1093: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: ExplicitPop][Source: WaitingForCommandCompletionAfterStop][Event : CommandComplete][Target: EnableCmdCompleteNotificationOnPowerUp]
1094: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: ImplicitPop][Source: EnableCmdCompleteNotificationOnPowerUp][Event : Stop][Target: PowerDownAndWaitForPowerUp]
1095: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: ImplicitPop][Source: PowerDownAndWaitForPowerUp][Event : Stop][Target: PpmOperational]
1096: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: ExplicitPop][Source: PpmOperational][Event : Stop][Target: Started]
1097: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: External][Source: Started][Event : Stop][Target: PurgingUnserviceCommandsDueToStop]
1098: UcmUcsiCx::Opm::EvtLogTransition - [OPM] SmFx transition [Transition: External][Source: PurgingUnserviceCommandsDueToStop][Event : Done][Target: StopCommandHandlerMachine]
1099: UcmUcsiCx::CommandHandler::EvtLogEventEnqueue - [COMMAND HANDLER] SmFx event: Stop
1100: UcmUcsiCx::CxDevice::UnrecoverableFailureEncountered - [WDFDEVICE: 0x000050706D0133C8] An unrecoverable failure has been encountered. The device will now restart.
---- end of log ----

To summarize the above failure, UcmUcsiCx driver sends SetNoficationEnable command to the firmware and the firmware fails to send command complete notification within the timeout period of 10 seconds. Instead it sends a Reset Complete notification.

When such a failure occurs, we advise an OEM with work with firmware developer to fix the problem.

Insecure Firmware Updates in Server Management Systems

( Original text by Eclypsium )

Supermicro BMC Case Study

Baseboard Management Controllers (BMCs) are high-value targets to attackers, who can use the out-of-band management features of the BMC to compromise servers even below the level of the host OS and system firmware. BMCs are currently an area of active research for both attackers and defenders. A compromised BMC can be used by malware to provide persistence that lasts beyond a complete wipe and reinstall of the operating system as well as the ability to cross security domains by moving laterally from host-side networks to management networks which are intended to be isolated. In addition to our research, multiple teams have discovered vulnerabilities in other BMCs, such as the recent HPE iLO4 Authentication Bypass and RCEpublication and the The Unbearable Lightness of BMC’s talk at Black Hat. Even back in 2013, Dan Farmer and HD Moore (penetration tester’s guideSupermicro IPMI) published about serious BMC vulnerabilities. Given the inherent privilege of the BMC in server architecture, a compromised BMC amounts to a significant compromise of the system.

Our research has uncovered vulnerabilities in the way that multiple vendors update their BMC firmware. These vendors typically leverage standard, off-the-shelf IPMI management tools instead of developing customized in-house management capabilities. In this case, we will go deep into the BMC update process on Supermicro systems, we found that the BMC code responsible for processing and applying firmware updates does not perform cryptographic signature verification on the provided firmware image before accepting the update and committing it to non-volatile storage. This effectively allows the attacker to load modified code onto the BMC.

After informing Supermicro of these issues, they have been working with us to improve the security of BMC firmware updates using cryptographic signatures. They have informed us that all new Supermicro products will be released with this feature, and customers who wish to upgrade existing X10 or X11 systems can contact Supermicro (details on Supermicro’s cryptographic signature page). Unfortunately, at the time of publication, updates that address this critical vulnerability are not publicly available for direct download. Also, since Supermicro’s tools do allow administrators to retrieve the firmware version and a dump of the image, Supermicro customers can check that the latest updates are installed and verify the integrity of BMC firmware from critical systems.

Impact

Using the vulnerabilities we discovered, it is possible to make arbitrary modifications to the BMC code and data. Using these modifications, an attacker can run malicious software within these highly privileged management controllers. This could be useful, for example, to survive operating system reinstallation or communicate covertly with the attacker’s infrastructure, similar to the PLATINUM malware that used manageability features to bypass detection. Alternatively, this vulnerability could be used to “brick” (permanently disable) the BMC or the entire system, creating an impact even more severe than the BlackEnergy KillDisk component. Malicious software could prevent any further firmware updates to the BMC or simply corrupt the host firmware. In both of these scenarios, the only option to recover the system is to physically attach reprogramming hardware to the SPI chips on the motherboard and restore the original image.

Because IPMI communications can be performed over the BMC LAN interface, this update mechanism could also be exploited remotely if the attacker has been able to capture the ADMIN password for the BMC. This requires access to the systems management network, which should be isolated and protected from the production network. However, the implicit trust of management networks and interfaces may generate a false sense of security, leading to otherwise-diligent administrators practicing password reuse for convenience. In such cases, an attacker who has compromised one system could capture this password and use it to spread remotely to other systems on the management network.

Many server platforms support a dedicated IPMI LAN interface with a separate physical port rather than sharing a single physical port with the host operating system through the use of the Network Controller-Sideband Interface (NC-SI) mechanism. These systems allow the use of a physically separate “management” network such that IPMI LAN traffic is isolated from normal host network traffic. When this type of network architecture is being used, we can encounter situations where hosts are isolated from talking to each other through network firewalling to reduce their attack surface, but they’re all on the same BMC management network. In this case, once we have compromised the BMC, we can use this management network to attack and compromise other BMCs when the host-side network interfaces can’t directly communicate with each other.

About the BMC

Modern server platforms include a Baseboard Management Controller (BMC) for out-of-band management via Intelligent Platform Management Interface (IPMI). the BMC contains a separate CPU with its own firmware, which can remain accessible even when the host is powered off. This system component is designed to provide administrative access to the platform and is highly privileged, allowing administrators to remotely manage the firmware and OS of the host in the case of a failure or as part of regular maintenance. As a result, compromise of the BMC can undermine trust in the system regardless of any operating system protections installed on the host processor.

Some of the features Supermicro advertises for their BMC firmware are:

  • IPMI 2.0 based management
  • Hardware health monitor
  • Remote power control
  • Keyboard, Video & Mouse (KVM) Console Redirection with multi language support
  • HTML5 web Console Redirection
  • Media Redirection
  • Web based configuration

The AST2400 and AST2500 are common BMC components used in many server platforms. They contain an ARM processor, graphics processor, and network hardware. In addition, they have their own SPI Flash and DRAM. This allows the BMC subsystem to operate independently from the main CPU(s) in the server. Most BMCs are then connected to a wide variety of system busses, including PCIe, USB, LPC, SMBUS and possibly a shared network adapter.

Vulnerabilities

We have discovered that the X8 through X11 generation Supermicro servers use insecure firmware update mechanisms for their BMC components. Using the existing update interface to the BMC, it is possible for host software to modify BMC firmware images and run arbitrary malicious code inside the BMC processor.

X8 and X9 BMC FW Update Process

The BMC firmware in both X8 and X9 generation systems uses a WPCM450 Boot Loader (WBL) from Winbond Limited, which takes just under 64kB at the beginning of the SPI flash image. The SPI flash also contains an “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

After the bootloader and this nvram section, each region in the firmware image is stored as a series of 32kB chunks where the end of the last chunk contains metadata about the region including load address and the name of the region, such as “1stFS”, “kernel”, and “2ndFS”. These correspond to two cramfs (rootfs and webfs) filesystems and a Linux kernel image which is zipped.

In the following screenshot, the 32kB chunk of this firmware image ending at offset 0x920000 contains a metadata footer with the magic value of 0xA0FFFF9F which indicates that this is a WBL-formatted image.

Further, the value of 0x40180000 indicates the “burn address” of this section of the file. For these types of images, the base of the flash region starts at 0x40000000 and the update tool uses this “burn address” minus the flash base address to determine how far into the file this region begins. The end of the region is calculated by simply taking the end of the chunk that contained the metadata footer. In this example, the start of this region is at offset 0x180000 and the end of the region is at 0x920000. The name for this region is “1stFS” and it contains the root filesystem for the Linux distribution running inside the BMC.

Using this information, we can carve the region out of the containing firmware image and extract the contents of the filesystem:

We can use the same process to extract the kernel region which contains a zipped Linux kernel:

Inside the kernel image, we found the default boot command line containing the mtdparts= argument which specifies how the flash regions are configured.

This flash layout declaration matches up with the 1stFS, kernel, and 2ndFS regions we discovered by scanning for the WBL (Winbond Boot Loader) magic value of 0xA0FFFF9F. In addition, it also shows us that the bootloader region starts at offset 0 with size 256kB and there’s an additional “nvram” region starting at offset 256kB with size 1280kB. This “nvram” region is used by the BMC operating system to store configuration settings, event logs, and other pieces of data that should persist across BMC resets.

The X9 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed. Note that the IPMI specification does not have a requirement for update authentication.

Additionally, no cryptographic signature verification is performed on the BMC firmware images before they are written to the SPI flash chip.

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, and replace the original filesystem sections in the firmware update image and flash these modified images to the X9 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X8DT3-LN4F using the X8DT3303.zip firmware image (latest available), X9DRi-LN4F+ using the SMT_X9_348.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X9-generation IPMI/BMC firmware images.

X10 BMC FW Update Process

In X10 generation systems, the BMC processor has been replaced with AST2x00 SoCs from ASPEED. The BMC firmware in these systems uses a U-Boot bootloader which takes up approximately 128kB at the beginning of the SPI flash image. The SPI flash also contains a “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

Like the X9 generation systems, the X10 BMC firmware images also contain three additional regions; two cramfs filesystems and a Linux kernel image. The BMC firmware image contains a u-boot header for the kernel image with a crc32 checksum as well as a text description of the flash regions containing start offset, size, crc32, and name for each region.

This example is from the REDFISH_X10_327.bin firmware image:

With a little formatting to make it easier to read:

[img]: 0 20fec cf74e74e u-boot.bin

[img]: 400000 d28000 ec25e35d out_rootfs_img.bin

[img]: 1400000 177620 64d09306 out_kernel.bin

[img]: 1700000 55f00a ed586d8c out_webfs_img.bin

[end]

The columns within each [img] tag are start offset, end offset, crc32, and section name.

The X10 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed.

CRC32 checksums are calculated and checked for each region in the flash image, but no cryptographic signature verification is performed before they are written to the SPI flash chip. The update process also checks two values near an “ATENs_FW” string, but this check does not provide any security protection. An attacker can replace sections of the firmware with malicious code because these checksums are not a security mechanism and only protect against accidental corruption.

Here’s what the ATENs_FW signature looks like in this firmware update image:

One thing that initially caused some confusion when analyzing this image was that the ec25d280qed5855f0 string looked very similar to the crc32 values from the [img] sections and we took a closer look to see how these sections are related. “ec25” are ascii hex values for the upper half of the crc32 from the out_rootfs_img.bin section, but “d280” is the the upper half of the *length* of the section rather than the lower 16-bits of the crc32. Likewise, ed58 and 55f0 are the upper halves of the crc32 and length from the out_rootfs_img.bin section.

When we unpacked the out_rootfs_img.bin cramsfs, one of the executable files we found was /bin/img_crc_check which checks these specific fields in the firmware image. However, the way that this check is performed can’t possibly work with this image.

Here’s the code from find_atens_signature() that is used to find the signature and extract the values:

And here’s the code from main() that uses those values:

However, there’s no hexdecimal conversion anywhere in this code so the length and checksum fields will be incorrect. As an example, the length for the rootfs will be 0x64323830 instead of 0xd28000. It appears that we’ve found a bug in the checksum generation for these images, but it turns out that this doesn’t matter.

When the BMC receives a new firmware image over the IPMI interface, the code that checks its signature doesn’t actually check the values of the checksum fields in this section before writing the received firmware image to the SPI flash, as shown in the following code:

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, replace the original filesystem sections in the firmware update image, recalculate the necessary CRC32 values in the [img] tags, and flash these modified images to the X10 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X10SLM-F system, the REDFISH_X10_327.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X10-generation IPMI/BMC firmware images.

X11 BMC FW Update Process

X11 generation systems continue the use of ASPEED AST2x00 BMC chips. However, with this generation, Supermicro made changes to the BMC firmware update file format. This appears to be an attempt to make it more difficult to extract and replace the BMC firmware for these systems.

When we first examined the X11 BMC firmware update images, we didn’t find the standard headers for embedded filesystems or the Linux kernel we expected, but we found many byte sequences that signified LZMA compressed data along with strings that looked like file and directory names:

This looked like a cramfs filesystem, except that we’d expect to see the standard signature bytes and other header fields at the beginning. Instead, we just had garbage. This seemed suspicious, so we bought an X11 motherboard so we could look at a real system.

Immediately after receiving the X11 platform, we hooked up a SPI programmer to physically read a copy of the SPI flash chip containing the BMC firmware before we even installed the CPU:

When we took a look at the dumped SPI flash image, we discovered that the live system contains the normal cramfs and Linux kernel headers precisely where we expected to find them.

After extracting the cramfs filesystem and examining libipmi.so, we found debugging messages and function symbols that made it clear what was going on:

j_console_log("[%s] img-size: %#x\n", "flash_decrypt_check", a2);
j_console_log("[%s] Flash encryption check ...\n", "flash_decrypt_check");
v3 = j_crypto_task1(v2, 2, 0x400000);
if ( v3 >= 0 )
{
v3 = j_crypto_task1(v2, 2, 0x1700000);
if ( v3 >= 0 )
{
v5 = j_crypto_task2(v2, 2);

A little more digging and we discovered hardcoded AES keys and IVs that were used to encrypt the first 96 or 192 bytes of each of the regions of the BMC firmware image we were interested in.

Because AES is a symmetric cipher, these keys can be used to decrypt an existing firmware update, modify the image, and then re-encrypt the image such that the X11 update mechanism will accept the modified image and apply it to the system.

These keys can easily be recovered by someone with physical access to an X11 system. We were able to extract the shipped firmware image from the system, find the keys, and successfully create a new firmware update image to connect back to our command and control server with a root shell running in the BMC in less than two hours of work. And this was all done before we installed the CPU in the new motherboard.

Because the keys are only used to obfuscate the header of the cramfs filesystem and not the contents of the compressed files, it’s also possible to find the compressed libipmi.so which contains the hardcoded keys and manually extract that from the firmware update image even without physical access to a system which contains the firmware image flashed in the clear.

Mitigation

The BMC firmware update mechanism must protect itself against potentially malicious updates including performing cryptographically secure signature verification before allowing new updates to be committed to the SPI flash. NIST has published Special Publication 800-193: Platform Firmware Resiliency Guidelines which describes important attributes for each component of a computer system. An Authenticated Update Mechanism is described in section 4.2.1.1 as follows:

One or more authenticated update mechanisms anchored in the RTU shall be the exclusive means for updating device firmware, absent unambiguous physical presence through a secure local update, as defined in Section 3.5.3. Authenticated update mechanisms shall meet the following authentication guidelines:

  1. Firmware update images shall be signed using an approved digital signature algorithm as specified in FIPS 186-4 [7], Digital Signature Standard, with security strength of at least 112 bits in compliance with SP 800-57, Recommendation for Key Management – Part 1: General [8].
  2. Each firmware update image shall be signed by an authorized entity – usually the device manufacturer, the platform manufacturer or a trusted third party – in conformance with SP 800-89, Recommendation for Obtaining Assurances for Digital Signature Applications [9].
  3. The digital signature of a new or recovery firmware update image shall be verified by an RTU or a CTU prior to the non-volatile storage completion of the update process. For example, this might be accomplished by verifying the contents of the update in RAM and then performing an update to the active flash. In another example, it could also be accomplished by loading the update into a region of flash, verifying it, and then selecting that region of flash as the active region.

While this requirement does not seem to appear in the IPMI specification, more attention on the area should drive wider adoption. After we reported these issues to Supermicro, they have implemented cryptographic signature verification for the BMC firmware update process. This appears to create a Root of Trust for Update (RTU) as described above, which is an important security feature for devices, including the BMC. Supermicro recommends reaching out to their support contacts for more information about the authentication feature on current systems.

In addition, Supermicro’s update tools provide the ability to check the firmware version which is installed on a system as well as dump a copy of the current firmware which can be used to check for known vulnerable versions of the firmware or check the integrity of the installed firmware.

Conclusion

BMCs provide highly privileged management access to servers, but vulnerabilities in these management components can undermine all of the existing protections in the host operating system and applications which are running on the server. As with any security issue, continued attention is needed for improvement. Given the recent BMC issues discovered by other researchers as well as the vulnerabilities discussed in this post, it is clear that more attention is needed. We believe that better visibility into the integrity of critical components like the BMC is an important improvement for many organizations, and we’re working to make that happen.

Brief reverse engineering work on FIMI A3

( Original text by Konrad Iturbe )

This is the start of a new series on reverse engineering consumer products, mainly to enhance their use but also to expose data leaks and vulnerabilities.

Something caught my eye last week. Xiaomi-backed FIMI, a Shenzhen company, released a drone. I tend to avoid most cheap drones since they tend to suck (bad quality camera, bad UX…) but this one is different.

This drone has a “DIY” port. This is an UART/PWM/GPIO port with what I assume are two ports for power. FIMI showcases how the user can attach a fireworks igniter or LEDs, but my mind went instantly to this DEF CON presentation from this year. The project is about a drone which is difficult to intercept but the author of the presentation also shows some offensive uses of drones. This drone can theoretically have a WiFi jammer or a promiscuous WiFi packet sniffer which can be activated from the ground. Possibilities are endless when you have this sort of port. The A3 also does not need a smartphone for operation, it includes a remote controller with an LCD panel. Having a smartphone controlling the drone opens a new vector for attackers (wifi network between the remote controller and cellphone can be brute forced, phone can have malware…). DJI does not yet have a all-in-one remote controller but Xiaomi has outsmarted them. Chinese innovation at its finest.

DIY port. Time to attach a RPI Zero with a pinneapple.
Some payloads that can be attached to drones. Source: David Melendez’s DEF CON talk PDF

On the camera side for those interested this drone is rocking the AMBA A12 chipset with a 1440p Sony CMOS sensor. Takes 8MP stills and can record 1080p video at 30FPS with a bitrate of 60 MB/s. The gimbal is 2 axis, just like the DJI Spark but at ~$250 this drone is a worthy competitor of the DJI Spark. If it only shot 4K video and had 3 axis gimbal, but we can’t have everything in life.


Part 1: The firmware(s):

FIMI makes the firmware available for downloading to anyone here.

The firmware is split into 3: The AMBA A12 firmware, the Drone Cortex A7 firmware and the remote controller firmware.

wget https://www.fimi.com/media/Productattachments//f/2/f21a-a-v010sp12rtm181027r16987-cn-rtm_u-release-741c119eb4d25878e21045e3f3c485d4.zip -P drone_fw/

wget https://www.fimi.com/media/Productattachments//f/i/firmware.zip -P cam_fw/

wget https://www.fimi.com/media/Productattachments//r/2/r21a-a-v010sp13rc181024r16900-cn-b_250k-release-ota-97b6c6c59241976086fabdc41472150c.zip -P remotecontrol_fw/

The firmwares are highly compressed. The filesize of each one is:

3.8M firmware.zip

488K f21a-a-v010sp12rtm181027r16987-cn-rtm_u-release-741c119eb4d25878e21045e3f3c485d4.zip

728K r21a-a-v010sp13rc181024r16900-cn-b_250k-release-ota-97b6c6c59241976086fabdc41472150c.zip

First step is to decompress each firmware zip file. After the decompression is done the remote firmware yields a 1.2M BFU file, the drone firmware is now a 492K BIN file and the camera firmware, which contains code relevant to the AMBA ISP yields 3 files: a 3.8M firmware.bin and two 0-byte files: rollback.txt and update.txt. Looking deeper at the firmware.bin file using binwalk 3 files are compressed: amba_ssp_svc.bin, dsp.bin.gz, rom.bin.gz

amba_ssp_svc.bin is a gz file, so the name should be amba_ssp_svc.bin.gz

Using extract_fw.sh we can get the gz files and their contents:

firmware.bin SHA512: 1cba74305d0491b957f1805c84e9b1cf5674002fc4f0de26905a16fb40303376187f1c35085b7455bff5c4de23cf8faa9479e4f78fd50dbf69947deb27f5d687

From here I used dd to extract the files. The “skip” flag is the location and the count is the next location — location.

dd if=[firmware.bin] of=out/amba_ssp_svc.bin.gz bs=1 skip=508 count=1812019
dd if=[firmware.bin] of=out/dsp.bin.gz bs=1 skip=1812527 count=1988127
dd if=[firmware.bin] of=out/rom.bin.gz bs=1 skip=3800654 count=143664

NOTE: All the files needed are on my github repository.

Now there is something to work with. Extrac each file with gunzip and we end up with:

4.3M amba_ssp_svc.bin
4.9M dsp.bin
2.3M rom.bin

Part 2: Ambarella chipset:

This is as far as we get. The 4.3M file is the AMBA chipset firmware. From here we can use some of the methods I used in reverse engineering GoPro camera firmwares:

strings amba_ssp_svc.bin | grep “c:\\\\”

The aim of this reverse engineering work is to:

  • See if we can flash our own images onto the drone
  • See what kind of resolutions and frame rates are available
  • See if we can run custom commands or get a telnet/RTOS session
  • Disable NFZ (No Fly Zones) and enable FCC (US 5GHz mode) in Europe
  • Can we root the drone?

It appears we can flash images from the SD card to the drone:

C:\version.txt
C:\update.txt
C:\rollback.txt
C:\firmware.bin

As per the resolutions, see cam_fw/out/README.md in the GitHub repository.

The AMBA A12 chipset accepts some commands, including what appears to be RTOS USB Shell.

The drone firmware and controller firmware don’t have such ways of getting into. The drone firmware is a bin file with no sections and the remote controller firmware is a bfu file.

It’d be interesting if the drone can fly on offline waypoints. Attaching a WiFi sniffer just got a whole lot easier.

I ordered this drone but it won’t arrive soon since it’s a pre-order.

Stay tuned for part 2!

GitHub repository: https://github.com/KonradIT/fimi_a3

Building the rCrumbl the Ultimate RaspberryPi Phone

( Original text by  Steward’s Notes )

About a year ago I set out to create a functional smartphone from a Raspberry Pi. Its been a fun adventure. I began this project with only a moderate amount of experience in working with electronics, and I’ve come a long way since that time. I am by no means the first person to create a raspberry pi phone there are one or two people who I am certain have come before me. In order to claim some sort of title for the work that I would be doing, I decided that I would attempt to create the smallest form factor phone possible given my knowledge and experience. This alone became quite the challege, but in the process I learned a great deal about product design, CAD, and 3D printing.

DSCF7527

A humble beginning

The majority of the electronic parts were purchased from Adafruit and Elmwood Electronics. I based the design of my phone off of the TyTelli that is several years old now. Unfortunately there are no wireless 2g networks currently operational here in Canada, so I will need to do a little bit more work on the software side to create a fully featured phone. In order to keep a “slim” form factor I needed to design some solutions to keep components from moving around to ensure that wires remained isolated from each other. In order to do this, I used a couple of JST connectors to allow the bottom half of the phone to separate from the top. I also created a small bracket to attach all of the extra circuit boards and camera. All that was left to complete was the wiring.

DSCF8009

On the left circuitboards are attached to a bracket and then screwed onto the bottom of a RaspberryPi B+. On the right you can see how the battery, power button and speaker all fit snugly  underneath the TFT.

The case itself took a huge amount of time to complete and went through several iterations. In the end I settled on the white case pictured below. Once I had settled on a case design it was time to put all of the final wiring into place. I’ve never had to solder in such a tight environment before, but the results were well worth it. I left a little bit of slack on the wires to prevent them from getting tugged too much and to enable servicing if required. It would save very little room to have them much shorter.

DSCF8014

All of the wires connected and the phone prior to final assembly.

Finally with all of the wires in place it was time to close up the case. I’d created a simple latching mechanism to keep the case from popping open. I’d also removed the small USB hole seen in the above photos. Also added the “rCrumbl” wordmark for good measure.

P_20181011_165747

The final product.

With the final product completed I just need to create the software to run on the phone. The TyTelli software will not run on a 3g network, so it will take some time to get a phone that operates outside of the command line. If anyone is interested in collaborating on the final stretch, let me know and I would love to have a conversation. Download the .stl file on the downloads page.

Reversing ESP8266 Firmware (Part 3)

( original text by @boredpentester )

What is it?

So, what is the ESP8266? Wikipedia describes it as follows:

The ESP8266 is a low-cost Wi-Fi microchip with full TCP/IP stack and microcontroller capability produced by Shanghai-based Chinese manufacturer, Espressif Systems.

Moreover, Wikipedia alludes to the processor specifics:

Processor: L106 32-bit RISC microprocessor core based on the Tensilica Xtensa Diamond Standard 106Micro running at 80 MHz”

At present, my version of IDA does not recognise this processor, but looking up “IDA Xtensa” unveils a processor module to support the instruction set, which is described as follows:

This is a processor plugin for IDA, to support the Xtensa core found in Espressif ESP8266.

With the above information, we’ve also answered our second question of “What is the processor?“.

Understanding the firmware format

Now that IDA can understand the instruction set of the processor, it’s time to learn how firmware images are comprised in terms of format, data and code. Indeed, what is the format of our firmware image?. To help answer this question, my first point of call was to analyse existing open source tools published by Expressif, in order to work with the ESP8266.

This leads us to ESPTool, an application written in Python capable of displaying some information about binary firmware images, amongst other things.

The manual for this tool also gives away some important information:

The elf2image command converts an ELF file (from compiler/linker output) into the binary executable images which can be flashed and then booted into.

From this, we can determine that compiled images, prior to their transformation into firmware, exist in the ELF-32 Xtensa format. This will be useful later on.

Moving back to the other features of ESPTool, we see it’s indeed able to present information about our firmware image:

1
2
3
4
5
6
7
josh@ioteeth:/tmp/reversing$ ~/esptool/esptool.py image_info recovered_file
esptool.py v2.4.0-dev
Image version: 1
Entry point: 4010f29c
1 segments
Segment 1: len 0x00568 load 0x4010f000 file_offs 0x00000008
Checksum: 2d (valid)

Clearly, this application understands the format of an image, so let’s take it apart and see how it works.

Browsing through the code, we come across:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Memory addresses
IROM_MAP_START = 0x40200000
IROM_MAP_END = 0x40300000
[...]
class ESPFirmwareImage(BaseFirmwareImage):
    """ 'Version 1' firmware image, segments loaded directly by the ROM bootloader. """
    ROM_LOADER = ESP8266ROM
    def __init__(self, load_file=None):
        super(ESPFirmwareImage, self).__init__()
        self.flash_mode = 0
        self.flash_size_freq = 0
        self.version = 1
        if load_file is not None:
            segments = self.load_common_header(load_file, ESPLoader.ESP_IMAGE_MAGIC)
            for _ in range(segments):
                self.load_segment(load_file)
            self.checksum = self.read_checksum(load_file)
[...]
class BaseFirmwareImage(object):
    SEG_HEADER_LEN = 8
    """ Base class with common firmware image functions """
    def __init__(self):
        self.segments = []
        self.entrypoint = 0
    def load_common_header(self, load_file, expected_magic):
            (magic, segments, self.flash_mode, self.flash_size_freq, self.entrypoint) = struct.unpack('<BBBBI', load_file.read(8))
            if magic != expected_magic or segments > 16:
                raise FatalError('Invalid firmware image magic=%d segments=%d' % (magic, segments))
            return segments
    def load_segment(self, f, is_irom_segment=False):
        """ Load the next segment from the image file """
        file_offs = f.tell()
        (offset, size) = struct.unpack('<II', f.read(8))
        self.warn_if_unusual_segment(offset, size, is_irom_segment)
        segment_data = f.read(size)
        if len(segment_data) < size:
            raise FatalError('End of file reading segment 0x%x, length %d (actual length %d)' % (offset, size, len(segment_data)))
        segment = ImageSegment(offset, segment_data, file_offs)
        self.segments.append(segment)
        return segment

All of the above code is notable. It allows us to discern the structure of the firmware image.

The function load_common_header() details the following format:

1
(magic, segments, self.flash_mode, self.flash_size_freq, self.entrypoint) = struct.unpack('<BBBBI', load_file.read(8))

Which represented as a structure would look like this:

1
2
3
4
5
6
7
typedef struct {
    uint8 magic;
    uint8 sect_count;
    uint8 flash_mode;
    uint8 flash_size_freq;
    uint32 entry_addr;
} rom_header;

We can see from the function load_segment() that following our image header are the image segment headers, followed immediately by the segment data itself, for each segment.

The following code parses a segment header:

1
(offset, size) = struct.unpack('<II', f.read(8))

Which again, represented as a structure would be as follows:

1
2
3
4
typedef struct {
    uint32 seg_addr;
    uint32 seg_size;
} segment_header;

This is helpful, we now know both the format of the firmware image and a number of the tools available to process such images. It’s worth noting that we haven’t considered elements such as checksums, but these aren’t important to us as we don’t intend on patching the firmware image.

Whilst a tangent, it’s worth noting that whilst in this case, our format has been documented and tools exist to parse such formats, often this is not the case. In such cases, I’d advise obtaining as many firmware images as you can from your target devices. At that point, a starting point could be to find commonalities between them, which could indicate what certain bytes mean within the format. Also of use would be to understand how an image is booted into, as the bootloader may act differently depending on certain values at fixed offsets.

Understanding the boot process

So, onto our next question, what is the boot process of the device? Understanding this is important as it will help to clarify our understanding of the image. Richard Aburton has very helpfully reverse engineered the boot loader and described the following key point:

It finds the flash address of the rom to boot. Rom 1 is always at 0×1000 (next sector after boot loader). Rom 2 is half the chip size + 0×1000 (unless the chip is above a 1mb when it’s position it kept down to to 0×81000).

Checking the 0×1000 offset within our firmware image, there is indeed a second image, as denoted by presence of the image magic signature (0xE9):

01
02
03
04
05
06
07
08
09
10
11
josh@ioteeth:/tmp/reversing$ hexdump -s 0x1000 -v -C recovered_file | head
00001000  e9 04 00 00 30 64 10 40  10 10 20 40 c0 ed 03 00  |....0d.@.. @....|
00001010  43 03 ab 83 1c 00 00 60  00 00 00 60 1c 0f 00 60  |C......`...`...`|
00001020  00 0f 00 60 41 fc ff 20  20 74 c0 20 00 32 24 00  |...`A..  t. .2$.|
00001030  30 30 75 56 33 ff 31 f8  ff 66 92 08 42 a0 0d c0  |00uV3.1..f..B...|
00001040  20 00 42 63 00 51 f5 ff  c0 20 00 29 03 42 a0 7d  | .Bc.Q... .).B.}|
00001050  c0 20 00 38 05 30 30 75  37 34 f4 31 f1 ff 66 92  |. .8.00u74.1..f.|
00001060  06 0c d4 c0 20 00 49 03  c0 20 00 29 03 0d f0 00  |.... .I.. .)....|
00001070  b0 ff ff 3f 24 10 20 40  00 ed fe 3f 80 6e 10 40  |...?$. @...?.n.@|
00001080  04 ed fe 3f 79 6e 10 40  fc ec fe 3f f8 ec fe 3f  |...?yn.@...?...?|
00001090  6b 6e 10 40 61 6e 10 40  f6 ec fe 3f 52 6e 10 40  |kn.@an.@...?Rn.@|

This second firmware image sits almost immediately after the padding bytes we observed earlier. Based on the format, we can see from the second byte (0×04) that this ROM has 4 segments and is likely to be user or custom ROM code, with the first ROM image potentially being the bootloader of the device, responsible for bootstrapping.

Whilst there are a lot of nuances to the boot process, the above is all we really need to be aware of at this time.

Understanding the physical memory layout

Next, understanding the physical memory layout will help us to differentiate between data and code segments, assuming consistency between images. Whilst not entirely accurate, the physical memory layout of the ESP8266 has been documented.

From the information within, we can conclude the following:

  • 0×40100000 – Instruction RAM. Used by bootloader to load SPI Flash <40000h.
  • 0x3FFE8000 – User data RAM. Available to applications.
  • 0x3FFFFFFF – Anything below this address appears to be data, not code
  • 0×40100000 – Anything above this address appears to be code, not data

Anything that doesn’t match an address exactly, we’ll mark as unknown and classify as either code or data based on the rules above.

It should be noted that simply loading the file as ‘binary’ within IDA, having set the appropriate processor, allows for limited understanding and doesn’t display any xrefs to strings that could guide our efforts:

With this in mind, we can write a simple loader for IDA to identify the firmware image and load the segments accordingly, which should yield better results. We’ll use the memory map above as a guide to name the segments and mark them as code or data accordingly.

Reversing ESP8266 Firmware (Part 1)

( original text by @boredpentester )

During my time with Cisco Portcullis, I wanted to learn more about reverse engineering embedded device firmware.

This six-part series was written both during my time with Cisco Portcullis, as well in my spare time (if the tagline of this blog didn’t give that away). This series intends to detail my analysis of an embedded device’s firmware, in this case, the ESP8266’s, present a methodology for analysing firmware more generally and of course, solve the challenge presented. It will be one of many series with a focus on firmware reverse engineering and security assessment of ‘smart’ devices.

We will cover the initial analysis, writing an IDA loader and recovering function symbols (names) via IDA’s FLAIR tools. Finally, we’ll reverse engineer the functionality required to solve the challenge, for extra points, without reliance upon string references.

I chose an ESP8266 firmware image supplied by a colleague as a contrived reversing challenge.  Mainly because this target made a good starting point due to the simplicity of the firmware format.

The challenge was described as follows:

We managed to obtain the firmware of an unknown device connected to our wireless access point. We’ve been told it’s connecting to a service and retrieving secrets, but we can’t reach the service. Can you?

Update: You can find the a slightly modified version of the supplied binary here (within the zip archive).

This series has been broken down into the following parts:

  • Part 1:
    • Introduction (you’re here now)
  • Part 2:
    • Initial analysis
    • Questions we need to answer
  • Part 3:
    • What is it?
    • Understanding the firmware format
    • Understanding the boot process
    • Understanding the physical memory layout
  • Part 4:
    • Writing an IDA loader
    • Performing library recognition
    • Fixing IDB2PAT
    • Generating our pattern file
  • Part 5:
    • Recognising VTABLE’s
    • Finding the port knock sequence
    • Understanding the Xtensa instruction set
    • Understanding the Xtensa calling convention
  • Part 6:
    • Reversing the loop function
    • Getting the secrets!
    • Conclusion
    • References
    • Tools
    • Feedback

Insecure Firmware Updates in Server Management Systems

( oirg text )

Supermicro BMC Case Study

Baseboard Management Controllers (BMCs) are high-value targets to attackers, who can use the out-of-band management features of the BMC to compromise servers even below the level of the host OS and system firmware. BMCs are currently an area of active research for both attackers and defenders. A compromised BMC can be used by malware to provide persistence that lasts beyond a complete wipe and reinstall of the operating system as well as the ability to cross security domains by moving laterally from host-side networks to management networks which are intended to be isolated. In addition to our research, multiple teams have discovered vulnerabilities in other BMCs, such as the recent HPE iLO4 Authentication Bypass and RCEpublication and the The Unbearable Lightness of BMC’s talk at Black Hat. Even back in 2013, Dan Farmer and HD Moore (penetration tester’s guideSupermicro IPMI) published about serious BMC vulnerabilities. Given the inherent privilege of the BMC in server architecture, a compromised BMC amounts to a significant compromise of the system.

Our research has uncovered vulnerabilities in the way that multiple vendors update their BMC firmware. These vendors typically leverage standard, off-the-shelf IPMI management tools instead of developing customized in-house management capabilities. In this case, we will go deep into the BMC update process on Supermicro systems, we found that the BMC code responsible for processing and applying firmware updates does not perform cryptographic signature verification on the provided firmware image before accepting the update and committing it to non-volatile storage. This effectively allows the attacker to load modified code onto the BMC.

After informing Supermicro of these issues, they have been working with us to improve the security of BMC firmware updates using cryptographic signatures. They have informed us that all new Supermicro products will be released with this feature, and customers who wish to upgrade existing X10 or X11 systems can contact Supermicro (details on Supermicro’s cryptographic signature page). Unfortunately, at the time of publication, updates that address this critical vulnerability are not publicly available for direct download. Also, since Supermicro’s tools do allow administrators to retrieve the firmware version and a dump of the image, Supermicro customers can check that the latest updates are installed and verify the integrity of BMC firmware from critical systems.

Impact

Using the vulnerabilities we discovered, it is possible to make arbitrary modifications to the BMC code and data. Using these modifications, an attacker can run malicious software within these highly privileged management controllers. This could be useful, for example, to survive operating system reinstallation or communicate covertly with the attacker’s infrastructure, similar to the PLATINUM malware that used manageability features to bypass detection. Alternatively, this vulnerability could be used to “brick” (permanently disable) the BMC or the entire system, creating an impact even more severe than the BlackEnergy KillDisk component. Malicious software could prevent any further firmware updates to the BMC or simply corrupt the host firmware. In both of these scenarios, the only option to recover the system is to physically attach reprogramming hardware to the SPI chips on the motherboard and restore the original image.

Because IPMI communications can be performed over the BMC LAN interface, this update mechanism could also be exploited remotely if the attacker has been able to capture the ADMIN password for the BMC. This requires access to the systems management network, which should be isolated and protected from the production network. However, the implicit trust of management networks and interfaces may generate a false sense of security, leading to otherwise-diligent administrators practicing password reuse for convenience. In such cases, an attacker who has compromised one system could capture this password and use it to spread remotely to other systems on the management network.

Many server platforms support a dedicated IPMI LAN interface with a separate physical port rather than sharing a single physical port with the host operating system through the use of the Network Controller-Sideband Interface (NC-SI) mechanism. These systems allow the use of a physically separate “management” network such that IPMI LAN traffic is isolated from normal host network traffic. When this type of network architecture is being used, we can encounter situations where hosts are isolated from talking to each other through network firewalling to reduce their attack surface, but they’re all on the same BMC management network. In this case, once we have compromised the BMC, we can use this management network to attack and compromise other BMCs when the host-side network interfaces can’t directly communicate with each other.

About the BMC

Modern server platforms include a Baseboard Management Controller (BMC) for out-of-band management via Intelligent Platform Management Interface (IPMI). the BMC contains a separate CPU with its own firmware, which can remain accessible even when the host is powered off. This system component is designed to provide administrative access to the platform and is highly privileged, allowing administrators to remotely manage the firmware and OS of the host in the case of a failure or as part of regular maintenance. As a result, compromise of the BMC can undermine trust in the system regardless of any operating system protections installed on the host processor.

Some of the features Supermicro advertises for their BMC firmware are:

  • IPMI 2.0 based management
  • Hardware health monitor
  • Remote power control
  • Keyboard, Video & Mouse (KVM) Console Redirection with multi language support
  • HTML5 web Console Redirection
  • Media Redirection
  • Web based configuration

The AST2400 and AST2500 are common BMC components used in many server platforms. They contain an ARM processor, graphics processor, and network hardware. In addition, they have their own SPI Flash and DRAM. This allows the BMC subsystem to operate independently from the main CPU(s) in the server. Most BMCs are then connected to a wide variety of system busses, including PCIe, USB, LPC, SMBUS and possibly a shared network adapter.

Vulnerabilities

We have discovered that the X8 through X11 generation Supermicro servers use insecure firmware update mechanisms for their BMC components. Using the existing update interface to the BMC, it is possible for host software to modify BMC firmware images and run arbitrary malicious code inside the BMC processor.

X8 and X9 BMC FW Update Process

The BMC firmware in both X8 and X9 generation systems uses a WPCM450 Boot Loader (WBL) from Winbond Limited, which takes just under 64kB at the beginning of the SPI flash image. The SPI flash also contains an “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

After the bootloader and this nvram section, each region in the firmware image is stored as a series of 32kB chunks where the end of the last chunk contains metadata about the region including load address and the name of the region, such as “1stFS”, “kernel”, and “2ndFS”. These correspond to two cramfs (rootfs and webfs) filesystems and a Linux kernel image which is zipped.

In the following screenshot, the 32kB chunk of this firmware image ending at offset 0x920000 contains a metadata footer with the magic value of 0xA0FFFF9F which indicates that this is a WBL-formatted image.

Further, the value of 0x40180000 indicates the “burn address” of this section of the file. For these types of images, the base of the flash region starts at 0x40000000 and the update tool uses this “burn address” minus the flash base address to determine how far into the file this region begins. The end of the region is calculated by simply taking the end of the chunk that contained the metadata footer. In this example, the start of this region is at offset 0x180000 and the end of the region is at 0x920000. The name for this region is “1stFS” and it contains the root filesystem for the Linux distribution running inside the BMC.

Using this information, we can carve the region out of the containing firmware image and extract the contents of the filesystem:

We can use the same process to extract the kernel region which contains a zipped Linux kernel:

Inside the kernel image, we found the default boot command line containing the mtdparts= argument which specifies how the flash regions are configured.

This flash layout declaration matches up with the 1stFS, kernel, and 2ndFS regions we discovered by scanning for the WBL (Winbond Boot Loader) magic value of 0xA0FFFF9F. In addition, it also shows us that the bootloader region starts at offset 0 with size 256kB and there’s an additional “nvram” region starting at offset 256kB with size 1280kB. This “nvram” region is used by the BMC operating system to store configuration settings, event logs, and other pieces of data that should persist across BMC resets.

The X9 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed. Note that the IPMI specification does not have a requirement for update authentication.

Additionally, no cryptographic signature verification is performed on the BMC firmware images before they are written to the SPI flash chip.

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, and replace the original filesystem sections in the firmware update image and flash these modified images to the X9 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X8DT3-LN4F using the X8DT3303.zip firmware image (latest available), X9DRi-LN4F+ using the SMT_X9_348.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X9-generation IPMI/BMC firmware images.

X10 BMC FW Update Process

In X10 generation systems, the BMC processor has been replaced with AST2x00 SoCs from ASPEED. The BMC firmware in these systems uses a U-Boot bootloader which takes up approximately 128kB at the beginning of the SPI flash image. The SPI flash also contains a “nvram” section which is used as a non-volatile storage region for configuration changes and event data.

Like the X9 generation systems, the X10 BMC firmware images also contain three additional regions; two cramfs filesystems and a Linux kernel image. The BMC firmware image contains a u-boot header for the kernel image with a crc32 checksum as well as a text description of the flash regions containing start offset, size, crc32, and name for each region.

This example is from the REDFISH_X10_327.bin firmware image:

With a little formatting to make it easier to read:

[img]: 0 20fec cf74e74e u-boot.bin

[img]: 400000 d28000 ec25e35d out_rootfs_img.bin

[img]: 1400000 177620 64d09306 out_kernel.bin

[img]: 1700000 55f00a ed586d8c out_webfs_img.bin

[end]

The columns within each [img] tag are start offset, end offset, crc32, and section name.

The X10 BMC firmware can be updated by sending OEM-specific IPMI messages to the BMC. When this is done from the host processor over the “keyboard controller style” (KCS) IPMI system interface, no authentication is performed.

CRC32 checksums are calculated and checked for each region in the flash image, but no cryptographic signature verification is performed before they are written to the SPI flash chip. The update process also checks two values near an “ATENs_FW” string, but this check does not provide any security protection. An attacker can replace sections of the firmware with malicious code because these checksums are not a security mechanism and only protect against accidental corruption.

Here’s what the ATENs_FW signature looks like in this firmware update image:

One thing that initially caused some confusion when analyzing this image was that the ec25d280qed5855f0 string looked very similar to the crc32 values from the [img] sections and we took a closer look to see how these sections are related. “ec25” are ascii hex values for the upper half of the crc32 from the out_rootfs_img.bin section, but “d280” is the the upper half of the *length* of the section rather than the lower 16-bits of the crc32. Likewise, ed58 and 55f0 are the upper halves of the crc32 and length from the out_rootfs_img.bin section.

When we unpacked the out_rootfs_img.bin cramsfs, one of the executable files we found was /bin/img_crc_check which checks these specific fields in the firmware image. However, the way that this check is performed can’t possibly work with this image.

Here’s the code from find_atens_signature() that is used to find the signature and extract the values:

And here’s the code from main() that uses those values:

However, there’s no hexdecimal conversion anywhere in this code so the length and checksum fields will be incorrect. As an example, the length for the rootfs will be 0x64323830 instead of 0xd28000. It appears that we’ve found a bug in the checksum generation for these images, but it turns out that this doesn’t matter.

When the BMC receives a new firmware image over the IPMI interface, the code that checks its signature doesn’t actually check the values of the checksum fields in this section before writing the received firmware image to the SPI flash, as shown in the following code:

We were able to successfully unpack the cramfs filesystem sections, replace files with arbitrary contents, add new files, replace the original filesystem sections in the firmware update image, recalculate the necessary CRC32 values in the [img] tags, and flash these modified images to the X10 BMC using the official Supermicro software update tools.

This vulnerability was confirmed with an X10SLM-F system, the REDFISH_X10_327.zip firmware image (the most recent version available for this motherboard), and the analysis of additional X10-generation IPMI/BMC firmware images.

X11 BMC FW Update Process

X11 generation systems continue the use of ASPEED AST2x00 BMC chips. However, with this generation, Supermicro made changes to the BMC firmware update file format. This appears to be an attempt to make it more difficult to extract and replace the BMC firmware for these systems.

When we first examined the X11 BMC firmware update images, we didn’t find the standard headers for embedded filesystems or the Linux kernel we expected, but we found many byte sequences that signified LZMA compressed data along with strings that looked like file and directory names:

This looked like a cramfs filesystem, except that we’d expect to see the standard signature bytes and other header fields at the beginning. Instead, we just had garbage. This seemed suspicious, so we bought an X11 motherboard so we could look at a real system.

Immediately after receiving the X11 platform, we hooked up a SPI programmer to physically read a copy of the SPI flash chip containing the BMC firmware before we even installed the CPU:

When we took a look at the dumped SPI flash image, we discovered that the live system contains the normal cramfs and Linux kernel headers precisely where we expected to find them.

After extracting the cramfs filesystem and examining libipmi.so, we found debugging messages and function symbols that made it clear what was going on:

j_console_log("[%s] img-size: %#x\n", "flash_decrypt_check", a2);
j_console_log("[%s] Flash encryption check ...\n", "flash_decrypt_check");
v3 = j_crypto_task1(v2, 2, 0x400000);
if ( v3 >= 0 )
{
v3 = j_crypto_task1(v2, 2, 0x1700000);
if ( v3 >= 0 )
{
v5 = j_crypto_task2(v2, 2);

A little more digging and we discovered hardcoded AES keys and IVs that were used to encrypt the first 96 or 192 bytes of each of the regions of the BMC firmware image we were interested in.

Because AES is a symmetric cipher, these keys can be used to decrypt an existing firmware update, modify the image, and then re-encrypt the image such that the X11 update mechanism will accept the modified image and apply it to the system.

These keys can easily be recovered by someone with physical access to an X11 system. We were able to extract the shipped firmware image from the system, find the keys, and successfully create a new firmware update image to connect back to our command and control server with a root shell running in the BMC in less than two hours of work. And this was all done before we installed the CPU in the new motherboard.

Because the keys are only used to obfuscate the header of the cramfs filesystem and not the contents of the compressed files, it’s also possible to find the compressed libipmi.so which contains the hardcoded keys and manually extract that from the firmware update image even without physical access to a system which contains the firmware image flashed in the clear.

Mitigation

The BMC firmware update mechanism must protect itself against potentially malicious updates including performing cryptographically secure signature verification before allowing new updates to be committed to the SPI flash. NIST has published Special Publication 800-193: Platform Firmware Resiliency Guidelines which describes important attributes for each component of a computer system. An Authenticated Update Mechanism is described in section 4.2.1.1 as follows:

One or more authenticated update mechanisms anchored in the RTU shall be the exclusive means for updating device firmware, absent unambiguous physical presence through a secure local update, as defined in Section 3.5.3. Authenticated update mechanisms shall meet the following authentication guidelines:

  1. Firmware update images shall be signed using an approved digital signature algorithm as specified in FIPS 186-4 [7], Digital Signature Standard, with security strength of at least 112 bits in compliance with SP 800-57, Recommendation for Key Management – Part 1: General [8].
  2. Each firmware update image shall be signed by an authorized entity – usually the device manufacturer, the platform manufacturer or a trusted third party – in conformance with SP 800-89, Recommendation for Obtaining Assurances for Digital Signature Applications [9].
  3. The digital signature of a new or recovery firmware update image shall be verified by an RTU or a CTU prior to the non-volatile storage completion of the update process. For example, this might be accomplished by verifying the contents of the update in RAM and then performing an update to the active flash. In another example, it could also be accomplished by loading the update into a region of flash, verifying it, and then selecting that region of flash as the active region.

While this requirement does not seem to appear in the IPMI specification, more attention on the area should drive wider adoption. After we reported these issues to Supermicro, they have implemented cryptographic signature verification for the BMC firmware update process. This appears to create a Root of Trust for Update (RTU) as described above, which is an important security feature for devices, including the BMC. Supermicro recommends reaching out to their support contacts for more information about the authentication feature on current systems.

In addition, Supermicro’s update tools provide the ability to check the firmware version which is installed on a system as well as dump a copy of the current firmware which can be used to check for known vulnerable versions of the firmware or check the integrity of the installed firmware.

Conclusion

BMCs provide highly privileged management access to servers, but vulnerabilities in these management components can undermine all of the existing protections in the host operating system and applications which are running on the server. As with any security issue, continued attention is needed for improvement. Given the recent BMC issues discovered by other researchers as well as the vulnerabilities discussed in this post, it is clear that more attention is needed. We believe that better visibility into the integrity of critical components like the BMC is an important improvement for many organizations, and we’re working to make that happen.

Aigo Chinese encrypted HDD − Part 1: taking it apart

Original post by Raphaël Rigo on syscall.eu ( under CC-BY-SA 4.0 )

Introduction

Analyzing and breaking external encrypted HDD has been a “hobby” of mine for quite some time. With my colleagues Joffrey Czarny and Julien Lenoir we looked at several models in the past:

  • Zalman VE-400
  • Zalman ZM-SHE500
  • Zalman ZM-VE500

Here I am going to detail how I had fun with one drive a colleague gave me: the Chinese Aigo “Patriot” SK8671, which follows the classical design for external encrypted HDDs: a LCD for information diplay and a keyboard to enter the PIN.

DISCLAIMER: This research was done on my personal time and is not related to my employer.

Patriot HDD front view with keyboard Patriot HDD package
Enclosure
Packaging

The user must input a password to access data, which is supposedly encrypted.

Note that the options are very limited:

  • the PIN can be changed by pressing F1 before unlocking
  • the PIN must be between 6 and 9 digits
  • there is a wrong PIN counter, which (I think) destroys data when it reaches 15 tries.

In practice, F2, F3 and F4 are useless.

Hardware design

Of course one of the first things we do is tear down everything to identify the various components.

Removing the case is actually boring, with lots of very small screws and plastic to break.

In the end, we get this (note that I soldered the 5 pins header):

disk

Main PCB

The main PCB is pretty simple:

main PCB

Important parts, from top to bottom:

  • connector to the LCD PCB (CN1)
  • beeper (SP1)
  • Pm25LD010 (datasheet) SPI flash (U2)
  • Jmicron JMS539 (datasheet) USB-SATA controller (U1)
  • USB 3 connector (J1)

The SPI flash stores the JMS539 firmware and some settings.

LCD PCB

The LCD PCB is not really interesting:

LCD view

LCD PCB

It has:

  • an unknown LCD character display (with Chinese fonts probably), with serial control
  • a ribbon connector to the keyboard PCB

Keyboard PCB

Things get more interesting when we start to look at the keyboard PCB:

Keyboard PCB, back

Here, on the back we can see the ribbon connector and a Cypress CY8C21434 PSoC 1 microcontroller (I’ll mostly refer to it as “µC” or “PSoC”):CY8C21434

The CY8C21434 is using the M8C instruction set, which is documented in the Assembly Language User Guide.

The product page states it supports CapSense, Cypress’ technology for capacitive keyboards, the technology in use here.

You can see the header I soldered, which is the standard ISSP programming header.

Following wires

It is always useful to get an idea of what’s connected to what. Here the PCB has rather big connectors and using a multimeter in continuity testing mode is enough to identify the connections:

hand drawn schematic

Some help to read this poorly drawn figure:

  • the PSoC is represented as in the datasheet
  • the next connector on the right is the ISSP header, which thankfully matches what we can find online
  • the right most connector is the clip for the ribbon, still on the keyboard PCB
  • the black square contains a drawing of the CN1 connector from the main PCB, where the cable goes to the LCD PCB. P11, P13 and P4 are linked to the PSoC pins 11, 13 and 4 through the LCD PCB.

Attack steps

Now that we know what are the different parts, the basic steps would be the same as for the drives analyzed in previous research :

  • make sure basic encryption functionnality is there
  • find how the encryption keys are generated / stored
  • find out where the PIN is verified

However, in practice I was not really focused on breaking the security but more on having fun. So, I did the following steps instead:

  • dump the SPI flash content
  • try to dump PSoC flash memory (see part 2)
  • start writing the blog post
  • realize that the communications between the Cypress PSoC and the JMS539 actually contains keyboard presses
  • verify that nothing is stored in the SPI when the password is changed
  • be too lazy to reverse the 8051 firmware of the JMS539
  • TBD: finish analyzing the overall security of the drive (in part 3 ?)

Dumping the SPI flash

Dumping the flash is rather easy:

  • connect probes to the CLKMOSIMISO and (optionally) EN pins of the flash
  • sniff the communications using a logic analyzer (I used a Saleae Logic Pro 16)
  • decode the SPI protocol and export the results in CSV
  • use decode_spi.rb to parse the results and get a dump

Note that this works very well with the JMS539 as it loads its whole firmware from flash at boot time.

$ decode_spi.rb boot_spi1.csv dump
0.039776 : WRITE DISABLE
0.039777 : JEDEC READ ID
0.039784 : ID 0x7f 0x9d 0x21
---------------------
0.039788 : READ @ 0x0
0x12,0x42,0x00,0xd3,0x22,0x00,
[...]
$ ls --size --block-size=1 dump
49152 dump
$ sha1sum dump
3d9db0dde7b4aadd2b7705a46b5d04e1a1f3b125  dump

Unfortunately it does not seem obviously useful as:

  • the content did not change after changing the PIN
  • the flash is actually never accessed after boot

So it probably only holds the firmware for the JMicron controller, which embeds a 8051 microcontroller.

Sniffing communications

One way to find which chip is responsible for what is to check communications for interesting timing/content.

As we know, the USB-SATA controller is connected to the screen and the Cypress µC through the CN1 connector and the two ribbons. So, we hook probes to the 3 relevant pins:

  • P4, generic I/O in the datasheet
  • P11, I²C SCL in the datasheet
  • P13, I²C SDA in the datasheet

probes

We then launch Saleae logic analyzer, set the trigger and enter “123456✓” on the keyboard. Which gives us the following view:

Saleae logic analyzer screenshot

You can see 3 differents types of communications:

  • on the P4 channel, some short bursts
  • on P11 and P13, almost continuous exchanges

Zooming on the first P4 burst (blue rectangle in previous picture), we get this :

P4 zoom

You can see here that P4 is almost 70ms of pure regular signal, which could be a clock. However, after spending some time making sense of this, I realized that it was actually a signal for the “beep” that goes off every time a key is touched… So it is not very useful in itself, however, it is a good marker to know when a keypress was registered by the PSoC.

However, we have on extra “beep” in the first picture, which is slightly different: the sound for “wrong pin” !

Going back to our keypresses, when zooming at the end of the beep (see the blue rectangle again), we get:end of beep zoom

Where we have a regular pattern, with a (probable) clock on P11 and data on P13. Note how the pattern changes after the end of the beep. It could be interesting to see what’s going on here.

2-wires protocols are usually SPI or I²C, and the Cypress datasheet says the pins correspond to I²C, which is apparently the case:i2c decoding of '1' keypress

The USB-SATA chipset constantly polls the PSoC to read the key state, which is ‘0’ by default. It then changes to ‘1’ when key ‘1’ was pressed.

The final communication, right after pressing “✓”, is different if a valid PIN is entered. However, for now I have not checked what the actual transmission is and it does not seem that an encryption key is transmitted.

Anyway, see part 2 to read how I did dump the PSoC internal flash.

Practical Reverse Engineering Part 4 — Dumping the Flash

Projects and learnt lessons on Systems Security, Embedded Development, IoT and anything worth writing about

  • Part 1: Hunting for Debug Ports
  • Part 2: Scouting the Firmware
  • Part 3: Following the Data
  • Part 4: Dumping the Flash
  • Part 5: Digging Through the Firmware

In Parts 1 to 3 we’ve been gathering data within its context. We could sniff the specific pieces of data we were interested in, or observe the resources used by each process. On the other hand, they had some serious limitations; we didn’t have access to ALL the data, and we had to deal with very minimal tools… And what if we had not been able to find a serial port on the PCB? What if we had but it didn’t use default credentials?

In this post we’re gonna get the data straight from the source, sacrificing context in favour of absolute access. We’re gonna dump the data from the Flash IC and decompress it so it’s usable. This method doesn’t require expensive equipment and is independent of everything we’ve done until now. An external Flash IC with a public datasheet is a reverser’s great ally.

Dumping the Memory Contents

As discussed in Part 3, we’ve got access to the datasheet for the Flash IC, so there’s no need to reverse its pinout:

Flash Pic Annotated Pinout

We also have its instruction set, so we can communicate with the IC using almost any device capable of ‘speaking’ SPI.

We also know that powering up the router will cause the Ralink to start communicating with the Flash IC, which would interfere with our own attempts to read the data. We need to stop the communication between the Ralink and the Flash IC, but the best way to do that depends on the design of the circuit we’re working with.

Do We Need to Desolder The Flash IC? [Theory]

The perfect way to avoid interference would be to simply desolder the Flash IC so it’s completely isolated from the rest of the circuit. It gives us absolute control and removes all possible sources of interference. Unfortunately, it also requires additional equipment, experience and time, so let’s see if we can avoid it.

The second option would be to find a way of keeping the Ralink inactive while everything else around it stays in standby. Microcontrollers often have a Reset pin that will force them to shut down when pulled to 0; they’re commonly used to force IC reboots without interrupting power to the board. In this case we don’t have access to the Ralink’s full datasheet (it’s probably distributed only to customers and under NDA); the IC’s form factor and the complexity of the circuit around it make for a very hard pinout to reverse, so let’s keep thinking…

What about powering one IC up but not the other? We can try applying voltage directly to the power pins of the Flash IC instead of powering up the whole circuit. Injecting power into the PCB in a way it wasn’t designed for could blow something up; we could reverse engineer the power circuit, but that’s tedious work. This router is cheap and widely available, so I took the ‘fuck it’ approach. The voltage required, according to the datasheet, is 3V; I’m just gonna apply power directly to the Flash IC and see what happens. It may power up the Ralink too, but it’s worth a try.

Flash Powered UART Connected

We start supplying power while observing the board and waiting for data from the Ralink’s UART port. We can see some LEDs light up at the back of the PCB, but there’s no data coming out of the UART port; the Ralink must not be running. Even though the Ralink is off, its connection to the Flash IC may still interfere with our traffic because of multiple design factors in both power circuit and the silicon. It’s important to keep that possibility in mind in case we see anything dodgy later on; if that was to happen we’d have to desolder the Flash IC (or just its data pins) to physically disconnect it from everything else.

The LEDs and other static components can’t communicate with the Flash IC, so they won’t be an issue as long as we can supply enough current for all of them. I’m just gonna use a bench power supply, with plenty of current available for everything. If you don’t have one you can try using the Master’s power lines, or some USB power adapter if you need some more current. They’ll probably do just fine.

Time to connect our SPI Master.

Connecting to the Flash IC

Now that we’ve confirmed there’s no need to desolder the Ralink we can connect any device that speaks SPI and start reading memory contents block by block. Any microcontroller will do, but a purpose-specific SPI-USB bridge will often be much faster. In this case I’m gonna be using a board based on the FT232H, which supports SPI among some other low level protocols.

We’ve got the pinout for both the Flash and my USB-SPI bridge, so let’s get everything connected.

Shikra and Power Connected to Flash

Now that the hardware is ready it’s time to start pumping data out.

Dumping the Data

We need some software in our computer that can understand the USB-SPI bridge’s traffic and replicate the memory contents as a binary file. Writing our own wouldn’t be difficult, but there are programs out there that already support lots of common Masters and Flash ICs. Let’s try the widely known and open source flashrom.

flashrom is old and buggy, but it already supports both the FT232H as Master and the FL064PIF as Slave. It gave me lots of trouble in both OSX and an Ubuntu VM, but ended up working just fine on a Raspberry Pi (Raspbian):

flashrom stdout

Success! We’ve got our memory dump, so we can ditch the hardware and start preparing the data for analysis.

Splitting the Binary

The file command has been able to identify some data about the binary, but that’s just because it starts with a header in a supported format. In a 0-knowledge scenario we’d use binwalk to take a first look at the binary file and find the data we’d like to extract.

Binwalk is a very useful tool for binary analysis created by the awesome hackers at /dev/ttyS0; you’ll certainly get to know them if you’re into hardware hacking.

binwalk spidump.bin

In this case we’re not in a 0-knowledge scenario; we’ve been gathering data since day 1, and we obtained a complete memory map of the Flash IC in Part 2. The addresses mentioned in the debug message are confirmed by binwalk, and it makes for much cleaner splitting of the binary, so let’s use it:

Flash Memory Map From Part 2

With the binary and the relevant addresses, it’s time to split the binary into its 4 basic segments. dd takes its parameters in terms of block size (bs, bytes), offset (skip, blocks) and size (count, blocks); all of them in decimal. We can use a calculator or let the shell do the hex do decimal conversions with $(()):

$ dd if=spidump.bin of=bootloader.bin bs=1 count=$((0x020000))
    131072+0 records in
    131072+0 records out
    131072 bytes transferred in 0.215768 secs (607467 bytes/sec)
$ dd if=spidump.bin of=mainkernel.bin bs=1 count=$((0x13D000-0x020000)) skip=$((0x020000))
    1167360+0 records in
    1167360+0 records out
    1167360 bytes transferred in 1.900925 secs (614101 bytes/sec)
$ dd if=spidump.bin of=mainrootfs.bin bs=1 count=$((0x660000-0x13D000)) skip=$((0x13D000))
    5386240+0 records in
    5386240+0 records out
    5386240 bytes transferred in 9.163635 secs (587784 bytes/sec)
$ dd if=spidump.bin of=protect.bin bs=1 count=$((0x800000-0x660000)) skip=$((0x660000))
    1703936+0 records in
    1703936+0 records out
    1703936 bytes transferred in 2.743594 secs (621060 bytes/sec)

We have created 4 different binary files:

  1. bootloader.bin: U-boot. The bootloader. It’s not compressed because the Ralink wouldn’t know how to decompress it.
  2. mainkernel.bin: Linux Kernel. The basic firmware in charge of controlling the bare metal. Compressed using lzma
  3. mainrootfs.bin: Filesystem. Contains all sorts of important binaries and configuration files. Compressed as squashfs using the lzma algorithm
  4. protect.bin: Miscellaneous data as explained in Part 3. Not compressed

Extracting the Data

Now that we’ve split the binary into its 4 basic segments, let’s take a closer look at each of them.

Bootloader

binwalk bootloader.bin

Binwalk found the uImage header and decoded it for us. U-Boot uses these headers to identify relevant memory areas. It’s the same info that the file command displayed when we fed it the whole memory dump because it’s the first header in the file.

We don’t care much for the bootloader’s contents in this case, so let’s ignore it.

Kernel

binwalk mainkernel.bin

Compression is something we have to deal with before we can make any use of the data. binwalk has confirmed what we discovered in Part 2, the kernel is compressed using lzma, a very popular compression algorithm in embedded systems. A quick check with strings mainkernel.bin | less confirms there’s no human readable data in the binary, as expected.

There are multiple tools that can decompress lzma, such as 7z or xz. None of those liked mainkernel.bin:

$ xz --decompress mainkernel.bin
xz: mainkernel.bin: File format not recognized

The uImage header is probably messing with tools, so we’re gonna have to strip it out. We know the lzma data starts at byte 0x40, so let’s copy everything but the first 64 bytes.

dd if=mainkernel of=noheader

And when we try to decompress…

$ xz --decompress mainkernel_noheader.lzma
xz: mainkernel_noheader.lzma: Compressed data is corrupt

xz has been able to recognize the file as lzma, but now it doesn’t like the data itself. We’re trying to decompress the whole mainkernel Flash area, but the stored data is extremely unlikely to be occupying 100% of the memory segment. Let’s remove any unused memory from the tail of the binary and try again:

Cut off the tail; decompression success

xz seems to have decompressed the data successfully. We can easily verify that using the strings command, which finds ASCII strings in binary files. Since we’re at it, we may as well look for something useful…

strings kernel grep key

The Wi-Fi Easy and Secure Key Derivation string looks promising, but as it turns out it’s just a hardcoded string defined by the Wi-Fi Protected Setup spec. Nothing to do with the password generation algorithm we’re interested in.

We’ve proven the data has been properly decompressed, so let’s keep moving.

Filesystem

binwalk mainrootfs.bin

The mainrootfs memory segment does not have a uImage header because it’s relevant to the kernel but not to U-Boot.

SquashFS is a very common filesystem in embedded systems. There are multiple versions and variations, and manufacturers sometimes use custom signatures to make the data harder to locate inside the binary. We may have to fiddle with multiple versions of unsquashfs and/or modify the signatures, so let me show you what the signature looks like in this case:

sqsh signature in hexdump

Since the filesystem is very common and finding the right configuration is tedious work, somebody may have already written a script to automate the task. I came across this OSX-specific fork of the Firmware Modification Kit, which compiles multiple versions of unsquashfs and includes a neat script called unsquashfs_all.sh to run all of them. It’s worth a try.

unsquashfs_all.sh mainrootfs.bin

Wasn’t that easy? We got lucky with the SquashFS version and supported signature, and unsquashfs_all.sh managed to decompress the filesystem. Now we’ve got every binary in the filesystem, every symlink and configuration file, and everything is nice and tidy:

tree unsquashed_filesystem

In the complete file tree we can see we’ve got every file in the system, (other than runtime files like those in /var/, of course).

Using the intel we have been gathering on the firmware since day 1 we can start looking for potentially interesting binaries:

grep -i -r '$INTEL' squashfs-root

If we were looking for network/application vulnerabilities in the router, having every binary and config file in the system would be massively useful.

Protected

binwalk protect.bin

As we discussed in Part 3, this memory area is not compressed and contains all pieces of data that need to survive across reboots but be different across devices. strings seems like an appropriate tool for a quick overview of the data:

strings protect.bin

Everything in there seems to be just the curcfg.xml contents, some logs and those few isolated strings in the picture. We already sniffed and analysed all of that data in Part 3, so there’s nothing else to discuss here.

Next Steps

At this point all hardware reversing for the Ralink is complete and we’ve collected everything there was to collect in ROM. Just think of what you may be interested in and there has to be a way to find it. Imagine we wanted to control the router through the UART debug port we found in Part 1, but when we try to access the ATP CLI we can’t figure out the credentials. After dumping the external Flash we’d be able to find the XML file in the protect area, and discover the credentials just like we did in Part 2 (The Rambo Approach to Intel Gatheringadmin:admin).

If you couldn’t dump the memory IC for any reason, the firmware upgrade files provided by the manufacturers will sometimes be complete memory segments; the device simply overwrites the relevant flash areas using code previously loaded to RAM. Downloading the file from the manufacturer would be the equivalent of dumping those segments from flash, so we just need to decompress them. They won’t have all the data, but it may be enough for your purposes.

Now that we’ve got the firmware we just need to think of anything we may be interested in and start looking for it through the data. In the next post we’ll dig a bit into different binaries and try to find more potentially useful data.