Original text by Barracuda

UNE 6^th, 2023:

ACTION NOTICE: Impacted ESG appliances must be immediately replaced regardless of patch version level. If you have not replaced your appliance after receiving notice in your UI, contact support now (support@barracuda.com).  

Barracuda’s remediation recommendation at this time is full replacement of the impacted ESG. 

JUNE 1^st, 2023:

Preliminary Summary of Key Findings

Document History

Barracuda Networks’ priorities throughout this incident have been transparency and to use this as an opportunity to strengthen our policies, practices, and technology to further protect against future attacks. Although our investigation is ongoing, the purpose of this document is to share preliminary findings, provide the known Indicators of Compromise (IOCs), and share YARA rules to aid our customers in their investigations, including with respect to their own environments.

Timeline

• On May 18, 2023, Barracuda was alerted to anomalous traffic originating from Barracuda Email Security Gateway (ESG) appliances.
• On May 18, 2023, Barracuda engaged Mandiant, leading global cyber security experts, to assist in the investigation.
• On May 19, 2023, Barracuda identified a vulnerability (CVE-2023-2868¹) in our Email Security Gateway appliance (ESG).
• On May 20, 2023, a security patch to remediate the vulnerability was applied to all ESG appliances worldwide.
• On May 21, 2023, a script was deployed to all impacted appliances to contain the incident and counter unauthorized access methods.
• A series of security patches are being deployed to all appliances in furtherance of our containment strategy.

Key Findings

While the investigation is still on-going, Barracuda has concluded the following:

• The vulnerability existed in a module which initially screens the attachments of incoming emails. No other Barracuda products, including our SaaS email security services, were subject to the vulnerability identified.
• Earliest identified evidence of exploitation of CVE-2023-2868 is currently October 2022.
• Barracuda identified that CVE-2023-2868 was utilized to obtain unauthorized access to a subset of ESG appliances.
• Malware was identified on a subset of appliances allowing for persistent backdoor access.
• Evidence of data exfiltration was identified on a subset of impacted appliances..

Users whose appliances we believe were impacted have been notified via the ESG user interface of actions to take. Barracuda has also reached out to these specific customers. Additional customers may be identified in the course of the investigation.

CVE-2023-2868

On May 19, 2023, Barracuda Networks identified a remote command injection vulnerability (CVE-2023-2868) present in the Barracuda Email Security Gateway (appliance form factor only) versions 5.1.3.001-9.2.0.006. The vulnerability stemmed from incomplete input validation of user supplied .tar files as it pertains to the names of the files contained within the archive. Consequently, a remote attacker could format file names in a particular manner that would result in remotely executing a system command through Perl’s qx operator with the privileges of the Email Security Gateway product.

Barracuda’s investigation to date has determined that a third party utilized the technique described above to gain unauthorized access to a subset of ESG appliances.

Malware

This section details the malware that has been identified to date, and to assist in tracking, codenames for the malware have been assigned.

SALTWATER

SALTWATER is a trojanized module for the Barracuda SMTP daemon (bsmtpd) that contains backdoor functionality. The capabilities of SALTWATER include the ability to upload or download arbitrary files, execute commands, as well as proxy and tunneling capabilities.

Identified at path: /home/product/code/firmware/current/lib/smtp/modules on a subset of ESG appliances.

The backdoor is implemented using hooks on the send, recv, close syscalls and amounts to five components, most of which are referred to as “Channels” within the binary. In addition to providing proxying capabilities, these components exhibit backdoor functionality.  The five (5) channels can be seen in the list below.

• DownloadChannel
• UploadChannel
• ProxyChannel
• ShellChannel
• TunnelArgs

Mandiant is still analyzing SALTWATER to determine if it overlaps with any other known malware families.

Table 1 below provides the file metadata related to a SALTWATER variant.

Table 1: SALTWATER variant metadata

SEASPY

SEASPY is an x64 ELF persistence backdoor that poses as a legitimate Barracuda Networks service and establishes itself as a PCAP filter, specifically monitoring traffic on port 25 (SMTP) and port 587. SEASPY contains backdoor functionality that is activated by a «magic packet».

Identified at path: /sbin/ on a subset of ESG appliances.

Mandiant analysis has identified code overlap between SEASPY and cd00r, a publicly available backdoor.

Table 2 below provides the file metadata related to a SEASPY variant.

Table 2: SEASPY variant metadata

SEASIDE

SEASIDE is a Lua based module for the Barracuda SMTP daemon (bsmtpd) that monitors SMTP HELO/EHLO commands to receive a command and control (C2) IP address and port which it passes as arguments to an external binary that establishes a reverse shell.

Table 3 below provides the file metadata related to a SEASIDE.

Table 3: SEASIDE metadata

Recommendations For Impacted Customers

1. Ensure your ESG appliance is receiving and applying updates, definitions, and security patches from Barracuda. Contact Barracuda support (support@barracuda.com) to validate if the appliance is up to date.
2. Discontinue the use of the compromised ESG appliance and contact Barracuda support (support@barracuda.com) to obtain a new ESG virtual or hardware appliance.
3. Rotate any applicable credentials connected to the ESG appliance:
   o  Any connected LDAP/AD
   o  Barracuda Cloud Control
   o  FTP Server
   o  SMB
   o  Any private TLS certificates
4. Review your network logs for any of the IOCs listed below and any unknown IPs. Contact compliance@barracuda.com if any are identified.

To support customers in the investigations of their environments, we are providing a list of all endpoint and network indicators observed over the course of the investigation to date. We have also developed a series of YARA rules that can be found in the section below.

Endpoint IOCs

Table 4 lists the endpoint IOCs, including malware and utilities, attributed to attacker activity during the investigation.

Table 4: Endpoint IOCs

Network IOCs

Table 5 lists the network IOCs, including IP addresses and domain names, attributed to attacker activity during the investigation.

Table 5: Network IOCs

YARA Rules

CVE-2023-2868

The following three (3) YARA rules can be used to hunt for the malicious TAR file which exploits CVE-2023-2868:

rule M_Hunting_Exploit_Archive_2
 {
     meta:
         description = "Looks for TAR archive with /tmp/ base64 encoded being part of filename of enclosed files"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $b64_tmp = "/tmp/" base64
     condition:
         filesize < 1MB and

         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $b64_tmp in (i * 512 .. i * 512 + 250)

         )
 }

rule M_Hunting_Exploit_Archive_3
 {
     meta:
         description = "Looks for TAR archive with openssl base64 encoded being part of filename of enclosed files"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $b64_openssl = "openssl" base64
     condition:

         filesize < 1MB and
         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $b64_openssl in (i * 512 .. i * 512 + 250)

         )
 }

rule M_Hunting_Exploit_Archive_CVE_2023_2868
 {
     meta:
         description = "Looks for TAR archive with single quote/backtick as start of filename of enclosed files. CVE-2023-2868"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $qb = "'`"
     condition:

         filesize < 1MB and
         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $qb at (@ustar[i] + 255)

         )
 }

SALTWATER

The following three (3) YARA rule can be used to hunt for SALTWATER:

rule M_Hunting_Linux_Funchook
 {
     strings:
         $f = "funchook_"
         $s1 = "Enter funchook_create()"
         $s2 = "Leave funchook_create() => %p"
         $s3 = "Enter funchook_prepare(%p, %p, %p)"
         $s4 = "Leave funchook_prepare(..., [%p->%p],...) => %d"
         $s5 = "Enter funchook_install(%p, 0x%x)"
         $s6 = "Leave funchook_install() => %d"
         $s7 = "Enter funchook_uninstall(%p, 0x%x)"
         $s8 = "Leave funchook_uninstall() => %d"
         $s9 = "Enter funchook_destroy(%p)"
         $s10 = "Leave funchook_destroy() => %d"
         $s11 = "Could not modify already-installed funchook handle."
         $s12 = "  change %s address from %p to %p"
         $s13 = "  link_map addr=%p, name=%s"
         $s14 = "  ELF type is neither ET_EXEC nor ET_DYN."
         $s15 = "  not a valid ELF module %s."
         $s16 = "Failed to protect memory %p (size=%"
         $s17 = "  protect memory %p (size=%"
         $s18 = "Failed to unprotect memory %p (size=%"
         $s19 = "  unprotect memory %p (size=%"
         $s20 = "Failed to unprotect page %p (size=%"
         $s21 = "  unprotect page %p (size=%"
         $s22 = "Failed to protect page %p (size=%"
         $s23 = "  protect page %p (size=%"
         $s24 = "Failed to deallocate page %p (size=%"
         $s25 = " deallocate page %p (size=%"
         $s26 = "  allocate page %p (size=%"
         $s27 = "  try to allocate %p but %p (size=%"
         $s28 = "  allocate page %p (size=%"
         $s29 = "Could not find a free region near %p"
         $s30 = "  -- Use address %p or %p for function %p"
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and (#f > 5 or 4 of ($s*))
 }

rule M_Hunting_Linux_SALTWATER_1
 {
     strings:
         $s1 = { 71 75 69 74 0D 0A 00 00 00 33 8C 25 3D 9C 17 70 08 F9 0C 1A 41 71 55 36 1A 5C 4B 8D 29 7E 0D 78 }
         $s2 = { 00 8B D5 AD 93 B7 54 D5 00 33 8C 25 3D 9C 17 70 08 F9 0C 1A 41 71 55 36 1A 5C 4B 8D 29 7E 0D 78 }
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and any of them
 }

rule M_Hunting_Linux_SALTWATER_2
 {
     strings:
         $c1 = "TunnelArgs"
         $c2 = "DownloadChannel"
         $c3 = "UploadChannel"
         $c4 = "ProxyChannel"
         $c5 = "ShellChannel"
         $c6 = "MyWriteAll"
         $c7 = "MyReadAll"
         $c8 = "Connected2Vps"
         $c9 = "CheckRemoteIp"
         $c10 = "GetFileSize"
         $s1 = "[-] error: popen failed"
         $s2 = "/home/product/code/config/ssl_engine_cert.pem"
         $s3 = "libbindshell.so"
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and (2 of ($s*) or 4 of ($c*))
 }

The following SNORT rule can be used to hunt for SEASPY magic packets:

alert tcp any any -> any [25,587] (msg:»M_Backdoor_SEASPY»; flags:S; dsize:>9; content:»oXmp»; offset:0; depth:4; threshold:type limit,track by_src,count 1,seconds 3600; sid:1000000; rev:1;)

The following SNORT rules require Suricata 5.0.4 or newer and can be used to hunt for SEASPY magic packets:

alert tcp any any -> any [25,587] (msg:»M_Backdoor_SEASPY_1358″; flags:S; tcp.hdr; content:»|05 4e|»; offset:22; depth:2; threshold:type limit,track by_src,count 1,seconds 3600; sid:1000001; rev:1;)

alert tcp any any -> any [25,587] (msg:»M_Backdoor_SEASPY_58928″; flags:S; tcp.hdr; content:»|e6 30|»; offset:28; depth:2; byte_test:4,>,16777216,0,big,relative; threshold:type limit,track by_src,count 1,seconds 3600; sid:1000002; rev:1;)

alert tcp any any -> any [25,587] (msg:»M_Backdoor_SEASPY_58930″; flags:S; tcp.hdr; content:»|e6 32|»; offset:28; depth:2; byte_test:4,>,16777216,0,big,relative; byte_test:2,>,0,0,big,relative; threshold:type limit,track by_src,count 1,seconds 3600; sid:1000003; rev:1;)

MAY 30^th, 2023:

Preliminary Summary of Key Findings

Barracuda Networks priorities throughout this incident have been transparency and to use this as an opportunity to strengthen our policies, practices, and technology to further protect against future attacks. Although our investigation is ongoing, the purpose of this document is to share preliminary findings, provide the known Indicators of Compromise (IOCs), and share YARA rules to aid our customers in their investigations, including with respect to their own environments.

Timeline

• On May 18, 2023, Barracuda was alerted to anomalous traffic originating from Barracuda Email Security Gateway (ESG) appliances.
• On May 18, 2023, Barracuda engaged Mandiant, leading global cyber security experts, to assist in the investigation.
• On May 19, 2023, Barracuda identified a vulnerability (CVE-2023-2868¹) in our Email Security Gateway appliance (ESG).
• On May 20, 2023, a security patch to remediate the vulnerability was applied to all ESG appliances worldwide.
• On May 21, 2023, a script was deployed to all impacted appliances to contain the incident and counter unauthorized access methods.
• A series of security patches are being deployed to all appliances in furtherance of our containment strategy.

Key Findings

While the investigation is still on-going, Barracuda has concluded the following:

• The vulnerability existed in a module which initially screens the attachments of incoming emails. No other Barracuda products, including our SaaS email security services, were subject to the vulnerability identified.
• Earliest identified evidence of exploitation of CVE-2023-2868 is currently October 2022.
• Barracuda identified that CVE-2023-2868 was utilized to obtain unauthorized access to a subset of ESG appliances.
• Malware was identified on a subset of appliances allowing for persistent backdoor access.
• Evidence of data exfiltration was identified on a subset of impacted appliances.

Users whose appliances we believe were impacted have been notified via the ESG user interface of actions to take. Barracuda has also reached out to these specific customers. Additional customers may be identified in the course of the investigation.

CVE-2023-2868

On May 19, 2023, Barracuda Networks identified a remote command injection vulnerability (CVE-2023-2868) present in the Barracuda Email Security Gateway (appliance form factor only) versions 5.1.3.001-9.2.0.006. The vulnerability stemmed from incomplete input validation of user supplied .tar files as it pertains to the names of the files contained within the archive. Consequently, a remote attacker could format file names in a particular manner that would result in remotely executing a system command through Perl’s qx operator with the privileges of the Email Security Gateway product.

Barracuda’s investigation to date has determined that a third party utilized the technique described above to gain unauthorized access to a subset of ESG appliances.

Malware

This section details the malware that has been identified to date.

SALTWATER

SALTWATER is a trojanized module for the Barracuda SMTP daemon (bsmtpd) that contains backdoor functionality. The capabilities of SALTWATER include the ability to upload or download arbitrary files, execute commands, as well as proxy and tunneling capabilities.

Identified at path: /home/product/code/firmware/current/lib/smtp/modules on a subset of ESG appliances.

The backdoor is implemented using hooks on the send, recv, close syscalls and amounts to five components, most of which are referred to as “Channels” within the binary. In addition to providing backdoor and proxying capabilities, these components exhibit classic backdoor functionality.  The five (5) channels can be seen in the list below.

• DownloadChannel
• UploadChannel
• ProxyChannel
• ShellChannel
• TunnelArgs

Mandiant is still analyzing SALTWATER to determine if it overlaps with any other known malware families. Table 1 below provides the file metadata related to a SALTWATER variant.

Table 1 below provides the file metadata related to a SALTWATER variant.

Table 1: SALTWATER variant metadata

SEASPY

SEASPY is an x64 ELF persistence backdoor that poses as a legitimate Barracuda Networks service and establishes itself as a PCAP filter, specifically monitoring traffic on port 25 (SMTP). SEASPY also contains backdoor functionality that is activated by a «magic packet».

Identified at path: /sbin/ on a subset of ESG appliances.

Mandiant analysis has identified code overlap between SEASPY and cd00r, a publicly available backdoor.

Table 2 below provides the file metadata related to a SEASPY variant.

Table 2: SEASPY variant metadata

SEASIDE

SEASIDE is a Lua based module for the Barracuda SMTP daemon (bsmtpd) that monitors SMTP HELO/EHLO commands to receive a command and control (C2) IP address and port which it passes as arguments to an external binary that establishes a reverse shell.

Table 3 below provides the file metadata related to a SEASIDE.

Table 3: SEASIDE metadata

Recommendations For Impacted Customers

1. Ensure your ESG appliance is receiving and applying updates, definitions, and security patches from Barracuda. Contact Barracuda support (support@barracuda.com) to validate if the appliance is up to date.
2. Discontinue the use of the compromised ESG appliance and contact Barracuda support (support@barracuda.com) to obtain a new ESG virtual or hardware appliance.
3. Rotate any applicable credentials connected to the ESG appliance:
   o  Any connected LDAP/AD
   o  Barracuda Cloud Control
   o  FTP Server
   o  SMB
   o  Any private TLS certificates
4. Review your network logs for any of the IOCs listed below and any unknown IPs. Contact compliance@barracuda.com if any are identified.

To support customers in the investigations of their environments, we are providing a list of all endpoint and network indicators observed over the course of the investigation to date. We have also developed a series of YARA rules that can be found in the section below.

Endpoint IOCs

Table 4 lists the endpoint IOCs, including malware and utilities, attributed to attacker activity during the investigation.

Table 4: Endpoint IOCs

Network IOCs

Table 5 lists the network IOCs, including IP addresses and domain names, attributed to attacker activity during the investigation.

Table 5: Network IOCs

YARA Rules

CVE-2023-2868

The following three (3) YARA rules can be used to hunt for the malicious TAR file which exploits CVE-2023-2868:

rule M_Hunting_Exploit_Archive_2
 {
     meta:
         description = "Looks for TAR archive with /tmp/ base64 encoded being part of filename of enclosed files"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $b64_tmp = "/tmp/" base64
     condition:
         filesize < 1MB and

         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $b64_tmp in (i * 512 .. i * 512 + 250)

         )
 }

rule M_Hunting_Exploit_Archive_3
 {
     meta:
         description = "Looks for TAR archive with openssl base64 encoded being part of filename of enclosed files"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $b64_openssl = "openssl" base64
     condition:

         filesize < 1MB and
         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $b64_openssl in (i * 512 .. i * 512 + 250)

         )
 }

rule M_Hunting_Exploit_Archive_CVE_2023_2868
 {
     meta:
         description = "Looks for TAR archive with single quote/backtick as start of filename of enclosed files. CVE-2023-2868"
         date_created = "2023-05-26"
         date_modified = "2023-05-26"
         md5 = "0d67f50a0bf7a3a017784146ac41ada0"
         version = "1.0"
     strings:
         $ustar = { 75 73 74 61 72 }
         $qb = "'`"
     condition:

         filesize < 1MB and
         $ustar at 257 and

         for any i in (0 .. #ustar) : (

             $qb at (@ustar[i] + 255)

         )
 }

SALTWATER

The following three (3) YARA rule can be used to hunt for SALTWATER:

rule M_Hunting_Linux_Funchook
 {
     strings:
         $f = "funchook_"
         $s1 = "Enter funchook_create()"
         $s2 = "Leave funchook_create() => %p"
         $s3 = "Enter funchook_prepare(%p, %p, %p)"
         $s4 = "Leave funchook_prepare(..., [%p->%p],...) => %d"
         $s5 = "Enter funchook_install(%p, 0x%x)"
         $s6 = "Leave funchook_install() => %d"
         $s7 = "Enter funchook_uninstall(%p, 0x%x)"
         $s8 = "Leave funchook_uninstall() => %d"
         $s9 = "Enter funchook_destroy(%p)"
         $s10 = "Leave funchook_destroy() => %d"
         $s11 = "Could not modify already-installed funchook handle."
         $s12 = "  change %s address from %p to %p"
         $s13 = "  link_map addr=%p, name=%s"
         $s14 = "  ELF type is neither ET_EXEC nor ET_DYN."
         $s15 = "  not a valid ELF module %s."
         $s16 = "Failed to protect memory %p (size=%"
         $s17 = "  protect memory %p (size=%"
         $s18 = "Failed to unprotect memory %p (size=%"
         $s19 = "  unprotect memory %p (size=%"
         $s20 = "Failed to unprotect page %p (size=%"
         $s21 = "  unprotect page %p (size=%"
         $s22 = "Failed to protect page %p (size=%"
         $s23 = "  protect page %p (size=%"
         $s24 = "Failed to deallocate page %p (size=%"
         $s25 = " deallocate page %p (size=%"
         $s26 = "  allocate page %p (size=%"
         $s27 = "  try to allocate %p but %p (size=%"
         $s28 = "  allocate page %p (size=%"
         $s29 = "Could not find a free region near %p"
         $s30 = "  -- Use address %p or %p for function %p"
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and (#f > 5 or 4 of ($s*))
 }

rule M_Hunting_Linux_SALTWATER_1
 {
     strings:
         $s1 = { 71 75 69 74 0D 0A 00 00 00 33 8C 25 3D 9C 17 70 08 F9 0C 1A 41 71 55 36 1A 5C 4B 8D 29 7E 0D 78 }
         $s2 = { 00 8B D5 AD 93 B7 54 D5 00 33 8C 25 3D 9C 17 70 08 F9 0C 1A 41 71 55 36 1A 5C 4B 8D 29 7E 0D 78 }
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and any of them
 }

rule M_Hunting_Linux_SALTWATER_2
 {
     strings:
         $c1 = "TunnelArgs"
         $c2 = "DownloadChannel"
         $c3 = "UploadChannel"
         $c4 = "ProxyChannel"
         $c5 = "ShellChannel"
         $c6 = "MyWriteAll"
         $c7 = "MyReadAll"
         $c8 = "Connected2Vps"
         $c9 = "CheckRemoteIp"
         $c10 = "GetFileSize"
         $s1 = "[-] error: popen failed"
         $s2 = "/home/product/code/config/ssl_engine_cert.pem"
         $s3 = "libbindshell.so"
     condition:
         filesize < 15MB and uint32(0) == 0x464c457f and (2 of ($s*) or 4 of ($c*))
 }

MAY 23rd, 2023:

Barracuda identified a vulnerability (CVE-2023-2868) in our Email Security Gateway appliance (ESG) on May 19, 2023. A security patch to eliminate the vulnerability was applied to all ESG appliances worldwide on Saturday, May 20, 2023. The vulnerability existed in a module which initially screens the attachments of incoming emails. No other Barracuda products, including our SaaS email security services, were subject to this vulnerability.

We took immediate steps to investigate this vulnerability. Based on our investigation to date, we’ve identified that the vulnerability resulted in unauthorized access to a subset of email gateway appliances. As part of our containment strategy, all ESG appliances have received a second patch on May 21, 2023. Users whose appliances we believe were impacted have been notified via the ESG user interface of actions to take. Barracuda has also reached out to these specific customers.

We will continue actively monitoring this situation, and we will be transparent in sharing details on what actions we are taking. Information gathering is ongoing as part of the investigation. We want to ensure we only share validated information with actionable steps for you to take. As we have information to share, we will provide updates via this product status page (https://status.barracuda.com) and direct outreach to impacted customers. Updates are also located on Barracuda’s Trust Center (https://www.barracuda.com/company/legal).

Barracuda’s investigation was limited to the ESG product, and not the customer’s specific environment. Therefore, impacted customers should review their environments and determine any additional actions they want to take.

Your trust is important to us. We thank you for your understanding and support as we work through this issue and sincerely apologize for any inconvenience it may cause. If you have any questions, please reach out to support@barracuda.com.