From the Spring 2021 Issue

The Key is Under the Mat, and We Left the Lights On

Scott Smurthwaite, PhD
Information System Security Engineer | mangoLabs

Russian Hackers

On December 8, 2020, when FireEye CEO, Kevin Mandia, announced in his company blog that a highly sophisticated state-sponsored adversary stole FireEye’s ‘Red Team’ tools1, it was a harbinger for the storm we would eventually know as, The SolarWinds Hack. The following week, when FireEye announced that they had identified a global campaign that introduced a compromise into the networks of public and private organizations through the software supply chain2, it sent shivers down the spines of the cybersecurity world.  Hackers had secretly broken into SolarWinds, a major U.S. information technology firm, and injected malicious code into the company’s Orion network monitoring product.

By the time the storm landed, FireEye was tracking the hack of the Orion product as UNC2452 (code name: Sunburst).3  “Based on my 25 years in cybersecurity and responding to incidents I’ve concluded, we are witnessing an attack by a nation with top-tier offensive capabilities,” Mr. Mandia said in the blog, adding that the hack was “different from the tens of thousands of incidents we have responded to throughout the years.”4

The following day, SolarWinds President and CEO, Kevin B. Thompson, reported to the Securities and Exchange Committee (SEC), that SolarWinds has been made aware of a cyberattack that inserted a vulnerability within its Orion monitoring products, and that this incident was likely the result of a highly sophisticated, targeted supply chain attack by an outside nation-state.  SolarWinds reported that up to 18,000 of its private and public customers may have been affected.5

The response from the U.S. Government was swift. A Joint Statement by the Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) confirmed the significant and ongoing cybersecurity campaign, and formed a Cyber Unified Coordination Group (UCG) to coordinate a whole-of-government response to this significant cyber incident.6 As the lead for asset response activities, CISA took immediate action and issued an Emergency Directive instructing federal and civilian agencies to immediately disconnect or power down affected SolarWinds Orion products from their network.7

The Joint Statement did not identify the attackers, but the Kaspersky Report stated that while looking at the Sunburst backdoor, they discovered several features that overlap with a previously identified backdoor hack called Kazuar, which had been linked to the Turla group.8  Both Sunburst and Kazuar include a near identical sleeping algorithm, and the extensive usage of the FNV-1a hash.9  The Turla group has been in operation for many years and was considered so lethal, that in 2019, the National Security Agency (NSA) released a Cybersecurity Advisory reporting that the Turla group (also known as Waterbug or VENOMOUS BEAR), was the Russian Advanced Persistent Threat (APT) actor that has used a range of tools and techniques to target government, military, technology, energy and commercial organizations for the purposes of intelligence collection.10

Security Assertion Markup Language (SAML)

The SolarWinds breach was particularly pernicious for organizations because once hackers were able to gain a foothold into IT infrastructure, legitimate certificates were created to mask normal user activity. The National Security Agency released a Cybersecurity Advisory, stating that, “Malicious cyber actors are abusing trust in federated authentication environments to access protected data. The exploitation occurs after the actors have gained initial access to a victim’s on-premises network.”11

The advisory added, “the actors compromise on-premises components of a federated [Single Sign-On] SSO infrastructure and steal the credential or private key that is used to sign Security Assertion Markup Language (SAML) tokens. Using the private keys, the actors then forge trusted authentication tokens to access cloud resources.”11

In order to authenticate web applications, computers communicate by the SAML protocol.  SAML tokens identify users, and then allows applications and organizations to communicate and trust each other.

The breach of the SAML tokens was confirmed by a CISA Alert stating that, “there are initial access vectors other than the SolarWinds Orion platform and has identified legitimate account abuse as one of these vectors.  Specifically, we are investigating incidents in which activity indicating abuse of Security Assertion Markup Language (SAML) tokens.”12

CISA added that, “the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment.  These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”12

The reports by the NSA and the CISA confirmed that after breaching the entrance, intruders created valid SAML tokens to impersonate normal activity.  Armed with valid SAML tokens, the imposters were able to roam freely throughout the premises.

Golden SAML Attack

The fact that SAML tokens could be falsified was no secret in the cybersecurity world. In 2017, on his company blog, Shaked Reiner, posted a new attack vector discovered by CyberArk Labs and dubbed “golden SAML.”   Reiner stated that, “The vector enables an attacker to create a golden SAML, which is basically a forged SAML “authentication object,” and authenticate across every service that uses SAML 2.0 protocol as an SSO mechanism.”13

In a golden SAML attack, attackers can gain access to any application that supports SAML authentication with any privileges they desire and be any user on the targeted application (even one that is non-existent in the application in some cases).13

Figure 1 below displays the sequence of transactions that occur between the Service Provider and the Identity Provider. In this example, a web browser (User Agent) is accessing an application (Service Provider) after being authenticated by the Identity Provider. 

Figure 1. SAML Web Browser Authentication Sequence14
  1. User attempts to access the applications on the server (Service Provider). The Service Provider (SP) in turn checks to see if the user is already authenticated within the system.
  2. The Service Provider redirects the user to that Identity Provider (IdP).
  3. The web browser sends an authentication request to the SSO service which identifies the user.
  4. The SSO service returns an XHTML document, which includes the authentication information needed by the Service Provider in a SAMLResponse parameter.
  5. The SAMLResponse parameter is passed on to the Service Provider.
  6. The Service Provider processes this response and grants or denies the user’s request.
  7. With this information, the user can gain access to the web site.
  8. User then interacts with the application.

In a golden SAML attack, the breach occurs when the SAMLResponse is sent. The SAMLResponse object is what the IdP sends to the SP. This data allows the SP to identify and authenticate the user.

As Reiner states, “Golden SAML is not a vulnerability per se, but it gives attackers the ability to gain unauthorized access to any service in a federation with any privileges and to stay persistent in this environment in a stealthy manner.”13

What Can Be Done?

The full extent of the SolarWinds Hack (with its multiple attack vectors), remains to be determined.  However, as the impact of the breach unfolds, it appears that the remediation and reconstitution of the identity and trust services will be very expensive. “If the adversary has compromised administrative level credentials in an environment — or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned.

“In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate.”

“In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”12

In a recent review of the SolarWinds breach, Shaked Reiner elaborated on the golden SAML attack employed by the adversary. “The threat actor (UNC2452) did a phenomenal job across all stages of the attack — from meticulously planting the backdoor code and making it look like yet another legitimate class, to avoiding almost every possible forensics/analysis tool you can name, and even hiding data in DNS queries and making all the traffic look as if it’s part of the SolarWinds Orion communication protocol. There’s no doubt the threat actor knew what they were doing and tried to do everything in the best way possible.”15

Reiner warned that the SAML certificate breach will be difficult to remediate. “But if an attacker steals your SAML token signing certificate, it’s a whole different ball game. First, if you will naively try to change your passwords, the attacker can easily continue to make SAML tokens that impersonate you, without the need to know the actual password. So, what you really need to do is to change the actual token signing certificate. Which basically means reestablishing the trust across your entire federation.”15


In 2013, Russian General, Valery Gerasimov, published a 2,000 word article, “The Value of Science Is in the Foresight,” in the weekly Russian trade paper Military-Industrial Kurier.16  The ‘Gerasimov Doctrine’ is considered to be the most useful articulation of Russia’s modern strategy for a ‘hybrid war.’ The document outlines a non-linear approach, and capitalizes on deception, psychological manipulation, and domination of the information domain. Gerasimov wrote: “The very ‘rules of war’ have changed. The role of nonmilitary means of achieving political and strategic goals has grown, and, in many cases, they have exceeded the power of force of weapons in their effectiveness. All this is supplemented by military means of a concealed character.”

The Gerasimov Doctrine builds a framework for new tools where chaos is the strategy.17  The objective is to achieve an environment of permanent unrest and conflict within an enemy state.18  By selectively amplifying targeted disinformation and misinformation on social media, Russia has attacked our core.  Democracy is sacred to our identity as Americans, and now, here we are, arguing about the legitimacy of the last two presidential elections.  We are fighting ourselves, and the enemy never fired a shot.  “The information space opens wide asymmetrical possibilities for reducing the fighting potential of the enemy,” Gerasimov writes.

Not all Russia-watchers agree on the Gerasimov Doctrine’s importance, or if it even exists.19 But there is no question that Russian intervention is systematic and multi-faceted.  It is a guerilla tactic, so we do not necessarily understand how it has been put into motion.

On February 23, 2021, the 117th Congress kicked off its First Session with, among other initiatives, oversight hearings on the SolarWinds hack.  The Senate Intelligence Committee held a hearing on the high profile breach, followed by a joint hearing in the House of Representatives held by the Oversight and Reform and Homeland Security Committees.  At both hearings, Sudhakar Ramakrishna, CEO of SolarWinds, and Kevin Mandia, CEO of FireEye, testified.

Eventually, the world would come to understand what happened.  According to SolarWinds CEO, Sudhakar Ramakrishna, an intern who worked at the company posted the “solarwinds123” password on a public website.  Ramakrishna testified that, “I believe that was a password that an intern used on one of his servers back in 2017, which was reported to our security team, and it was immediately removed.”20  

We would hear from FireEye CEO, Kevin Mandia, who would testify that his FireEye ‘Red Team’ had created powerful tools to help our nation defend itself, but the tools had been stolen and were now being used against his security firm.  He stated, “The bottom line:  We may never know the full range and extent of damage, and we may never know the full range and extent as to how the stolen information is benefitting an adversary.”20

Unfortunately, all of this closed-door testimony would have to wait.  The original hearings scheduled on January 6, 2021, were interrupted by another breach.  This time it was not the intern. lock
Scott Smurthwaite


Mandia, Kevin. “Unauthorized Access of FireEye Red Team Tools”. December 8, 2020.

Mandia, Kevin. “Global Intrusion Campaign Leverages Software Supply Chain Compromise”. December 13, 2020.

Mandia, Kevin. “Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor”. December 13, 2020.

Mandia, Kevin. “FireEye Shares Details of Recent Cyber Attack, Actions to Protect Community”. December 8, 2020.

Securities and Exchange Commission. “Form 8-K”. December 14, 2020.

Department of Homeland Security. “Joint Statement by the Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) [Press release]”.  December 17, 2020.

Cybersecurity and Infrastructure Security Agency.  Emergency Directive 21-01, “Mitigate SolarWinds Orion Code Compromise”. December 13, 2020.

MITRE ATT&CK. “Turla.” May 31, 2017.

Kucherin, G., Kuznetsov I., Raiu, C. “Sunburst backdoor – code overlaps with Kazuar”. January 11, 2021.

National Security Agency. “Turla Group Exploits Iranian APT To Expand Coverage of Victims”.  Cybersecurity Advisory. October 18, 2019.

National Security Agency. “Detecting Abuse of Authentication Mechanisms”. Cybersecurity Advisory. December, 2020.

CISA Alert (AA20-352A). “Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations”. December 17, 2020.

Reiner, Shaked. “Golden SAML: Newly Discovered Attack Technique Forges Authentication to Cloud Apps.” November 21, 2017.

Fruhlinger, Josh. “What is SAML? How it works and how it enables single sign on.” October 12, 2017.

Reiner, Shaked. “Golden SAML Revisited: The Solorigate Connection.” Security Boulevard.  December 29, 2020.

Gerasimov, Valery. “The value of science in foresight.”  February 26, 2013.

Chivvus, Christopher. “Understanding Russian ‘Hybrid Warfare.’” March 22, 2017.

Helmus, T. C., Bodine-Baron, E., Radin, A., Magnuson, M., Mendelsohn, J., Marcellino, W., Bega, A., Winkelman Z. “Russian Social Media Influence.” Rand Corporation 2018.

Galeotti, Mark. “I’m Sorry for Creating the Gerasimov Doctrine”. March 5, 2018.

House Committee on Oversight and Reform. “Oversight and Homeland Security Committees Discussed Next Steps for Government and Private Tech Following SolarWinds Breach.” February 26, 2021.

Leave a Comment