The Cyber Safety Review Board (CSRB), created in 2021 to review major cyber events, released a report last summer recapping the 2021 discovery of the Log4j vulnerability. Its disclosure triggered a global race between malicious hackers seeking to exploit a flaw in Java code and the security experts seeking to stop them. Among its findings, the report revealed that the response to Log4j by one federal cabinet department entailed 33,000 hours of staff time – the equivalent of engaging more than 16 full-time employees for a year. It’s a large figure, but hardly shocking. Many federal agencies don’t have the tools to easily detect and remediate IT vulnerabilities, including the Log4j exploitation. With so many applications relying on the logging framework, this vulnerability had the capacity to disrupt entire software supply chains – and it did.
Using customer data, a Veracode report found that nearly 58 percent of organizations were using a vulnerable version of Log4j following the zero-day vulnerability discovery on Dec. 9, 2021. (The “lucky” remaining 42 percent weren’t saved by proper hygiene practices, necessarily. Many were simply using an older version of Log4j which had not yet introduced the vulnerability into the codebase.)
With a library this ubiquitous, finding all instances of direct use is hard; ferreting out linked dependencies is an even more cumbersome challenge. The CSRB report was a wake-up call – a reminder for agencies to improve cyber hygiene practices before the next Log4j-like vulnerability emerges. How? The route to remediation begins with patching.
The reported time and cost of patching vulnerable areas affected by Log4j (after countless scans to find what exactly needed patching) didn’t surprise cybersecurity professionals. We have known for some time that widespread vulnerabilities like Log4j can be difficult to contain.
As software goes, Log4j is incredibly widespread. It has been around for a while, and everyone was using it, either directly or indirectly (by using something dependent on Log4j), hence the monumental impact on the software supply chain following the discovery.
In addition, the Log4j vulnerability was particularly easy to exploit. Its short and simple code was ideally suited to exploiting in a tweet, enabling malicious actors to automate and rapidly scale the exploitation. The stars were aligned for hopeful hackers in the form of Log4j’s prevalence and the ease with which it was exploited.
Following the discovery of the Log4j vulnerability, agencies scrambled to uncover its use in their systems. For some organizations, this was a multi-phase process: determining the best tool for the job, procuring licenses, and assembling a team to complete the mission. Agencies that hadn’t invested in software testing before the discovery of Log4j faced a daunting task.
On the flip side, agencies that had diligently built capacity to inventory software were able to consult a list to sift through for affected areas.
The Software Supply Chain: A Not-So-Hidden Culprit
Though certainly a labor-intensive and costly remediation, fixing the Log4j vulnerability itself was likely not the cause of the massive employee labor figure. The real issue stemmed from the complexity of the software supply chain. Stopping the flaw was fairly straightforward for those who knew exactly where patching was needed. Unsurprisingly, most didn’t.
The subject of various legislation pieces, agencies have long been urged to enhance the security and integrity of their software supply chains. When the Executive Order on Improving the Nation’s Cybersecurity was released in May 2021, organizations should have strived to get ahead of the threat curve and avoid vulnerability issues before they occurred.
To do this, agencies should have been aware of the components of the software their organizations use. This starts with taking advantage of software bills of material (SBOMs) — the IT equivalent of labels that list ingredients in packaged foods. SBOMs give organizations the ability to ensure they are utilizing software that has been built following common cybersecurity practices.
A key component of “shifting left,” organizations using tools to generate SBOMs in December 2021 would have found and patched Log4j vulnerabilities in much less time than those who had to scour their supply chains for affected areas before beginning patching efforts. The tool of choice? Software Composition Analysis.
Software Composition Analysis: The Key to Staying Ahead of Vulnerabilities
Using an SCA tool enables agencies to take care of the first part of the equation: inventory. A robust tool providing SCA will keep users up to date with constantly evolving open-source libraries by automating the discovery and fixing of vulnerabilities. Take care to ensure that SCA tooling is applied across your agency’s entire portfolio.
Once you understand the scope of your engineering work, you’re ready to get started on the to-do list. In the case of Log4j, users that had applied appropriate SCA tooling across their portfolios were able to see exactly where they were using Log4j, so no time was wasted backtracking and determining where to start with their mitigation efforts. Having a curated list to prioritize and act on is key to keeping up with, and staying ahead of, security debt.
While open-source software (OSS) in many cases saves developers from having to reinvent the wheel, we need to realize and safeguard against the inherent risk of OSS. Knowing that vulnerabilities are inevitable allows us to sharpen and build cyber hygiene before the next Log4j happens. Keeping inventory of code and updating software will give you an upper hand in the fight against future attacks.
Moving Forward: Preparing for the Next Log4j
We can’t solve 2021’s challenge, but there are steps agencies can take now to avoid the effects of another Log4j-like catastrophe.
Keeping software up to date is paramount. According to Veracode’s State of Software Security v11: Open Source Edition, an alarming 79 percent of developers never update third-party libraries after including them in the codebase. The more out of date your software is, the harder it will be to patch.
It’s important to take note of inventory and to update the list as needed, but simple awareness of inventory is not enough. With new vulnerabilities discovered daily, agencies must take steps now to minimize security debt piling up in code. This will save effort, time, and funds down the line.
Robust software development practices should be encouraged. Engineering teams will appreciate the rigor when the next major vulnerability comes to light, so it’s important to make the investment as soon as possible.
Now is the time to invest in tools that can analyze the extent to which a given vulnerability will affect your agency. Back in December 2021, having a tool that provided SCA scanning could have helped agencies more quickly remedy the effects of the Log4j vulnerability. Instead, many scrambled to pinpoint the use of the flawed library across their systems, wasting precious time and resources.
Agencies should secure their software supply chains through regular scanning and rapid patching, when needed, before it’s too late. Though the task may seem intimidating, building an inventory list will save your agency time, money, and stress in the future – a worthwhile effort given today’s constantly evolving cyber threat landscape.
The views expressed here are the writer’s and are not necessarily endorsed by Homeland Security Today, which welcomes a broad range of viewpoints in support of securing our homeland. To submit a piece for consideration, email [email protected]