We wrote tests for our third-party security libraries, and you won’t believe what happened next! (CVE-2017-8028)

On the importance of thorough testing

Much of modern software development revolves around the concept of “quality”. As with all abstract concepts, “quality” is somewhat difficult to pin down, but for this article we can define it as “how well software conforms to its requirements”. Less formally, we can say that “high-quality software does what it’s supposed to”.

One way to ensure that you consistently deliver high-quality software is to have automated tests. I’m not a Test-Driven Development fundamentalist insisting on 100% test coverage; but if some part of your software is important to your users, there should be at least one automated test demonstrating that it works. This is obviously true for your software’s functional requirements, but it applies equally to its non-functional requirements such as auditing and performance.

So what non-functional requirements are important to the users of Basefarm’s internally-developed applications?
One thing that immediately springs to (at least my) mind is security. So we need some tests for that, right?

Think like a hacker

When developing secure applications, it is always useful to try to think like an attacker. If I were to try to get unauthorized access to this application, how would I go about it?

Obviously, you’d first need to log in. For Basefarm’s internal applications we use username/password logins authenticating against our internal LDAP server; this means that an attacker must obtain a set of working credentials somehow.

This suggests three tests:

  1. An existing user can log in with the correct password (this is what we call the happy-path test, which demonstrates that in fair weather conditions the software works as intended);
  2. A non-existent user is not allowed access;
  3. An existing user logging in with an incorrect password is not allowed access

As part of these tests, it is also useful to think about things like auditing and information leakage; login attempts — successful or not — should be logged, and the error message(s) presented on failure should not give an attacker information she does not already have. In particular, authentication failures should not say “no such user” or “incorrect password”; “Invalid credentials” is a safe, neutral way to put it.

So, in pseudo-code, we have the following tests:

authenticate("existinguser", "correctpassword") must succeed

authenticate("nonexistentuser", "anypassword") must fail

authenticate("existinguser", "wrongpassword") must fail

Don’t reinvent the wheel

The developers at Basefarm, as in most other places, don’t have the time (or, to be honest, the expertise) to write everything from scratch. It is hard enough to implement our specific functionality; implementing and maintaining code to talk to databases and LDAP, handle authentication and transactions and all the other things a large-scale (dare I say Enterprise?) application needs would be impossible. So like many other teams on the JVM platform we lean heavily on the Spring framework and its attendant ecosystem of libraries; in particular, we use Spring Security for authentication and authorization.

Setting the scene

Since our LDAP server (as all good LDAP servers should) requires authentication and encrypted communication using STARTTLS, our Spring Security configuration looks something like this (ignoring some details on how, exactly, things are wired together):

<bean id="tlsAuthStrategy" 
      class="org.springframework.ldap.core.support.DefaultTlsDirContextAuthenticationStrategy" />

<ldap:context-source 
  url="ldap://monkeymachine:389/dc=springframework,dc=org" 
  username="cn=Manager,dc=springframework,dc=org" password="secret" 
  authentication-strategy-ref="tlsAuthStrategy" />

<security:authentication-manager>
  <security:ldap-authentication-provider 
    user-search-base="ou=People,dc=springframework,dc=org" 
    user-search-filter="(uid={0})" />
</security:authentication-manager>   

The idea is that the security framework, authenticating with the manager credentials, searches for a user to really authenticate as.

We believe this to be a fairly typical LDAP-auth setup; in fact, the actual configuration is more or less copy-pasted from the Spring Security documentation.

We originally implemented this in 2011; and the configuration worked well, all the tests passed, and we were happy.

Let’s dance!

Fast-forward six years, to 2017. As part of our regular maintenance cycle, I went through all of our application’s dependencies and updated them to the latest version, to get the benefit of whatever new features and fixes are available.

This is usually a cushy job: Update some version numbers; recompile everything; drink coffee while the tests pass; and then ship it.

So I upgrade Spring Security to 4.2.1, and start the test run.

Imagine my surprise when I returned from my coffee binge to see

Tests failed: 1

Waltz Tango Foxtrot?

And the test that failed?

authenticate("existinguser", "wrongpassword") must fail

Whisky Tequila Fernet? In short, WTF???

Finding the problem

I’ll spare you the details of how we debugged the issue; it’s not really interesting, involving as it did copious amounts of coffee, creative swearing, scratching of heads, wailing and gnashing of teeth, and poring over code, logs and packet captures.

It turned out that the problem was not in Spring Security per se, but in Spring LDAP; a change in Spring Security’s use of that library exposed a weakness in DefaultTlsDirContextAuthenticationStrategy, which never actually performs an LDAP “bind” operation using the provided credentials.

The desired sequence of events when authenticating in our setup is something like this:

  1. Open a new LDAP connection
  2. Secure it using STARTTLS
  3. Bind with the search credentials (username/password from the context-source)
  4. Perform an LDAP search for the DN to bind as for the real authentication step
  5. Open another new LDAP connection
  6. Secure it using STARTTLS
  7. Bind using the DN from step 4 and the provided password

The logs from the LDAP server show what really happens (I’ve removed some entries from the log; the full log can be seen here):

-- step 1
59ceb606 conn=1000 fd=15 ACCEPT from IP=172.17.0.1:33378 (IP=0.0.0.0:389)
-- step 2
59ceb606 conn=1000 op=0 STARTTLS
-- step 3
59ceb606 conn=1000 op=1 BIND dn="cn=admin,dc=example,dc=org" mech=SIMPLE ssf=0
-- step 4
59ceb606 conn=1000 op=2 SRCH base="ou=People,dc=example,dc=org" scope=2 deref=3 filter="(cn=user)"
59ceb606 conn=1000 fd=15 closed
-- step 5
59ceb606 conn=1001 fd=15 ACCEPT from IP=172.17.0.1:33380 (IP=0.0.0.0:389)
-- step 6
59ceb606 conn=1001 op=0 STARTTLS
59ceb606 conn=1001 fd=15 closed (connection lost)

Step 7 never happens; as long as you have a username, all passwords are accepted.

Further investigation showed that this had been noticed as far back as 2013 by one mwebb and reported as a bug in November 2016, but not recognized as a security issue.

We had also noticed it before, earlier in 2017, but due to time pressures we did not properly investigate the issue at that time.

Houston, you may want to look at this

The Spring Framework is currently maintained by Pivotal Software, Inc., so we notified their security team of the problem. They, in turn, notified the maintainers of Spring LDAP, who initially concluded that this is not a bug because anonymous LDAP access was involved, as it was in the original example project exhibiting the bug.

However, a slight modification of the example project showed that the erroneous behaviour persists even when anonymous bind is not used; it is clear from the log that there is no bind attempt with the user credentials at all.

After a little back and forth, the Spring LDAP team released a fixed version of Spring LDAP on October 6, 2017, nearly a year after the initial bug report.

Timeline

2016-11-11 (approximately)
Tobias Schneider discovers the issue
2016-11-18
Schneider reports the bug to Spring LDAP, with a pull request
2017-01-11
We initially notice the issue, but due to time pressures do not investigate and roll back to an earlier version
2017-08-10
We attempt to upgrade again; again, we roll back to the earlier, working version, but this time (probably due to higher blood caffeine levels) a full investigation is scheduled
2017-09-26
We isolate the problem and discover the earlier bug report
2017-09-28
We notify Pivotal’s security team that we consider this a security vulnerability
2017-09-29
The Spring LDAP team determines that this is not a vulnerability
We demonstrate that the issue exists without involving anonymous search
2017-10-06
The Spring LDAP team commits a fix and releases a working version
2017-10-11
We request a CVE via Dell EMC and Pivotal
2017-10-16
Pivotal publishes CVE-2017-8028

The moral of the story (a.k.a TL;DR)

  • Security is an important part of your application; write tests for it
  • All non-trivial software has bugs, and some of these will be security-related
  • Report security issues to the security contact for the involved software

BF-SIRT Newsletter 2017-43

This weeks top stories is that Bad Rabbit, a new Petya-like ransomware is spreading, and Reaper, a new Mirai-like Iot botnet, has been detected and is many times larger.

A recent report concludes that cybercriminals focus on the shipping and cloud storage sectors, and Kaspersky is in hard weather against the US intelligence industry, but says NSA contractor leaked US hacking tools by mistake.

Dell lost control of a key customer support domain for a month in 2017 and Google Play Protect is literally the worst at detecting malware on Android, according to test results.

Top 5 Security links
Bad Rabbit: A new Petya-like ransomware that’s spreading, but beatable
After quietly infecting a million devices, Reaper botnet set to be worse than Mirai
NSA contractor leaked US hacking tools by mistake, Kaspersky says
Cybercriminals Focus on the Shipping and Cloud Storage Sectors
Dell Lost Control of Key Customer Support Domain for a Month in 2017

BF-SIRT Newsletter 2017-42

This weeks top stories is that a serious flaw in the WPA2 protocol lets attackers intercept network traffic (KRACK), and a factorization flaw in TPM chips makes attacks on RSA private keys feasible (ROCA).

You can also read about how Oracle fixes 20 remotely exploitable Java SE vulnerabilities in their latest patch, and that Flash is hit by another zero-day vulnerability that is actively exploited.

Top 5 Security links
Serious flaw in WPA2 protocol lets attackers intercept passwords and much more
Factorization Flaw in TPM Chips Makes Attacks on RSA Private Keys Feasible
Oracle Fixes 20 Remotely Exploitable Java SE Vulns
Flash hit by another zero-day vulnerability
Third-Party Code Hack Leads to Compromised Equifax Site Serving Fake Flash Install

BF-SIRT Newsletter 2017-41

This weeks top stories is that Kaspersky reportedly modified its AV to help Russia Government spy, and in the latest string of AWS S3 bucket embarrassments Accenture left four servers of sensitive data completely unprotected.

You can also read about how the Equifax website is found to be hacked again, and that hackers steals 60 million USD from Taiwanese bank in tailored SWIFT attack.

Top 5 Security links
Kaspersky reportedly modified its AV to help Russia steal NSA secrets
Accenture left four servers of sensitive data completely unprotected
Equifax website hacked again, this time to redirect to fake Flash update
Hackers nick $60m from Taiwanese bank in tailored SWIFT attack
Outlook bug meant S/MIME emails were sent unencrypted for months

IS THE FINANCE INDUSTRY AFRAID OF THE CLOUD?

The cloud promises flexibility and efficiency, allowing the service to continue to grow in popularity. In our cloud report we see huge differences when it comes to adoption rates in different industries. The banking and finance sector are well behind other industries, and yet they are relatively dependent on technology – why?

BF-SIRT Newsletter 2017-40

This weeks top stories is that Yahoo says all 3 billion accounts was hit by the 2013 hack and Google Security researchers have discovered seven serious vulnerabilities in Dnsmasq.

Security researchers have developed a variant of the Rowhammer attack that is able to bypass all the current countermeasures proposed for such an attack, and Apache Tomcat patches important remote code execution flaw.

Top 5 Security links
Yahoo says all 3 billion accounts hit by 2013 hack
Widely used DNS forwarder and DHCP server Dnsmasq riddled with flaws
Rowhammer Variant Bypasses Countermeasures
Apache Tomcat Patches Important Remote Code Execution Flaw
3 Zero-Day Plugin Vulnerabilities Being Exploited In The Wild

Seven ways to survive the cloud

Staying relevant means a digital transition. One way to achieve this is to work in an agile manner, and to use the possibilities offered by the cloud.