We wrote tests for our third-party security libraries, and you won’t believe what happened next! (CVE-2017-8028)

On the importance of thorough testing

Much of modern software development revolves around the concept of “quality”. As with all abstract concepts, “quality” is somewhat difficult to pin down, but for this article we can define it as “how well software conforms to its requirements”. Less formally, we can say that “high-quality software does what it’s supposed to”.

One way to ensure that you consistently deliver high-quality software is to have automated tests. I’m not a Test-Driven Development fundamentalist insisting on 100% test coverage; but if some part of your software is important to your users, there should be at least one automated test demonstrating that it works. This is obviously true for your software’s functional requirements, but it applies equally to its non-functional requirements such as auditing and performance.

So what non-functional requirements are important to the users of Basefarm’s internally-developed applications?
One thing that immediately springs to (at least my) mind is security. So we need some tests for that, right?

Think like a hacker

When developing secure applications, it is always useful to try to think like an attacker. If I were to try to get unauthorized access to this application, how would I go about it?

Obviously, you’d first need to log in. For Basefarm’s internal applications we use username/password logins authenticating against our internal LDAP server; this means that an attacker must obtain a set of working credentials somehow.

This suggests three tests:

  1. An existing user can log in with the correct password (this is what we call the happy-path test, which demonstrates that in fair weather conditions the software works as intended);
  2. A non-existent user is not allowed access;
  3. An existing user logging in with an incorrect password is not allowed access

As part of these tests, it is also useful to think about things like auditing and information leakage; login attempts — successful or not — should be logged, and the error message(s) presented on failure should not give an attacker information she does not already have. In particular, authentication failures should not say “no such user” or “incorrect password”; “Invalid credentials” is a safe, neutral way to put it.

So, in pseudo-code, we have the following tests:

authenticate("existinguser", "correctpassword") must succeed

authenticate("nonexistentuser", "anypassword") must fail

authenticate("existinguser", "wrongpassword") must fail

Don’t reinvent the wheel

The developers at Basefarm, as in most other places, don’t have the time (or, to be honest, the expertise) to write everything from scratch. It is hard enough to implement our specific functionality; implementing and maintaining code to talk to databases and LDAP, handle authentication and transactions and all the other things a large-scale (dare I say Enterprise?) application needs would be impossible. So like many other teams on the JVM platform we lean heavily on the Spring framework and its attendant ecosystem of libraries; in particular, we use Spring Security for authentication and authorization.

Setting the scene

Since our LDAP server (as all good LDAP servers should) requires authentication and encrypted communication using STARTTLS, our Spring Security configuration looks something like this (ignoring some details on how, exactly, things are wired together):

<bean id="tlsAuthStrategy" 
      class="org.springframework.ldap.core.support.DefaultTlsDirContextAuthenticationStrategy" />

<ldap:context-source 
  url="ldap://monkeymachine:389/dc=springframework,dc=org" 
  username="cn=Manager,dc=springframework,dc=org" password="secret" 
  authentication-strategy-ref="tlsAuthStrategy" />

<security:authentication-manager>
  <security:ldap-authentication-provider 
    user-search-base="ou=People,dc=springframework,dc=org" 
    user-search-filter="(uid={0})" />
</security:authentication-manager>   

The idea is that the security framework, authenticating with the manager credentials, searches for a user to really authenticate as.

We believe this to be a fairly typical LDAP-auth setup; in fact, the actual configuration is more or less copy-pasted from the Spring Security documentation.

We originally implemented this in 2011; and the configuration worked well, all the tests passed, and we were happy.

Let’s dance!

Fast-forward six years, to 2017. As part of our regular maintenance cycle, I went through all of our application’s dependencies and updated them to the latest version, to get the benefit of whatever new features and fixes are available.

This is usually a cushy job: Update some version numbers; recompile everything; drink coffee while the tests pass; and then ship it.

So I upgrade Spring Security to 4.2.1, and start the test run.

Imagine my surprise when I returned from my coffee binge to see

Tests failed: 1

Waltz Tango Foxtrot?

And the test that failed?

authenticate("existinguser", "wrongpassword") must fail

Whisky Tequila Fernet? In short, WTF???

Finding the problem

I’ll spare you the details of how we debugged the issue; it’s not really interesting, involving as it did copious amounts of coffee, creative swearing, scratching of heads, wailing and gnashing of teeth, and poring over code, logs and packet captures.

It turned out that the problem was not in Spring Security per se, but in Spring LDAP; a change in Spring Security’s use of that library exposed a weakness in DefaultTlsDirContextAuthenticationStrategy, which never actually performs an LDAP “bind” operation using the provided credentials.

The desired sequence of events when authenticating in our setup is something like this:

  1. Open a new LDAP connection
  2. Secure it using STARTTLS
  3. Bind with the search credentials (username/password from the context-source)
  4. Perform an LDAP search for the DN to bind as for the real authentication step
  5. Open another new LDAP connection
  6. Secure it using STARTTLS
  7. Bind using the DN from step 4 and the provided password

The logs from the LDAP server show what really happens (I’ve removed some entries from the log; the full log can be seen here):

-- step 1
59ceb606 conn=1000 fd=15 ACCEPT from IP=172.17.0.1:33378 (IP=0.0.0.0:389)
-- step 2
59ceb606 conn=1000 op=0 STARTTLS
-- step 3
59ceb606 conn=1000 op=1 BIND dn="cn=admin,dc=example,dc=org" mech=SIMPLE ssf=0
-- step 4
59ceb606 conn=1000 op=2 SRCH base="ou=People,dc=example,dc=org" scope=2 deref=3 filter="(cn=user)"
59ceb606 conn=1000 fd=15 closed
-- step 5
59ceb606 conn=1001 fd=15 ACCEPT from IP=172.17.0.1:33380 (IP=0.0.0.0:389)
-- step 6
59ceb606 conn=1001 op=0 STARTTLS
59ceb606 conn=1001 fd=15 closed (connection lost)

Step 7 never happens; as long as you have a username, all passwords are accepted.

Further investigation showed that this had been noticed as far back as 2013 by one mwebb and reported as a bug in November 2016, but not recognized as a security issue.

We had also noticed it before, earlier in 2017, but due to time pressures we did not properly investigate the issue at that time.

Houston, you may want to look at this

The Spring Framework is currently maintained by Pivotal Software, Inc., so we notified their security team of the problem. They, in turn, notified the maintainers of Spring LDAP, who initially concluded that this is not a bug because anonymous LDAP access was involved, as it was in the original example project exhibiting the bug.

However, a slight modification of the example project showed that the erroneous behaviour persists even when anonymous bind is not used; it is clear from the log that there is no bind attempt with the user credentials at all.

After a little back and forth, the Spring LDAP team released a fixed version of Spring LDAP on October 6, 2017, nearly a year after the initial bug report.

Timeline

2016-11-11 (approximately)
Tobias Schneider discovers the issue
2016-11-18
Schneider reports the bug to Spring LDAP, with a pull request
2017-01-11
We initially notice the issue, but due to time pressures do not investigate and roll back to an earlier version
2017-08-10
We attempt to upgrade again; again, we roll back to the earlier, working version, but this time (probably due to higher blood caffeine levels) a full investigation is scheduled
2017-09-26
We isolate the problem and discover the earlier bug report
2017-09-28
We notify Pivotal’s security team that we consider this a security vulnerability
2017-09-29
The Spring LDAP team determines that this is not a vulnerability
We demonstrate that the issue exists without involving anonymous search
2017-10-06
The Spring LDAP team commits a fix and releases a working version
2017-10-11
We request a CVE via Dell EMC and Pivotal
2017-10-16
Pivotal publishes CVE-2017-8028

The moral of the story (a.k.a TL;DR)

  • Security is an important part of your application; write tests for it
  • All non-trivial software has bugs, and some of these will be security-related
  • Report security issues to the security contact for the involved software