Rarely does anything have a defined turning point in its history, a single day where people can point and say that was the day everything changed.
For OpenSSL, that day was April 7, 2014, the day that Heartbleed became part of the security lexicon. Heartbleed was a critical vulnerability in the venerable crypto library. OpenSSL is everywhere, in tens of thousands of commercial and homespun software projects. And so too, as of last April, was Heartbleed, an Internet-wide bug that leaked enough memory that a determined hacker could piece together anything from credentials to encryption keys.
It was a bad day for server admins worldwide and an equally terrible day for the small crew of folks who kept OpenSSL afloat. Like natural disasters in the real world, people donât understand the shaky footing some things are balancing upon until an earthquake or hurricane tears through town. That certainly applies to OpenSSL, for Heartbleed made people take notice that, despite the library living seemingly in everything everywhere, it was barely funded and sadly under-resourced.
âTwo years ago, it was a night-and-day difference. Two years ago, aside from our loyal user community, we were invisible. No one knew we existed,â says Steve Marquess, cofounder, president and business manager of the OpenSSL Foundation, the corporate entity that handles commercial contracting for OpenSSL. âOpenSSL is used everywhere: hundreds, thousands of vendors use it; every smartphone uses it. Everyone took that for granted; most companies have no clue they even used it.â
To say OpenSSL has been flipped on its headâin a good wayâis an understatement.
Heartbleed made the tech world realize that the status quo wasnât healthy to the security and privacy of ecommerce transactions and communication worldwide. Shortly after Heartbleed, the Core Infrastructure Initiative was created, uniting The Linux Foundation, Microsoft, Facebook, Amazon, Dell, Google and other large technology companies in funding various open source projects. OpenSSL was the first beneficiary, getting enough money to hire Dr. Steve Henson and Andy Polyakov as its first full-timers. Henson, who did not return a request to be interviewed for this article, is universally known as the one steady hand that kept OpenSSL together, an unsung hero of the project who along with other volunteers handled bug reports, code reviews and changes.
Now with steady money flowing in, Henson and Polyakov are flanked by a battery of compensated help that is executing on a road map to clean up the OpenSSL codebase, institute processes for critical changes, and how to communicate those changes. A laborious refactoring is under way, as is a crucial overhaul of the TLS state machine, super important projects that could see the light of day by the summerâjust in time for the results of an audit of the current OpenSSL code being conducted by NCC Group Cryptography Services, the same outfit that recently completed an audit of TrueCrypt.
** OpenSSL Past: Pre-Heartbleed**
âOpenSSL pre-Heartbleed was kind of a mess; there was one guy working on it with a few others putting in contributions around the edges,â said Matthew Green, a cryptography expert and professor at Johns Hopkins University. âOver the years, OpenSSL accumulated a lot of cruft. The quality of code was really low and no one was interested in refactoring it. The thing about OpenSSL is that itâs low level code that lives in the critical path of every machine that wants to communicate securely.
âIt lives next to the network stack and that means a lot of data is going through it, and not all of it is friendly data,â Green said. âWhat happened was that OpenSSL was like UNIX or the TCP/IP stack back in the day; it was critical and not mature, and no one had looked at it since the â80s.â
All of that is changing, given the CII funding, and awareness. Rich Salz, a member of the current OpenSSL development team and an engineer at Akamai, has been a longtime contributor. He remembers times he would send patches in to OpenSSL for review and they would languish in limbo because there were few resources to address them in a timely manner.
âOver the years, OpenSSL accumulated a lot of cruft. The quality of code was really low and no one was interested in refactoring it.â
-Matthew Green
âBefore, youâd throw something over the wall and maybe get a response three months later,â Salz said. âNow, thereâs a defined growth path, defined [criticality] rankings, and a process for who coordinates things if thereâs a high-level bug. We have policies and procedures that made the project healthy and got the community more involved. Itâs been a really good thing.â
Kenneth White, a security researcher and director of the Open Crypto Audit Project, said that OpenSSL isnât alone in its shaky standing.
âEven talented technical people with a grasp on Linux or network security donât appreciate how fragile the whole stack is, not just OpenSSL, but the whole stack,â White said. With Heartbleed, POODLE, FREAK and other bugs cutting across a swathe of technologies, White hopes many start paying serious attention.
âYouâve got things like DNS or libcurl everywhere under the hood. So many pieces of the stack that people assume are solid and mature code bases that are still maintained by one or two people. A lot of the foundation of our entire infrastructure in terms of global networking is built on so many layers of trust. If you pull back the covers, there are really so many layers of legacy cruft that needs scrutiny,â White said.
The attention on OpenSSL, for example, shed harsh light on simple things such as code structure. Already, thousands of lines of code have been cleaned of bad indents, for example, using a code cleaner to create a consistent style so that code can be read and haphazard mistakes avoided.
âSome projects contain integration hooks that you canât check if it doesnât follow structure. That sounds silly, but thatâs exactly how gotofail happened,â White said. âItâs a minor footnote, trivial pieces, but theyâre risky because theyâre touching every source file in the project in an automated way.â
OpenSSL Present: Spring Cleaning
The OpenSSL makeover currently under way promises to be a complicated and busy time for the development team. Priority number one is the rewriting of the TLS state machine, which, like much of the rest of the OpenSSL codebase, has been a patchwork effort.
âThe [TLS] state machine is a big thing,â Salz said. âThe whole mechanism, the kindest thing we can say, is that it grew organically. It started with SSL v2, grew from there and things got out of control. Thatâs one of our big areas, the source of many vulnerabilities and bugs in OpenSSL.â
Salz and Green point out that the FREAK vulnerabilities that were disclosed in early March are examples of bugs caused by the fragility and complexity of the state machine. Some SSL clients, OpenSSL included, contained the FREAK vulnerability, which is short for Factoring Related Attack on RSA Keys. The vulnerability allows an attacker to force clients to downgrade to weakened 512-bit RSA keys that are breakable and put encrypted communication at risk through a man-in-the-middle attack.
âItâs going to be brutally exercised in a way that hasnât been done in such a coordinated fashion.â
-Kenneth White
âOpenSSL and TLS are complicated protocols,â Green said. âThey have to know what is coming in at the right time, and reject anything else. These are thousands of lines of code that should be broken up into manageable pieces versus spaghetti-ing the code. Theyâve added more spaghetti through the years, and no one has gone in there with a scalpel and ripped it out. Now theyâre in there designing the right state machine.â
The refactoring, meanwhile, involves overhauling other key pieces of the library such as memory management, the EVP library which provides an interface to crypto functions, OpenSSL input-output abstraction layers, as well as the construction of an entire testing harness, White said, that will allow for a lot of the analysis being done to be repeated in an automated way. Thereâs also a high computation aspect to the refactoring and subsequent audit of OpenSSL.
âItâs going to be brutally exercised in a way that hasnât been done in such a coordinated fashion,â White said. âThis is one of the most ambitious security projects done in a while, thatâs absolutely for sure. We have high ambitions.â
For all its faults, OpenSSL is a solid crypto library, âbattle-hardenedâ as White put it, and one whose core library hasnât changed much through the years.
âIronically OpenSSL is amazingly good at crypto,â Green said. âItâs amazing, but amazing in messy ways, kind of like the brilliant prodigy who can play concertos but wonât clean up his room. The crypto code is amazingly good, but high level stuff is sloppy like parsing packages that didnât get Heartbleed.â
OpenSSL Future: Audit Time
The OpenSSL Audit, sponsored by the Core Infrastructure Initiative, is under way and the first set of results could trickle in by early summer. Like TrueCrypt, OpenSSL developers are curious to see the vulnerabilities dredged up during the inspection, and like its file encryption cousin, have fingers crossed that a backdoor isnât lurking.
âWhat I worry about is entire classes of vulnerabilities,â White said. âAuditing source code is useful, but we donât run source code, we run compiled binaries. What worries me are classes of things that are systematically an issue, not the whack-a-mole thing. With this number of eyes and the caliber of crypto engineers looking at the code, itâs inevitable they will turn over interesting bugs. Occamâs razor has utility in these things. Itâs far more likely to be things in forgotten code or esoteric things, rarely used options than anything malicious.â
âWeâre definitely treating [the audit] as a source of advance QA.â
-Rich Salz
While the audit will provide value to the developers, it is however, just a point-in-time snapshot. Thatâs why things like development of the test harness and continued attention thrust on OpenSSL have engineers most excited.
âWeâre definitely treating [the audit] as a source of advance QA,â Salz said. âThe value to the community and to us will be folks who are really good at it trying to break things. We donât have a dedicated test team and we want to show that this thing works well and that if I give you a million-byte certificate, it wonât fall over and die.â
While OpenSSL continues to collect from the Core Infrastructure Initiative, individual donations have dwindled down to pre-Heartbleed levels.
âWe were getting around $2,000 a year in donations in the years heading up to Heartbleed, and weâre back to that now,â Marquess said. âWe cherish those because individual people think OpenSSL is that important. Itâs also interesting how much we get from parts world where privacy is a life-and-death matter with messages of how important crypto is to them.â
threatpost.com/apple-fixes-certificate-validation-flaw-in-ios/104427
threatpost.com/group-backed-by-google-microsoft-to-help-fund-openssl-and-other-open-source-projects/105672
threatpost.com/new-freak-attack-threatens-many-ssl-clients/111390
threatpost.com/openssl-fixes-tls-vulnerability/105300
threatpost.com/openssl-security-audit-ready-to-start/111538
www.openssl.org/docs/crypto/evp.html