XZ Utils and the future of Open Source

In the following post I would like to give a quick background on the recently discovered exploit in a widely used open source compression library/utility that shook the software world and the very core of OSS development and follow-up with my analysis and thoughts on the root problem that caused this and potential solutiosn.

Background

Let's start with a short list of events to summarize what happened:

1. Andres Freund (Postgres developer hired by Microsoft) finds out strange performance issues in Debian, more specifically regarding SSH login.
2. After more introspection he pinpoints to root cause of the problem - the latest version of xz Utils, a very popular tool and library for loseless compression.
3. Eventually he discovers that the problem is caused by a backdoor that was introduced intentionally by one of the developers of xz going by the username JiaT75.
4. Soon it is discovered that most probably this was a longterm attempt at introducing a RCE exploint into major RPM-based distributions and it was possibly orchestrated by multiple people or even a goverment agency.
5. The exploit was found by chance and all of the contributions of JiaT75 are being analyzed for potential holes and malicious code.

A more technical focused analysis can be found e.g. here but my main focus will be on the project, people involved and OSS as a whole - as this incident raised a lot of questions and doubts, people still wondering how much of these kinds of exploits might have flown under the radar and are still in the wild waiting to be used and what is the future of open source after such a blunder.

The woe of the maintainer

One of the selling points of OSS security is the "many-eyes principle" which comes down to the notion that the code safety is provided by it being accessible to anyone and the scrutiny of many different people and parties analyzing it which protects it from attempts of introducing an exploit. Where on the other side of the spectrum we have little idea what kind of backdoors exist in closed-source software and need to rely fully on the goodwill of the creators and companies that develop it - most of which have quite conflicting interests to our own as users.

But the above is false to a large degree, even for widely used and important projects that are used in many critical infrastructure areas and businesses worth billions of USD and it lies in the fundamental model upon which most projects are based on.

The truth is that there is no mechanism that would guarantee appriopriate staffing and resource allocation in a OSS project as it is based on people investing their private time on a task that will most probably provide little or none financial reimbursment  for the work they put it. This contrasts with prioprietary projects where staffing and resource allocation are contracted responsibilities of someone to guarantee continuity and unhindered development. Some (usually big) projects like Linux operate in a sort of "hybrid" mode where a company hires full or part-time developers to help develop and maintain the code which one could say provides "the best of both worlds" but is quite rare in practice and mostly reserved to widely known and "sexy" projects - like Linux, Blender, Postgres and such.

This effectively means that even critical projects might become abandonware or, like in the case of xz, be maintained by an overworked person with mental health issues becoming a ripe target for hostile takover by a malicious person or party with the goal of introducing a backdoor.

Is there any remedy for that? One might think of a few different solutions:

• Donations - gather funds for development and mainatance of the project via donations.
• Hybrid model - a private company helps out either financially pouring funds into the project and funding the mainainters or by assigning employees to help out.
• "Open Core" - this applies only a certain class of software, essentially an open-source "core" is provided to the community while on top of it a closed-source solution is developed and sold.
• SaaS/Cloud on OSS - the software is fully open source but around it a commercial offering is built to provide e.g. a hosted solution.
• Pay-for-commercial - any commercial use of the software requires some form payment (e.g. to a foundation or company built around the project).

The above can be mixed and each comes with its own set of advantages and disadvantages. But the main goal remains the same - have some sort of established mechanism to allocate funds for development, protect it from abandonment and enables the project to remain open source.

The last three can't be effectively applied to a large portion of OSS (e.g. libraries) and many of those are extremely critical pieces of software where the risks of it being no longer developed or an exploit introduced should be unnaceptable to many.

So we are effectively left with donations - either private or by companies, financial or otherwise (e.g. private or employee time spent on development). And in my opinion the most optimal solution that should work for almost all projects would be the consortium model - a foundation that is composed of people and companies where the "heavy-lifting" is done by companies that have vested interest in the software and use it to generate income. A great example of that would be the Linux and Apache foundations. So there are strong, real-life examples of this approach working and producing great results.

What we need is a push to pull more projects from being maintained and developed solely by an unstable group of volunteers to big foundations that gather companies that are users of the software and have (or at least should) interest in them being development, exploit free. To that one could easily add goverment agencies of many countries which would prefer not to have to rely on closed source software under the umbrella and control of the US goverment (and the NSA/CIA/etc.).

We should try to push companies to pay for what they use - they've built multi-billion dollar businesses around a plethora of OSS projects but in many cases none of that revenue ever reaches the people upon whose work they make their money. This should stop and in the end it would be greatly beneficial to the companies themselves - moving to another library as the old one got abandoned or fighting with the aftermath of an exploit like we saw with xz most probably costs them more than some modest investment into a shared development model. Some big projects are funded but many smaller ones are completely left on their own and should be integrated into a larger organization to gather funds and drive development.

Security

According to many the myth of OSS being more secure because of many people watching and analyzing the code has crumbled and we should enter some kind of permanent state of paranoia as over the years multiple exploits might have been introduced into many projects.

That is incorrect as the issues that have enabled the introduction of the exploit into xz don't apply to most widely used projects - there are actually many engaged maintainers and developers where in the case of xz the "many-eyes principle" simply didn't apply and that's most probably why it was chosen for exploitation.

And what is the alternative? Closed source software might have a plethora of different backdoors and we have no way of knowing or fixing them up as we don't have any control over the code. And they most probably have them baked in by default - there is evidence of goverment agencies requiring companies to install them for later use, enforced by the heavy hand of the state and legislature. For some that is not a problem (e.g. US goverment agencies probably don't really care about being spied upon by the NSA) but for others that is (or at least should be) a big no-no when it comes to using black-box code, self-hosted or cloud.

And we have evidence where malicious actors were introduced into companies to e.g. steal their IP and secrets like in the recent case of Tesla so employee vetting won't really fully protect you from hostile actors having the ability to modify the code you use, just you'll never be able to find it out yourself.

So all in all - OSS is still the best approach to provide and maintain security and privacy and this exploit doesn't change that. We just need to learn from the mistakes made and introduce positive changes that will protect us in the future from it happening again. Burying our heads in the sand and crossing fingers for it to not happen again won't really work as it almost certainly will.

To close it up

My hope is that the recent events will shine some light on the problems many OSS projects face and that it will result in companies actually stepping up to their responsibilities as commercial users of the software. There is a big need to expand the number of foundations that maintain projects and/or put them into existing ones with an increase in funding. Without that another xz exploit is just a matter of time and this time the stars might not align for another Andres to find it out before it explodes in the faces of many.

Hope you liked what you've read and have a nice day!