How Do We Stop the Next OpenSSL Disaster?
Open Source hasn’t had good press in recent weeks.
The OpenSSL Heartbleed bug led to many articles revealing how badly run OpenSSL was. One typical example read, “The Internet Is Being Protected By Two Guys Named Steve“.
It’s clear that some major open source projects have serious funding problems. Open source has become so successful so quickly that the infrastructure behind projects hasn’t caught up.
Many projects are not able to ensure that all essential work gets done. There are more OpenSSL disasters waiting to happen.
Fortunately, I think there is a successful answer out there.
Before we get to the answer, I’m going to describe the main ways that open source development is funded right now.
#1. Volunteers
The classic stereotype of an open source developer is a volunteer. In this view, everyone is a hobbyist or moonlighting from the day-job. No-one is paid anything and they’re all doing it for the love of the community and the code.
In reality, it’s never really been quite like that. There are many true volunteers but many also have day jobs that are directly related to the code their working on.
It is hard to reliably run a 100% volunteer project. Contributors get busy at their day job, or they get burned out by juggling paid work and volunteer work.
Contributors need a successful infrastructure that allows them to make the most of the time they have.
#2. Developers Paid By The Company Running the Project
Magento is a great example of an open source project run a by a company.
Magento was created by a private company called Varien that was then purchased by eBay.
Yes, Magento was and remains an open source project. It also grew very quickly by adding commercial success to the open source mix.
However, Magento is also an example of the potential downsides to such an arrangement. Magento’s co-founder walked out in 2012, saying that is was very difficult for a large company such as eBay to manage an open source project. He said that the new Magento owners:
“don’t really understand the meaning of open and have a hard time explaining and defining it to them selves and to others. As such, long term, it would be very interesting to see if Magento will continue to stay open in the manor the people behind Magento and I meant it to be.”
100% commercially-run open source projects always run a large risk of putting their business ahead of the project.
#3. Developers Paid By External Companies
This is the solution is common in Drupal, Joomla and WordPress.
Large companies who have a material interest in the success of the project, will donate developer time.
Several WordPress companies such as 10Up and Dreamhost dedicate large amounts of developer time to WordPress core development. The same is true of Acquia and other companies in Drupal.
The potential downside, as with option #1, is that some companies may have an undue influence over the direction of the project.
Going down this route requires a strong, transparent core team with the ability to set an independent roadmap.
#4. Developers Paid by a Non-Profit
The most famous example of this is Mozilla. Mozilla makes over $300 million annually from it’s search agreement from Google.
The Linux Foundation is another good example. They pay developers thanks to donations from HP, IBM, Red Hat, Intel, Oracle and others.
With new projects, this is the route taken by Ghost. They recently blogged about their journey so far: blog.ghost.org/a-year-to-the-day:
“we’ve grown to a paying user base of 2,300 customers generating slightly over $175,000 in annual recurring revenue — growing at roughly 30% month on month. That last stat is arguably the most important, as it secures the long term future of Ghost. “
That money goes to pay the developers:
“We are now a full-time team of 5 and, as a non-profit, we aren’t able to take Venture Capital as would be the norm for a tech startup (though many have contacted us to try, all the same). In order for Ghost to continue to evolve and develop, we’re relying on growing slowly and organically. Much like the first iteration of Ghost was made possible by our users via Kickstarter — the future of Ghost is made possible by our users on Ghost(Pro).”
This approach is the one the taken by OpenSSL, although their income came from doing contract work.
The downside to this route is that non-profits aren’t always known for their good management practices. It takes a smart and experienced team to turn a hobby project into a successful non-profit. The skills needed to run a non-profit are very different from the skills need to write the original code. OpenSSL couldn’t manage it, so they called in the Linux Foundation.
#5. Developers Fundraising For Themselves
Perhaps the most famous example of open source platform fundraising was Ghost.
When Ghost launched, they asked for £25,000 on Kickstarter and ended up with nearly £200k.
However, there have not been many more such examples. And, there have been very mixed results when it comes to fundraising for on-going development, rather than a launch.
Hannes Papenberg did raise over €4,000 for his project to improve Joomla’s URLs:
And the Drupalfund.us site has helped some people raise money for Drupal projects:
However, between Joomla and Drupalfund, there were only 6 successfully funded projects and quite a few more failures.
Alex Pott is probably the most prominent example of some who tried fundraising.
He left a very well-paid job in London with Capgemini and became a Drupal 8 core maintainer. When he started, he was enthusiastic about fundraising:
“I’m trying to give the community fundraising a shot. I believe that if we can start a culture of supporting each other through crowdfunding sites like gittip, www.drupalfund.us, and other simliar sites then the community will be healthier for it. Why? Well, just because Acquia employs many well-known contributors, some have jumped to the conclusion that Acquia controls Drupal core. This concern is unfounded. Drupal 8 has over 1500 contributors, many more developers than any single company can hope to employ. However if we, as a community, want to mitigate any threats (real or imagined) from the enterprise or big companies then we have to step up.” Link.
Alex wasn’t able to make his fundraising goals and came to believe a different approach might be needed:
“I think we need to seriously consider extending the Drupal Association’s remit to be able to coordinate the collection and distribution of funds from major Drupal users for Drupal core development. ” Link.
I agree. Not only has fundraising failed to succeed in most cases, but it places a large added burden onto developers. The coders have to organize and promote the fundraising. Then what happens when the money? We’ve haven’t built a sustainable resource.
Fundraising is a solution that works in some situations, but it is not a sustainable answer. Fundraising does not set up our projects for long-term success.
Update: Alex Pott has been hired by Chapter Three, a Drupal company who will pay him to work on Drupal 8 full-time.
#6. Hybrid Arrangements
We’ve seen that Ghost has got it’s start through a mix of fundraising and a non-profit able to pay developers.
It’s worth noting that the most common way for open source projects is a mix of several solutions.
WordPress is probably the perfect example of a hybrid arrangement.
We’ve seen that some WordPress companies donate developer time. However, there’s a large parent company and a non-profit in the mix.
Jane Wells, who has worked in several roles with Automattic and WordPress, produced the chart below in 2010. There is a video version of the talk here.
This chart gives a really interesting insight into the mixed-up nature of the funding arrangements in 2010. The key element to all of it is Matt Mullenweg:
- Matt runs Automattic which employed about half of the core team.
- Matt had some other core teams members on his pay roll with other companies.
Matt is also the head of the non-profit WordPress Foundation which collects donations and supports some WordPress activities.
This post helps untangle things to some extent.
However, suffice it to say, this is a hodgepodge arrangement that works for Matt and WordPress, but it can’t serve as model for other projects.
So what is the best answer?
There are clear pros and cons to all the current methods used to support open source.
In the long run, any project is going to need to balance independence and commercial success:
- Too much independence and not enough commercialism, you become OpenSSL.
- Too much commercialism and not enough independence, you become a corporate footnote.
Mozilla and the Linux Foundation are two of the biggest success stories in open source and perhaps are the best examples for other projects to follow. They have at least three things in common:
- A strong non-profit which take the lead in running the project and also pays developers.
- A powerful commercial ecosystem which allows successful companies to fund the non-profit.
- An enthusiastic group of volunteers whose contributions are made easier and more effective due to the work done by the non-profit.
In my opinion, those three attributes together can produce independent, sustainable open source projects.
It’s over to you now. Do you agree with my conclusions? Are there other ways that we can sustain open source over the long-term?
First, OpenSSL with Heartbleed being discovered rather late by the public teaches more than business lessons. It ought to be seen as a belated success for the public interest and open source. If OpenSSL was closed, proprietary software it would probably have had ways for government agencies to defeat it built in from day one and nobody would know without whistleblowers risking their lives.
What open source project could possibly be adequate to the task of providing a key part of security infrastructure if the US intelligence community and its allies want to compromise it? The main lesson is you can’t expect all bugs to be found quickly if they are in open source software that is widely adopted and caught up in the interests of extraordinarily powerful organizations that most people trust implicitly.
Second, from the common open source business and developer perspective, the concern is not that you will become OpenSSL–it’s failing to ever achieve that kind of success in the first place. The focus is on gaining sufficient adoption and quality participation to grow, not planning or explaining to anyone how you will transition to a commercial model to your success if you get that far. It seems like an inherent and fundamental contradiction that can’t be avoided.
So while I agree with your conclusions, the problem is you can’t just build those three pillars at once unless you start with a lot of money. You begin with the volunteers, and if you have success, the commercial ecosystem emerges and you grow the non-profit to help sustain a balance between the commercial and non-commercial interests. The balancing act that unfolds is always on the edge of risk and necessity. I doubt there is any structural feature or rule for how to do it. A hybrid model will likely emerge like WordPress or Acquia where a founder with a proprietary interest in the core non-commercial product runs the non-profit that holds its trademark but money is made through commercial products whose development and revenue can be contributed back into the non-commercial product. This becomes a model and carrot/stick to get other for-profit companies to do the same thing and take a more proprietary attitude toward the core non-commercial product. Conflicts will emerge from that so executive leadership and decisionmaking must be wise and/or lucky. There is no recipe for wisdom or luck, just people who learn from adversity and make their own luck.
As with anything, anything that works, works. Healthy organizational cultures succeed for reasons that can’t be reduced to structure; often they succeed and are healthy in spite of their structure. The reverse is also true: ineffective, dysfunctional organizations exist that use the same structure as effective, healthy organizations. This is because all human groups have a tribal, herding model at bottom that precedes and overrides any modern rationalized notion we have of how a group is supposed to work. It’s not that primal passions and instincts dominate — they won’t except in very unhealthy groups — it’s more about the inevitability of an intergenerational, dynastic flow that creates natural friction points and triangular conflict relationships in the mix of junior and senior members, conservers of tradition and iconoclastic revolutionaries, the more and the less powerful, perceived insiders and outsiders.