Interviews/Brian Behlendorf

Brian Behlendorf is the former President of Apache Software Foundation and Mozilla Foundation Board Member.

Highlights

• Apache Software Foundation has about 450 members, the majority of whom are being paid by different companies to work on Apache.
• In order to grow a volunteer developer community you need to move the development itself to the public sphere. Make the source code repository public, the issue tracker, the roadmap. Make these tools transparent. Get your developers to use those tools so that it's transparent to the community.
• Some ideas to grow the volunteer developer community would be to consider reaching out to universities. Google Summer of Code is a great program. Are there interesting partnerships that can be formed? Find a computer science department that would be willing to work on projects.
• Having people dedicated to community engagement is really important. One approach is to make sure that the lead developers recognize their roles as not just writers of code but as leaders of the community. Other projects do have people who are identified as community managers. They take on much more of that strategic thinking / marketing role rather than writing code. That can work if that person is technical enough. If not, they won't necessarily get the respect of the developers. It's ideal to have the community building skill set as part of the core developers, but you can certainly get there faster and try to stoke those fires with someone else.

Interview

Brian: What are the Wikimedia Foundation's goals and challenges?

Expanding reach beyond global north, males, etc. What types of changes would need to happen in MediaWiki in order to get that increase in reach and participation?

Brian: How many people are invested full-time in supporting MediaWiki?

Less than 10.

Brian: There's an ideal in open source projects. Great projects have a universe of contributors, lots of investment from lots of different parties. Any project can survive departure of any org. That's an ideal. It's not always achieved. Many successful projects, the majority of contributions come from a single vendor. OpenOffice, majority of contributions have come from Sun. Sun invests in hundreds of developers as a way to challenge Microsoft. At the same time, Sun has maintained tight control over the process, which has made it harder for other companies to participate (for example, Novell). It would have been hard for them to let go of the reins completely because the development of an Office system is complex. Just like a web browser, with lots of moving parts.

Another example would be MySQL. They didn't do much to try to expand the core developer community. They did a lot to create great APIs. Even to the degree that people submitted patches, would go about and rewrite those.

Do we need that external investment? Do we need more people helping with the core? I Don't know if MediaWiki is complex at the same level.

What are the pain points? Does the community know about these objectives?

Current developers and editors are comfortable with MediaWiki as it is, so they don't have the drive or see the need to make it more usable. Usability tests have shown lots of barriers to editing.

Brian: How do we set the direction for the project? Whose role is that?

There's one major installation that a lot of people care about. There's almost two different communities that we're talking about here: the contributors -- this is a large pool, and if only 1 percent were willing to tinker with the code, that's still a lot of people -- and MediaWiki users.

Different strategies for encouraging them to contribute to the code. Are my contributions going to end up in the main site? Can we talk about that? How frequently do we update the site? What's the desired roadmap? Help people understand how their investment to the site makes a difference.

What steps would you take to create a more open developer community?

The big one, but the hard one: Move the development itself to the public sphere. Make the source code repository public, the issue tracker, the roadmap. Make these tools transparent. Get your developers to use those tools so that it's transparent to the community. Sun made the development public, but not the process. Didn't tie this to any public conversation. Judging the cake, not judging the recipes.

Bring people over the hump from beginning to expert users/developers. One way to get them over that is a standard customer service/complaint process. The way many open source projects work is not customer service mode, but hey, help us figure it out.

IRC -- paying attention to time of day. Are we bringing in new blood to the project?

Consider reaching out to universities. Google Summer of Code is a great program. Are there interesting partnerships that can be formed? Find a computer science department that would be willing to work on projects.

Community engagement is an important part of any successful open source company. Subversion -- hired the first three engineers. Then they made it clear that they weren't enough to make this work and left it up to them to be developers and community managers. Karl Fogel emerged. You need to constantly looked at the development process from the perspective of an outsider. One approach is to make sure that the lead developers recognize their roles as not just write code but lead the community. Other projects do have people who are identified as community managers. Take on much more of that strategic thinking / marketing role rather than writing code. That can work if that person is technical enough. If not, they won't necessarily get the respect of the developers. It's ideal to have the community building skill set as part of the core developers, but you can certainly get there faster and try to stoke those fires with someone else. Karl Fogel's book is great.

Overview of why Apache Software Foundation was created? Mission and structure?

Started in 1995 when a bunch of people on mailing list associated with web tech recognized unmet need. NCSA developers all left to go to Netscape. Didn't want to pay the licensing fees from Netscape but were also idealists about keeping the playing field level and respecting standards. Decided to build a good enough web server and build the code as widely as should. Really it was because we were lazy. Had pretty pragmatic goals. None of us wanted to be commercial web server companies. Just wanted better web server software.

Project started multi-participant, with a single person acting as lead. Identity associated with the server rather than the company.

Three years into the project, we realized that this amorphous hack was running 65% of the web. We recognized opportunity but also liabilities. Developers could be sued for infringement. We weren’t really scaling our own efforts very well. Lots of people wanted to build add-on modules. We had pretty good APIs, so people could do that. How would people distribute them? We needed a neutral ground community.

The processes that we used to write software could be templatized. What would it mean to start a nonprofit around our activities? Protection from liability was appealing. Companies nosing around Apache would have legal framework to use it.

Foundation launched as 501(c)3 membership organization. Membership based on original core developers. A couple of times a year, we vote in new members. Today, there are about 450 members. Almost like the old guild kind of model. There are a larger number of people who are committers and there are 60 different Apache projects. Each project has project management committee made up of the senior developers. The Board of the foundation is elected by membership and doesn't make technical decisions. The Board makes sure that people follow the law and that community grows and is healthy. An unhealthy community is when two or three people run away with the project and block progress, or when the project is run by one company.

We tried to make it clear that the foundation is built by individuals. Corporations don't have formal roles but they are sponsors to help us cover basic costs. This pays for some consultants to do system administration work but there are no paid executive staff. The nature of the foundation is probably different from Wikimedia and certainly different from Mozilla. Mozilla is almost the opposite extreme. Employs 250 people and derives revenue from search engines.

Wikimedia seems to be in the middle slot. That seems appropriate. You have a major runtime operation that can't go down.

MediaWiki did start for Wikipedia and hoping that others find it useful. Is it a community-managed process?

Some of the other projects we've talked to (like Gnome) have all volunteer developer, but many of those developers are being paid to work full-time on Gnome. Does Apache have those kinds of relationships?

I'd imagine the majority of those members are paid to work on Apache. Probably a reasonable percentage (maybe 20-30%?) working full-time. Many people willing to take a reduced pay rate to work on open source.

Rishab Ghosh's seminal study from a few years ago shows that most of the development in open source projects is paid for. Don't know the effect of the recession. If anything, being an open source developer where your name is associated with a successful project is better than any resume builder. Imagine that the majority of people working on Apache are paid.

How does the Foundation decide to accept / shut down projects?

There's an incubator (incubator.apache.org) where projects first land when they've been proposed and accepted by the incubator PMC (project management committee). While the project is there, a couple of things happen. Paperwork happens so that all the contributors have authorized their work to be released under the Apache license. Also there is a fair bit of a trial process for community to go through to show they can work together and release code. This phase is the only time we care about who people work for because for a project to be a full-fledged project, it need to show that there's no majority of core developers working for a single organization.

That's been the process for pretty much all of the projects beyond the initial set of projects.

There is an Apache attic -- user bases have moved on to other tools, or it's mature enough so there's not a steady stream of bug reports, it's moved to the attic. It's still available, people can file bug reports, but it's not considered an actively maintained project.

Why have an attic?

Project's original developers are saying there might be better solutions. If you leave everything up and give the appearance that it can start up again any time, you create an expectation that you will.

Is there a best practice for how direction is set / governing process for incorporating?

No one practice. Really a collection of anecdotes about what works and doesn't work. Sometimes it comes down to the peculiarities of the developers and the user base. The user base of OpenOffice or Firefox is predominantly not technical. The user base of Apache web server is much more willing to edit a config file. Therefore, the expectation of how to engage developers is different. It's hard to be specific about that.

What's common is that they need to figure out a way to scale the trust of the community, to put themselves in the shoes of person who's on the verge of becoming a developer. I found a bug. What do I do next? If I ask a question, will I get an answer quickly? Is there a community to tap into and search for the answer? Is it worth my time to try to fix the issue I have? Is there a path for my contribution to end up in the codebase? It's a leap of faith you're asking people to go through.

The more they can see real professionals at the helm and a regular release cycle. On Subversion, developers went through the bug database and tagged issues they felt were small-sized -- solutions were obvious but would require time ("bite-sized issues") -- and encouraged people new on the mailing list to search for bite-sized issues as a good way to get over learning curve. That was remarkably successful at bringing in new developers.

What does it take for open source projects to create highly usable software?

Good question. Don't have a good answer. People have thought about it. Gnome in particular. KDE (Celeste Paul) who is professional user experience developer. She's thought deeply about it. Most of the projects I've worked on have not had much of a UI. When it comes to usability of the wiki editor interface, one has to think about what has to drive typical usability designer. The typical back-and-forth way to discuss debate over which algorithm to use is different from how usability designers are used to working. Tend to dislike design-by-committee. Frankly, arguments about aesthetics are tougher to resolve than what's algorithmically easy to solve the problem.

Give people freedom to create skins. WordPress and Drupal have allowed designers to have thin veneer over backend machinery. Given them ability to compete with one another. Designers don't have to fight with the world to get their interfaces.

How many options do we provide to the end user? What level of technical proficiency do we assume?