Lenz Grimmer's blog

I recently received a question from Robin Schumacher at Calpont, the makers of the InfiniDB analytics database engine for MySQL: "How would you recommend we try and get bundled in with the various Linux distros?"

Since this question has come up several times before, I thought it might make sense to blog about my take on this.

First of all, please note that there is a difference between "being part of the core distribution" and "being available from a distributor's package repository". The latter one is relatively easy, the former can be hard, as you need to convince the distributor that your application is worth devoting engineering resources to maintain and support your application as part of their product. It's also a space issue – distributions need to make sure that the core packages still fit on the installation media (e.g. CD-ROMs or a DVD). Therefore they take a very close look at each package and if it's really needed to be part of the installation medium or if it's fine to provide it for download from a package repository instead.

Distributors prefer to keep their core product small and restricted to the "basic OS building blocks". While MySQL might still be considered to be a part of this, this probably does not apply to the various plugins and extensions that are available for it. Therefore the best approach is to invest some engineering time and start doing the packaging yourself, either by hiring an engineer capable of creating and maintaining the packages, or by finding someone in your community who has the required experiences and is willing to do it.

While it's of course possible to set up and maintain your own build and package hosting infrastructure for that, I recommend to make use of the existing services provided by the distributors.

The top tier distributors all provide means of offloading the maintenance of "non-core" packages to their community, offering various options for packages to be made available. For example, Novell/openSUSE provide the free "Build Service", which is capable of building packages for other distributions as well (e.g. Fedora, Mandriva, Debian/Ubuntu, etc.). In addition to automating the builds, the Build Service also takes care of the distribution via their download mirror network and ensures that your application can be found via their package search interface.

Red Hat/Fedora provide something similar, named "Koji" – but it's "Fedora only". Here's a HOWTO that outlines the process of becoming a Fedora package maintainer.

Ubuntu/Canonical have "Personal Package Archives (PPAs) – if your project is hosted on Launchpad already, that might be something to look into for providing Debian/Ubuntu packages. Alternatively you could join the Debian project and start building and maintaining your package there. They maintain a list of "Work-Needing and Prospective Packages", a description of the process on how to become a new maintainer is outlined here.

If you'd like to target Solaris/OpenSolaris as well, there is the OpenSolaris Source Juicer – a web service which allows OpenSolaris community developers to build packages (using RPM spec files) and publish them for review, so they will be included in an official package repository. The Software Porters Community Group coordinates, advocates, encourages and helps with the porting of Software from multiple Platforms to the OpenSolaris Platform.

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

There have been several similar articles to this one, but I find it pretty amazing to see all these emerging and established Solaris/OpenSolaris distributions. It's a good sign of a healthy community!

So here is my collection - please let me know if I forgot one.

Maintained by Sun Microsystems:

• Solaris 10: The commercial, production-ready and full supported version, similar to Red Hat Enterprise Linux (RHEL) or SUSE Linux Enterprise Server (SLES).
• OpenSolaris: This is the "official" OpenSolaris distribution, developed & supported by Sun. Can be compared to Fedora or openSUSE Linux - this is the version which will become the basis of future "Enterprise" releases with a different support and maintenance model. The developers aim on publishing a release roughly every six months.
• Solaris Express: Community Edition (SX:CE): is Sun's periodically built unsupported distribution of OpenSolaris (along with many additional open-source and a few closed-source components necessary to make a complete installable system). However, it looks as if Sun plans to discontinue the production of this version soon.
• Solaris Express: Sun's official distribution of OpenSolaris. These are provided primarily for customers interested in trying out new features of the Solaris release currently under development without the total cutting-edge-ness of the Solaris Express: Community Edition

Third-party distributions (in alphabetical order):

• AuroraUX: A core operating system for high integrity scientific computing. AuroraUX is a Solaris-derived kernel- and user- land. The core of the project are its utilities written in Ada. When necessary, poorly implemented features get fixed or rewritten, as well.
• Belenix: BeleniX is an OpenSolaris Distribution with a Live CD (runs directly off the CD). It can be installed to harddisk and is free to use, modify and distribute.
• Embedded Operating system/Networking (EON): A RAM-based live ZFS NAS appliance (CIFS/NFS/Samba) that boots from USB.
• Jaris: An OpenSolaris derivative suitable for Japanese users, providing a windows-like environment. (The web site is available in Japanese only)
• Korona: A live DVD distribution based on OpenSolaris, but using KDE 4.3 as the default desktop.
• Martux: A barebones OpenSolaris distribution for SPARC
• MilaX: MilaX is a small size Live CD distribution which runs completely off a miniCD, bootable business card or USB flash drive.
• Nexenta Core: Basically an OpenSolaris Kernel with an Ubuntu/GNU userland (including an improved version of apt-get that utilizes ZFS snapshots to perform rollbacks of failed upgrades)
• SchilliX: The very first independent OpenSolaris-based distribution (seems like it's not maintained anymore)
• StormOS: an OpenSolaris desktop distribution, based on Nexenta Core

XtraBackup is an Open Source online (non-blockable) backup solution for the InnoDB and XtraDB storage engines. It works with both MySQL 5.0 and 5.1 (and possibly 5.4 as well) and is distributed under the GPLv2.

Some weeks ago Vadim announced the availability of xtrabackup-0.7, stating that they consider it stable enough now to label this version a "Release Candidate". I've been maintaining RPM packages of xtrabackup on the fine openSUSE Build Service for quite some time now, RPMs of 0.7 for a number of distributions are now available for download. Please report any bug reports via the bug tracker on Launchpad.

Tomorrow (Thursday, 11th of September) at 9:00 PST/16:00 UTC/17:00 GMT/18:00 CET, there will be an new free MySQL University Session. MySQL University started as an internal training program for MySQL engineers, to share and spread knowledge about their areas of expertise and has been available to the public for quite some time now. It covers a wide range of technical topics around the MySQL Server and usually takes place once per week.

For the first time, the presentation will not be performed by (former) MySQL employees/developers, but by two of our "Sun Classic" colleagues: Jyri Virkki (OpenSolaris Web Stack community lead) and Murthy Chintalapati (Sr Engineering Manager, Web Stack development) will talk about the OpenSolaris Web Stack:

OpenSolaris Web Stack is an OpenSolaris project and community building an integrated stack of popular open source web tier infrastructure technologies such as Apache HTTP server, MySQL, memcached, PHP and Ruby On Rails optimized for Solaris platform. This session introduces OpenSolaris Web Stack, its status and future development including addition of newer technologies such as lighttpd, Varnish etc., as well as the ease of use features for developers and deployers. We will also be discussing an experimental web stack IPS package repository and it could be leveraged to build and make available popular end user applications such as Drupal.

MySQL University sessions are free to attend - all you need is an IRC client (to post your questions and comments) and an audio player capable of playing back an OGG audio stream, so you can listen to what is being said. See the Instructions for Attendees on the MySQL University pages for more information on how to log in and attend. The audio stream will be recorded and published on the MySQL University pages for later consumption, in case you can't make it or want to listen to a previous session.

 

I recently added two new packages to my repository on the openSUSE Build Service:

• Maatkit is a collection of essential command-line utilities for MySQL. Each is completely stand-alone, without dependencies other than core Perl and the DBI drivers needed to connect to MySQL, and doesn't need to be "installed" - you can just execute the scripts. This makes the tools easy to use on systems where you can't install anything extra, such as customer sites or ISPs.
• protobuf - Protocol Buffers - Google's data interchange format. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!

In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.

However, Matt seems to reduce "hosting" to "providing downloads" only:

Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
[...]
It can tell you many things, but the key is to be able to glean insight from the earliest stage of your interaction with a potential customer, and that means you have to host your own downloads. Otherwise, you have no idea how or when a would-be customer downloads your code, which makes the "why" they download it less interesting, because it becomes less actionable.

I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.

Project hosting is not just about downloads

First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.

But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.

I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.

There is more to distribute than releases

At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.

Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.

Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.

I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.

I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:

Downloads stats are overrated

Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.

Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:

As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.

I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.

 Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).

So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.

 

While we're on the topic of Bazaar - this week I got informed by the organizers of the FrOSCon 2008 conference that they accepted two of my talk proposals: one session will be an introduction to this source code management system (what a coincidence), the other one will be an introduction to OpenSolaris for Linux users, explaining some of the underlying technologies and how they differ from what a seasoned Linux user may be accustomed to.

And no, I have not given up on using Linux - quite the contrary! I have been very impressed by the latest OpenSUSE 11.0 release and already run it for since quite some time on several of my work systems. In fact, I already convinced several colleagues of mine to give it a try as well! I am amazed by the speed and "out of the box experience" of this version and I actually plan to install it on my Genesi Pegasos PowerPC machine as well, replacing Debian on there. But as a Sun employee, I of course have to familiarize myself with the other products and projects that we're involved in. And on the Server side, Solaris does have a few interesting features that Linux currently lacks. But I digress.

I look forward to speaking at FrOSCon again - it has been a great conference in the past two years. Very well organized, nice venue, a relaxed atmosphere and excellent technical sessions and speakers.

Other MySQLers submitted talks as well - for example, Giuseppe will give a presentation titled "MySQL Community How To", Susanne will give a PostgreSQL tutorial and others will participate in the separate PHP subconference. Don't miss it - this year's FrOSCon will take place on August 23rd&24th in St. Augustin, Germany (close to Bonn). For the first time, we will also try to set up a MySQL project table. So if you are there, make sure to stop by and have a chat with us!

 

Back when I still worked at SuSE, I was in charge of maintaining a number or packages of the distribution (actually, you should still be able to find traces of my work in the RPM changelogs). Nowadays, I maintain a number of packages for openSUSE and other distributions on the openSUSE Build Service, which is just brilliant for this purpose.

If you happen to live in northern Germany and are interested to learn more about the RPM package manager and how to build packages, consider coming to the TU Harburg this coming Thursday (March 13th). At 19:00, I will give a presentation about this topic in building, D, room D1023 (in cooperation with the Hamburg branch of the German Unix User Group). More information (in German) can be obtained from here.

See you there!

 

 

In addition to the binary downloads that we provide from our site, Linux RPM builds of the MySQL Proxy (both the latest stable version as well as SVN snapshot releases, named mysql-proxy-snapshot) are now available for download from the server:/database repository of the openSUSE build service. It provides packages for a number of Linux distributions, e.g. Fedora 5/6, SLES 9/10 SuSE/openSUSE 10.x. By the way, this repository also contains RPMs of the current 5.0.45 MySQL Community Server for the distributions mentioned above. Thanks a lot to Darix for the initial checkin of the proxy package!

 

I took this picture at Hamburg Airport while waiting for Kaj to arrive for our meeting last week. I was browsing through the magazines in a news shop in the arrival area and was pretty impressed about the amount of available magazines about Linux and Open Source Software. It amazes me how they all manage to produce enough content to fill a magazine on a frequent basis. I take this as a very positive sign - Linux has become a mainstream OS. I wonder how many people consider switching to Linux instead of going with MS Vista - the numerous CDs/DVDs that are included in these magazines significantly reduce the acceptance threshold for new users. Live CDs make it easy to give Linux a try without having to touch the installed operating system. The support and automatic detection and configuration of hardware in Linux has become much better nowadays, too. Granted, there is still a lot of room for improvement (especially when it comes to video card and WLAN chipset support, it seems) - but Linux distributions have reached a level of maturity that make them a viable alternative to Mac OS X or MS Vista for the most common use cases. And all the required applications are already included out of the box and are free (as in beer)! So what is missing to world domination?

So far, the MySQL Server RPM packages as provided by MySQL AB used to automatically start the mysqld process after the package has been installed. It has been like that since the very beginning and we think of it as a convenience for our users when they want to get up and running quickly.

However, Kristian raised an interesting point in BUG#27072 where he points out that automatically starting mysqld during the RPM installation might not always be the desired behaviour, especially in automatic installation environments or during a fresh installation (where the system might not be fully configured yet). Therefore he proposes to change this behaviour to not start mysqld as part of the installation.

While I personally agree with his proposal, this is of course a tricky decision: our users are familiar with this behaviour and it's never a good idea to cause surprises. Therefore this change could only be done in future versions of MySQL, where they can be properly announced and documented and don't cause too much confusion.

Then again, there are several options here. Should we just disable it for the Enterprise Linux RPMs (RHEL/SLES) and keep it enabled in the "generic" RPMs? This would be inconsistent and harder to document/explain. Should the server never be started at all, or should we keep the behaviour for updates at least (when the server was already running before the package update)?

We would like to get your take on this. Please post your comments and suggestions to the bug report (preferred) or leave a comment on my blog, so we can get an impression on what you would expect to be the correct behaviour here. Thanks!

Thanks to Duncan Mac-Vicar for the friendly reminder: I finally bit the bullet and started using the openSUSE Build Service for building and hosting the RPMs that I build for SUSE Linux by myself. I've now moved the KeyJnote and mercurial RPMs for SUSE Linux 10.1 to my own repository there. Future RPMs will also be provided from there exclusively. Kudos to the openSUSE team for providing this service!

Yesterday I received a sample copy of the "SUSE Linux 10.1 OSS - vorkon" DVD, which was assembled by Nicolaus Millin. While I have not installed it yet (as I by now have a very well running installation of SUSE Linux 10.1 on my laptop), the content of the DVD sounds very promising. In addition to incorporating all the updates that have been published since SUSE Linux 10.1 was released (around 100), this version also includes additional drivers like the 3D graphics drivers from ATI and nVidia, network card drivers like ndiswrapper, madwifi and rt2500 as well as drivers for AVM products and Logitech QuickCams. So the base system should already run very smooth and provides a lot of stuff that one usually has to download and install after the initial installation of SUSE Linux OSS.

But the fun doesn't stop here! Nicolaus also added a number of interesting applications, usually not included in SUSE Linux OSS, such as Adobe Reader 7.0, moneyplex, RealPlayer, FlashPlayer, Cinepaint, FlightGear and several extensions for Mozilla Firefox. He also included many LAMP applications (based on the famous Linux/Apache/MySQL/PHP stack), that have already been preconfigured and are ready to use right after installing the packages! This is where the name "vorkon" comes from - "vorkonfiguriert" means "preconfigured" in German. Some of the LAMP apps included are eGroupware, Gallery2, Joomla!, MediaWiki, nagios, phpBB, Serendipity, Typo3 and several others.

All in all this looks like a well-rounded package - it reminds me a lot of how SUSE Linux used to be when people referred to it as the "everything but the kitchen sink" distribution. Unfortunately it seems as if the legal restrictions of being owned by a US company have caused a lot of the former SUSE benefits to vanish - nowadays one has to hunt for a lot of stuff like drivers and uncrippled applications after the initial installation just as with any other Linux distribution. Thanks to Nicolaus for scratching that itch!

You can purchase SUSE Linux 10.1 "vorkon" via amazon.de - for just 19,95 EUR this is a very attractive product, especially for Linux Newcomers. Give it a try!