Lenz Grimmer's blog

I recently received a question from Robin Schumacher at Calpont, the makers of the InfiniDB analytics database engine for MySQL: "How would you recommend we try and get bundled in with the various Linux distros?"

Since this question has come up several times before, I thought it might make sense to blog about my take on this.

First of all, please note that there is a difference between "being part of the core distribution" and "being available from a distributor's package repository". The latter one is relatively easy, the former can be hard, as you need to convince the distributor that your application is worth devoting engineering resources to maintain and support your application as part of their product. It's also a space issue – distributions need to make sure that the core packages still fit on the installation media (e.g. CD-ROMs or a DVD). Therefore they take a very close look at each package and if it's really needed to be part of the installation medium or if it's fine to provide it for download from a package repository instead.

Distributors prefer to keep their core product small and restricted to the "basic OS building blocks". While MySQL might still be considered to be a part of this, this probably does not apply to the various plugins and extensions that are available for it. Therefore the best approach is to invest some engineering time and start doing the packaging yourself, either by hiring an engineer capable of creating and maintaining the packages, or by finding someone in your community who has the required experiences and is willing to do it.

While it's of course possible to set up and maintain your own build and package hosting infrastructure for that, I recommend to make use of the existing services provided by the distributors.

The top tier distributors all provide means of offloading the maintenance of "non-core" packages to their community, offering various options for packages to be made available. For example, Novell/openSUSE provide the free "Build Service", which is capable of building packages for other distributions as well (e.g. Fedora, Mandriva, Debian/Ubuntu, etc.). In addition to automating the builds, the Build Service also takes care of the distribution via their download mirror network and ensures that your application can be found via their package search interface.

Red Hat/Fedora provide something similar, named "Koji" – but it's "Fedora only". Here's a HOWTO that outlines the process of becoming a Fedora package maintainer.

Ubuntu/Canonical have "Personal Package Archives (PPAs) – if your project is hosted on Launchpad already, that might be something to look into for providing Debian/Ubuntu packages. Alternatively you could join the Debian project and start building and maintaining your package there. They maintain a list of "Work-Needing and Prospective Packages", a description of the process on how to become a new maintainer is outlined here.

If you'd like to target Solaris/OpenSolaris as well, there is the OpenSolaris Source Juicer – a web service which allows OpenSolaris community developers to build packages (using RPM spec files) and publish them for review, so they will be included in an official package repository. The Software Porters Community Group coordinates, advocates, encourages and helps with the porting of Software from multiple Platforms to the OpenSolaris Platform.

CMake is a cross-platform, open-source build system, maintained by Kitware, Inc.

From the CMake.org home page:

CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice.

It has been used for building the MySQL Server on Windows since MySQL 5.0 – the initial CMake build support was added in August 2006.

For building MySQL on all other platforms, the GNU autotools (autoconf, automake and libtool) are currently being used.

CMake is used in some other MySQL projects as well, e.g.

• MySQL Connector/C
• MySQL Connector/C++
• MySQL Proxy

On February 22nd, Vladislav Vaintroub pushed the changes required to implement WorkLog#5161 "CMake-based unified build system" into the "mysql-next-mr" branch (aka the "Celosia" mile stone).

From this version on, CMake can also be used to build MySQL on Linux and other Unix platforms. For the time being, the autoconf/automake files are still available as well, but will be phased out once the CMake build enviroment has reached the desired level of maturity. The change was announced on February 28th on our "internals" developer discussion list.

The purpose of WL#5161 is to simplify the MySQL build system. It is much easier and less error-prone to maintain a unified build system for all platforms than two separate ones.

CMake has been chosen because of several reasons; the worklog description lists a few pro-CMake arguments (slightly rephrased):

• CMake works on Windows. The GNU buildsystem does not really work and likely never will work natively on Windows (Using Cygwin is not really an option).
• Traditionally, new MySQL features that required changes in the build environment (e.g. the plugin system, unit tests, most recently googletest integration) were always implemented on Unix first, leaving Windows behind (sometimes for years). This would not happen with a unified build system.
• MySQL already uses CMake since 2006 on Windows, so we do not need to start from scratch, only port what we have to Unix.
• CMake runs on every OS and compiler we support.
• It is simple to obtain and install on a wide range of platforms. It is available in all major Linux package repositories (e.g. Ubuntu, Fedora, OpenSUSE). It is also in the OpenSolaris repository, known as SUNWCmake. It's in FreeBSD ports and available for Mac OS X. It is also very simple to compile it from source, the single prerequisite is a working C++ compiler and make utility.
• CMake has support for features we need and might need, e.g. system checks or cross-compiling.
• CMake provides integrated support for packaging. It can handle both simple packages (tar.gz or zip archives) and more complex things like DEB and RPM without much extra coding.
• Good integration with the popular IDEs (Visual Studio, Xcode, Eclipse CDT, KDevelop). Developing in an IDE makes the development process more enjoyable, and potentially it lowers the barrier for external contributors. Of course, CMake can generate traditional Unix Makefiles, which appear to be are superior to the ones generated by GNU autotools (for example, they have progress indicators, colored output and working dependencies).
• The scripting language used by CMake is simpler than m4 used by autotools.
• CMake is a single small tool, not a bunch of different tools as in GNU system (autoconf, autoheader, automake, libtool)

I'd like to mention a few additional reasons:

• Out-of-source builds – CMake can separate the build directory from the source directory. This is convenient, as your working source tree is not cluttered with object files and other fragments of the build process.
• Build configuration using a GUI. The cmake-gui package (based on Nokia/TrollTech's Qt library) provides a convenient way of enabling and configuring the various available build options. This is much better than having to memorize all the required defines and configuration flags.
• Integrated support for creating a wide range of package formats.

The CMake Wiki lists a number of other "nice to have" features.

From a developer perspective, I hope that it will make it much easier to finally implement two things that many developers working with MySQL have been waiting for (now that the build code has been cleaned up):

• Compiling the embedded MySQL Server (libmysqld) as a shared library
• Better support for cross-compilation

Building MySQL with CMake is quite simple and straighforward – the process is outlined on the MySQL Forge Wiki. The document is still work in progress and we'd like to encourage you to take a look at it, try to follow the steps and update/improve the Wiki page, if needed! Your feedback on the build process is appreciated. Feel free to join our internals mailing list to discuss your impressions and observations or submit a bug report via the Bug Database. It's likely that the build still has a few rough edges that we'd like to fix quickly (e.g. BUG#51502 – a fix for this one is already commited to the mysql-next-mr-bugfixing source tree and will be merged into the mysql-next-mr trunk soon).

If you're new to CMake, you might want to take a look at the "Getting Started With CMake (An End-User's Perspective) For Cross-Platform Building" screencast or the "Running CMake" article.

Happy hacking!

Shortly after I posted my last summary of MySQL releases, our son Mats was born and I went on a 2.5-week vacation. Our developers did not rest in the meanwhile and I'd like to give you a quick update of what's new since then:

MySQL Connector/Net 6.3.0

• Visual Studio 2010 RC support
• Nested transaction scope support

MySQL Workbench 5.2.16 Beta 6

• Fixed 67 bugs
• Saving your profile/connection passwords in OSX keychain, gnome-keyring or in an encrypted password-vault-file.
• New rapid development features for generating complete SQL Select/DML statements or names for selected objects in Query Editor to either the query area or clipboard.
• The ability to set a preference for the placement  (left or right side) of the sidebar in the Query Editor (Currently on Windows only, coming to Mac and Linux soon).
• Further optimization and stabilization of the administrator components

MySQL Server 5.1.44

• This release now includes InnoDB Plugin version 1.0.6, wich is considered to be of Release Candidate (RC) quality.
• Lots of bug fixes - see the ChangeLog for details.

MySQL Server 5.5.2 Milestone 2

• Also includes the updated InnoDB Plugin 1.0.6 and several fixes - see the ChangeLog for details. For an overview of new features in the 5.5 code base, check out the "What's new in 5.5" page in the reference manual.

Please note that the MySQL downloads section has been split into two parts. As usual, you will find downloads of both GA and development versions of all MySQL products and releases on the MySQL DevZone. In addition to that, we've now added a pointer to the downloads of officially released (GA) versions to the main web site on http://www.mysql.com/downloads/.

Last week, my colleagues Giuseppe, Kai and myself attended the SAPO Codebits event in Lisbon, Portugal. Codebits is an annual, invite-only hacking event, which went on for three days. The venue they chose this year was the "Cordoaria", a former rope factory located in the Belém district, close to the 25 de Abril Bridge (which is an impressive sight!). I have been told that the Cordoaria is the longest building in Portugal and I have no doubts about that! The building is so long that the crew used bicycles to get from one end to the other. I've taken a number of pictures from the event as well as from Lisbon itself, you can find them in this flickr set.

The organizers described this year's event as follows:

3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience.

I wholeheartedly agree, we had a great time! The conference started with sessions and presentations on a wide range of topics on the first two days. Afterwards, a 24-hour programming contest was held. I was invited to give two talks, one being my all-time favourite about "MySQL High Availability solutions" (slides, video), the other one was titled "Why you should be using a distributed version control system (DVCS) for your project" (video, slides). Both went quite well and the feedback I received was pretty positive. Giuseppe talked about "MySQL Schema Migration" (slides, video) and gave an "Introduction to Gearman" (video). Kai's talk was titled "Think before you develop" (video) and gave a nice roundup of tips and best practices for setting up and developing new web projects.

The Codebits session schedule was filled with amazing and interesting talks in four parallel tracks. Sometimes it was hard to choose – some other talks I attended and enjoyed:

• Robert Nyman: JavaScript - From Birth To Closure (slides, video)
• Glenn Jones: Re-using social media data (slides)
• Brian LeRoux: PhoneGap: Mobile App Developer Zero to Hero (video)
• Brian LeRoux: The State of JavaScript on Mobile Devices
• Walter Belgers: Becoming a lockpicker

Walter gave a lockpicking workshop after his presentation, which I attended as well. I was quite impressed (and a bit shocked) to find out how easy many locks can be opened this way! Later that evening there even was a live band named "Pornophonique" playing (one guy with a guitar, the other one using an Nintendo Game Boy for making music), but I missed that show as I was too busy opening more locks... Fortunately the concert and most of the sessions were recorded on video (in excellent quality) and are already available from the SAPO video pages. Kudos for this speedy service!

But this just matches my overall conclusion of this event: very well organized, great speakers and venue. Thanks to the organizers for having us, we really enjoyed our stay!

This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!

Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

With IntelliJ now being available under an Open Source license, developers have another option to choose from when it comes to Java-based IDEs/Frameworks (Eclipse and NetBeans being the other two prominent ones). Choice is always good, and being an Open Source enthusiast, I of course welcome JetBrain's move!

However, as I'm not really a heavy GUI-based IDE user myself, I can't really comment on which one is the best. These kind of discussions tend to turn into a Holy War anyway... In the end it's likely that each of them gets the job done and you have to come to your own conclusions, based on your personal preference and requirements.

I personally would be interested in seeing how their support for PHP or Python compares to the one in NetBeans. Their plugin repository lists more that 560 plugins, including many for database connectivity/modeling/navigation (incl. support for MySQL). I'm also glad to see that they have a plugin for Bazaar, something that I'm desperately missing from NetBeans!

Interestingly, they decided to keep a few parts proprietary, it's going to be interesting to see how this will turn out for them and if developers will be willing to pay for these extra features, considering that most of this is available for free from the other two projects.

Their Contributor License Agreement looks like it has been derived from the Sun Contributor Agreement (SCA), which is always nice to see. I assume this can be attributed to Roman Strobl - I was positively surprised to notice that he joined their team as a technology evangelist in June! Roman did a great job in spreading the NetBeans and OpenSolaris gospel at Sun before and I briefly met him at this year's FOSDEM conference in Brussels. Congratulations!

Last week I gave a MySQL University Presentation about how to contribute code to MySQL. This time DimDim did not fail to record the session, even though there is a funky overlap of audio from Stefan Hinz (the moderator) and myself at the beginning. I had a bit of a slow start into the presentation, because of a very nasty headache that plagued me that day. But we had a lively discussion at the end and I hope it was useful to the participants.

In case you have missed it, you can now watch the playback or download the session slides:

• Presentation slides: MySQL Code Contributions (PDF)
• Session recording (Adobe Flash Player required)
• Chat transcript

As you may have heard, we're switching to a new release model with the upcoming MySQL 5.4 release.

If you are curious to learn more about what will change in the way in which future versions MySQL will be developed and released, make sure to attend our next MySQL University session about The New MySQL Release Model on Thursday, 11th of June, 14:00 UTC. Tomas Ulin, our director of MySQL server development will go through the planned changes and would also like to get your input and feedback on these changes.

We're using DimDim for broadcasting this session, which allows you to listen to the audio while watching the slides with your web browser. You can comment and discuss via a chat function, too! We're looking forward to your input. To attend, point your browser to this address (Adobe flash player required).

The session will be recorded and posted on the MySQL Forge Wiki, so you can watch the presentation later as well. You can also provide your feedback on the release model by posting on the MySQL Internals mailinglist.

The MySQL Conference & Expo 2009 in Santa Clara is now in full swing and Karen Padir just gave the opening keynote, talking about Sun's continued and improved commitment to Open Source and the upcoming MySQL products like the MySQL 5.4 performance release or MySQL Cluster 7.0. One of the activities that she mentioned in her keynote is our ongoing activity to improve the acceptance and incorporation of patches contributed by the community.

We've scheduled a BoF about this topic tonight (7:30pm in Ballroom A), where we would like to talk about the recent changes that we've made and discuss a new way in how to produce future releases of the MySQL Server on a shorter and more predictable schedule. We've invited Tomas Ulin (Director of MySQL Server) to join us and explain the proposed changes to the MySQL release model and how they will help us to incorporate patch contributions and make them available to the community at a faster rate.

Please join us and let us know what you think of these changes and what else we can do to make it easier and attractive to contribute patches to the MySQL Server! There will be free T-Shirts as well

While Colin beat me in blogging about Project Kenai, I think I can still provide some additional background information about this new project hosting service from Sun.

If you are a maintainer of an Open Source project, you currently have plenty of choice when it comes to getting your project hosted for free. One criterion could be your software configuration management system (SCM) of choice.

Some of the hosting services that I am currently aware of and the choice of SCM they offer include:

• BitBucket (Mercurial)
• Collab.net (Subversion)
• github (Git)
• GoogleCode (Subversion)
• Launchpad (Bazaar)
• Sourceforge.net (CVS, Subversion)

As disclosed by Tim Bray some days ago, there now is another option - Kenai is open for project hosting (currently by invitation only)! In his blog post, he interviews Nick Sieger, one of the developers behind this project about their motivation and intentions:

We need to demonstrate credibility in building on top of more traditional LAMP/SAMP web stacks (not just Java EE); and we need to show viability of Sun technologies and hardware for next-generation web applications.

In a nutshell, Kenai is a platform for:

• Developer collaboration
• Communities of connected developers
• Integrated collaboration services stack

Some of the features that are currently available include:

• SCM services using Subversion and Mercurial
• Bug Tracking (Bugzilla)
• Forums
• Wikis
• Mailing Lists (using Sympa)

Reading the interview with Nick and looking at some presentations slides for RailsConf from Fernando Castano (a jRuby and Database performance engineer at Sun and another member of the project team),  I was able to gather a list of the tools and technologies they used to build Kenai:

• Apache2, mod_proxy_balancer
• Ruby on Rails (running on jRuby)
• Glassfish Java application server
• MySQL 5.0
• Apache Solr for search
• memcached

I found it interesting that they decided to deploy and run the Rails application as a war file within the Glassfish application server (using Warbler). By the way, the fabolous OpenSUSE Build Service is a Rails application, too! So far, the entire site is powered by a single MySQL instance with query cache enabled.

The project is hosted on the following infrastructure:

• Sun Fire T2000 for (for web and application serving)
• Sun Fire X4500 for Storage
• OpenSolaris Nevada (Zones/Containers, SMF)
• ZFS (9.7 TB RAIDz, Snapshots, NFS)
• Coolstack & Blastwave packages

You should check out Fernando's presentation for more technical details, tuning info and how they benchmarked the setup - it contains a number of useful tuning hints and performance graphs.

Last time I checked, 27 Projects have joined so far (e.g. jRuby, xVM Server). Kenai itself is developed on Kenai. It's going to be interesting what other projects will find their home there.

Nick also talked a bit about their future near term plans: to improve the usability and feature set, incrementally improve the site navigation and layout and adding support for hosting files/release downloads. They also consider offering Jira as an option to Bugzilla for bug tracking and Git as another SCM option.

There is an IRC channel #projectkenai on freenode.net, to get in touch with the developers directly. The mailing list for the Project Kenai site itself, is users@help.kenai.com - you can subscribe to this list here.

One of the sessions at DrupalCon I attended was Larry Garfield's talk about "Drupal Databases: The Next Generation", which gave me a good insight into the current state of the Drupal database layer and how they plan to overhaul it for Drupal 7. The key points that I took away:

• A new API based on PDO
• Object-oriented, requiring PHP5
• Support for using prepared statements
• A unified access API
• A query builder
• More support for other database systems (currently Drupal supports MySQL and PostgreSQL only). In particular, they are keen on adding SQLite support, to ease local development.
• Support for master-slave replication (by randomly distributing reads among the hosts)
• Support for using different database types in parallel (e.g. using SQLite for read-only tables, MySQL for everything else)

The slides and a video of the presentation are available, if you want to check it out. There is a task list on the Drupal.org web site that keeps track of the ongoing activities.

Being able to use an Open Source DBMS to manage your data is nice, but what good would it be if you can't easily access it from your applications? One key factor to the popularity of MySQL is probably its wide range of available language bindings, which started with support for C, PHP and Perl from early on.

I've tried to gather a list of languages and their respective MySQL drivers/modules below. It's by no means complete or exhaustive, but I think I covered quite a lot of popular as well as exotic programming languages.

There is a number of connectors which are actually developed by the Sun Database Group (aka MySQL) itself and that are ready to use:

• Connector/ODBC - Standardized database driver Windows, Linux, Mac OS X, and Unix platforms.
• Connector/J - Standardized database driver for Java platforms and development.
• Connector/Net - Standardized database driver for .NET platforms and development.
• Connector/MXJ - MBean for embedding the MySQL server in Java applications.
• MySQL native driver for PHP - mysqlnd - The MySQL native driver for PHP is an additional, alternative way to connect from PHP 6 to the MySQL Server 4.1 or newer.
• libmysql - The original implementation of the MySQL Client/Server protocol (in C). This library is the basis for a large number of client libraries for other languages.

In addition to the above, there are several other connectors developed by Sun/MySQL, which are still under development:

• MySQL Connector/OpenOffice.org - a driver that allows you to connect to MySQL with OpenOffice.org natively, without having to use JDBC/ODBC.
• PHP PDO mysqlnd - a PHP/PDO driver based on the mysqlnd driver
• Connector C++: a driver for C++ applications that uses an API similar to the JDBC 3.0 API.

But it's not only us who develop language bindings for the MySQL server. There is an abundance of drivers that are developed and maintained by the Community, independently from Sun/MySQL (but sometimes with support or guidance from MySQL engineers). The list below is not sorted in any particular order other than the sequence in how I found them over time:

• DBD::mysql - a MySQL driver for the DBI database abstraction layer of Perl scripting language. There used to be a a native driver named "Msql-Mysql-modules", but it does not seem to be maintained anymore.
• Net::MySQL - an implementation of the MySQL Client/Server Protocol in Perl (does not require the MySQL client library). There also is MySQL::Packet, a Perl module that can encode and decode the binary protocol.
• MySQLdb - a MySQL driver for the Python scripting language.
• mysqli, ext/mysqli - MySQL Improved Extension for PHP
• mysql, ext/mysql - MySQL Extension for PHP
• PDO_MYSQL - MySQL PDO driver for PHP
• MySQL/Ruby - the MySQL API module for Ruby, based on the libmysql C library.
• Ruby/MySQL - the MySQL Ruby API, implemented in Ruby.
• MySQL++ - a C++ wrapper for MySQL's C API. It is built around the same principles as the Standard C++ Library, to make dealing with the database as easy as dealing with STL containers.
• MySQL wrapped - another C++ wrapper for the MySQL C API.
• MySQLtcl - a simple API for accessing a MySQL database server from the Tcl programming language.
• MySQL-Cocoa - An Objective-C Cocoa API for MySQL (based on libmysql).
• Eiffel MySQL - an interface to the MySQL database server using the Eiffel programming language.
• Bryan O'Sullivan's pure Haskell MySQL bindings
• hsql-mysql - A MySQL driver for Haskell
• MySQL Driver for the programming language D
• MyDB module for the Kaya programming language (included in the distribution)
• MySQL bindings for the Objective Caml language
• MySQL bindings for GNU Ada
• The SPL programming language ships with a MySQL module included in the distribution
• Database bindings (incl. MySQL) for GNU Octave
• LuaSQL is a simple database interface for the Lua programming language
• erlang-mysql-driver - a revamped MySQL driver for Erlang
• Myscsh - a Scheme implementation of the MySQL protocol

I probably forgot some other drivers/bindings - if you have any more to add, please let me know!

And if you'd like to create your own implementation for your favourite language: the protocol is documented here and here. Jan's additional notes may also be helpful to get you started.

I recently added two new packages to my repository on the openSUSE Build Service:

• Maatkit is a collection of essential command-line utilities for MySQL. Each is completely stand-alone, without dependencies other than core Perl and the DBI drivers needed to connect to MySQL, and doesn't need to be "installed" - you can just execute the scripts. This makes the tools easy to use on systems where you can't install anything extra, such as customer sites or ISPs.
• protobuf - Protocol Buffers - Google's data interchange format. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!

In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.

However, Matt seems to reduce "hosting" to "providing downloads" only:

Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
[...]
It can tell you many things, but the key is to be able to glean insight from the earliest stage of your interaction with a potential customer, and that means you have to host your own downloads. Otherwise, you have no idea how or when a would-be customer downloads your code, which makes the "why" they download it less interesting, because it becomes less actionable.

I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.

Project hosting is not just about downloads

First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.

But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.

I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.

There is more to distribute than releases

At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.

Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.

Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.

I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.

I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:

Downloads stats are overrated

Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.

Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:

As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.

I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.

 Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).

So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.