Lenz Grimmer's blog

I recently received a question from Robin Schumacher at Calpont, the makers of the InfiniDB analytics database engine for MySQL: "How would you recommend we try and get bundled in with the various Linux distros?"

Since this question has come up several times before, I thought it might make sense to blog about my take on this.

First of all, please note that there is a difference between "being part of the core distribution" and "being available from a distributor's package repository". The latter one is relatively easy, the former can be hard, as you need to convince the distributor that your application is worth devoting engineering resources to maintain and support your application as part of their product. It's also a space issue – distributions need to make sure that the core packages still fit on the installation media (e.g. CD-ROMs or a DVD). Therefore they take a very close look at each package and if it's really needed to be part of the installation medium or if it's fine to provide it for download from a package repository instead.

Distributors prefer to keep their core product small and restricted to the "basic OS building blocks". While MySQL might still be considered to be a part of this, this probably does not apply to the various plugins and extensions that are available for it. Therefore the best approach is to invest some engineering time and start doing the packaging yourself, either by hiring an engineer capable of creating and maintaining the packages, or by finding someone in your community who has the required experiences and is willing to do it.

While it's of course possible to set up and maintain your own build and package hosting infrastructure for that, I recommend to make use of the existing services provided by the distributors.

The top tier distributors all provide means of offloading the maintenance of "non-core" packages to their community, offering various options for packages to be made available. For example, Novell/openSUSE provide the free "Build Service", which is capable of building packages for other distributions as well (e.g. Fedora, Mandriva, Debian/Ubuntu, etc.). In addition to automating the builds, the Build Service also takes care of the distribution via their download mirror network and ensures that your application can be found via their package search interface.

Red Hat/Fedora provide something similar, named "Koji" – but it's "Fedora only". Here's a HOWTO that outlines the process of becoming a Fedora package maintainer.

Ubuntu/Canonical have "Personal Package Archives (PPAs) – if your project is hosted on Launchpad already, that might be something to look into for providing Debian/Ubuntu packages. Alternatively you could join the Debian project and start building and maintaining your package there. They maintain a list of "Work-Needing and Prospective Packages", a description of the process on how to become a new maintainer is outlined here.

If you'd like to target Solaris/OpenSolaris as well, there is the OpenSolaris Source Juicer – a web service which allows OpenSolaris community developers to build packages (using RPM spec files) and publish them for review, so they will be included in an official package repository. The Software Porters Community Group coordinates, advocates, encourages and helps with the porting of Software from multiple Platforms to the OpenSolaris Platform.

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

I admit it — I'm a fan of simulation software, particularly flight simulators. Probably the best Open Source Flight Simulator out there is FlightGear — it provides an impressive level of reality and you can download and install many additional plane models and terrains. There are packages of FlightGear 1.0.0 in the games repository of the openSUSE Build Service, which works quite well and I have been enjoying it a lot. However, the FlightGear project released version 1.9.x quite a while ago (1.9.1 was published in January 2009) and I was itching on giving the new version a try (just take a look at the screenshots and you know what I mean). However, building FlighGear on Linux is quite a complex task with many dependencies, and so held off from doing it myself, waiting for someone else to perform the update...

Well, this weekend I finally bit the bullet and did it myself - FlightGear 1.9.1 has now been added to my home:LenzGr build repository. I based my packages on the ones included in the games repository, but I plan on cleaning them up a bit and splitting them into separate packages (currently the FlightGear source RPM contains SimGear and fgrun as well). I also "borrowed" the OpenSceneGraph sources and spec file from the PackMan repository, in order to have a functional build. Unfortunately FlightGear currently only builds on a very limited list of distributions so far (namely OpenSUSE 11.0, just what I needed) — I haven't had time to adapt the spec files for FlightGear and OpenSceneGraph to match the appropriate build dependencies for the other distributions yet and "02-check-gcc-output" gives me some grief on platforms where it actually builds but generates compiler warnings (but patches are welcome!)...

Shortly after I created the initial packages of embedded InnoDB on the OpenSUSE Build Service, Oracle/Innobase released an updated version (1.0.3.5325). In addition to many improvements and bug fixes, they slightly changed the versioning scheme to better indicate what version of the InnodDB plugin their code is based on (see Vasil's posting on the InnoDB Forums for more information).

I've now updated my InnoDB packages on the Build Service to this version as well - please note that the naming scheme of the shared library package has been changed from "embedded_innodb1" to "libinnodb2" — RPM will take care of replacing the old package during update, even though the name has changed.

Oracle/InnoBase announced the availability of the embedded version of InnoDB at this year's MySQL Conference & Expo, but I have not seen a lot of comments or reviews about it so far. Which surprises me, because I think this is a very interesting piece of technology!

In my opinion it might actually hit the sweet spot for application developers seeking an alternative embedded database solution. SQLite is nice and popular, but it seems to have concurrency issues when used in multi-threaded applications. An embedded MySQL server would be an alternative - this is what the Amarok developers decided to go with, for example. But this approach has its issues, too, especially the lack of a shared library version of libmysqld poses some challenges when distributing binaries.

This is where I think the embedded version of InnoDB might have an edge. It's pretty lightweight in comparison to a full-blown MySQL server, provides excellent crash-recovery (which is essential for desktop applications), transactions (useful in environments with high concurrency) and foreign key constraints. I'm not sure how important these are for embedded use cases, it probably depends on the complexity of the data to be stored. On the downside, Embedded InnoDB does not "speak" SQL. In order to store and retrieve values, you need to use the InnoDB API. See the chapter Concepts and Architecture for more details and an overview.

Another possible reason for the low popularity might be that it's currently not part of any Linux distribution (yet) and that Oracle only provides binary tarball packages for Linux and a Windows binary for download from the web site.

Therefore I've now created a spec file to build RPMs of Embedded InnoDB and added it to my repository on the openSUSE Build Service, which now provides Embedded InnoDB packages for a wide range of RPM-based Linux distributions. I hope that the spec file will be included in the next source distribution. I've posted it (and a patch to fix a few problems with the examples) to the newly created InnoDB mailing list, but to be sure I added a note to the Embedded InnoDB Forum as well.

XtraBackup is an Open Source online (non-blockable) backup solution for the InnoDB and XtraDB storage engines. It works with both MySQL 5.0 and 5.1 (and possibly 5.4 as well) and is distributed under the GPLv2.

Some weeks ago Vadim announced the availability of xtrabackup-0.7, stating that they consider it stable enough now to label this version a "Release Candidate". I've been maintaining RPM packages of xtrabackup on the fine openSUSE Build Service for quite some time now, RPMs of 0.7 for a number of distributions are now available for download. Please report any bug reports via the bug tracker on Launchpad.

Today I attended the Drizzle Developer Day which took place in the auditorium of the Sun Campus in Santa Clara.

Many of the the Drizzle core hackers as well as several other people interested in the development attended this event, hacking away and discussing various issues. Jeremy Zawodny gave a presentation about Craigslist's needs for Drizzle, Jay Pipes gave an overview over Google's protocol buffers library. I took a number of pictures, which you can find in my Flickr photo set.

I joined a group of people that haven't built Drizzle from source by themselves so far, helping them with installing Bazaar and the required libraries. As Drizzle requires several third-party libraries that sometimes are not included in the common linux distributions (or only in outdated versions), we spent some time in getting these build requirements fulfilled.

One of the requirements for building Drizzle is libdrizzle - the client & protocol library. So one first has to download and compile this one, before the actual build of the server can proceed. I noticed that the libdrizzle source distribution contained an RPM spec file already, so I've been working on adding libdrizzle to the openSUSE build service today. The packages for various distributions (Fedora, openSUSE, RHEL, Mandriva) will be available for download shortly. Along the way I also fixed several small issues in the spec file and created a libdrizzle-devel subpackage. The patches are now proposed for merging on Launchpad, I hope Eric will take a look at these shortly.

Now that MySQL 5.1 has finally been labeled and released as "GA", we have noticed quite a significant increase in the download numbers for 5.1. I find this quite promising, despite the comments from some people that suggested to stay away from it for now. I performed updates of 5.0 to 5.1 several times already, and did not observe any serious problem so far. But then again, I am by no means a power user, so your mileage may vary...

I assume that many of the users are already using some older versions of MySQL and will have to perform an upgrade from the previous version to the new one. As many things have been changed and improved between 5.0 and 5.1, an upgrade needs to be planned carefully. The MySQL reference manual has an entire chapter devoted to this subject. The best general advice is probably to create a backup of your data first before attempting to upgrade!

I performed an upgrade test on an openSUSE 11.0 system that was running on the MySQL version shipped with the distribution (5.0.51a). I first loaded it with some databases that I copied from other live systems (via SQL dumps). Then I grabbed the Server, Client and shared library RPMs the "Linux x86 generic RPM (dynamically linked)" section of the 5.1 downloads page and installed them with rpm -Uhv MySQL-server-5.1.30-0.glibc23.i386.rpm MySQL-client-5.1.30-0.glibc23.i386.rpm MySQL-shared-5.1.30-0.glibc23.i386.rpm. The packages were installed without a hitch and properly replaced the existing 5.0 packages (even though they use different package names). I then restarted the server and ran mysql_upgrade afterwards.

Success! I was pretty impressed how smooth this upgrade was performed. So far, I did not notice any regression or incompatibility. But I would be interested in hearing stories and comments from other users about their upgrade experiences!

So here are some questions I would like you to answer — I would appreciate your feedback either via this blog or by email to firstname at sun dot com:

• What are your expectations and most important requirements for performing MySQL upgrades?
• Have you upgraded to MySQL 5.1 yet?
• If not, why not?
• If yes, what was your upgrade experience like? Any serious issues or problems? What platform and package format did you use?
• Did you use the documentation in our reference manual to perform the update? Was it helpful? What could be improved?
• Do you have any other comments/suggestions on the topic of upgrading MySQL?

I am looking forward to your comments. Thanks and Happy Holidays!

I recently added two new packages to my repository on the openSUSE Build Service:

• Maatkit is a collection of essential command-line utilities for MySQL. Each is completely stand-alone, without dependencies other than core Perl and the DBI drivers needed to connect to MySQL, and doesn't need to be "installed" - you can just execute the scripts. This makes the tools easy to use on systems where you can't install anything extra, such as customer sites or ISPs.
• protobuf - Protocol Buffers - Google's data interchange format. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!

Back when I still worked at SuSE, I was in charge of maintaining a number or packages of the distribution (actually, you should still be able to find traces of my work in the RPM changelogs). Nowadays, I maintain a number of packages for openSUSE and other distributions on the openSUSE Build Service, which is just brilliant for this purpose.

If you happen to live in northern Germany and are interested to learn more about the RPM package manager and how to build packages, consider coming to the TU Harburg this coming Thursday (March 13th). At 19:00, I will give a presentation about this topic in building, D, room D1023 (in cooperation with the Hamburg branch of the German Unix User Group). More information (in German) can be obtained from here.

See you there!

Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.

In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!

• Added a new rsync backup type. This is very useful if you want to use mylvmbackup to create the initial state for your slave servers. Instead of creating a .tar.gz archive, the data directory is copied into a timestamped archive directory. (Robin)
• Added support for a trailing argument to tar, which can be used for excluding files. (Robin)
• Separated out the suffix of the tarball (Preperation for rsync and users that want to use bzip2 or no compression on the tarball.) (Robin)
• While the backup is performed, a temporary suffix at the end of the tar backup file name (or the rsync target directory name) now indicates that it is incomplete. (Robin)
• The my.cnf configuration file is now included in the backup. (Robin)
• Added the ability to run an extra FLUSH TABLES on busy databases where lvcreate might take a long time (and may overrun the interactivity timeout on the connection, losing the lock). (Robin)
• Added option "--pidfile" to provide an alternative PID file location for the second server instance that is started to perform the InnoDB recovery on the snapshot prior to backing it up (Otherwise it may default to using the same pid file location that the running server uses and safe_mysqld will abort) - thanks to Kristian Köhntopp for making me aware of this problem.
• Before discarding the snapshot LV, the output of "lvs <snapshot>" is now printed out for diagnostics. It contains useful information like "how much percent of the backing store was used", which helps tuning the size of the snapshot LV.
• Added option "--skip-flush-tables" that performs the snapshot without flushing the tables to disk beforehand (which is not supported by InnoDB tables anyway) - this would save time, as the flushing can take a while, depending on the buffer sizes. (Thanks to Peter Zaitsev for the suggestion)

Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.

Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.

After getting very annoyed about the behaviour of pinepg in combination with gpg2 on my openSUSE 10.3 beta test system, I have now scratched my itch and switched to an alternative tool: pine-gpg-filter:

The distinguishing characteristic of this package (when compared against similar pine and gpg wrappers) is its ability to handle multiple roles or identities (i.e. different keys for different email addresses). Unlike some of the other pine and gpg wrappers, this one performs no passphrase caching (consider using gpg-agent in gnupg2).

They provide a "noarch" RPM directly from that site, but I've now also added it to my home:LenzGr repository in the openSUSE build service. Enjoy! 

In addition to the binary downloads that we provide from our site, Linux RPM builds of the MySQL Proxy (both the latest stable version as well as SVN snapshot releases, named mysql-proxy-snapshot) are now available for download from the server:/database repository of the openSUSE build service. It provides packages for a number of Linux distributions, e.g. Fedora 5/6, SLES 9/10 SuSE/openSUSE 10.x. By the way, this repository also contains RPMs of the current 5.0.45 MySQL Community Server for the distributions mentioned above. Thanks a lot to Darix for the initial checkin of the proxy package!

JFYI, I recently updated several packages in my home:LenzGr and the devel:tools:scm repositories of the openSUSE Build Service:

• bzr (Version 0.18)
• darcs (Version 1.0.9)
• google-perftools (Version 0.92)
• KeyJnote (Version 0.10.0)
• Mercurial (Version 0.9.4)

Enjoy! And Kudos to the folks that develop and maintain the build service, this is a great service to the Community.