Lenz Grimmer's blog

I've been working in a fully distributed work environment for almost 8 years now (I joined MySQL AB in April, 2002). Therefore I've been reading Toni Schneider's blog post about the "5 reasons why your company should be distributed" with great interest – he raised several points that I fully agree with and which I covered in my talks about "Working for a virtual company - how we do it at MySQL" at last year's next09 conference (slides, video) and at FrOSCon 2009 (video).

However, Toni draws a profusely positive picture here, or, as my dear colleague Dean pointed out "The blog overly simplifies the realities of a distributed workforce, making it sound like it's all ponies and rainbows".

I recently received a question from Robin Schumacher at Calpont, the makers of the InfiniDB analytics database engine for MySQL: "How would you recommend we try and get bundled in with the various Linux distros?"

Since this question has come up several times before, I thought it might make sense to blog about my take on this.

First of all, please note that there is a difference between "being part of the core distribution" and "being available from a distributor's package repository". The latter one is relatively easy, the former can be hard, as you need to convince the distributor that your application is worth devoting engineering resources to maintain and support your application as part of their product. It's also a space issue – distributions need to make sure that the core packages still fit on the installation media (e.g. CD-ROMs or a DVD). Therefore they take a very close look at each package and if it's really needed to be part of the installation medium or if it's fine to provide it for download from a package repository instead.

Distributors prefer to keep their core product small and restricted to the "basic OS building blocks". While MySQL might still be considered to be a part of this, this probably does not apply to the various plugins and extensions that are available for it. Therefore the best approach is to invest some engineering time and start doing the packaging yourself, either by hiring an engineer capable of creating and maintaining the packages, or by finding someone in your community who has the required experiences and is willing to do it.

While it's of course possible to set up and maintain your own build and package hosting infrastructure for that, I recommend to make use of the existing services provided by the distributors.

The top tier distributors all provide means of offloading the maintenance of "non-core" packages to their community, offering various options for packages to be made available. For example, Novell/openSUSE provide the free "Build Service", which is capable of building packages for other distributions as well (e.g. Fedora, Mandriva, Debian/Ubuntu, etc.). In addition to automating the builds, the Build Service also takes care of the distribution via their download mirror network and ensures that your application can be found via their package search interface.

Red Hat/Fedora provide something similar, named "Koji" – but it's "Fedora only". Here's a HOWTO that outlines the process of becoming a Fedora package maintainer.

Ubuntu/Canonical have "Personal Package Archives (PPAs) – if your project is hosted on Launchpad already, that might be something to look into for providing Debian/Ubuntu packages. Alternatively you could join the Debian project and start building and maintaining your package there. They maintain a list of "Work-Needing and Prospective Packages", a description of the process on how to become a new maintainer is outlined here.

If you'd like to target Solaris/OpenSolaris as well, there is the OpenSolaris Source Juicer – a web service which allows OpenSolaris community developers to build packages (using RPM spec files) and publish them for review, so they will be included in an official package repository. The Software Porters Community Group coordinates, advocates, encourages and helps with the porting of Software from multiple Platforms to the OpenSolaris Platform.

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

The summer break seems to be over and the event season is heating up again! There is a number of conferences and events coming up in the next months — here is a quick summary of the events that I plan to attend.

This Friday I will attend an event here in Hamburg: the "Silpion Sommerfest", organized by Silpion (a local IT solutions provider which is a partner of Sun Microsystems as well). I will be there to network and talk about MySQL.

This coming weekend (2009-09-12/2009-09-13), there will be the PHP Unconference here in Hamburg, Germany . It will consist of two days of Barcamp-style sessions about PHP. Sun/MySQL are sponsors of the event and I expect several of my team mates to be there as well. With more than 180 participants, the event is already sold out.

The following week I will be attending the openSUSE Conference in Nuremberg, Germany on 2009-09-17/2009-09-20. I will give the opening keynote on Thursday morning. Titled "Working in a Virtual Community", I will talk about the pros and cons of working in a virtual organization, giving an overview about some of the technical and social aspects that play a role in working with virtual communities.

On December 10th and 11th I will be attending the 4. Secure Linux Administration Conference 2009 (SLAC) in Berlin, Germany. I've been invited to talk about MySQL and will give two sessions about MySQL Backup & Security as well as MySQL High Availability Solutions. The Call for Papers for this event is still open, so if you have a technical, "best practice" talk that might be relevant for system administrators, consider submitting your proposal!

I admit it — I'm a fan of simulation software, particularly flight simulators. Probably the best Open Source Flight Simulator out there is FlightGear — it provides an impressive level of reality and you can download and install many additional plane models and terrains. There are packages of FlightGear 1.0.0 in the games repository of the openSUSE Build Service, which works quite well and I have been enjoying it a lot. However, the FlightGear project released version 1.9.x quite a while ago (1.9.1 was published in January 2009) and I was itching on giving the new version a try (just take a look at the screenshots and you know what I mean). However, building FlighGear on Linux is quite a complex task with many dependencies, and so held off from doing it myself, waiting for someone else to perform the update...

Well, this weekend I finally bit the bullet and did it myself - FlightGear 1.9.1 has now been added to my home:LenzGr build repository. I based my packages on the ones included in the games repository, but I plan on cleaning them up a bit and splitting them into separate packages (currently the FlightGear source RPM contains SimGear and fgrun as well). I also "borrowed" the OpenSceneGraph sources and spec file from the PackMan repository, in order to have a functional build. Unfortunately FlightGear currently only builds on a very limited list of distributions so far (namely OpenSUSE 11.0, just what I needed) — I haven't had time to adapt the spec files for FlightGear and OpenSceneGraph to match the appropriate build dependencies for the other distributions yet and "02-check-gcc-output" gives me some grief on platforms where it actually builds but generates compiler warnings (but patches are welcome!)...

Shortly after I created the initial packages of embedded InnoDB on the OpenSUSE Build Service, Oracle/Innobase released an updated version (1.0.3.5325). In addition to many improvements and bug fixes, they slightly changed the versioning scheme to better indicate what version of the InnodDB plugin their code is based on (see Vasil's posting on the InnoDB Forums for more information).

I've now updated my InnoDB packages on the Build Service to this version as well - please note that the naming scheme of the shared library package has been changed from "embedded_innodb1" to "libinnodb2" — RPM will take care of replacing the old package during update, even though the name has changed.

Oracle/InnoBase announced the availability of the embedded version of InnoDB at this year's MySQL Conference & Expo, but I have not seen a lot of comments or reviews about it so far. Which surprises me, because I think this is a very interesting piece of technology!

In my opinion it might actually hit the sweet spot for application developers seeking an alternative embedded database solution. SQLite is nice and popular, but it seems to have concurrency issues when used in multi-threaded applications. An embedded MySQL server would be an alternative - this is what the Amarok developers decided to go with, for example. But this approach has its issues, too, especially the lack of a shared library version of libmysqld poses some challenges when distributing binaries.

This is where I think the embedded version of InnoDB might have an edge. It's pretty lightweight in comparison to a full-blown MySQL server, provides excellent crash-recovery (which is essential for desktop applications), transactions (useful in environments with high concurrency) and foreign key constraints. I'm not sure how important these are for embedded use cases, it probably depends on the complexity of the data to be stored. On the downside, Embedded InnoDB does not "speak" SQL. In order to store and retrieve values, you need to use the InnoDB API. See the chapter Concepts and Architecture for more details and an overview.

Another possible reason for the low popularity might be that it's currently not part of any Linux distribution (yet) and that Oracle only provides binary tarball packages for Linux and a Windows binary for download from the web site.

Therefore I've now created a spec file to build RPMs of Embedded InnoDB and added it to my repository on the openSUSE Build Service, which now provides Embedded InnoDB packages for a wide range of RPM-based Linux distributions. I hope that the spec file will be included in the next source distribution. I've posted it (and a patch to fix a few problems with the examples) to the newly created InnoDB mailing list, but to be sure I added a note to the Embedded InnoDB Forum as well.

XtraBackup is an Open Source online (non-blockable) backup solution for the InnoDB and XtraDB storage engines. It works with both MySQL 5.0 and 5.1 (and possibly 5.4 as well) and is distributed under the GPLv2.

Some weeks ago Vadim announced the availability of xtrabackup-0.7, stating that they consider it stable enough now to label this version a "Release Candidate". I've been maintaining RPM packages of xtrabackup on the fine openSUSE Build Service for quite some time now, RPMs of 0.7 for a number of distributions are now available for download. Please report any bug reports via the bug tracker on Launchpad.

Today I attended the Drizzle Developer Day which took place in the auditorium of the Sun Campus in Santa Clara.

Many of the the Drizzle core hackers as well as several other people interested in the development attended this event, hacking away and discussing various issues. Jeremy Zawodny gave a presentation about Craigslist's needs for Drizzle, Jay Pipes gave an overview over Google's protocol buffers library. I took a number of pictures, which you can find in my Flickr photo set.

I joined a group of people that haven't built Drizzle from source by themselves so far, helping them with installing Bazaar and the required libraries. As Drizzle requires several third-party libraries that sometimes are not included in the common linux distributions (or only in outdated versions), we spent some time in getting these build requirements fulfilled.

One of the requirements for building Drizzle is libdrizzle - the client & protocol library. So one first has to download and compile this one, before the actual build of the server can proceed. I noticed that the libdrizzle source distribution contained an RPM spec file already, so I've been working on adding libdrizzle to the openSUSE build service today. The packages for various distributions (Fedora, openSUSE, RHEL, Mandriva) will be available for download shortly. Along the way I also fixed several small issues in the spec file and created a libdrizzle-devel subpackage. The patches are now proposed for merging on Launchpad, I hope Eric will take a look at these shortly.

On early Tuesday morning, I made a quick trip to Paris, France, to attend and speak at the Solutions Linux / Open Source 2009 Conference. I've never been to this conference before and was quite surprised about its size - it's actually the largest Open Source event in France and it reminded me a lot of LinuxTag in Germany. Many well-known vendors (e.g. Sun, Novell, Canonical, Bull, etc.) were exhibiting. The also was a large "DotOrg" section for various Open Source projects and I was very happy to see that LeMUG.fr, the official MySQL User Group of France, had a table there, too! A big Thank You goes to Pascal Borghino, who manned that table on his own most of the time and answered questions about MySQL. I walked around the exhibition floor and took some pictures, which I have now posted to my Flickr account.

In addition to the exhibition, there were several parallel tracks with sessions. I was invited to speak about MySQL HA Solutions in the "Aquarium". Unfortunately I had the last slot at that day and they were running a bit behind schedule, so I had only 15-20 people in the audience. But I still had a great time and I received several positive comments about my presentation. I travelled back home early the next day - I wish I had scheduled some more time to attend the conference. I look forward to going there again next year, it was a nice event.

Sorry for the downtime of this site - until around a week ago I hosted my home page on a trusty Genesi Pegasos II system (powered by a PowerPC G4 Processor clocked at 1GHz, using Debian 4.0 PPC with 512 MB of RAM), serving these pages from my home DSL connection. Unfortunately this system provided no means of redundancy - the hard disk drive died.

Luckily I perform frequent backups, so I moved most parts of the site to a shared hosting space now - the picture gallery is unfortunately too big to fit into the space that I have there. I'll try to move the pictures into my Flickr account instead, but this will take some time.

Note that the primary domain name of this site is now lenzg.net - lenzg.org, (the domain that I tried to promote as the official domain for my site) used to redirect to the home machine at lenz.homelinux.org. Both now redirect to the new address instead. I've initiated the move of the lenzg.org domain to the other provider as well, so soon this site will be available from both the .org and .net domain. Please don't link to lenz.homelinux.org anymore, as that site will eventually go out of service. Until then, a small openSUSE Linux box (Intel PIII, 500 MHz, 192 MB of RAM) running lighttpd will perform the URL redirection.

Now that MySQL 5.1 has finally been labeled and released as "GA", we have noticed quite a significant increase in the download numbers for 5.1. I find this quite promising, despite the comments from some people that suggested to stay away from it for now. I performed updates of 5.0 to 5.1 several times already, and did not observe any serious problem so far. But then again, I am by no means a power user, so your mileage may vary...

I assume that many of the users are already using some older versions of MySQL and will have to perform an upgrade from the previous version to the new one. As many things have been changed and improved between 5.0 and 5.1, an upgrade needs to be planned carefully. The MySQL reference manual has an entire chapter devoted to this subject. The best general advice is probably to create a backup of your data first before attempting to upgrade!

I performed an upgrade test on an openSUSE 11.0 system that was running on the MySQL version shipped with the distribution (5.0.51a). I first loaded it with some databases that I copied from other live systems (via SQL dumps). Then I grabbed the Server, Client and shared library RPMs the "Linux x86 generic RPM (dynamically linked)" section of the 5.1 downloads page and installed them with rpm -Uhv MySQL-server-5.1.30-0.glibc23.i386.rpm MySQL-client-5.1.30-0.glibc23.i386.rpm MySQL-shared-5.1.30-0.glibc23.i386.rpm. The packages were installed without a hitch and properly replaced the existing 5.0 packages (even though they use different package names). I then restarted the server and ran mysql_upgrade afterwards.

Success! I was pretty impressed how smooth this upgrade was performed. So far, I did not notice any regression or incompatibility. But I would be interested in hearing stories and comments from other users about their upgrade experiences!

So here are some questions I would like you to answer — I would appreciate your feedback either via this blog or by email to firstname at sun dot com:

• What are your expectations and most important requirements for performing MySQL upgrades?
• Have you upgraded to MySQL 5.1 yet?
• If not, why not?
• If yes, what was your upgrade experience like? Any serious issues or problems? What platform and package format did you use?
• Did you use the documentation in our reference manual to perform the update? Was it helpful? What could be improved?
• Do you have any other comments/suggestions on the topic of upgrading MySQL?

I am looking forward to your comments. Thanks and Happy Holidays!

I recently added two new packages to my repository on the openSUSE Build Service:

• Maatkit is a collection of essential command-line utilities for MySQL. Each is completely stand-alone, without dependencies other than core Perl and the DBI drivers needed to connect to MySQL, and doesn't need to be "installed" - you can just execute the scripts. This makes the tools easy to use on systems where you can't install anything extra, such as customer sites or ISPs.
• protobuf - Protocol Buffers - Google's data interchange format. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!

While we're on the topic of Bazaar - this week I got informed by the organizers of the FrOSCon 2008 conference that they accepted two of my talk proposals: one session will be an introduction to this source code management system (what a coincidence), the other one will be an introduction to OpenSolaris for Linux users, explaining some of the underlying technologies and how they differ from what a seasoned Linux user may be accustomed to.

And no, I have not given up on using Linux - quite the contrary! I have been very impressed by the latest OpenSUSE 11.0 release and already run it for since quite some time on several of my work systems. In fact, I already convinced several colleagues of mine to give it a try as well! I am amazed by the speed and "out of the box experience" of this version and I actually plan to install it on my Genesi Pegasos PowerPC machine as well, replacing Debian on there. But as a Sun employee, I of course have to familiarize myself with the other products and projects that we're involved in. And on the Server side, Solaris does have a few interesting features that Linux currently lacks. But I digress.

I look forward to speaking at FrOSCon again - it has been a great conference in the past two years. Very well organized, nice venue, a relaxed atmosphere and excellent technical sessions and speakers.

Other MySQLers submitted talks as well - for example, Giuseppe will give a presentation titled "MySQL Community How To", Susanne will give a PostgreSQL tutorial and others will participate in the separate PHP subconference. Don't miss it - this year's FrOSCon will take place on August 23rd&24th in St. Augustin, Germany (close to Bonn). For the first time, we will also try to set up a MySQL project table. So if you are there, make sure to stop by and have a chat with us!

Back when I still worked at SuSE, I was in charge of maintaining a number or packages of the distribution (actually, you should still be able to find traces of my work in the RPM changelogs). Nowadays, I maintain a number of packages for openSUSE and other distributions on the openSUSE Build Service, which is just brilliant for this purpose.

If you happen to live in northern Germany and are interested to learn more about the RPM package manager and how to build packages, consider coming to the TU Harburg this coming Thursday (March 13th). At 19:00, I will give a presentation about this topic in building, D, room D1023 (in cooperation with the Hamburg branch of the German Unix User Group). More information (in German) can be obtained from here.

See you there!