Lenz Grimmer's blog

There has been a lot of buzz about the MySQL 5.5 GA release and its new features and other user-visible improvements. In this blog post, I'd like to touch on a less noticeable, but still important change.

CMake has already been used to build the MySQL Server on Windows for a long time, while the GNU autotools were used on all other platforms. Since MySQL 5.5, all builds on all platforms are now performed using the same tool chain. With the latest release of MySQL 5.5, we've made an important step to clean up and simplify the MySQL build system: the support for autoconf/automake has now been removed completely. We've been performing the release builds of MySQL 5.5 using CMake exclusively for quite some time already. It became obvious that maintaining two separate build systems simply had become too much of a burden for our engineers, especially since the autotools-based builds were no longer exhaustively tested. This change was outlined in WorkLog#5665 - Removal of the autotools-based build system. We've made this step in close cooperation with our community of packagers (e.g. the maintainers of MySQL packages on the various Linux distributions).

By moving to CMake, we are giving our developers one common build mechanism for all platforms and there is a lot of new useful functionality such as out-of-source builds or a GUI for configuring the build options. And they can now build MySQL in the very same way that we do it for our own binaries! I've covered the advantages in my previous blog post about Building MySQL Server with CMake on Linux/Unix already. We've also created a general article about CMake and MySQL as well as an Autotools to CMake Transition Guide on the MySQL Forge Wiki. The description of the source build process in the reference manual has also been updated to reflect this change.

A big “thank you” should to go to Vladislav Vaintroub and Davi Arnaut for implementing and pushing the transition to CMake forward, and to Paul DuBois for creating and improving the documentation! Wearing my former build and release engineer hat, I am very happy about this change.

Other CMake-related articles that are worth reading:

• Giuseppe Maxia: Building MySQL 5.5 with CMake
• Jonathan Perkin: How To Build MySQL Releases
• Santo Leto: Step by Step Guide on How to Compile MySQL 5.5 on Microsoft Windows

CMake is a cross-platform, open-source build system, maintained by Kitware, Inc.

From the CMake.org home page:

CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice.

It has been used for building the MySQL Server on Windows since MySQL 5.0 – the initial CMake build support was added in August 2006.

For building MySQL on all other platforms, the GNU autotools (autoconf, automake and libtool) are currently being used.

CMake is used in some other MySQL projects as well, e.g.

• MySQL Connector/C
• MySQL Connector/C++
• MySQL Proxy

On February 22nd, Vladislav Vaintroub pushed the changes required to implement WorkLog#5161 "CMake-based unified build system" into the "mysql-next-mr" branch (aka the "Celosia" mile stone).

From this version on, CMake can also be used to build MySQL on Linux and other Unix platforms. For the time being, the autoconf/automake files are still available as well, but will be phased out once the CMake build enviroment has reached the desired level of maturity. The change was announced on February 28th on our "internals" developer discussion list.

The purpose of WL#5161 is to simplify the MySQL build system. It is much easier and less error-prone to maintain a unified build system for all platforms than two separate ones.

CMake has been chosen because of several reasons; the worklog description lists a few pro-CMake arguments (slightly rephrased):

• CMake works on Windows. The GNU buildsystem does not really work and likely never will work natively on Windows (Using Cygwin is not really an option).
• Traditionally, new MySQL features that required changes in the build environment (e.g. the plugin system, unit tests, most recently googletest integration) were always implemented on Unix first, leaving Windows behind (sometimes for years). This would not happen with a unified build system.
• MySQL already uses CMake since 2006 on Windows, so we do not need to start from scratch, only port what we have to Unix.
• CMake runs on every OS and compiler we support.
• It is simple to obtain and install on a wide range of platforms. It is available in all major Linux package repositories (e.g. Ubuntu, Fedora, OpenSUSE). It is also in the OpenSolaris repository, known as SUNWCmake. It's in FreeBSD ports and available for Mac OS X. It is also very simple to compile it from source, the single prerequisite is a working C++ compiler and make utility.
• CMake has support for features we need and might need, e.g. system checks or cross-compiling.
• CMake provides integrated support for packaging. It can handle both simple packages (tar.gz or zip archives) and more complex things like DEB and RPM without much extra coding.
• Good integration with the popular IDEs (Visual Studio, Xcode, Eclipse CDT, KDevelop). Developing in an IDE makes the development process more enjoyable, and potentially it lowers the barrier for external contributors. Of course, CMake can generate traditional Unix Makefiles, which appear to be are superior to the ones generated by GNU autotools (for example, they have progress indicators, colored output and working dependencies).
• The scripting language used by CMake is simpler than m4 used by autotools.
• CMake is a single small tool, not a bunch of different tools as in GNU system (autoconf, autoheader, automake, libtool)

I'd like to mention a few additional reasons:

• Out-of-source builds – CMake can separate the build directory from the source directory. This is convenient, as your working source tree is not cluttered with object files and other fragments of the build process.
• Build configuration using a GUI. The cmake-gui package (based on Nokia/TrollTech's Qt library) provides a convenient way of enabling and configuring the various available build options. This is much better than having to memorize all the required defines and configuration flags.
• Integrated support for creating a wide range of package formats.

The CMake Wiki lists a number of other "nice to have" features.

From a developer perspective, I hope that it will make it much easier to finally implement two things that many developers working with MySQL have been waiting for (now that the build code has been cleaned up):

• Compiling the embedded MySQL Server (libmysqld) as a shared library
• Better support for cross-compilation

Building MySQL with CMake is quite simple and straighforward – the process is outlined on the MySQL Forge Wiki. The document is still work in progress and we'd like to encourage you to take a look at it, try to follow the steps and update/improve the Wiki page, if needed! Your feedback on the build process is appreciated. Feel free to join our internals mailing list to discuss your impressions and observations or submit a bug report via the Bug Database. It's likely that the build still has a few rough edges that we'd like to fix quickly (e.g. BUG#51502 – a fix for this one is already commited to the mysql-next-mr-bugfixing source tree and will be merged into the mysql-next-mr trunk soon).

If you're new to CMake, you might want to take a look at the "Getting Started With CMake (An End-User's Perspective) For Cross-Platform Building" screencast or the "Running CMake" article.

Happy hacking!

Last week, my colleagues Giuseppe, Kai and myself attended the SAPO Codebits event in Lisbon, Portugal. Codebits is an annual, invite-only hacking event, which went on for three days. The venue they chose this year was the "Cordoaria", a former rope factory located in the Belém district, close to the 25 de Abril Bridge (which is an impressive sight!). I have been told that the Cordoaria is the longest building in Portugal and I have no doubts about that! The building is so long that the crew used bicycles to get from one end to the other. I've taken a number of pictures from the event as well as from Lisbon itself, you can find them in this flickr set.

The organizers described this year's event as follows:

3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience.

I wholeheartedly agree, we had a great time! The conference started with sessions and presentations on a wide range of topics on the first two days. Afterwards, a 24-hour programming contest was held. I was invited to give two talks, one being my all-time favourite about "MySQL High Availability solutions" (slides, video), the other one was titled "Why you should be using a distributed version control system (DVCS) for your project" (video, slides). Both went quite well and the feedback I received was pretty positive. Giuseppe talked about "MySQL Schema Migration" (slides, video) and gave an "Introduction to Gearman" (video). Kai's talk was titled "Think before you develop" (video) and gave a nice roundup of tips and best practices for setting up and developing new web projects.

The Codebits session schedule was filled with amazing and interesting talks in four parallel tracks. Sometimes it was hard to choose – some other talks I attended and enjoyed:

• Robert Nyman: JavaScript - From Birth To Closure (slides, video)
• Glenn Jones: Re-using social media data (slides)
• Brian LeRoux: PhoneGap: Mobile App Developer Zero to Hero (video)
• Brian LeRoux: The State of JavaScript on Mobile Devices
• Walter Belgers: Becoming a lockpicker

Walter gave a lockpicking workshop after his presentation, which I attended as well. I was quite impressed (and a bit shocked) to find out how easy many locks can be opened this way! Later that evening there even was a live band named "Pornophonique" playing (one guy with a guitar, the other one using an Nintendo Game Boy for making music), but I missed that show as I was too busy opening more locks... Fortunately the concert and most of the sessions were recorded on video (in excellent quality) and are already available from the SAPO video pages. Kudos for this speedy service!

But this just matches my overall conclusion of this event: very well organized, great speakers and venue. Thanks to the organizers for having us, we really enjoyed our stay!

This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!

Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.

With IntelliJ now being available under an Open Source license, developers have another option to choose from when it comes to Java-based IDEs/Frameworks (Eclipse and NetBeans being the other two prominent ones). Choice is always good, and being an Open Source enthusiast, I of course welcome JetBrain's move!

However, as I'm not really a heavy GUI-based IDE user myself, I can't really comment on which one is the best. These kind of discussions tend to turn into a Holy War anyway... In the end it's likely that each of them gets the job done and you have to come to your own conclusions, based on your personal preference and requirements.

I personally would be interested in seeing how their support for PHP or Python compares to the one in NetBeans. Their plugin repository lists more that 560 plugins, including many for database connectivity/modeling/navigation (incl. support for MySQL). I'm also glad to see that they have a plugin for Bazaar, something that I'm desperately missing from NetBeans!

Interestingly, they decided to keep a few parts proprietary, it's going to be interesting to see how this will turn out for them and if developers will be willing to pay for these extra features, considering that most of this is available for free from the other two projects.

Their Contributor License Agreement looks like it has been derived from the Sun Contributor Agreement (SCA), which is always nice to see. I assume this can be attributed to Roman Strobl - I was positively surprised to notice that he joined their team as a technology evangelist in June! Roman did a great job in spreading the NetBeans and OpenSolaris gospel at Sun before and I briefly met him at this year's FOSDEM conference in Brussels. Congratulations!

Some time ago I posted a compilation of applications and programming languages that provide an API to connect to the MySQL Server. As it turned out, I forgot a few that I would like to mention here:

• Apache DBD API: a MySQL driver for mod_apr_dbd is not included in the official distribution, but can be obtained seperately from here. Some distributions (e.g. openSUSE) actually provide installable packages of this driver module.
• GRASS MySQL driver: The MySQL database driver in GRASS enables GRASS applications to store vector attributes in a MySQL server. Alternatively, data can be stored in an embedded version of the MySQL Server.
• HDBC: HDBC provides an abstraction layer between Haskell programs and SQL relational databases. This lets you write database code once, in Haskell, and have it work with any number of backend SQL databases.
• libdrizzle: This is the client and protocol library for Drizzle and MySQL that provides both client and server interfaces. A number of APIs (e.g. Java, Lua, Perl and Ruby as well as Python) have been built on top of it, but I'm not quite sure if they all support the MySQL client/server protocol as well.
• MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. This means you don't have to compile anything - the MySQL client library doesn't even have to be installed on your system.

Do you know of any other MySQL Connectors/APIs? Let me know!

Shortly after I created the initial packages of embedded InnoDB on the OpenSUSE Build Service, Oracle/Innobase released an updated version (1.0.3.5325). In addition to many improvements and bug fixes, they slightly changed the versioning scheme to better indicate what version of the InnodDB plugin their code is based on (see Vasil's posting on the InnoDB Forums for more information).

I've now updated my InnoDB packages on the Build Service to this version as well - please note that the naming scheme of the shared library package has been changed from "embedded_innodb1" to "libinnodb2" — RPM will take care of replacing the old package during update, even though the name has changed.

After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.

Some notable highlights from the ChangeLog:

• Removed the absolute path names to external tools (make sure $PATH is correct)
• Added --log-err to the startup options of the recovery instance to avoid cluttering the server's error log
• Added support for hooks written as Perl Modules. (Matthew Boehm)
• Added support for date/time-formatted path names for backupdir and mountdir (Matthew Boehm)
• Backupdir and mountdir are now created automatically (Matthew Boehm)
• Added new hook "logerr" when an error is logged. (Matthew Boehm)
• Added Option --keep-mount... (Tim Stoop)
• Removed the bind mount, now requires LVMv2
• Support reading login/password from ~/.my.cnf (Baron Schwartz)
• Documentation fixes and improvements (Ville Skyttä) (Bug #302144)

Version 1.0.3-alpha of the MySQL Connector/C++ has just been announced by Lawrin Novitsky and is now available for download. This driver is licensed under the GPL and is a new implementation of the MySQL Client/Server protocol. Instead of wrapping the C API calls in C++ methods, the implementation mimics the JDBC API, which hopefully feels much more "native" to a seasoned C++ developer.

The driver has been ported to a wide range of platforms and is about to hit the beta test phase. So if you're writing a C++ application that needs to connect to a MySQL Server, give it a try! The developers are always looking for feedback.

More information and examples on how to use it can be found on this detailed Wiki page on the MySQL Forge.

Some days ago, I released version 0.11 of mylvmbackup a Perl script that performs consistent backups of a MySQL server by using LVM filesystem snapshots. The source archive as well as a generic RPM can be found on the project home page, packages for many Linux distributions are available on the openSUSE Build service.

This release includes some new functionality as well as numerous bug fixes and improvements, most notably:

• Added support for using rsnap as a backup backend (Matt Lohier)
• The documentation is now maintained in POD style instead of asciidoc (Matthew Boehm)
• Support using non-GNU tar and additional compression methods (e.g. bzip2, lzma) (Alexander Skwar)
• Code cleanups, improved error handling and logging

I would like to thank all the contributors for their support! More details about the changes in this release, directly from the ChangeLog:

• Added new option "--quiet" that suppresses informal logging output (warnings and errors will still be printed/logged)
• Applied patch from "kjetilho" that makes mylvmbackup more robust and paranoid when it comes to handling errors/failures (Bug #298175) Now the script aborts cleanly in case of any error.
• Fixed error handling in case of a failed DB connection (patch submitted by Matthew Boehm, Bug #280989)
• Merged changes from Alexander Skwar: Fix for Bug #278478 and implemented Blueprint "Improve the tar backup backend of mylvmbackup"
• Fixed Bug #271671: "overloading parameters does not work" by removing the default values for host and port from the configuration file and removing the unnecessary check for passing both host and socket at the same time. Updated documentation and configuration file comments accordingly.
• Applied patch suggested by "Jonas" to fix Bug #267944: "backup returns successfully when snapshot creation fails"
• Code cleanup: moved flushing of tables in a separate subroutine flush_tables()
• Code cleanup: moved log messages into the respective subroutines
• Code cleanup: use return values of subroutines instead of updating global variables
• Improved some log messages to explain what was DONE or FAILED
• Code cleanup: build up long command strings in a $command variable before passing it to system()
• Renamed subroutine create_snapshot() to create_lvm_snapshot()
• Merged patch from Matthew Boehm: Removed old asciidoc documentation in favor of POD style. This removes the dependency on the external program a2x for creating documentation and uses the 'built-in' pod2html and pod2man instead. Updated the Makefile to accommodate the change.
• Applied patch from Matthew Boehm to make the backup file name suffix configurable via a "--suffix" option. Updated the man page accordingly.
• Applied patch from Matt Lohier to support rsnap as a backup backend 
• Moved the list of contributors from the man page into a separate CREDITS file, added missing names

Enjoy!

This is more of an "behind the scenes" update and I hope that you won't see any (negative) changes on the PlanetMySQL front page or the RSS feeds: I just finished and commited the conversion of the backend script that performs the parsing and aggregation of feeds from requiring MagpieRSS to SimplePie.

This will provide better support for a wider range of feed types and should also fix a few quirks, e.g. that some postings (for example the one from Kevin Burton) only showed up as an "A" in the Planet's RSS feed. It hopefully also fixes a weirdness with time zones that some people were reporting, but this requires further investigation.

The size and complexity of the script was reduced significantly because of this change - SimplePie is a breeze to use in comparison to MagpieRSS and it's very well-documented. The developers provide some more reasons and a comparison on why you should also make this switch!

And in case you notice anything broken or weird on PlanetMySQL that might be related to the change, please let me know! This change was an important step for future improvements of the site.

Tomorrow (Thursday, 11th of September) at 9:00 PST/16:00 UTC/17:00 GMT/18:00 CET, there will be an new free MySQL University Session. MySQL University started as an internal training program for MySQL engineers, to share and spread knowledge about their areas of expertise and has been available to the public for quite some time now. It covers a wide range of technical topics around the MySQL Server and usually takes place once per week.

For the first time, the presentation will not be performed by (former) MySQL employees/developers, but by two of our "Sun Classic" colleagues: Jyri Virkki (OpenSolaris Web Stack community lead) and Murthy Chintalapati (Sr Engineering Manager, Web Stack development) will talk about the OpenSolaris Web Stack:

OpenSolaris Web Stack is an OpenSolaris project and community building an integrated stack of popular open source web tier infrastructure technologies such as Apache HTTP server, MySQL, memcached, PHP and Ruby On Rails optimized for Solaris platform. This session introduces OpenSolaris Web Stack, its status and future development including addition of newer technologies such as lighttpd, Varnish etc., as well as the ease of use features for developers and deployers. We will also be discussing an experimental web stack IPS package repository and it could be leveraged to build and make available popular end user applications such as Drupal.

MySQL University sessions are free to attend - all you need is an IRC client (to post your questions and comments) and an audio player capable of playing back an OGG audio stream, so you can listen to what is being said. See the Instructions for Attendees on the MySQL University pages for more information on how to log in and attend. The audio stream will be recorded and published on the MySQL University pages for later consumption, in case you can't make it or want to listen to a previous session.

One of the sessions at DrupalCon I attended was Larry Garfield's talk about "Drupal Databases: The Next Generation", which gave me a good insight into the current state of the Drupal database layer and how they plan to overhaul it for Drupal 7. The key points that I took away:

• A new API based on PDO
• Object-oriented, requiring PHP5
• Support for using prepared statements
• A unified access API
• A query builder
• More support for other database systems (currently Drupal supports MySQL and PostgreSQL only). In particular, they are keen on adding SQLite support, to ease local development.
• Support for master-slave replication (by randomly distributing reads among the hosts)
• Support for using different database types in parallel (e.g. using SQLite for read-only tables, MySQL for everything else)

The slides and a video of the presentation are available, if you want to check it out. There is a task list on the Drupal.org web site that keeps track of the ongoing activities.

Being able to use an Open Source DBMS to manage your data is nice, but what good would it be if you can't easily access it from your applications? One key factor to the popularity of MySQL is probably its wide range of available language bindings, which started with support for C, PHP and Perl from early on.

I've tried to gather a list of languages and their respective MySQL drivers/modules below. It's by no means complete or exhaustive, but I think I covered quite a lot of popular as well as exotic programming languages.

There is a number of connectors which are actually developed by the Sun Database Group (aka MySQL) itself and that are ready to use:

• Connector/ODBC - Standardized database driver Windows, Linux, Mac OS X, and Unix platforms.
• Connector/J - Standardized database driver for Java platforms and development.
• Connector/Net - Standardized database driver for .NET platforms and development.
• Connector/MXJ - MBean for embedding the MySQL server in Java applications.
• MySQL native driver for PHP - mysqlnd - The MySQL native driver for PHP is an additional, alternative way to connect from PHP 6 to the MySQL Server 4.1 or newer.
• libmysql - The original implementation of the MySQL Client/Server protocol (in C). This library is the basis for a large number of client libraries for other languages.

In addition to the above, there are several other connectors developed by Sun/MySQL, which are still under development:

• MySQL Connector/OpenOffice.org - a driver that allows you to connect to MySQL with OpenOffice.org natively, without having to use JDBC/ODBC.
• PHP PDO mysqlnd - a PHP/PDO driver based on the mysqlnd driver
• Connector C++: a driver for C++ applications that uses an API similar to the JDBC 3.0 API.

But it's not only us who develop language bindings for the MySQL server. There is an abundance of drivers that are developed and maintained by the Community, independently from Sun/MySQL (but sometimes with support or guidance from MySQL engineers). The list below is not sorted in any particular order other than the sequence in how I found them over time:

• DBD::mysql - a MySQL driver for the DBI database abstraction layer of Perl scripting language. There used to be a a native driver named "Msql-Mysql-modules", but it does not seem to be maintained anymore.
• Net::MySQL - an implementation of the MySQL Client/Server Protocol in Perl (does not require the MySQL client library). There also is MySQL::Packet, a Perl module that can encode and decode the binary protocol.
• MySQLdb - a MySQL driver for the Python scripting language.
• mysqli, ext/mysqli - MySQL Improved Extension for PHP
• mysql, ext/mysql - MySQL Extension for PHP
• PDO_MYSQL - MySQL PDO driver for PHP
• MySQL/Ruby - the MySQL API module for Ruby, based on the libmysql C library.
• Ruby/MySQL - the MySQL Ruby API, implemented in Ruby.
• MySQL++ - a C++ wrapper for MySQL's C API. It is built around the same principles as the Standard C++ Library, to make dealing with the database as easy as dealing with STL containers.
• MySQL wrapped - another C++ wrapper for the MySQL C API.
• MySQLtcl - a simple API for accessing a MySQL database server from the Tcl programming language.
• MySQL-Cocoa - An Objective-C Cocoa API for MySQL (based on libmysql).
• Eiffel MySQL - an interface to the MySQL database server using the Eiffel programming language.
• Bryan O'Sullivan's pure Haskell MySQL bindings
• hsql-mysql - A MySQL driver for Haskell
• MySQL Driver for the programming language D
• MyDB module for the Kaya programming language (included in the distribution)
• MySQL bindings for the Objective Caml language
• MySQL bindings for GNU Ada
• The SPL programming language ships with a MySQL module included in the distribution
• Database bindings (incl. MySQL) for GNU Octave
• LuaSQL is a simple database interface for the Lua programming language
• erlang-mysql-driver - a revamped MySQL driver for Erlang
• Myscsh - a Scheme implementation of the MySQL protocol

I probably forgot some other drivers/bindings - if you have any more to add, please let me know!

And if you'd like to create your own implementation for your favourite language: the protocol is documented here and here. Jan's additional notes may also be helpful to get you started.