Lenz Grimmer's blog

It's my great pleasure to announce the release of mylvmbackup version 0.14.

This release includes a large number of improvements, code cleanups, and new functionality.

I would like to thank Ask Bjørn Hansen, Ben Bonnel, Norbert Tretkowski, Neil Wilson, Klaus Ethgen and Alexandre Anriot for their feedback and contributions to this release.

The release is available as a source tarball and generic RPM package. Packages for other distributions are available from the openSUSE Build Service.

Some notable highlights from the ChangeLog (please check the bzr commit log for a more detailed history of changes):

• Improved error handling: send out an email if mail_report_on has been set to "errors" and a log message with log level LOG_ERR was logged.
• Abort and clean up if the snapshot volume could not be mounted
• Abort and clean up if the backup creation failed
• Exit with a non-zero return code if errorstate was set
• Enabled timestr() formatting sequences for snapshot volume names, to create snapshot names which contain a dynamic date value. This can be useful when thin provisioned snapshots are used as the actual backup by enabling the keep_snapshot option
• Added "thin" option to mylvmbackup.conf (default=0), to enable thin provisioned LVM snapshots
• Added new configuration option "recoveryopts" that allows to modify the command line options that are passed to the MySQL instance that performs the InnoDB recovery (BUG#544295 and BUG#1091950). The current default options are "--skip-networking --skip-grant --bootstrap --skip-syslog --skip-slave-start" (which better accommodates MySQL 5.1)
• Merged patches from Alexandre Anriot to provide a mail reporting method which sends the log output via email and a new "purge" action that allows to only keep a defined number of backups in the backup directory
• Applied patch contributed by Klaus Ethgen that removes the annoying LVM warnings about a leaked file descriptor (DBD:mysql does not add FD_CLOEXEC on the MySQL socket)
• BUG#1086313: mvlvmbackup lacks support for 'thin' LVM snapshots (Thanks to Neil Wilson for the patch)
• BUG#787063: Ensure to clean up temporary .pos files when the snapshot mount is kept, too
• Applied small patch from Norbert Tretkowski: change umask to 077 before creating the tarball backup files to prevent them from being readable by everyone
• BUG#593220: Fixed the compressarg option for cat (removed the comment)
• Applied patch contributed by Ben Bonnel to add a new configuration option "rsnaprsyncarg" that allows one to pass special options to the rsync process when using the rsnap backend.
• Applied patch provided by Ask Bjørn Hansen to allow targets not to have a timestamp (e.g. for rsync targets)

Congratulations to the Drupal community for getting version 7.0 released! This is a major mile stone and an excellent reason to celebrate!

If you want to give Drupal 7 a try without having to install anything, I've now updated my Drupal 7 appliances on SuSE Studio to the latest release. The appliance is based on openSUSE Linux 11.3 and is available in two variants:

• A text-mode only appliance to which you connect using your local web browser via the network.
• A GUI version that starts up the Firefox browser in a minimalistic GNOME desktop to perform the installation locally. Remote network access is available, too.

The database backend is MySQL 5.1, with the InnoDB plugin and strict mode enabled. phpMyAdmin has been added to support web-based administration of the MySQL server. You can access it via http://localhost/phpMyAdmin/. I also added drush, the Drupal command line shell and scripting interface and some additional packages (yast2-http-server, bind-utils, php5-ctype, patch). I also bumped up the appliance's version number to 7.0.0, to match the version number of Drupal included.

The appliance is available in various formats:

• A live raw disk image, ready to be written to an USB stick or flash drive
• A live ISO image, to be burned on a CD or used in a virtual machine
• A hard disk image, to be dumped on a hard disk drive
• Various virtual disk formats, e.g. OVF, VMWare/VirtualBox/KVM and Xen

Please see the installation instructions provided on the download pages for details on how to use the various image formats.

So congratulations to the Drupal developer community for reaching this goal and thanks to SuSE/Novell/Attachmate for providing the infrastructure for creating such appliances. I also would like to especially thank Richard Bos for the testing and many suggestions for improvement of these appliances!

There has been a lot of buzz about the MySQL 5.5 GA release and its new features and other user-visible improvements. In this blog post, I'd like to touch on a less noticeable, but still important change.

CMake has already been used to build the MySQL Server on Windows for a long time, while the GNU autotools were used on all other platforms. Since MySQL 5.5, all builds on all platforms are now performed using the same tool chain. With the latest release of MySQL 5.5, we've made an important step to clean up and simplify the MySQL build system: the support for autoconf/automake has now been removed completely. We've been performing the release builds of MySQL 5.5 using CMake exclusively for quite some time already. It became obvious that maintaining two separate build systems simply had become too much of a burden for our engineers, especially since the autotools-based builds were no longer exhaustively tested. This change was outlined in WorkLog#5665 - Removal of the autotools-based build system. We've made this step in close cooperation with our community of packagers (e.g. the maintainers of MySQL packages on the various Linux distributions).

By moving to CMake, we are giving our developers one common build mechanism for all platforms and there is a lot of new useful functionality such as out-of-source builds or a GUI for configuring the build options. And they can now build MySQL in the very same way that we do it for our own binaries! I've covered the advantages in my previous blog post about Building MySQL Server with CMake on Linux/Unix already. We've also created a general article about CMake and MySQL as well as an Autotools to CMake Transition Guide on the MySQL Forge Wiki. The description of the source build process in the reference manual has also been updated to reflect this change.

A big “thank you” should to go to Vladislav Vaintroub and Davi Arnaut for implementing and pushing the transition to CMake forward, and to Paul DuBois for creating and improving the documentation! Wearing my former build and release engineer hat, I am very happy about this change.

Other CMake-related articles that are worth reading:

• Giuseppe Maxia: Building MySQL 5.5 with CMake
• Jonathan Perkin: How To Build MySQL Releases
• Santo Leto: Step by Step Guide on How to Compile MySQL 5.5 on Microsoft Windows

If you wonder why there hasn't been an update from me for quite a while — I just returned from two months of paternal leave, in which I actually managed to stay away from the PC most of the time. In the meanwhile, I've officially become an Oracle employee and there is a lot of administrative things to take care of... But it feels good to be back!

During my absence, Giuseppe and Felix kicked off the Call for Papers for this year's European OpenSQL Camp, which will again take place in parallel to FrOSCon in St. Augustin (Germany) on August 21st/22nd. We've received a number of great submissions, now we would like to ask our community about your favourites!

Basically it's "one vote per person per session" and you can cast your votes in two ways, either by twittering @opensqlcamp or via the opensqlcamp mailing list. The procedure is outlined in more detail on this wiki page.

As we need to finalize the schedule and inform the speakers, the voting period will close this coming Sunday, 18th of July. So don't hesitate, cast your votes now! Based on your feedback we will compile the session schedule for this year's camp. Thanks for your help!

Shortly after I posted my last summary of MySQL releases, our son Mats was born and I went on a 2.5-week vacation. Our developers did not rest in the meanwhile and I'd like to give you a quick update of what's new since then:

MySQL Connector/Net 6.3.0

• Visual Studio 2010 RC support
• Nested transaction scope support

MySQL Workbench 5.2.16 Beta 6

• Fixed 67 bugs
• Saving your profile/connection passwords in OSX keychain, gnome-keyring or in an encrypted password-vault-file.
• New rapid development features for generating complete SQL Select/DML statements or names for selected objects in Query Editor to either the query area or clipboard.
• The ability to set a preference for the placement  (left or right side) of the sidebar in the Query Editor (Currently on Windows only, coming to Mac and Linux soon).
• Further optimization and stabilization of the administrator components

MySQL Server 5.1.44

• This release now includes InnoDB Plugin version 1.0.6, wich is considered to be of Release Candidate (RC) quality.
• Lots of bug fixes - see the ChangeLog for details.

MySQL Server 5.5.2 Milestone 2

• Also includes the updated InnoDB Plugin 1.0.6 and several fixes - see the ChangeLog for details. For an overview of new features in the 5.5 code base, check out the "What's new in 5.5" page in the reference manual.

Please note that the MySQL downloads section has been split into two parts. As usual, you will find downloads of both GA and development versions of all MySQL products and releases on the MySQL DevZone. In addition to that, we've now added a pointer to the downloads of officially released (GA) versions to the main web site on http://www.mysql.com/downloads/.

I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).

From the ChangeLog: 

• Deleted sample precleanup.pm hook as it has potential to cause harm and is too specialized on a particular use case (BUG#394668)
• Added support for rsync via SSH (BUG#392462)
• Fixed InnoDB recovery in case a relative path to the MySQL data directory is defined (BUG#38337), improved the documentation of relpath in the man page.

 

Some time ago I posted a compilation of applications and programming languages that provide an API to connect to the MySQL Server. As it turned out, I forgot a few that I would like to mention here:

• Apache DBD API: a MySQL driver for mod_apr_dbd is not included in the official distribution, but can be obtained seperately from here. Some distributions (e.g. openSUSE) actually provide installable packages of this driver module.
• GRASS MySQL driver: The MySQL database driver in GRASS enables GRASS applications to store vector attributes in a MySQL server. Alternatively, data can be stored in an embedded version of the MySQL Server.
• HDBC: HDBC provides an abstraction layer between Haskell programs and SQL relational databases. This lets you write database code once, in Haskell, and have it work with any number of backend SQL databases.
• libdrizzle: This is the client and protocol library for Drizzle and MySQL that provides both client and server interfaces. A number of APIs (e.g. Java, Lua, Perl and Ruby as well as Python) have been built on top of it, but I'm not quite sure if they all support the MySQL client/server protocol as well.
• MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. This means you don't have to compile anything - the MySQL client library doesn't even have to be installed on your system.

Do you know of any other MySQL Connectors/APIs? Let me know!

We've now published a new set of binary packages including the extended GIS functionality from the mysql-5.1-wl1326 source tree.

This release is based on the MySQL 5.1.35 code base and fixes the bugs mentioned below. It includes some improvements to the GIS functionality as well, so please use these packages for future testing of the MySQL GIS functionality.

The following GIS-related bugs were fixed in this version:

• Bug#31753: Buffer/area functions only return first row of set
• Bug#32032: Contains() does not work on MultiPolygons, may force a disconnect and/or result in extremely long query times.
• Bug#32100: contains, intersects functions never return. Query disconnects or times out.
• Bug#33035: Intersection function returns 'Function doesn't exist' error
• Bug#40874: Buffer function doesn't work with LINESTRING
• Bug#41481: Buffer function never returns, cpu 100%, system locks up
• Bug#43493: union function returns polygon not multipolygon for non-intersecting polygons
• Bug#44753: nan error in union function

I'd like to explicitly thank John Powell for his extensive testing and bug reporting so far, he helped us quite a lot to improve the quality of the GIS code.

Please help us by by testing this new release and reporting bugs at our public bug database. For convenience, please tag your Bug reports with "gis" and make sure to put them in the "Server: GIS" category!

I admit it — I'm a fan of simulation software, particularly flight simulators. Probably the best Open Source Flight Simulator out there is FlightGear — it provides an impressive level of reality and you can download and install many additional plane models and terrains. There are packages of FlightGear 1.0.0 in the games repository of the openSUSE Build Service, which works quite well and I have been enjoying it a lot. However, the FlightGear project released version 1.9.x quite a while ago (1.9.1 was published in January 2009) and I was itching on giving the new version a try (just take a look at the screenshots and you know what I mean). However, building FlighGear on Linux is quite a complex task with many dependencies, and so held off from doing it myself, waiting for someone else to perform the update...

Well, this weekend I finally bit the bullet and did it myself - FlightGear 1.9.1 has now been added to my home:LenzGr build repository. I based my packages on the ones included in the games repository, but I plan on cleaning them up a bit and splitting them into separate packages (currently the FlightGear source RPM contains SimGear and fgrun as well). I also "borrowed" the OpenSceneGraph sources and spec file from the PackMan repository, in order to have a functional build. Unfortunately FlightGear currently only builds on a very limited list of distributions so far (namely OpenSUSE 11.0, just what I needed) — I haven't had time to adapt the spec files for FlightGear and OpenSceneGraph to match the appropriate build dependencies for the other distributions yet and "02-check-gcc-output" gives me some grief on platforms where it actually builds but generates compiler warnings (but patches are welcome!)...

Shortly after I created the initial packages of embedded InnoDB on the OpenSUSE Build Service, Oracle/Innobase released an updated version (1.0.3.5325). In addition to many improvements and bug fixes, they slightly changed the versioning scheme to better indicate what version of the InnodDB plugin their code is based on (see Vasil's posting on the InnoDB Forums for more information).

I've now updated my InnoDB packages on the Build Service to this version as well - please note that the naming scheme of the shared library package has been changed from "embedded_innodb1" to "libinnodb2" — RPM will take care of replacing the old package during update, even though the name has changed.

After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.

Some notable highlights from the ChangeLog:

• Removed the absolute path names to external tools (make sure $PATH is correct)
• Added --log-err to the startup options of the recovery instance to avoid cluttering the server's error log
• Added support for hooks written as Perl Modules. (Matthew Boehm)
• Added support for date/time-formatted path names for backupdir and mountdir (Matthew Boehm)
• Backupdir and mountdir are now created automatically (Matthew Boehm)
• Added new hook "logerr" when an error is logged. (Matthew Boehm)
• Added Option --keep-mount... (Tim Stoop)
• Removed the bind mount, now requires LVMv2
• Support reading login/password from ~/.my.cnf (Baron Schwartz)
• Documentation fixes and improvements (Ville Skyttä) (Bug #302144)

XtraBackup is an Open Source online (non-blockable) backup solution for the InnoDB and XtraDB storage engines. It works with both MySQL 5.0 and 5.1 (and possibly 5.4 as well) and is distributed under the GPLv2.

Some weeks ago Vadim announced the availability of xtrabackup-0.7, stating that they consider it stable enough now to label this version a "Release Candidate". I've been maintaining RPM packages of xtrabackup on the fine openSUSE Build Service for quite some time now, RPMs of 0.7 for a number of distributions are now available for download. Please report any bug reports via the bug tracker on Launchpad.

As you may have heard, we're switching to a new release model with the upcoming MySQL 5.4 release.

If you are curious to learn more about what will change in the way in which future versions MySQL will be developed and released, make sure to attend our next MySQL University session about The New MySQL Release Model on Thursday, 11th of June, 14:00 UTC. Tomas Ulin, our director of MySQL server development will go through the planned changes and would also like to get your input and feedback on these changes.

We're using DimDim for broadcasting this session, which allows you to listen to the audio while watching the slides with your web browser. You can comment and discuss via a chat function, too! We're looking forward to your input. To attend, point your browser to this address (Adobe flash player required).

The session will be recorded and posted on the MySQL Forge Wiki, so you can watch the presentation later as well. You can also provide your feedback on the release model by posting on the MySQL Internals mailinglist.

Version 1.0.3-alpha of the MySQL Connector/C++ has just been announced by Lawrin Novitsky and is now available for download. This driver is licensed under the GPL and is a new implementation of the MySQL Client/Server protocol. Instead of wrapping the C API calls in C++ methods, the implementation mimics the JDBC API, which hopefully feels much more "native" to a seasoned C++ developer.

The driver has been ported to a wide range of platforms and is about to hit the beta test phase. So if you're writing a C++ application that needs to connect to a MySQL Server, give it a try! The developers are always looking for feedback.

More information and examples on how to use it can be found on this detailed Wiki page on the MySQL Forge.

Now that MySQL 5.1 has finally been labeled and released as "GA", we have noticed quite a significant increase in the download numbers for 5.1. I find this quite promising, despite the comments from some people that suggested to stay away from it for now. I performed updates of 5.0 to 5.1 several times already, and did not observe any serious problem so far. But then again, I am by no means a power user, so your mileage may vary...

I assume that many of the users are already using some older versions of MySQL and will have to perform an upgrade from the previous version to the new one. As many things have been changed and improved between 5.0 and 5.1, an upgrade needs to be planned carefully. The MySQL reference manual has an entire chapter devoted to this subject. The best general advice is probably to create a backup of your data first before attempting to upgrade!

 

I performed an upgrade test on an openSUSE 11.0 system that was running on the MySQL version shipped with the distribution (5.0.51a). I first loaded it with some databases that I copied from other live systems (via SQL dumps). Then I grabbed the Server, Client and shared library RPMs the "Linux x86 generic RPM (dynamically linked)" section of the 5.1 downloads page and installed them with rpm -Uhv MySQL-server-5.1.30-0.glibc23.i386.rpm MySQL-client-5.1.30-0.glibc23.i386.rpm MySQL-shared-5.1.30-0.glibc23.i386.rpm. The packages were installed without a hitch and properly replaced the existing 5.0 packages (even though they use different package names). I then restarted the server and ran mysql_upgrade afterwards.

 

Success! I was pretty impressed how smooth this upgrade was performed. So far, I did not notice any regression or incompatibility. But I would be interested in hearing stories and comments from other users about their upgrade experiences!

So here are some questions I would like you to answer — I would appreciate your feedback either via this blog or by email to firstname at sun dot com:

• What are your expectations and most important requirements for performing MySQL upgrades?
• Have you upgraded to MySQL 5.1 yet?
• If not, why not?
• If yes, what was your upgrade experience like? Any serious issues or problems? What platform and package format did you use?
• Did you use the documentation in our reference manual to perform the update? Was it helpful? What could be improved?
• Do you have any other comments/suggestions on the topic of upgrading MySQL?

I am looking forward to your comments. Thanks and Happy Holidays!