Lenz Grimmer's blog

JFYI, I now placed a PDF of my MySQL Conference talk slides about "Performing MySQL backups using Linux LVM Snapshots" on my MySQL talks page. Enjoy!

 

While reading Colin's post about LugRadio Live, I stumbled over the Zumastor Linux Storage Project. Going through the project home page and their HOWTO got me curious - could this eventually become an alternative to using DRBD (for replicating data) and LVM snapshots (for performing backups)?

Zumastor is Free software that adds enterprise storage features (primarily improved snapshots and remote replication) to Linux.

Snapshots

LVM already lets administrators create snapshots, but its design has the surprising property that every block you change on the original volume consumes one block for each snapshot. The resulting speed and space penalty usually makes the use of more than one or two snapshots at a time impractical.

Zumastor keeps all snapshots for a particular volume in a common snapshot store, and shares blocks the way one would expect. Thus making a change to one block of a file in the original volume only uses one block in the snapshot store no matter how many snapshots you have.

Replication

Andrew Tridgell's rsync is a wonderful tool for replicating files remotely. However, when doing periodic replication of large numbers of infrequently changing files, the overhead for figuring out what files need to be sent can be extreme.

Zumastor keeps track of which block change between one snapshot and the next, and can easily send just the changed blocks. Thus Zumastor can do frequent replication of large filesystems much more efficiently than rsync can.

I assume it's not ready for production use yet, but it would sure be interesting to investigate on how to utilize it for the purpose of running MySQL on top of it...

I will keep an eye on this project, I wonder if I will have to add support for Zumastor snapshots to mylvmbackup at some point?

 

 

 

I am happy to announce the release of mylvmbackup version 0.8. mylvmbackup is a tool for quickly creating backups of a MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.

Below is the list of changes since version 0.6. You may wonder what happened to version 0.7 - it had a rather short life cycle as I was informed about a bug that I fixed quickly before I made a wider release announcement of 0.7.

• Fixed a bug in the InnoDB recovery function: the second mysqld process clobbered the socket file of the primary MySQL instance (thanks to Alain Hoang for reporting this)
• Updated the man page, noted some other limitations of the InnoDB recovery function
• Bug fix: use the correct mysqld parameter to provide an alternative PID file (--pid-file instead of --pidfile) - thanks to Guillaume Boddaert and Jim Wilson for reporting this!
• Added option "--skip_mycnf" to skip including a copy of the MySQL configuration file in the backup, added a safety check that the file actually exists prior to backing it up.

Updated package are available from the home page and via the openSUSE Build Service as usual. Updated packages for Debian/Ubuntu and Gentoo Linux should also be available shortly. Enjoy!

Speaking of LVM snapshot backups: I will be giving a talk about this subject at our MySQL Conference 2008 in Santa Clara, CA next week. If you are curious about how MySQL can be backed up using this technology, please consider to stop by!

 

I recently was interviewed by Packt Publishing for their Impackt '08 web pages:

Ever since the formal adoption of the term in 1998, Open Source has experienced growth and adoption rates that defy pressures and suggestions that it’s a viable option for enthusiasts and geeks only. Governments, corporations as well as small businesses have begun to choose Open Source over proprietary software. However, with the global economy facing an uncertain future, how will open source be impacted? Can it continue to grow despite this?

With these questions in mind and more, Packt approached some people at the heart of this movement to understand their take on the future of open source.

The interview has now been published. Enjoy!

 

 

As Kaj already pointed out, the Sun/MySQL Meetup Mashup Tour will also make a stop in Munich, Germany this Friday. I will be there as well, we will meet at 14:00 at the Hilton Munich Park Hotel. Kaj and some other Sun people will join us a bit later. If you are in the area, make sure to stop by!

By the way, the Meetup Mashup Tour will make at least one more stop in Germany - I am organizing an event in Hamburg, Germany which will take place on Monday, April 7th (19:00). This was initially planned as another regular MySQL Meetup, but I offered to expand the scope a bit. We will now meet at the Sun offices, Sun will sponsor some food and drinks! In addition to the usual MySQL Meetup crowd, we expect participation from various Sun communities (e.g. OpenOffice/StarOffice, Java or OpenSolaris). As usual, there will be a MySQL tech talk (this time held by Giuseppe).

I look forward to this event - it will be exciting to mingle with the people from these other communities and to exchange experiences and make new contacts. If you live somewhere around the Hambur area and would like to participate, please RSVP via Meetup.com or Xing.com soon!

 

Last week at CeBit I once again had a conversation with an application developer that was not aware that it's possible to embed the MySQL Server inside an application instead of running it as a separate process. I experienced something similar at FOSDEM recently, while talking with the Amarok developers - how come that this feature is so little known? In fact, there are two possibilities for embedding MySQL (and both of them are licensed under the GPL): the first one is using the libmysqld embedded library, which is suitable if you are developing C/C++ applications:

The embedded MySQL server library makes it possible to run a full-featured MySQL server inside a client application. The main benefits are increased speed and more simple management for embedded applications.

The embedded server library is based on the client/server version of MySQL, which is written in C/C++. Consequently, the embedded server also is written in C/C++.

The API is identical for the embedded MySQL version and the client/server version.

Especially consider the last sentence - it's actually fairly easy to create an application that supports connecting to both an embedded or a standalone server instance! Here are some related links to get you going:

• libmysqld, the Embedded MySQL Server Library
• MySQL C API functions
• Embedding the MySQL Server Library in a C Program by Russell Dyer
• Embedded MySQL Discussion Forum

If you're a Java developer, you should consider taking a look at our Connector/MXJ product - this allows you to embed a MySQL server in your Java application:

MySQL Connector/MXJ is a Java Utility package for deploying and managing a MySQL database. Deploying and using MySQL can be as easy as adding an additional parameter to the JDBC connection url, which will result in the database being started when the first connection is made. This makes it easy for Java developers to deploy applications which require a database by reducing installation barriers for their end-users.

MySQL Connector/MXJ makes the MySQL database appear to be a java-based component. It does this by determining what platform the system is running on, selecting the appropriate binary, and launching the executable. It will also optionally deploy an initial database, with any specified parameters.

Included are instructions for use with a JDBC driver and deploying as a JMX MBean to JBoss.

Some related links to further information/help:

• MySQL Connector/MXJ documentation
• MySQL Connector/MXJ Downloads
• JDBC and Java discussion forum
• MySQL and Java Mailing list

Does your application make use of an embedded MySQL Server already? Please let me know! I'd love to learn more about the use cases and experiences you have made.

 

 

 

Back when I still worked at SuSE, I was in charge of maintaining a number or packages of the distribution (actually, you should still be able to find traces of my work in the RPM changelogs). Nowadays, I maintain a number of packages for openSUSE and other distributions on the openSUSE Build Service, which is just brilliant for this purpose.

If you happen to live in northern Germany and are interested to learn more about the RPM package manager and how to build packages, consider coming to the TU Harburg this coming Thursday (March 13th). At 19:00, I will give a presentation about this topic in building, D, room D1023 (in cooperation with the Hamburg branch of the German Unix User Group). More information (in German) can be obtained from here.

See you there!

 

 

I have not been at CeBit for quite a while, but this year I will be there as a regular visitor this Thursday (6th of March). If you would like to meet with me, please send me an email or ping me via IM/Skype! I look forward to walking around the hallways, visiting my new employer's booth and finding out what other Open Source presences and activities there will be.

 

Since almost a year now, we host a weekly training session for our engineers on Thursday (14:00 UTC winter time), coined the "MySQL University". While it's primary purpose is to share and distribute knowledge about a wide variety of topics relevant to our own developers, many of the sessions are of general interest for developers on other projects as well.

Therefore we hold this sessions in the public and everybody is welcome to attend! You can listen to the presentation via an OGG Audio stream, questions can be posted via IRC on the #mysql-university channel on freenode.net. The audio file and IRC log will be saved, so you can also listen to past university sessions at a later point in time again.

Tomorrow's session will be about using the Valgrind memory checker, held by Stewart Smith, a colleague from Australia who is a member of the MySQL Cluster development team.

The next upcoming sessions include topics more closely related to MySQL Server development:

• February 28: New Optimizer Features in MySQL 6.0 (Sergey Petrunia)
• March 6: How to Add a Collation (Alexander Barkov)
• March 13: Checking Threading and Locking With Helgrind (Stewart Smith)
• March 20: Building MySQL Client Applications (Hartmut Holzgraefe)

See the MySQL University page on the MySQL Forge Wiki for more details and follow the instructions for attendees to get connected.

 

I recently received a review copy of the book "Xen Virtualization" by Prabhakar Chaganti (Packt Publishing) and finished reading it a few days ago.

The subtitle "A fast and practical guide" is a matching description - I managed to read the ~130 pages over the course of a week. The book is by no means an exhaustive reference manual, but it gives the reader a good overview about Xen and assists with performing the first steps and getting started.

 

Continue reading "Book review: "Xen Virtualization" by Prabhakar Chaganti (Packt Publishing)"

Like in the years before, I (and some other colleagues from MySQL) will be attending FOSDEM 2008 in Brussels, Belgium on February 23rd and 24th.

The schedule is packed as usual and lists several MySQL-related topics and sessions:

• Main tracks: Unicoding With PHP 6 by Andrei Zmievski and the talk about CakePHP by Garrett J Woodworth are certainly relevant for web developers using PHP.
• Lightning talks: Jan Kneschke will speak about MySQL Proxy (not yet listed in the schedule, but likely between 15-16 o'clock on Sunday). Other sessions with MySQL relevance/relations are Kettle: extracting, transforming and loading data by Matt Casters and Alfresco: fresh document management approach by Philippe Dubois.
• DevRooms: The Drupal people have set up a DevRoom with a session track - Kris Buytaert will talk about Drupal and MySQL High Availability. Another DevRoom worth visiting will be the one organized by the Belgian Ruby User Group - they set up a Ruby on Rails Developer Room. My friends from the openSUSE project also have a great number of sessions lined up for their openSUSE Developer Room: several talks about the Build Service (proudly powered by MySQL) and a session about the openSUSE redirector (a tool to set up and manage a content delivery network with MySQL) by Peter Poeml.

If you also plan to attend, please ping me if you would like to meet!

 

 

 

When it comes to handling my work-related email, many people will probably conclude that I am an old fart in Linux terms: I still use the console-based Pine as my MUA of choice, as my fingers have been hardcoded to its key combinations over the years (my first encounter with Pine was around 1994 on a shared DEC Ultrix box in my university). So far, I have not found any other application that allows me to process email as quickly as by using this tool.

I admit that I do use Thunderbird for my personal email, though, to gather experience with it (and to toy around with the various extensions, especially Nostalgy is a gift from heaven for people like me!). And of course because the mail volume there is less critical to cope with! But I am not going to start a holy war here - this article is supposed to explain how I have configured my mail setup to rotate the log files that keep track of all my incoming mails.

Continue reading "Linux tip: Automatically rotate/archive your fetchmail/procmail log files"

Since the very early days, the MySQL distribution packages contain a very useful commandline-tool named replace. As the name implies, it can be used to replace strings in text files. From the documentation:

The replace utility program changes strings in place in files or on the standard input. Invoke replace in one of the following ways:

shell> replace from to [from to] ... -- file [file] ...
shell> replace from to [from to] ... < file

from represents a string to look for and to represents its replacement. There can be one or more pairs of strings. Use the -- option to indicate where the string-replacement list ends and the filenames begin. In this case, any file named on the command line is modified in place, so you may want to make a copy of the original before converting it. replace prints a message indicating which of the input files it actually modifies. If the -- option is not given, replace reads the standard input and writes to the standard output. replace uses a finite state machine to match longer strings first. It can be used to swap strings. For example, the following command swaps a and b in the given files, file1 and file2:

shell> replace a b b a -- file1 file2 ...

This comes in very handy, especially if you need to replace a similar text in a large number of files, or if you need to replace multiple different strings in a file at the same time! So next time you need to replace strings in text files, take a look at this hidden gem!

Unfortunately the string matching does not support regular expressions - but the source code actually documents some additional options:

Special characters in from string:
  ^  Match start of line.
  \$  Match end of line.
  \b  Match space-character, start of line or end of line.
        For end \b the next replace starts locking at the end space-character.
        An \b alone or in a string matches only a space-character.
  \r, \t, \v as in C.

The programs make a DFA-state-machine of the strings and the speed isn't
dependent on the count of replace-strings (only of the number of replaces).
A line is assumed ending with \n or \0.
There are no limit exept memory on length of strings.

 

 

 

 

Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.

In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!

• Added a new rsync backup type. This is very useful if you want to use mylvmbackup to create the initial state for your slave servers. Instead of creating a .tar.gz archive, the data directory is copied into a timestamped archive directory. (Robin)
• Added support for a trailing argument to tar, which can be used for excluding files. (Robin)
• Separated out the suffix of the tarball (Preperation for rsync and users that want to use bzip2 or no compression on the tarball.) (Robin)
• While the backup is performed, a temporary suffix at the end of the tar backup file name (or the rsync target directory name) now indicates that it is incomplete. (Robin)
• The my.cnf configuration file is now included in the backup. (Robin)
• Added the ability to run an extra FLUSH TABLES on busy databases where lvcreate might take a long time (and may overrun the interactivity timeout on the connection, losing the lock). (Robin)
• Added option "--pidfile" to provide an alternative PID file location for the second server instance that is started to perform the InnoDB recovery on the snapshot prior to backing it up (Otherwise it may default to using the same pid file location that the running server uses and safe_mysqld will abort) - thanks to Kristian Köhntopp for making me aware of this problem.
• Before discarding the snapshot LV, the output of "lvs <snapshot>" is now printed out for diagnostics. It contains useful information like "how much percent of the backing store was used", which helps tuning the size of the snapshot LV.
• Added option "--skip-flush-tables" that performs the snapshot without flushing the tables to disk beforehand (which is not supported by InnoDB tables anyway) - this would save time, as the flushing can take a while, depending on the buffer sizes. (Thanks to Peter Zaitsev for the suggestion)

Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.

Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.

 

Better late then never: I've now uploaded a few pics that I've taken during this year's FrOSCon to my gallery. Have fun!