Lenz Grimmer's blog

JFYI, I now placed a PDF of my MySQL Conference talk slides about "Performing MySQL backups using Linux LVM Snapshots" on my MySQL talks page. Enjoy!

 

While reading Colin's post about LugRadio Live, I stumbled over the Zumastor Linux Storage Project. Going through the project home page and their HOWTO got me curious - could this eventually become an alternative to using DRBD (for replicating data) and LVM snapshots (for performing backups)?

Zumastor is Free software that adds enterprise storage features (primarily improved snapshots and remote replication) to Linux.

Snapshots

LVM already lets administrators create snapshots, but its design has the surprising property that every block you change on the original volume consumes one block for each snapshot. The resulting speed and space penalty usually makes the use of more than one or two snapshots at a time impractical.

Zumastor keeps all snapshots for a particular volume in a common snapshot store, and shares blocks the way one would expect. Thus making a change to one block of a file in the original volume only uses one block in the snapshot store no matter how many snapshots you have.

Replication

Andrew Tridgell's rsync is a wonderful tool for replicating files remotely. However, when doing periodic replication of large numbers of infrequently changing files, the overhead for figuring out what files need to be sent can be extreme.

Zumastor keeps track of which block change between one snapshot and the next, and can easily send just the changed blocks. Thus Zumastor can do frequent replication of large filesystems much more efficiently than rsync can.

I assume it's not ready for production use yet, but it would sure be interesting to investigate on how to utilize it for the purpose of running MySQL on top of it...

I will keep an eye on this project, I wonder if I will have to add support for Zumastor snapshots to mylvmbackup at some point?

 

 

 

I am happy to announce the release of mylvmbackup version 0.8. mylvmbackup is a tool for quickly creating backups of a MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.

Below is the list of changes since version 0.6. You may wonder what happened to version 0.7 - it had a rather short life cycle as I was informed about a bug that I fixed quickly before I made a wider release announcement of 0.7.

• Fixed a bug in the InnoDB recovery function: the second mysqld process clobbered the socket file of the primary MySQL instance (thanks to Alain Hoang for reporting this)
• Updated the man page, noted some other limitations of the InnoDB recovery function
• Bug fix: use the correct mysqld parameter to provide an alternative PID file (--pid-file instead of --pidfile) - thanks to Guillaume Boddaert and Jim Wilson for reporting this!
• Added option "--skip_mycnf" to skip including a copy of the MySQL configuration file in the backup, added a safety check that the file actually exists prior to backing it up.

Updated package are available from the home page and via the openSUSE Build Service as usual. Updated packages for Debian/Ubuntu and Gentoo Linux should also be available shortly. Enjoy!

Speaking of LVM snapshot backups: I will be giving a talk about this subject at our MySQL Conference 2008 in Santa Clara, CA next week. If you are curious about how MySQL can be backed up using this technology, please consider to stop by!

 

While browsing the many blog entries on blogs.sun.com about the MySQL Acquisition (thanks a lot for the very warm welcome!), I stumbled over this (Python-based) utility: SnapBack, a tool that uses ZFS snapshots to perform physical backups of MySQL databases on Solaris. Very cool! This is actually something I was wanting to add to the mylvmbackup script, too - I have to take a closer look at how this is done (I tried to install OpenSolaris on a VirtualBox instance, but it caused it to crash the emulator).
 

 

I am happy to announce that I have been selected to speak at the MySQL Conference & Expo 2008 on April 14-17, 2008 in Santa Clara, CA. The topic of my talk will be "Performing MySQL Backups using LVM Snapshots" - and I will of course include a plug for the mylvmbackup tool

From the abstract:

Modern Linux distributions provide a very advanced technology to maintain and administer storage devices, called “Logical Volume Manager” (LVM). Among providing several benefits over using plain hard disk partitions, one of the unique features of LVM is the capability to create snapshots of file systems on the fly. This functionality comes in handy for performing consistent physical backups of MySQL tables without having to take down the MySQL server for the entire backup operation.

In this session, Lenz will give an introduction to the underlying technology, the actual steps that have to be performed and which caveats there may be. Additionally, he will introduce “mylvmbackup”, a script created to automate and ease the described procedure.

My talk has been scheduled as part of the "Security and Database Administration" track for 17:15 on Wednesday, April 16th in Ballroom B - the last slot before the infamous Quiz Show!

 

 

Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.

In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!

• Added a new rsync backup type. This is very useful if you want to use mylvmbackup to create the initial state for your slave servers. Instead of creating a .tar.gz archive, the data directory is copied into a timestamped archive directory. (Robin)
• Added support for a trailing argument to tar, which can be used for excluding files. (Robin)
• Separated out the suffix of the tarball (Preperation for rsync and users that want to use bzip2 or no compression on the tarball.) (Robin)
• While the backup is performed, a temporary suffix at the end of the tar backup file name (or the rsync target directory name) now indicates that it is incomplete. (Robin)
• The my.cnf configuration file is now included in the backup. (Robin)
• Added the ability to run an extra FLUSH TABLES on busy databases where lvcreate might take a long time (and may overrun the interactivity timeout on the connection, losing the lock). (Robin)
• Added option "--pidfile" to provide an alternative PID file location for the second server instance that is started to perform the InnoDB recovery on the snapshot prior to backing it up (Otherwise it may default to using the same pid file location that the running server uses and safe_mysqld will abort) - thanks to Kristian Köhntopp for making me aware of this problem.
• Before discarding the snapshot LV, the output of "lvs <snapshot>" is now printed out for diagnostics. It contains useful information like "how much percent of the backing store was used", which helps tuning the size of the snapshot LV.
• Added option "--skip-flush-tables" that performs the snapshot without flushing the tables to disk beforehand (which is not supported by InnoDB tables anyway) - this would save time, as the flushing can take a while, depending on the buffer sizes. (Thanks to Peter Zaitsev for the suggestion)

Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.

Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.

 

Eric Bergen from Proven Scaling (which I had the pleasure to meet in person during the MySQL Conference & Expo in Santa Clara last month) was kind enough to send me a patch for the mylmbackup tool, which justifies a new release:

Attached is a patch file for mylvmbackup that adds the ability to use
lvm version 2 and perform innodb recovery on the snapshot prior to
creating a tar ball. The option is named --innodb-recover.

I've also fixed a bug with default value handling for command line
options. In version 0.4 if a config file was specified default values
in the script were all changed to blank. This means that the config
file had to supply values for every variable instead of just the
values that need to be changed from default.

This is a very useful addition, as it significantly reduces the time required to recover a MySQL server from a snapshot backup. Thank you, Eric! Note that you need to use LVM2, as LVM1 does not support the required writing to snapshot volumes.

Version 0.5 is now available for download from the mylvmbackup project page.

I also would like to announce that I have set up a dedicated mailing list for this tool: if you want to discuss the usage/future of mylvmbackup, propose patches or ask for help, there now is a mailinglist, hosted on FreeLists.org. To subscribe, either enter your email address on the mylvmbackup list information page or send an email with the subject "subscribe" to mylvmbackup-request@freelists.org. The list is archived here. See you there!

I am happy to announce version 0.4 of mylvmbackup, a tool to perform consistent backups of a MySQL server's tables using Linux LVM snapshots.

For this release I'd like to especially thank Robin H. Johnson from the Gentoo project, who contributed another batch of useful changes and informed me that mylvmbackup is now in productive use to perform backups of the MySQL databases that power the project's Bugzilla bug tracking system. I am always glad to read about such use cases - how do you utilize mylvmbackup in your environment?

• The option handling has been improved. mylvmbackup now starts by using the builtin defaults, followed by the default configuration file (/etc/mylvmbackup.conf, followed by an alternative configuration file (specified via CLI arguments), followed by the remainer of the CLI arguments.
• Changed the capture of the position file and tarball creation to not use absolute path names, and instead use them relative to $mountdir. This means that the content of the backup tar archives has changed with regards to the directory structure, please keep this in mind when restoring from backups!
• Now utilizes a bindmount to get the position directory near the $mountdir for the tarball creation.
• Directory names are sanitized for excessive whitespace and trailing slashes
• Code cleanup: use my $variable instead of use vars

In addition to Robin's improvements, the following changes were made by me:

• Fixed DSN string handling (thanks to Peter Zaitsev for spotting this)
• Fixed logging to the console (log levels were not printed)
• Removed a few currently unused configuration options from the config file

Recently I stumbled over a posting on the German MySQL Forum from a user that accidentally removed all table files from a MySQL Server's data directory with a misbehaving shell script. He was surprised to find out that the server happily continued to serve requests and his web site was still fully operational, even though /var/lib/mysql/<database> was completely emtpy! The reason for this in a nutshell: the rm command only removed the reference to the table files from the database directory, the files itself were not removed from the file system yet as the mysqld process still had the files opened. So as long as a process keeps a file open, the kernel will not release the disk space occupied by the file and it will remain intact, albeit no longer visible.

Of course, the user was now desperate to recover the deleted tables files and was asking for help. Fortunately the recovery in this case is pretty simple. You should first shut down your application to avoid further activitiy on the affected database. Important: you must not shut down the MySQL Server, as this would close the last open reference to the table files! Now you can simply use mysqldump --opt <database> > database.sql to perform an SQL dump of the deleted tables. As the MySQL server still can access the open table files, the dump will contain the entire content and can then be used to restore the database again. Now you should restart the MySQL server so it closes the still open file descriptors of the deleted tables files. Alternatively, you could use DROP TABLE <table> or DROP DATABASE <database> to properly remove the references, in case you don't want to shut down the entire server. Now you can restore your missing tables from the SQL dump as usual and can restart your application!

Note that this trick only works on table files that were removed on the file system level, not after you used DROP TABLE/DATABASE, so it's not a magic undo function for these commands - only restoring from a recent backup (e.g. performed with mylvmbackup, hint, hint) will help in this case. In addition to that the MySQL server must have had opened the tables before. A freshly started MySQL server has not opened any table files apart from the ones in the mysql system database.

By the way, there is a related article "Bring back deleted files with lsof" on Linux.com that covers the subject of recovering deleted (but still open) files on a more general level and also provides some more background information about the Linux internals. Worth a read!

I am happy to announce version 0.3 of mylvmbackup, a tool that performs consistent backups of a MySQL server's tables using Linux LVM snapshots.

Special thanks go to Fred Blaise, who contributed the majority of the new features that have been added to this new release:

• It is now possible to use an external configuration file /etc/mylvmbackup.conf to store the options. This is probably more convenient than having to pass a slew of options on the command line or having to hack the script itself to change the default values. This new feature requires the Config::IniFiles Perl module to be installed, a sample configuration file is included in the package.
• The logging to the console has been visually enhanced by including a time stamp and the message category (e.g. Info, Warning or Error). In addition to that, it is now possible to log messages to a local or remote syslog server. This feature requires the Sys::Syslog Perl module.
• The man page has now been converted into an asciidoc file, which makes it easier to generate other document formats as well, e.g. a HTML version.
• Several small bugs have been fixed, too: see the ChangeLog for details.

If you are looking for a convenient backup tool to create fast and consistent MySQL backups, please give mylvmbackup a try! You feedback is appreciated. A tarball and RPM are now available for download from the project's home page. Thanks!

I am happy to announce that version 0.2 of the mylvmbackup tool is now available!

mylvmbackup is a Perl script for quickly performing backups of a MySQL server's databases using the Linux Logical Volume Manager (LVM). It creates a consistent LVM snapshot of the server's data directory which is then backed up without further blocking the server's operation.

After version 0.1 was published in May this year, I did not really get much feedback about it. I had some ideas for improvements (see the TODO file included in the package), but never got around to actually start working on them.

Thanks to Robin H. Johnson from the Gentoo project for contributing a number of new options and features as well as some code cleanups. His changes motivated me to make a few more modifications and improvements by myself, which have now been rolled into a new release.

The new options provide some more flexibility in the way the script handles the logical volumes and how the backup files are being created. I also overhauled the building and packaging and added a Makefile to automate these procedures. For details, please refer to the ChangeLog and check the manual page and the README for additional info.

A tarball and RPM of version 0.2 can now be downloaded from the project page.

The SVN repository can now be browsed using WebSVN as well.

Please give it a try! Your feedback is very welcome.

Long time no post! Last Sunday I spotted the following email in my junk mailfolder:

Date: Sun, 21 May 2006 19:08:39 +0200
From: root <root@metis.lenznet>
To:  <lenz@localhost.metis.lenznet>
Subject: SMART error (OfflineUncorrectableSector) detected on host: metis

This email was generated by the smartd daemon running on:

   host name: metis
  DNS domain: lenznet
  NIS domain:

The following warning/error was logged by the smartd daemon:

Device: /dev/hda, 1 Offline uncorrectable sectors

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.

Which did not sound too good. In addition to that, the system had failed to resume from a suspend to disk earlier that day - the kernel experienced disk read errors while trying to load the suspended image from the swap partition. Fortunately a fresh reboot still worked and I ran a more thorough analysis of the disk drive by using smartctl -t long /dev/hda. Using various Open Source tools from a SUSE Linux 10.1 rescue system (which boots off the first installation CD) helped me to backup and restore my data without losing anything (except for some time, of course).

With all the recent buzz on Planet MySQL about using LVM to perform "semi-hot" backups of MySQL on Linux, I remembered that we once created such a Perl script for a server project that we worked on in cooperation with a hardware vendor. As this particular backup method was also mentioned and recommended in my talk about MySQL backup and security on Linux and the script hasn't actually been publicly available so far, I offered to take over the ownership and release it under the GPL. The first tarball release (0.1) is now available from this web page (still work in progress), the files are available via SVN from here as well. Make sure to see the README and TODO files for the current status. A man page is included, too. I hope you will find it useful - feedback is very welcome!