Lenz Grimmer's blog

I am happy to announce that I have been selected to speak at the MySQL Conference & Expo 2008 on April 14-17, 2008 in Santa Clara, CA. The topic of my talk will be "Performing MySQL Backups using LVM Snapshots" - and I will of course include a plug for the mylvmbackup tool

From the abstract:

Modern Linux distributions provide a very advanced technology to maintain and administer storage devices, called “Logical Volume Manager” (LVM). Among providing several benefits over using plain hard disk partitions, one of the unique features of LVM is the capability to create snapshots of file systems on the fly. This functionality comes in handy for performing consistent physical backups of MySQL tables without having to take down the MySQL server for the entire backup operation.

In this session, Lenz will give an introduction to the underlying technology, the actual steps that have to be performed and which caveats there may be. Additionally, he will introduce “mylvmbackup”, a script created to automate and ease the described procedure.

My talk has been scheduled as part of the "Security and Database Administration" track for 17:15 on Wednesday, April 16th in Ballroom B - the last slot before the infamous Quiz Show!

 

 

Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.

In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!

• Added a new rsync backup type. This is very useful if you want to use mylvmbackup to create the initial state for your slave servers. Instead of creating a .tar.gz archive, the data directory is copied into a timestamped archive directory. (Robin)
• Added support for a trailing argument to tar, which can be used for excluding files. (Robin)
• Separated out the suffix of the tarball (Preperation for rsync and users that want to use bzip2 or no compression on the tarball.) (Robin)
• While the backup is performed, a temporary suffix at the end of the tar backup file name (or the rsync target directory name) now indicates that it is incomplete. (Robin)
• The my.cnf configuration file is now included in the backup. (Robin)
• Added the ability to run an extra FLUSH TABLES on busy databases where lvcreate might take a long time (and may overrun the interactivity timeout on the connection, losing the lock). (Robin)
• Added option "--pidfile" to provide an alternative PID file location for the second server instance that is started to perform the InnoDB recovery on the snapshot prior to backing it up (Otherwise it may default to using the same pid file location that the running server uses and safe_mysqld will abort) - thanks to Kristian Köhntopp for making me aware of this problem.
• Before discarding the snapshot LV, the output of "lvs <snapshot>" is now printed out for diagnostics. It contains useful information like "how much percent of the backing store was used", which helps tuning the size of the snapshot LV.
• Added option "--skip-flush-tables" that performs the snapshot without flushing the tables to disk beforehand (which is not supported by InnoDB tables anyway) - this would save time, as the flushing can take a while, depending on the buffer sizes. (Thanks to Peter Zaitsev for the suggestion)

Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.

Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.

 

Eric Bergen from Proven Scaling (which I had the pleasure to meet in person during the MySQL Conference & Expo in Santa Clara last month) was kind enough to send me a patch for the mylmbackup tool, which justifies a new release:

Attached is a patch file for mylvmbackup that adds the ability to use
lvm version 2 and perform innodb recovery on the snapshot prior to
creating a tar ball. The option is named --innodb-recover.

I've also fixed a bug with default value handling for command line
options. In version 0.4 if a config file was specified default values
in the script were all changed to blank. This means that the config
file had to supply values for every variable instead of just the
values that need to be changed from default.

This is a very useful addition, as it significantly reduces the time required to recover a MySQL server from a snapshot backup. Thank you, Eric! Note that you need to use LVM2, as LVM1 does not support the required writing to snapshot volumes.

Version 0.5 is now available for download from the mylvmbackup project page.

I also would like to announce that I have set up a dedicated mailing list for this tool: if you want to discuss the usage/future of mylvmbackup, propose patches or ask for help, there now is a mailinglist, hosted on FreeLists.org. To subscribe, either enter your email address on the mylvmbackup list information page or send an email with the subject "subscribe" to mylvmbackup-request@freelists.org. The list is archived here. See you there!

I am happy to announce version 0.4 of mylvmbackup, a tool to perform consistent backups of a MySQL server's tables using Linux LVM snapshots.

For this release I'd like to especially thank Robin H. Johnson from the Gentoo project, who contributed another batch of useful changes and informed me that mylvmbackup is now in productive use to perform backups of the MySQL databases that power the project's Bugzilla bug tracking system. I am always glad to read about such use cases - how do you utilize mylvmbackup in your environment?

• The option handling has been improved. mylvmbackup now starts by using the builtin defaults, followed by the default configuration file (/etc/mylvmbackup.conf, followed by an alternative configuration file (specified via CLI arguments), followed by the remainer of the CLI arguments.
• Changed the capture of the position file and tarball creation to not use absolute path names, and instead use them relative to $mountdir. This means that the content of the backup tar archives has changed with regards to the directory structure, please keep this in mind when restoring from backups!
• Now utilizes a bindmount to get the position directory near the $mountdir for the tarball creation.
• Directory names are sanitized for excessive whitespace and trailing slashes
• Code cleanup: use my $variable instead of use vars

In addition to Robin's improvements, the following changes were made by me:

• Fixed DSN string handling (thanks to Peter Zaitsev for spotting this)
• Fixed logging to the console (log levels were not printed)
• Removed a few currently unused configuration options from the config file

I am happy to announce version 0.3 of mylvmbackup, a tool that performs consistent backups of a MySQL server's tables using Linux LVM snapshots.

Special thanks go to Fred Blaise, who contributed the majority of the new features that have been added to this new release:

• It is now possible to use an external configuration file /etc/mylvmbackup.conf to store the options. This is probably more convenient than having to pass a slew of options on the command line or having to hack the script itself to change the default values. This new feature requires the Config::IniFiles Perl module to be installed, a sample configuration file is included in the package.
• The logging to the console has been visually enhanced by including a time stamp and the message category (e.g. Info, Warning or Error). In addition to that, it is now possible to log messages to a local or remote syslog server. This feature requires the Sys::Syslog Perl module.
• The man page has now been converted into an asciidoc file, which makes it easier to generate other document formats as well, e.g. a HTML version.
• Several small bugs have been fixed, too: see the ChangeLog for details.

If you are looking for a convenient backup tool to create fast and consistent MySQL backups, please give mylvmbackup a try! You feedback is appreciated. A tarball and RPM are now available for download from the project's home page. Thanks!

I am happy to announce that version 0.2 of the mylvmbackup tool is now available!

mylvmbackup is a Perl script for quickly performing backups of a MySQL server's databases using the Linux Logical Volume Manager (LVM). It creates a consistent LVM snapshot of the server's data directory which is then backed up without further blocking the server's operation.

After version 0.1 was published in May this year, I did not really get much feedback about it. I had some ideas for improvements (see the TODO file included in the package), but never got around to actually start working on them.

Thanks to Robin H. Johnson from the Gentoo project for contributing a number of new options and features as well as some code cleanups. His changes motivated me to make a few more modifications and improvements by myself, which have now been rolled into a new release.

The new options provide some more flexibility in the way the script handles the logical volumes and how the backup files are being created. I also overhauled the building and packaging and added a Makefile to automate these procedures. For details, please refer to the ChangeLog and check the manual page and the README for additional info.

A tarball and RPM of version 0.2 can now be downloaded from the project page.

The SVN repository can now be browsed using WebSVN as well.

Please give it a try! Your feedback is very welcome.

With all the recent buzz on Planet MySQL about using LVM to perform "semi-hot" backups of MySQL on Linux, I remembered that we once created such a Perl script for a server project that we worked on in cooperation with a hardware vendor. As this particular backup method was also mentioned and recommended in my talk about MySQL backup and security on Linux and the script hasn't actually been publicly available so far, I offered to take over the ownership and release it under the GPL. The first tarball release (0.1) is now available from this web page (still work in progress), the files are available via SVN from here as well. Make sure to see the README and TODO files for the current status. A man page is included, too. I hope you will find it useful - feedback is very welcome!