Lenz Grimmer's blog

I tend to switch between Linux and OpenSolaris as my desktop operating system from time to time. To be more flexible in this setup, I store most of my work-related data (e.g. source trees, VirtualBox images) on an external 320GB USB disk drive, using the ZFS file system. While OpenSolaris supports ZFS natively, I can access the file system on Linux using zfs-fuse and I could even mount these file systems on a FreeBSD system, if needed. There aren't that many file systems that allow an easy exchange of data between (Open)Solaris and Linux – the other ones that I am aware of are FAT and UFS, which both don't give me the confidence and flexibility I need.

A while ago, I purchased a second external drive of the same size and now use both of them in a mirrored configuration. This gives me several benefits:

• Redundancy: external disk drives have a higher risk of getting physically damaged, so having a mirrored copy of my data ensures that I won't lose anything important if one of the drives dies.
• Increased performance: ZFS is capable of distributing reads across both devices, thus I get twice the speed of a single USB 2.0 port, if I connect the drives to ports attached to separate USB host controllers.
• Automatic resyncing: When I'm on the road, I only take one half of the mirror with me. In case this drive gets lost/stolen, I still have a second copy of the data at home. ZFS complains about the pool being in degraded state, but continues to work normally. When I return home, I simply attach the second drive again and ZFS automatically resyncs (resilvers) anything that has been modified in the meanwhile.
• I can use snapshots for backup purposes. I have a small Intel Atom based PC (running OpenSolaris) that acts as my central file server (using two 1TB disks in a ZFS mirror) and CUPS print server. If I want to create a backup of my external USB disks, I create ZFS snapshots of the file systems in question and transfer these to the home server using "zfs send/receive". This works both locally by connecting the drives to the server box directly or via SSH over the network. I wrote a small shell script to automate this process. The transfer is done in an incremental fashion – only the differences between the current and the last snapshot are being propagated. To save disk space on the external drives, I usually discard all older snapshots except for the last one or two. On the file server, I maintain snapshots for a longer time period. The snapshots are named by using a simple date/time timestamp (filesystem@YYYY-MM-DD-HH:MM), this makes it easy to go back to a particular snapshot in case I'm looking for something I may have removed by accident, but I still recall when it was last used.

From my experience, ZFS is a very solid and reliable solution, providing impressive functionality with a very user-friendly UI (you only need use two commands, zfs and zpool).

If you want to learn more about ZFS and how to use it in practice, consider attending my upcoming talk (in German) about this subject at FrOSCon 2010 in Sankt Augustin, Germany!

I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).

From the ChangeLog: 

• Deleted sample precleanup.pm hook as it has potential to cause harm and is too specialized on a particular use case (BUG#394668)
• Added support for rsync via SSH (BUG#392462)
• Fixed InnoDB recovery in case a relative path to the MySQL data directory is defined (BUG#38337), improved the documentation of relpath in the man page.

After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.

Some notable highlights from the ChangeLog:

• Removed the absolute path names to external tools (make sure $PATH is correct)
• Added --log-err to the startup options of the recovery instance to avoid cluttering the server's error log
• Added support for hooks written as Perl Modules. (Matthew Boehm)
• Added support for date/time-formatted path names for backupdir and mountdir (Matthew Boehm)
• Backupdir and mountdir are now created automatically (Matthew Boehm)
• Added new hook "logerr" when an error is logged. (Matthew Boehm)
• Added Option --keep-mount... (Tim Stoop)
• Removed the bind mount, now requires LVMv2
• Support reading login/password from ~/.my.cnf (Baron Schwartz)
• Documentation fixes and improvements (Ville Skyttä) (Bug #302144)

XtraBackup is an Open Source online (non-blockable) backup solution for the InnoDB and XtraDB storage engines. It works with both MySQL 5.0 and 5.1 (and possibly 5.4 as well) and is distributed under the GPLv2.

Some weeks ago Vadim announced the availability of xtrabackup-0.7, stating that they consider it stable enough now to label this version a "Release Candidate". I've been maintaining RPM packages of xtrabackup on the fine openSUSE Build Service for quite some time now, RPMs of 0.7 for a number of distributions are now available for download. Please report any bug reports via the bug tracker on Launchpad.

I just realized that I haven't blogged for more than a month! Shame on me. But I will blame it on being away on conferences and vacation for quite some time And if you are following me on twitter, you may have noticed what I was going on in my life and that I did't get hit by a bus...

So what was going on since I returned back home from the MySQL Conference? First off, I uploaded und sorted my pictures from the conference and the Drizzle developer day on Flickr. I also uploaded the slides (PDF) from Colin and myself speaking about "MySQL Server Backup, Restoration, and Disaster Recovery Planning" to the MySQL Conference site and they are now available for download from the session page.

On May 4th and 5th I attended the amoocon in Rostock, Germany - a conference primarily about Open Source Telephony and VoIP, where I gave two talks (in German) about MySQL HA Solutions and MySQL Backup and Security Best Practices. There were several other MySQL-related sessions at this conference (e.g. Geert speaking about MySQL Cluster) and I was quite impressed to learn about how widely used it is in the VoIP/Telephony sector. The PDFs of my slides for both talks are available from the conference web site as well as on my slidespace on SlideShare (a very convenient service that I recently started using). There even is an MP3 and video recording of the HA talk, which is also linked from the session page mentioned above.

Right after the amoocon, I attended the next09 conference, which took place in Hamburg, Germany. On the first day I helped manning the Sun booth for some time (Sun Startup Essentials Germany was a sponsor of the event) and listened to some talks, on Wednesday I gave a presentation about "Working for a Virtual Company: How do we do it at MySQL?". A video recording of my talk is available from sevenload. Last time I checked the video was truncated, but I hope they will publish a complete version of it soon. Unfortunately I had a very bad speaking slot - the very first one in the morning (9:00am), after there was a party/social event going on the night before... But luckily there still were ~20-30 people in the audience. This was the first time that I gave this presentation and talked about something less technical, which was actually quite fun! As for the other recent talks, the slides are on SlideShare.

From May 14th to the 16th I was in Verona, Italy to speak about MySQL Backup and Security and bzr - The Bazaar source revision control system at the Italian phpDay. This was quite a nice event with ~150 attendees and Rasmus Lerdorf giving two talks as well. However, many of the sessions were in Italian, so I decided to spend some time walking around the city and taking lots of pictures. I also took a number pictures from the conference, but the light was pretty dim in the room and many shots turned out to be too blurry. Next year they plan to have the phpDay in Rimini - I definitely plan to be there!

Hyatt Regency, Santa Clara
Originally uploaded by Lenz Grimmer

After a long and uneventful flight from HAM to SFO via FRA, I arrived safely in Santa Clara yesterday. Today we spent the day in San Francisco, for some sightseeing (Downtown, Chinatown, Pier 39 and Fishermen's Wharf) and a bit of shopping. It was a very nice and sunny day, the sunlight helped a lot with getting over the jetlag for a while.

But now I'm pretty tired - I look forward to meeting friends, colleagues and members of our community at the MySQL Conference, which starts next week. I will also give a talk together with Colin Charles about MySQL Server Backup, Restoration, and Disaster Recovery Planning as well as moderating a BoF about MySQL Code Contributions, where we will talk about the recent developments and changes that we've implemented (or plan to put in place) to make it easier for developers to contribute code to the MySQL project.

I will try to take lots of pictures during the conference, which will be posted to my MySQL Conference Photo Set on Flickr every once in a while.

My travel schedule is getting quite crowded for the next months - I just received confirmation that I will be speaking at the phpDay in Verona, Italy on May 15-16th. I'll be talking about bzr - The Bazaar source revision control system as well as MySQL Backup and Security - Best practices in the developer track of the conference. I've never been to Verona, but it seems like it's a beautiful city. I look forward to being there!

Today I gave my first MySQL University session as a speaker, talking about Backing up MySQL using file system snapshots. The talk went quite well (at least that was my impression) and we had ~10 people attending. The slides (PDF) and a recording of the session are now available from the Wiki page. Unfortunately the recording lacks the audio track, which is a bit of a bummer. We've submitted a support request with the DimDim folks, so hopefully they can provide us with a complete recording.

There was one question during the session that I was not able to answer myself, so I'm asking for your insights here:

Consider we're using InnoDB and MyISAM tables on a file system that can be snapshotted (e.g. Linux LVM or ZFS) and we're performing the following operations:

1. FLUSH TABLES WITH READ LOCK (yes, this won't help for the InnoDB tables)
2. Create the snapshot
3. Store the output of SHOW MASTER/SLAVE status in a file to be part of the backup
4. UNLOCK TABLES
5. Mount the snapshot
6. Start a second MySQL instance that accesses the tables on the snapshot, let InnoDB perform its table recovery
7. Shut down the second instance and perform the backup of the snapshot

The question that came up was if this actually still is a consistent backup, considering that InnoDB rolled back uncommited transactions. Does the state of the tables still match the binary log positions we noted before? I assume yes, as long as the transaction does not involve modifications non-transactional tables.

Another suggestion that came up was to change InnoDB's configuration variable innodb_max_dirty_pages_pct to "0" prior to performing the snapshot, to minimize the amount of dirty pages that have not been written to disk (and thus reducing the time required for recovering later). I wonder if this would make a difference...

What other InnoDB variables might have a noteworthy effect in the context of snapshot backups? I am looking forward to your comments.

Sorry for the downtime of this site - until around a week ago I hosted my home page on a trusty Genesi Pegasos II system (powered by a PowerPC G4 Processor clocked at 1GHz, using Debian 4.0 PPC with 512 MB of RAM), serving these pages from my home DSL connection. Unfortunately this system provided no means of redundancy - the hard disk drive died.

Luckily I perform frequent backups, so I moved most parts of the site to a shared hosting space now - the picture gallery is unfortunately too big to fit into the space that I have there. I'll try to move the pictures into my Flickr account instead, but this will take some time.

Note that the primary domain name of this site is now lenzg.net - lenzg.org, (the domain that I tried to promote as the official domain for my site) used to redirect to the home machine at lenz.homelinux.org. Both now redirect to the new address instead. I've initiated the move of the lenzg.org domain to the other provider as well, so soon this site will be available from both the .org and .net domain. Please don't link to lenz.homelinux.org anymore, as that site will eventually go out of service. Until then, a small openSUSE Linux box (Intel PIII, 500 MHz, 192 MB of RAM) running lighttpd will perform the URL redirection.

Some days ago, I released version 0.11 of mylvmbackup a Perl script that performs consistent backups of a MySQL server by using LVM filesystem snapshots. The source archive as well as a generic RPM can be found on the project home page, packages for many Linux distributions are available on the openSUSE Build service.

This release includes some new functionality as well as numerous bug fixes and improvements, most notably:

• Added support for using rsnap as a backup backend (Matt Lohier)
• The documentation is now maintained in POD style instead of asciidoc (Matthew Boehm)
• Support using non-GNU tar and additional compression methods (e.g. bzip2, lzma) (Alexander Skwar)
• Code cleanups, improved error handling and logging

I would like to thank all the contributors for their support! More details about the changes in this release, directly from the ChangeLog:

• Added new option "--quiet" that suppresses informal logging output (warnings and errors will still be printed/logged)
• Applied patch from "kjetilho" that makes mylvmbackup more robust and paranoid when it comes to handling errors/failures (Bug #298175) Now the script aborts cleanly in case of any error.
• Fixed error handling in case of a failed DB connection (patch submitted by Matthew Boehm, Bug #280989)
• Merged changes from Alexander Skwar: Fix for Bug #278478 and implemented Blueprint "Improve the tar backup backend of mylvmbackup"
• Fixed Bug #271671: "overloading parameters does not work" by removing the default values for host and port from the configuration file and removing the unnecessary check for passing both host and socket at the same time. Updated documentation and configuration file comments accordingly.
• Applied patch suggested by "Jonas" to fix Bug #267944: "backup returns successfully when snapshot creation fails"
• Code cleanup: moved flushing of tables in a separate subroutine flush_tables()
• Code cleanup: moved log messages into the respective subroutines
• Code cleanup: use return values of subroutines instead of updating global variables
• Improved some log messages to explain what was DONE or FAILED
• Code cleanup: build up long command strings in a $command variable before passing it to system()
• Renamed subroutine create_snapshot() to create_lvm_snapshot()
• Merged patch from Matthew Boehm: Removed old asciidoc documentation in favor of POD style. This removes the dependency on the external program a2x for creating documentation and uses the 'built-in' pod2html and pod2man instead. Updated the Makefile to accommodate the change.
• Applied patch from Matthew Boehm to make the backup file name suffix configurable via a "--suffix" option. Updated the man page accordingly.
• Applied patch from Matt Lohier to support rsnap as a backup backend 
• Moved the list of contributors from the man page into a separate CREDITS file, added missing names

Enjoy!

I am happy to announce that mylvmbackup version 0.10 has been released.

You can download the updated package from the project home page or via the openSUSE Build Service.

This version fixes some bugs and includes new functionality:

• Applied patch from Marc Haber: added option --keep_snapshot that will skip the removal of the backup snapshot before terminating the script. Providing the option --backuptype=none will now skip creating a backup using the builtin backup modules. Both options provide more flexibility when using hooks for performing the actual backup tasks or when the snapshot is considered to be the actual backup.
• Added two new hooks: "backupsuccess" and "backupfailure" which are called respectively upon success of failure of the backup operation (Bug #264089)
• Make sure that binaries are being found ($PATH may not include /sbin when called from cron), added missing entry for "lvs" to mylvmbackup.conf (Bug #255703)
• Updated documentation

Hello and greetings from DrupalCon 2008 in Szeged, Hungary!

We (Thierry Manfé, Scott Mattoon and myself) are having a great time manning our booth and talking about Drupal, MySQL and Open Source@Sun with the nice crowd of Drupal Users and Developers here. Sun is a gold sponsor of the event and we're giving a number of sessions as well.

Today I gave my first presentation about MySQL Backup and Security - Best practices - unfortunately I ran a tad bit out of time at the end... The slides have already been attached to the session page, so you can read up on the last few things I was going to talk about. Feel free to contact me, if you have further questions!

Tomorrow I'll be talking about High availability solutions for MySQL: An Overview and practical demo, which will also include a practical demonstration of a two-Node Linux Cluster, performed by Jakub Suchy. In the afternoon, I will also hold a BoF about bzr - The Bazaar source revision control system

I've also uploaded some pictures from the event (and some impressions from the city) on my gallery (more will follow later). Enjoy!

I am happy to announce that a new version (0.9) of mylvmbackup has been released. This is the first release since the source code has been moved from Subversion to Bazaar and is now hosted on Launchpad.net. I would like to thank Robin H. Johnson and Patrick Hahn for providing the patches that contributed to this new release!

mylvmbackup is a tool for quickly creating backups of MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.

From the ChangeLog:

• Applied patch from Patrick Hahn: provide an option to call external scripts/applications (hooks) at various stages of the backup process. See the man page for instructions
• Added options "--skip_hooks" to disable the execution of hooks and "--hooksdir" to define the location for these (default is /usr/share/mylvmbackup)
• Updated documentation: added new options and instructions on how to use hooks
• Applied patch from Robin H. Johnson: Full support for an rsync:// service as the backup destination. If you include any path fragments with the rsync module name, they must already exist!
• Updated documentation to reflect these changes
• Updated TODO

You can download a source tarball or RPM from the project home page. Additional packages for various Linux distributions can be obtained from the openSUSE Build Service. Packages for Gentoo and Debian should appear shortly, too.

Enjoy! Please let me know how mylvmbackup works for you, either by posting to our mailing list or by submitting a bug report. Thanks!

I've just been informed that a PDF of my slides as well as a recording of yesterday's webinar about MySQL Backup and Security (in German) is already available for replay/review. You can access the files from our on demand webinars page (free registration required). Enjoy!

If you are new to MySQL and would like to get an overview about some best practices for securing a MySQL server and some commonly used backup techniques, consider attending this webinar (in german), held by yours truly. It will take place this Tuesday (2008-06-17) at 15:00 CEST - participation is free of charge! This is my first attempt to perform a webinar, I usually give talks in front of a live audience... Let's see how it goes.