Lenz Grimmer's blog

Please note that I won't be updating this blog anymore. I will keep the content around for the time being, but future posts will only appear on the new site. I tried to articulate my reasons for doing so in my post Why the switch to a new blog?

I'm happy to announce the release of mylvmbackup version 0.16. The source package is now available for download from http://lenzg.net/mylvmbackup/ and https://launchpad.net/mylvmbackup.

Installation packages for a number of platforms can be obtained from the openSUSE Build Service.

Version 0.16 adds support for sending out SNMP traps in case of backup successes or failures. I'd like to thank Alexandre Anriot for contributing this new feature and his patience with me.

Please see the ChangeLog and bzr history for more details.

Since the beginning of this year, I'm a happy Lenovo ThinkPad T440 user. All components are supported on Linux (currently running Ubuntu 14.04 on it) — everything Just Worked "out of the box" after the installation. This has been my experience with all previous ThinkPad models so far (I also owned a T42, T60 and T420 before).

The laptop came with Windows 8 pre-installed, which I found very confusing to use (I haven't had actively worked with Windows for quite some time). But since I replaced the built-in hard disk drive with an SSD anyway (a 500GB Samsung EVO 840), that problem was solved quite quickly. Installing Linux via an USB stick was a simple affair, it most cases you can simply download and dd the ISO image on the USB stick and select it as the boot device on the next reboot by pressing F12 at the right moment.

However, not having Windows installed on another partition and not owning an external CD-ROM/DVD-drive, I now faced one problem: how to update the BIOS? Lenovo only provides a Windows-based tool and an ISO image that contains the BIOS update utility and the new firmware. However, this ISO-Image does not boot from an USB stick directly, so the dd method does not work.

Fortunately, like many times before, the ThinkWiki came to the rescue — their page about BIOS update without optical disk gave me the right clues. I downloaded the ISO image containing the BIOS update from the Lenovo Support web site and plugged in an empty USB stick, which appeared as /dev/sdc in the device list (the T440 has a built-in 16GB SSD on /dev/sdb.

The ISO image actually contains a (bootable) hard disk image, which can be dumped on an USB stick directly. To extract the image, I used geteltorito from the genisoimagepackage:

$ geteltorito -o bios.img gjuj13us.iso
Booting catalog starts at sector: 20 
Manufacturer of CD: NERO BURNING ROM VER 12
Image architecture: x86
Boot media type is: harddisk
El Torito image starts at sector 27 and has 32768 sector(s) of 512 Bytes

Image has been written to file "bios.img".
$ sudo dd if=bios.img of=/dev/sdc
32768+0 records in
32768+0 records out
16777216 bytes (17 MB) copied, 0,641393 s, 26,2 MB/s

$ sudo fdisk -l /dev/sdc

Disk /dev/sdc: 16.0 GB, 16013942784 bytes
64 heads, 32 sectors/track, 15272 cylinders, total 31277232 sectors
Units = sectors of 1  512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1             32       32767       16368    1  FAT12

There's my bootable USB stick. Rebooting the laptop and choosing the USB stick from the boot menu (F12) properly booted into the BIOS update utility and allowed me to update the BIOS to the latest version. No Windows required!

I'm happy to announce the release of mylvmbackup version 0.15. It is now available for download from http://lenzg.net/mylvmbackup/ and https://launchpad.net/mylvmbackup

Probably the biggest highlight of 0.15 is the addition of zbackup as an additional backup type. I'd like to thank Ivan Korjavin for contributing this new feature.

Additionally, this release provides several improvements and bug fixes. For example, it's now possible to back up more than a single my.cnf configuration file by providing a directory name like /etc/mysql instead.

Please see the ChangeLog and bzr history for more details.

It's my great pleasure to announce the release of mylvmbackup version 0.14.

This release includes a large number of improvements, code cleanups, and new functionality.

I would like to thank Ask Bjørn Hansen, Ben Bonnel, Norbert Tretkowski, Neil Wilson, Klaus Ethgen and Alexandre Anriot for their feedback and contributions to this release.

The release is available as a source tarball and generic RPM package. Packages for other distributions are available from the openSUSE Build Service.

Some notable highlights from the ChangeLog (please check the bzr commit log for a more detailed history of changes):

• Improved error handling: send out an email if mail_report_on has been set to "errors" and a log message with log level LOG_ERR was logged.
• Abort and clean up if the snapshot volume could not be mounted
• Abort and clean up if the backup creation failed
• Exit with a non-zero return code if errorstate was set
• Enabled timestr() formatting sequences for snapshot volume names, to create snapshot names which contain a dynamic date value. This can be useful when thin provisioned snapshots are used as the actual backup by enabling the keep_snapshot option
• Added "thin" option to mylvmbackup.conf (default=0), to enable thin provisioned LVM snapshots
• Added new configuration option "recoveryopts" that allows to modify the command line options that are passed to the MySQL instance that performs the InnoDB recovery (BUG#544295 and BUG#1091950). The current default options are "--skip-networking --skip-grant --bootstrap --skip-syslog --skip-slave-start" (which better accommodates MySQL 5.1)
• Merged patches from Alexandre Anriot to provide a mail reporting method which sends the log output via email and a new "purge" action that allows to only keep a defined number of backups in the backup directory
• Applied patch contributed by Klaus Ethgen that removes the annoying LVM warnings about a leaked file descriptor (DBD:mysql does not add FD_CLOEXEC on the MySQL socket)
• BUG#1086313: mvlvmbackup lacks support for 'thin' LVM snapshots (Thanks to Neil Wilson for the patch)
• BUG#787063: Ensure to clean up temporary .pos files when the snapshot mount is kept, too
• Applied small patch from Norbert Tretkowski: change umask to 077 before creating the tarball backup files to prevent them from being readable by everyone
• BUG#593220: Fixed the compressarg option for cat (removed the comment)
• Applied patch contributed by Ben Bonnel to add a new configuration option "rsnaprsyncarg" that allows one to pass special options to the rsync process when using the rsnap backend.
• Applied patch provided by Ask Bjørn Hansen to allow targets not to have a timestamp (e.g. for rsync targets)

Congratulations to the Drupal community for getting version 7.0 released! This is a major mile stone and an excellent reason to celebrate!

If you want to give Drupal 7 a try without having to install anything, I've now updated my Drupal 7 appliances on SuSE Studio to the latest release. The appliance is based on openSUSE Linux 11.3 and is available in two variants:

• A text-mode only appliance to which you connect using your local web browser via the network.
• A GUI version that starts up the Firefox browser in a minimalistic GNOME desktop to perform the installation locally. Remote network access is available, too.

The database backend is MySQL 5.1, with the InnoDB plugin and strict mode enabled. phpMyAdmin has been added to support web-based administration of the MySQL server. You can access it via http://localhost/phpMyAdmin/. I also added drush, the Drupal command line shell and scripting interface and some additional packages (yast2-http-server, bind-utils, php5-ctype, patch). I also bumped up the appliance's version number to 7.0.0, to match the version number of Drupal included.

The appliance is available in various formats:

• A live raw disk image, ready to be written to an USB stick or flash drive
• A live ISO image, to be burned on a CD or used in a virtual machine
• A hard disk image, to be dumped on a hard disk drive
• Various virtual disk formats, e.g. OVF, VMWare/VirtualBox/KVM and Xen

Please see the installation instructions provided on the download pages for details on how to use the various image formats.

So congratulations to the Drupal developer community for reaching this goal and thanks to SuSE/Novell/Attachmate for providing the infrastructure for creating such appliances. I also would like to especially thank Richard Bos for the testing and many suggestions for improvement of these appliances!

There has been a lot of buzz about the MySQL 5.5 GA release and its new features and other user-visible improvements. In this blog post, I'd like to touch on a less noticeable, but still important change.

CMake has already been used to build the MySQL Server on Windows for a long time, while the GNU autotools were used on all other platforms. Since MySQL 5.5, all builds on all platforms are now performed using the same tool chain. With the latest release of MySQL 5.5, we've made an important step to clean up and simplify the MySQL build system: the support for autoconf/automake has now been removed completely. We've been performing the release builds of MySQL 5.5 using CMake exclusively for quite some time already. It became obvious that maintaining two separate build systems simply had become too much of a burden for our engineers, especially since the autotools-based builds were no longer exhaustively tested. This change was outlined in WorkLog#5665 - Removal of the autotools-based build system. We've made this step in close cooperation with our community of packagers (e.g. the maintainers of MySQL packages on the various Linux distributions).

By moving to CMake, we are giving our developers one common build mechanism for all platforms and there is a lot of new useful functionality such as out-of-source builds or a GUI for configuring the build options. And they can now build MySQL in the very same way that we do it for our own binaries! I've covered the advantages in my previous blog post about Building MySQL Server with CMake on Linux/Unix already. We've also created a general article about CMake and MySQL as well as an Autotools to CMake Transition Guide on the MySQL Forge Wiki. The description of the source build process in the reference manual has also been updated to reflect this change.

A big “thank you” should to go to Vladislav Vaintroub and Davi Arnaut for implementing and pushing the transition to CMake forward, and to Paul DuBois for creating and improving the documentation! Wearing my former build and release engineer hat, I am very happy about this change.

Other CMake-related articles that are worth reading:

• Giuseppe Maxia: Building MySQL 5.5 with CMake
• Jonathan Perkin: How To Build MySQL Releases
• Santo Leto: Step by Step Guide on How to Compile MySQL 5.5 on Microsoft Windows

If you wonder why there hasn't been an update from me for quite a while — I just returned from two months of paternal leave, in which I actually managed to stay away from the PC most of the time. In the meanwhile, I've officially become an Oracle employee and there is a lot of administrative things to take care of... But it feels good to be back!

During my absence, Giuseppe and Felix kicked off the Call for Papers for this year's European OpenSQL Camp, which will again take place in parallel to FrOSCon in St. Augustin (Germany) on August 21st/22nd. We've received a number of great submissions, now we would like to ask our community about your favourites!

Basically it's "one vote per person per session" and you can cast your votes in two ways, either by twittering @opensqlcamp or via the opensqlcamp mailing list. The procedure is outlined in more detail on this wiki page.

As we need to finalize the schedule and inform the speakers, the voting period will close this coming Sunday, 18th of July. So don't hesitate, cast your votes now! Based on your feedback we will compile the session schedule for this year's camp. Thanks for your help!

I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).

From the ChangeLog: 

• Deleted sample precleanup.pm hook as it has potential to cause harm and is too specialized on a particular use case (BUG#394668)
• Added support for rsync via SSH (BUG#392462)
• Fixed InnoDB recovery in case a relative path to the MySQL data directory is defined (BUG#38337), improved the documentation of relpath in the man page.

 

Some time ago I posted a compilation of applications and programming languages that provide an API to connect to the MySQL Server. As it turned out, I forgot a few that I would like to mention here:

• Apache DBD API: a MySQL driver for mod_apr_dbd is not included in the official distribution, but can be obtained seperately from here. Some distributions (e.g. openSUSE) actually provide installable packages of this driver module.
• GRASS MySQL driver: The MySQL database driver in GRASS enables GRASS applications to store vector attributes in a MySQL server. Alternatively, data can be stored in an embedded version of the MySQL Server.
• HDBC: HDBC provides an abstraction layer between Haskell programs and SQL relational databases. This lets you write database code once, in Haskell, and have it work with any number of backend SQL databases.
• libdrizzle: This is the client and protocol library for Drizzle and MySQL that provides both client and server interfaces. A number of APIs (e.g. Java, Lua, Perl and Ruby as well as Python) have been built on top of it, but I'm not quite sure if they all support the MySQL client/server protocol as well.
• MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. This means you don't have to compile anything - the MySQL client library doesn't even have to be installed on your system.

Do you know of any other MySQL Connectors/APIs? Let me know!

We've now published a new set of binary packages including the extended GIS functionality from the mysql-5.1-wl1326 source tree.

This release is based on the MySQL 5.1.35 code base and fixes the bugs mentioned below. It includes some improvements to the GIS functionality as well, so please use these packages for future testing of the MySQL GIS functionality.

The following GIS-related bugs were fixed in this version:

• Bug#31753: Buffer/area functions only return first row of set
• Bug#32032: Contains() does not work on MultiPolygons, may force a disconnect and/or result in extremely long query times.
• Bug#32100: contains, intersects functions never return. Query disconnects or times out.
• Bug#33035: Intersection function returns 'Function doesn't exist' error
• Bug#40874: Buffer function doesn't work with LINESTRING
• Bug#41481: Buffer function never returns, cpu 100%, system locks up
• Bug#43493: union function returns polygon not multipolygon for non-intersecting polygons
• Bug#44753: nan error in union function

I'd like to explicitly thank John Powell for his extensive testing and bug reporting so far, he helped us quite a lot to improve the quality of the GIS code.

Please help us by by testing this new release and reporting bugs at our public bug database. For convenience, please tag your Bug reports with "gis" and make sure to put them in the "Server: GIS" category!

I admit it — I'm a fan of simulation software, particularly flight simulators. Probably the best Open Source Flight Simulator out there is FlightGear — it provides an impressive level of reality and you can download and install many additional plane models and terrains. There are packages of FlightGear 1.0.0 in the games repository of the openSUSE Build Service, which works quite well and I have been enjoying it a lot. However, the FlightGear project released version 1.9.x quite a while ago (1.9.1 was published in January 2009) and I was itching on giving the new version a try (just take a look at the screenshots and you know what I mean). However, building FlighGear on Linux is quite a complex task with many dependencies, and so held off from doing it myself, waiting for someone else to perform the update...

Well, this weekend I finally bit the bullet and did it myself - FlightGear 1.9.1 has now been added to my home:LenzGr build repository. I based my packages on the ones included in the games repository, but I plan on cleaning them up a bit and splitting them into separate packages (currently the FlightGear source RPM contains SimGear and fgrun as well). I also "borrowed" the OpenSceneGraph sources and spec file from the PackMan repository, in order to have a functional build. Unfortunately FlightGear currently only builds on a very limited list of distributions so far (namely OpenSUSE 11.0, just what I needed) — I haven't had time to adapt the spec files for FlightGear and OpenSceneGraph to match the appropriate build dependencies for the other distributions yet and "02-check-gcc-output" gives me some grief on platforms where it actually builds but generates compiler warnings (but patches are welcome!)...

Shortly after I created the initial packages of embedded InnoDB on the OpenSUSE Build Service, Oracle/Innobase released an updated version (1.0.3.5325). In addition to many improvements and bug fixes, they slightly changed the versioning scheme to better indicate what version of the InnodDB plugin their code is based on (see Vasil's posting on the InnoDB Forums for more information).

I've now updated my InnoDB packages on the Build Service to this version as well - please note that the naming scheme of the shared library package has been changed from "embedded_innodb1" to "libinnodb2" — RPM will take care of replacing the old package during update, even though the name has changed.

After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.

Some notable highlights from the ChangeLog:

• Removed the absolute path names to external tools (make sure $PATH is correct)
• Added --log-err to the startup options of the recovery instance to avoid cluttering the server's error log
• Added support for hooks written as Perl Modules. (Matthew Boehm)
• Added support for date/time-formatted path names for backupdir and mountdir (Matthew Boehm)
• Backupdir and mountdir are now created automatically (Matthew Boehm)
• Added new hook "logerr" when an error is logged. (Matthew Boehm)
• Added Option --keep-mount... (Tim Stoop)
• Removed the bind mount, now requires LVMv2
• Support reading login/password from ~/.my.cnf (Baron Schwartz)
• Documentation fixes and improvements (Ville Skyttä) (Bug #302144)