Lenz Grimmer's blog

CMake is a cross-platform, open-source build system, maintained by Kitware, Inc.

From the CMake.org home page:

CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice.

It has been used for building the MySQL Server on Windows since MySQL 5.0 – the initial CMake build support was added in August 2006.

For building MySQL on all other platforms, the GNU autotools (autoconf, automake and libtool) are currently being used.

CMake is used in some other MySQL projects as well, e.g.

• MySQL Connector/C
• MySQL Connector/C++
• MySQL Proxy

On February 22nd, Vladislav Vaintroub pushed the changes required to implement WorkLog#5161 "CMake-based unified build system" into the "mysql-next-mr" branch (aka the "Celosia" mile stone).

From this version on, CMake can also be used to build MySQL on Linux and other Unix platforms. For the time being, the autoconf/automake files are still available as well, but will be phased out once the CMake build enviroment has reached the desired level of maturity. The change was announced on February 28th on our "internals" developer discussion list.

The purpose of WL#5161 is to simplify the MySQL build system. It is much easier and less error-prone to maintain a unified build system for all platforms than two separate ones.

CMake has been chosen because of several reasons; the worklog description lists a few pro-CMake arguments (slightly rephrased):

• CMake works on Windows. The GNU buildsystem does not really work and likely never will work natively on Windows (Using Cygwin is not really an option).
• Traditionally, new MySQL features that required changes in the build environment (e.g. the plugin system, unit tests, most recently googletest integration) were always implemented on Unix first, leaving Windows behind (sometimes for years). This would not happen with a unified build system.
• MySQL already uses CMake since 2006 on Windows, so we do not need to start from scratch, only port what we have to Unix.
• CMake runs on every OS and compiler we support.
• It is simple to obtain and install on a wide range of platforms. It is available in all major Linux package repositories (e.g. Ubuntu, Fedora, OpenSUSE). It is also in the OpenSolaris repository, known as SUNWCmake. It's in FreeBSD ports and available for Mac OS X. It is also very simple to compile it from source, the single prerequisite is a working C++ compiler and make utility.
• CMake has support for features we need and might need, e.g. system checks or cross-compiling.
• CMake provides integrated support for packaging. It can handle both simple packages (tar.gz or zip archives) and more complex things like DEB and RPM without much extra coding.
• Good integration with the popular IDEs (Visual Studio, Xcode, Eclipse CDT, KDevelop). Developing in an IDE makes the development process more enjoyable, and potentially it lowers the barrier for external contributors. Of course, CMake can generate traditional Unix Makefiles, which appear to be are superior to the ones generated by GNU autotools (for example, they have progress indicators, colored output and working dependencies).
• The scripting language used by CMake is simpler than m4 used by autotools.
• CMake is a single small tool, not a bunch of different tools as in GNU system (autoconf, autoheader, automake, libtool)

I'd like to mention a few additional reasons:

• Out-of-source builds – CMake can separate the build directory from the source directory. This is convenient, as your working source tree is not cluttered with object files and other fragments of the build process.
• Build configuration using a GUI. The cmake-gui package (based on Nokia/TrollTech's Qt library) provides a convenient way of enabling and configuring the various available build options. This is much better than having to memorize all the required defines and configuration flags.
• Integrated support for creating a wide range of package formats.

The CMake Wiki lists a number of other "nice to have" features.

From a developer perspective, I hope that it will make it much easier to finally implement two things that many developers working with MySQL have been waiting for (now that the build code has been cleaned up):

• Compiling the embedded MySQL Server (libmysqld) as a shared library
• Better support for cross-compilation

Building MySQL with CMake is quite simple and straighforward – the process is outlined on the MySQL Forge Wiki. The document is still work in progress and we'd like to encourage you to take a look at it, try to follow the steps and update/improve the Wiki page, if needed! Your feedback on the build process is appreciated. Feel free to join our internals mailing list to discuss your impressions and observations or submit a bug report via the Bug Database. It's likely that the build still has a few rough edges that we'd like to fix quickly (e.g. BUG#51502 – a fix for this one is already commited to the mysql-next-mr-bugfixing source tree and will be merged into the mysql-next-mr trunk soon).

If you're new to CMake, you might want to take a look at the "Getting Started With CMake (An End-User's Perspective) For Cross-Platform Building" screencast or the "Running CMake" article.

Happy hacking!

Today I gave my first MySQL University session as a speaker, talking about Backing up MySQL using file system snapshots. The talk went quite well (at least that was my impression) and we had ~10 people attending. The slides (PDF) and a recording of the session are now available from the Wiki page. Unfortunately the recording lacks the audio track, which is a bit of a bummer. We've submitted a support request with the DimDim folks, so hopefully they can provide us with a complete recording.

There was one question during the session that I was not able to answer myself, so I'm asking for your insights here:

Consider we're using InnoDB and MyISAM tables on a file system that can be snapshotted (e.g. Linux LVM or ZFS) and we're performing the following operations:

1. FLUSH TABLES WITH READ LOCK (yes, this won't help for the InnoDB tables)
2. Create the snapshot
3. Store the output of SHOW MASTER/SLAVE status in a file to be part of the backup
4. UNLOCK TABLES
5. Mount the snapshot
6. Start a second MySQL instance that accesses the tables on the snapshot, let InnoDB perform its table recovery
7. Shut down the second instance and perform the backup of the snapshot

The question that came up was if this actually still is a consistent backup, considering that InnoDB rolled back uncommited transactions. Does the state of the tables still match the binary log positions we noted before? I assume yes, as long as the transaction does not involve modifications non-transactional tables.

Another suggestion that came up was to change InnoDB's configuration variable innodb_max_dirty_pages_pct to "0" prior to performing the snapshot, to minimize the amount of dirty pages that have not been written to disk (and thus reducing the time required for recovering later). I wonder if this would make a difference...

What other InnoDB variables might have a noteworthy effect in the context of snapshot backups? I am looking forward to your comments.

As noted in my previous blog posting, I manged to revive my old Logitech TrackMan Marble FX on Linux (openSuSE 11.1b4), using a Serial-to-USB dongle with a Prolific PL2303 chip. But I also use OpenSolaris on my Laptop quite frequently (currently testing the upcoming 2008.11 release), so I investigated if it would be possible to enable the trackball there as well.

Luckily, the Driver Manager listed the plugged in adapter and the correct driver (usbsprl) was loaded already. Now the real challenge was finding out which device node to use. Some research revealed that the driver actually comes with a manual page , which indicated that /dev/term/0 was the correct device name.

Lo and behold, I copied the InputDevice section from my Linux xorg.conf file into the OpenSolaris one, replaced the Device parameter with the appropriate one and restarted the X server. Immediate success! Now I can enjoy using my most favourite input device on OpenSolaris as well.

I am probably different than most users, but I am a a fan of unusual input devices. I prefer Laptops with trackpoints - I immediately disabled the touchpads on my Lenovo laptops (a T61 and T42) in the BIOS when I received them. My first Laptop (a Toshiba Portege 3440CT) didn't even have a touchpad to begin with. It's a pity that trackpoints seem to a dying breed.

And I don't like using regular mice on my desktop, either! Actually, my most favourite input device is a trackball - I purchased a Logitech TrackMan Marble FX a long time ago, and used it for years. Then computers stopped having serial or PS/2 connectors, and I replaced the trackball with an USB mouse. I never really got the hang of using mice, but Logitech (or other vendors) somehow never came up with a suitable replacement model for the Marble FX with a USB port. I recently looked at the Logitech Trackman Optical, but was not convinced by the reviews I read, and the fact that it requires batteries (a trackball is a stationary device, so a cable does not really interfere here!). The Microsoft Trackball Explorer might have been an option, but it seems to be impossible to get nowadays.

After experimenting with several mouse models (Cherry, Microsoft), I decided to revive the TrackMan Marble again. It comes with a PS/2-connector by default, so I first tried to connect it to my PC using a PS/2-to-USB converter dongle. This actually worked without any tweaking, but had two limitations: the fourth mouse button was not detected anymore (I could have lived with that) and the Trackball stopped responding after it had been idle for a while, requiring me to restart the X server to get it working again.

So using the PS/2-to-USB dongle was ruled out and I tried an Serial-to-USB dongle instead:

lenz@thebe:~> lsusb|grep Serial
Bus 002 Device 006: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port

When plugged in, udev creates a new serial device /dev/ttyUSB0 which I needed to inform the X server about. Fortunately it's possible to define multiple input devices in the xorg.conf configuration file. Using the serial port actually had another advantage - I was able to add some trackball-specific tweaks that would have collided with the settings of the "regular" PS/2 mouse section that I needed for the builtin trackpoint of my Thinkpad. It required some tweaking and testing, but this is what I added to xorg.conf to be able to use the TrackMan Marble FX in addition to the builtin pointing device:

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    InputDevice    "Trackball" "SendCoreEvents"
    Option         "Clone" "off"
    Option         "Xinerama" "off"
EndSection

Section "InputDevice"
    Driver       "mouse"
    Identifier   "Trackball"
    Option       "Device" "/dev/ttyUSB0"
    Option       "Name" "TrackMan Marble FX"
    Option       "Protocol" "Intellimouse"
    Option       "Vendor" "Logitech"
    Option       "AngleOffset" "10"
    Option       "Buttons" "8"
    Option       "Emulate3Buttons" "off"
    Option       "EmulateWheel" "true"
    Option       "EmulateWheelButton" "8"
    Option       "YAxisMapping" "4 5"
    Option       "EmulateWheelInertia" "8"
EndSection

You may wonder about the 8 mouse buttons, as the device only has four physical buttons. Interestingly, the fourth button on the TrackMan reported itself as "button 8" when I probed it with "xev", so I needed to make sure the server is aware of it. When pressed, the trackball now acts like a mouse wheel and allows me to quickly scroll across long documents - very handy! Now I just hope that the button switches in the TrackMan last for another while - until some vendor eventually comes up with a suitable replacement...

Today, the root file system on our public svn server nearly ran out of disk space. The reason? The /tmp directory was quickly filling up with temporary files created by websvn, which I set up parallel to the FishEye repository browser for testing purposes. A quick investigation of the apache log files revealed the culprit - a crawler from Microsoft was running haywire and decided to ignore the rules in the robots.txt file, even though it did actually looked at the file before!

Here is how robots.txt looked like (I now changed it to disallow everything):

User-agent: *
Disallow: /fisheye/
Disallow: /websvn/

If I am not mistaken, no crawler should actually consider going into the SVN browser directories. Some snippets from the apache log:

$ grep robots.txt /var/log/apache2/access_log | grep msn
65.55.208.178 - - [03/Aug/2008:16:58:35 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [03/Aug/2008:19:05:55 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [03/Aug/2008:22:14:47 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:00:31:32 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:00:57:38 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [04/Aug/2008:06:49:33 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:07:16:21 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:09:29:17 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.104.156 - - [04/Aug/2008:11:08:24 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [04/Aug/2008:11:29:34 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [05/Aug/2008:13:30:20 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.208.178 - - [05/Aug/2008:16:17:59 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

Good boy, it checks the robots.txt file. But what is this?

$ grep msnbot /var/log/apache2/access_log | tail -20
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/filedetails.php?repname=MySQL+Documentation&path=%2Fworkbench%2Fall-entities.ent&rev=9981&sc=1 HTTP/1.1" 200 6408 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/dl.php?repname=MySQL+Connector%2FJ&path=%2Fbranches%2Fbranch_5_0%2Fconnector-j%2F&rev=6600&isdir=1 HTTP/1.1" 200 40960 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:19 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Fproto-doc%2F&rev=9994&sc=1&isdir=1 HTTP/1.1" 200 36907 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8323&sc=0&isdir=1 HTTP/1.1" 200 15278 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Proxy&path=%2Ftrunk%2FDoxyfile&rev=365&sc=1&isdir=0 HTTP/1.1" 200 4162 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=Eventum&path=%2Feventum%2Freports%2F&rev=3542&sc=1&isdir=1 HTTP/1.1" 200 90591 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Fndbapi%2F&rev=9749&sc=0&isdir=1 HTTP/1.1" 200 21440 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8511&sc=0&isdir=1 HTTP/1.1" 200 18541 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

As you can see, it is happily crawling everything below /websvn/, which also includes links named "Tarball" - guess what they are good for? Yes, they create tarballs of a given SVN directory, using /tmp to build up the archive file... Within a very short amount of time, it used up more than 6 GB of disk space, as it seems as if websvn leaves these temporary directories behind, if the connection gets aborted or times out. We do have a cron job that wipes /tmp from files older than a certain amount of days, but it currently fills up much faster than what the cron job usually discards. I need to investigate if it is actually is a bug in websvn to leave these temporary dirs behind.

Hello Microsoft? Can you please fix your bots so they not only read but honor robots.txt files and stop DOSing our site? Thanks

This article describes how to install the Drupal 6.2 CMS on MySQL 6.0, using the Falcon Storage Engine. The operating system is a default Ubuntu 8.04 "Hardy Heron" (x86) installation.

I will make a few assumptions here, in order to keep the instructions simple: a fresh OS install, no other MySQL databases or web services are running or have already been installed. Both MySQL and the web server are installed on the same host. You should be able to become root to install packages and to have access to the local file system and the system configuration.

This article will explain how to install and configure Apache/PHP, MySQL 6.0 and Drupal 6.2.

When it comes to handling my work-related email, many people will probably conclude that I am an old fart in Linux terms: I still use the console-based Pine as my MUA of choice, as my fingers have been hardcoded to its key combinations over the years (my first encounter with Pine was around 1994 on a shared DEC Ultrix box in my university). So far, I have not found any other application that allows me to process email as quickly as by using this tool.

I admit that I do use Thunderbird for my personal email, though, to gather experience with it (and to toy around with the various extensions, especially Nostalgy is a gift from heaven for people like me!). And of course because the mail volume there is less critical to cope with! But I am not going to start a holy war here - this article is supposed to explain how I have configured my mail setup to rotate the log files that keep track of all my incoming mails.

So far, the MySQL Server RPM packages as provided by MySQL AB used to automatically start the mysqld process after the package has been installed. It has been like that since the very beginning and we think of it as a convenience for our users when they want to get up and running quickly.

However, Kristian raised an interesting point in BUG#27072 where he points out that automatically starting mysqld during the RPM installation might not always be the desired behaviour, especially in automatic installation environments or during a fresh installation (where the system might not be fully configured yet). Therefore he proposes to change this behaviour to not start mysqld as part of the installation.

While I personally agree with his proposal, this is of course a tricky decision: our users are familiar with this behaviour and it's never a good idea to cause surprises. Therefore this change could only be done in future versions of MySQL, where they can be properly announced and documented and don't cause too much confusion.

Then again, there are several options here. Should we just disable it for the Enterprise Linux RPMs (RHEL/SLES) and keep it enabled in the "generic" RPMs? This would be inconsistent and harder to document/explain. Should the server never be started at all, or should we keep the behaviour for updates at least (when the server was already running before the package update)?

We would like to get your take on this. Please post your comments and suggestions to the bug report (preferred) or leave a comment on my blog, so we can get an impression on what you would expect to be the correct behaviour here. Thanks!

Yesterday I received a sample copy of the "SUSE Linux 10.1 OSS - vorkon" DVD, which was assembled by Nicolaus Millin. While I have not installed it yet (as I by now have a very well running installation of SUSE Linux 10.1 on my laptop), the content of the DVD sounds very promising. In addition to incorporating all the updates that have been published since SUSE Linux 10.1 was released (around 100), this version also includes additional drivers like the 3D graphics drivers from ATI and nVidia, network card drivers like ndiswrapper, madwifi and rt2500 as well as drivers for AVM products and Logitech QuickCams. So the base system should already run very smooth and provides a lot of stuff that one usually has to download and install after the initial installation of SUSE Linux OSS.

But the fun doesn't stop here! Nicolaus also added a number of interesting applications, usually not included in SUSE Linux OSS, such as Adobe Reader 7.0, moneyplex, RealPlayer, FlashPlayer, Cinepaint, FlightGear and several extensions for Mozilla Firefox. He also included many LAMP applications (based on the famous Linux/Apache/MySQL/PHP stack), that have already been preconfigured and are ready to use right after installing the packages! This is where the name "vorkon" comes from - "vorkonfiguriert" means "preconfigured" in German. Some of the LAMP apps included are eGroupware, Gallery2, Joomla!, MediaWiki, nagios, phpBB, Serendipity, Typo3 and several others.

All in all this looks like a well-rounded package - it reminds me a lot of how SUSE Linux used to be when people referred to it as the "everything but the kitchen sink" distribution. Unfortunately it seems as if the legal restrictions of being owned by a US company have caused a lot of the former SUSE benefits to vanish - nowadays one has to hunt for a lot of stuff like drivers and uncrippled applications after the initial installation just as with any other Linux distribution. Thanks to Nicolaus for scratching that itch!

You can purchase SUSE Linux 10.1 "vorkon" via amazon.de - for just 19,95 EUR this is a very attractive product, especially for Linux Newcomers. Give it a try!

A few days ago I received a neat little gadget: a Yakumo QuickStick DVB-T, which allows me to watch TV via the USB-2 port on my Thinkpad. Amazon sold it for 52 EUR, so I could not resist, after making sure that it was listed in the list of supported devices on the very informative LinuxTV Wiki pages. All what was left to be done for me was to download the firmware and putting it into /lib/firmware. The driver and firmware were loaded automatically when I plugged in the device:

Mar 20 19:44:05 metis kernel: usb 1-4: new high speed USB device using ehci_hcd and address 8
Mar 20 19:44:05 metis kernel: dvb-usb: found a 'WideView WT-220U PenType Receiver (and clones)' in cold state, will try to load a firmware
Mar 20 19:44:05 metis kernel: dvb-usb: downloading firmware from file 'dvb-usb-wt220u-01.fw' to the 'Cypress FX2'
Mar 20 19:44:05 metis kernel: dvb-usb: WideView WT-220U PenType Receiver (and clones) successfully initialized and connected.
Mar 20 19:44:07 metis kernel: usb 1-4: USB disconnect, address 8
Mar 20 19:44:07 metis kernel: dvb-usb: generic DVB-USB module successfully deinitialized and disconnected.
Mar 20 19:44:08 metis kernel: usb 1-4: new high speed USB device using ehci_hcd and address 9
Mar 20 19:44:08 metis kernel: dvb-usb: found a 'WideView WT-220U PenType Receiver (and clones)' in warm state.
Mar 20 19:44:08 metis kernel: dvb-usb: will use the device's hardware PID filter (table count: 15).
Mar 20 19:44:08 metis kernel: DVB: registering new adapter (WideView WT-220U PenType Receiver (and clones)).
Mar 20 19:44:08 metis kernel: DVB: registering frontend 0 (WideView USB DVB-T)...
Mar 20 19:44:08 metis kernel: dvb-usb: schedule remote query interval to 300 msecs.
Mar 20 19:44:08 metis kernel: dvb-usb: WideView WT-220U PenType Receiver (and clones) successfully initialized and connected.

Now I fired up the KDE video player Kaffeine 0.7.1 and configured it to scan for available channels. That's all there was to it! I am positively surprised.

I have been struggling with that for ages: how can I configure my slides to not display all bullet points at once when I switch to the next slide, but rather display one at a time?

I prefer not not distract my audience by having them read through the entire slide in advance instead of listening to me still elaborating on the previous bullet points. Some Google research finally pointed me to this article, which also mentions how to accomplish this:

• Mark the frame that holds the bullet points
• In the Tasks pane, select "Custom Animation"
• Click "Add" and select the Entrance Effect you want to use. For displaying one item after another, either the plain "Appear" or "Fade In" effect is probably least distracting. Apply the Effect with OK.
• Small progress: now all bullet points will be displayed at once, but after the slide itself has been displayed.
• This can be changed with the "Effect Options" Button in the "Custom Animation -> Modify Effect" dialogue of the Tasks pane (it's well hidden as the button is only labelled as "..."). Click on the button and choose the "Text Animation" tab in the appearing dialogue. In there, change the "Group Text" option from "As one object" to "By 1st level paragraphs". If you have sublevels of bullets, choose the appropriate level that suits your needs.
• Now click OK and test the presentation again. Each bullet should now be appear individually. You're done!

The release of MySQL 5.0 introduced the MySQL Instance Manager (IM), which intends to replace the mysqld_safe wrapper script as well as the mysqld_multi script that keeps track of multiple MySQL instances running on the same machine.

This article will qive you a quick overview about the IM and how to enable and configure a minimal setup that uses the IM to manage the default mysqld instance. I used MySQL 5.1.7-beta for my tests, some of this may work differently on other versions of MySQL.