My plan is to provide an overview over the most popular utilities and applications that a MySQL DBA should be aware of to make his life easier. The focus will be on Linux/Unix applications available under opensource licenses that ease tasks related to user administration, setting up and administering replication setups, performing backups and security audits.
Of course I will cover the usual suspects (e.g. Maatkit), some of these are actually collections of different utilities by themselves. As it's impossible to go over each individual component in the given time frame, I will try to pick out the most popular/useful parts related to the scopes mentioned above. But I will also cover some lesser known gems that migh be worth taking a look at. What's your the most valued tool in your toolchest? I am still looking for more inspiration.
I look forward to being at the conference again and meeting with colleagues and friends in the MySQL community. Judging from the current schedule, it will be a very interesting mix of talks.
If you're interested in attending, you should consider registering soon! The early registration ends on March 15th. Until then, I encourage you to make use of this "Friend of Speaker" discount code (25% off): mys10fsp
I recently received a question from Robin Schumacher at Calpont, the makers of the InfiniDB analytics database engine for MySQL: "How would you recommend we try and get bundled in with the various Linux distros?"
Since this question has come up several times before, I thought it might make sense to blog about my take on this.
First of all, please note that there is a difference between "being part of the core distribution" and "being available from a distributor's package repository". The latter one is relatively easy, the former can be hard, as you need to convince the distributor that your application is worth devoting engineering resources to maintain and support your application as part of their product. It's also a space issue – distributions need to make sure that the core packages still fit on the installation media (e.g. CD-ROMs or a DVD). Therefore they take a very close look at each package and if it's really needed to be part of the installation medium or if it's fine to provide it for download from a package repository instead.
Distributors prefer to keep their core product small and restricted to the "basic OS building blocks". While MySQL might still be considered to be a part of this, this probably does not apply to the various plugins and extensions that are available for it. Therefore the best approach is to invest some engineering time and start doing the packaging yourself, either by hiring an engineer capable of creating and maintaining the packages, or by finding someone in your community who has the required experiences and is willing to do it.
While it's of course possible to set up and maintain your own build and package hosting infrastructure for that, I recommend to make use of the existing services provided by the distributors.
The top tier distributors all provide means of offloading the maintenance of "non-core" packages to their community, offering various options for packages to be made available. For example, Novell/openSUSE provide the free "Build Service", which is capable of building packages for other distributions as well (e.g. Fedora, Mandriva, Debian/Ubuntu, etc.). In addition to automating the builds, the Build Service also takes care of the distribution via their download mirror network and ensures that your application can be found via their package search interface.
Red Hat/Fedora provide something similar, named "Koji" – but it's "Fedora only". Here's a HOWTO that outlines the process of becoming a Fedora package maintainer.
Ubuntu/Canonical have "Personal Package Archives (PPAs) – if your project is hosted on Launchpad already, that might be something to look into for providing Debian/Ubuntu packages. Alternatively you could join the Debian project and start building and maintaining your package there. They maintain a list of "Work-Needing and Prospective Packages", a description of the process on how to become a new maintainer is outlined here.
If you'd like to target Solaris/OpenSolaris as well, there is the OpenSolaris Source Juicer – a web service which allows OpenSolaris community developers to build packages (using RPM spec files) and publish them for review, so they will be included in an official package repository. The Software Porters Community Group coordinates, advocates, encourages and helps with the porting of Software from multiple Platforms to the OpenSolaris Platform.
CMake is a cross-platform, open-source build system, maintained by Kitware, Inc.
From the CMake.org home page:
CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice.
On February 22nd, Vladislav Vaintroub pushed the changes required to implement WorkLog#5161 "CMake-based unified build system" into the "mysql-next-mr" branch (aka the "Celosia" mile stone).
From this version on, CMake can also be used to build MySQL on Linux and other Unix platforms. For the time being, the autoconf/automake files are still available as well, but will be phased out once the CMake build enviroment has reached the desired level of maturity. The change was announced on February 28th on our "internals" developer discussion list.
The purpose of WL#5161 is to simplify the MySQL build system. It is much easier and less error-prone to maintain a unified build system for all platforms than two separate ones.
CMake has been chosen because of several reasons; the worklog description lists a few pro-CMake arguments (slightly rephrased):
CMake works on Windows. The GNU buildsystem does not really work and likely never will work natively on Windows (Using Cygwin is not really an option).
Traditionally, new MySQL features that required changes in the build environment (e.g. the plugin system, unit tests, most recently googletest integration) were always implemented on Unix first, leaving Windows behind (sometimes for years). This would not happen with a unified build system.
MySQL already uses CMake since 2006 on Windows, so we do not need to start from scratch, only port what we have to Unix.
CMake runs on every OS and compiler we support.
It is simple to obtain and install on a wide range of platforms. It is available in all major Linux package repositories (e.g. Ubuntu, Fedora, OpenSUSE). It is also in the OpenSolaris repository, known as SUNWCmake. It's in FreeBSD ports and available for Mac OS X. It is also very simple to compile it from source, the single prerequisite is a working C++ compiler and make utility.
CMake has support for features we need and might need, e.g. system checks or cross-compiling.
CMake provides integrated support for packaging. It can handle both simple packages (tar.gz or zip archives) and more complex things like DEB and RPM without much extra coding.
Good integration with the popular IDEs (Visual Studio, Xcode, Eclipse CDT, KDevelop). Developing in an IDE makes the development process more enjoyable, and potentially it lowers the barrier for external contributors. Of course, CMake can generate traditional Unix Makefiles, which appear to be are superior to the ones generated by GNU autotools (for example, they have progress indicators, colored output and working dependencies).
The scripting language used by CMake is simpler than m4 used by autotools.
CMake is a single small tool, not a bunch of different tools as in GNU system (autoconf, autoheader, automake, libtool)
I'd like to mention a few additional reasons:
Out-of-source builds – CMake can separate the build directory from the source directory. This is convenient, as your working source tree is not cluttered with object files and other fragments of the build process.
Build configuration using a GUI. The cmake-gui package (based on Nokia/TrollTech's Qt library) provides a convenient way of enabling and configuring the various available build options. This is much better than having to memorize all the required defines and configuration flags.
Integrated support for creating a wide range of package formats.
The CMake Wiki lists a number of other "nice to have" features.
From a developer perspective, I hope that it will make it much easier to finally implement two things that many developers working with MySQL have been waiting for (now that the build code has been cleaned up):
Compiling the embedded MySQL Server (libmysqld) as a shared library
Building MySQL with CMake is quite simple and straighforward – the process is outlined on the MySQL Forge Wiki. The document is still work in progress and we'd like to encourage you to take a look at it, try to follow the steps and update/improve the Wiki page, if needed! Your feedback on the build process is appreciated. Feel free to join our internals mailing list to discuss your impressions and observations or submit a bug report via the Bug Database. It's likely that the build still has a few rough edges that we'd like to fix quickly (e.g. BUG#51502 – a fix for this one is already commited to the mysql-next-mr-bugfixing source tree and will be merged into the mysql-next-mr trunk soon).
Last week, my colleagues Giuseppe, Kai and myself attended the SAPO Codebits event in Lisbon, Portugal. Codebits is an annual, invite-only hacking event, which went on for three days. The venue they chose this year was the "Cordoaria", a former rope factory located in the Belém district, close to the 25 de Abril Bridge (which is an impressive sight!). I have been told that the Cordoaria is the longest building in Portugal and I have no doubts about that! The building is so long that the crew used bicycles to get from one end to the other. I've taken a number of pictures from the event as well as from Lisbon itself, you can find them in this flickr set.
The organizers described this year's event as follows:
3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience.
The Codebits session schedule was filled with amazing and interesting talks in four parallel tracks. Sometimes it was hard to choose – some other talks I attended and enjoyed:
Walter gave a lockpicking workshop after his presentation, which I attended as well. I was quite impressed (and a bit shocked) to find out how easy many locks can be opened this way! Later that evening there even was a live band named "Pornophonique" playing (one guy with a guitar, the other one using an Nintendo Game Boy for making music), but I missed that show as I was too busy opening more locks... Fortunately the concert and most of the sessions were recorded on video (in excellent quality) and are already available from the SAPO video pages. Kudos for this speedy service!
But this just matches my overall conclusion of this event: very well organized, great speakers and venue. Thanks to the organizers for having us, we really enjoyed our stay!
This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!
Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.
So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!
Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.
This time of the year is usually a very busy one, as there are plenty of events and conferences to attend. Just take a look at our calendar of OSS events on the MySQL Forge to see what I mean! Here's a quick summary of the ones that I will attend and speak at until the end of this year:
On November 14-15, I'll attend the openSQL Camp in Portland (OR), USA. I missed the first one that took place in Charlottesville (VA) in 2008, but had a lot of fun organizing the European Edition earlier this year. The upcoming one will be more like an unconference again - the list of proposed sessions looks very interesting and the attendee list reads like a "who is who" list of the OSS database community.
On December 3-5, I'll be joining Giuseppe at SAPO Codebits in Lisbon, Portugal, which is going to be a very cool event: "3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience". I will be talking about my favorite topic of MySQL High Availability (I'm currently working on revising my slides based on several excellent discussions about MySQL HA that happened on Planet MySQL in the past weeks) and about the benefits (both social and technical) of using a distributed revision control system (DVCS) like bazaar, git or mercurial for your open source project.
Shortly after Codebits, I will attend SLAC 09, the "Secure Linux Administration Conference" in Berlin, Germany (December 10-11), where I will give two MySQL-related talks (in German) - my usual suspects, but in revised and extended form: MySQL High Availability solutions and MySQL Backup & Security best practices.
FOSDEM, the Free and Opensource Conference, will again take place in Brussels, Belgium on Saturday and Sunday (6th and 7th February, 2010). Now happening for the 10th time (congratulations!), it is one of the largest Open Source conferences in Europe, with a strong focus on developers. Sun/MySQL have been regular sponsors of and contributors to the event in the past and it is alway a great experience to be there. It's very rare to meet so many well-known and bright people from such a wide range of OSS projects.
As for the last conference, we plan to apply for a MySQL developer room (and maybe a project stand, anybody interested to join? Please contact me!). However, the web site currently states that they are "slightly reworking the concept of developer rooms", so it remains to be seen what this will turn into. In any case, we will set up our own call for papers, once the developer room allocations have been finalized (and we were lucky). I was initially thinking about running another OpenSQL Camp, but it's probably too short after the upcoming one in Portland...
If you have an interesting talk about a MySQL-related subject, consider submitting your proposal now! We will gladly review and comment on your proposal in advance, if you would like to get our input or need suggestions about topics! Thanks.
With IntelliJ now being available under an Open Source license, developers have another option to choose from when it comes to Java-based IDEs/Frameworks (Eclipse and NetBeans being the other two prominent ones). Choice is always good, and being an Open Source enthusiast, I of course welcome JetBrain's move!
However, as I'm not really a heavy GUI-based IDE user myself, I can't really comment on which one is the best. These kind of discussions tend to turn into a Holy War anyway... In the end it's likely that each of them gets the job done and you have to come to your own conclusions, based on your personal preference and requirements.
I personally would be interested in seeing how their support for PHP or Python compares to the one in NetBeans. Their plugin repository lists more that 560 plugins, including many for database connectivity/modeling/navigation (incl. support for MySQL). I'm also glad to see that they have a plugin for Bazaar, something that I'm desperately missing from NetBeans!
Interestingly, they decided to keep a few parts proprietary, it's going to be interesting to see how this will turn out for them and if developers will be willing to pay for these extra features, considering that most of this is available for free from the other two projects.
Their Contributor License Agreement looks like it has been derived from the Sun Contributor Agreement (SCA), which is always nice to see. I assume this can be attributed to Roman Strobl - I was positively surprised to notice that he joined their team as a technology evangelist in June! Roman did a great job in spreading the NetBeans and OpenSolaris gospel at Sun before and I briefly met him at this year's FOSDEM conference in Brussels. Congratulations!
The summer break seems to be over and the event season is heating up again! There is a number of conferences and events coming up in the next months — here is a quick summary of the events that I plan to attend.
This Friday I will attend an event here in Hamburg: the "Silpion Sommerfest", organized by Silpion (a local IT solutions provider which is a partner of Sun Microsystems as well). I will be there to network and talk about MySQL.
This coming weekend (2009-09-12/2009-09-13), there will be the PHP Unconference here in Hamburg, Germany . It will consist of two days of Barcamp-style sessions about PHP. Sun/MySQL are sponsors of the event and I expect several of my team mates to be there as well. With more than 180 participants, the event is already sold out.
The following week I will be attending the openSUSE Conference in Nuremberg, Germany on 2009-09-17/2009-09-20. I will give the opening keynote on Thursday morning. Titled "Working in a Virtual Community", I will talk about the pros and cons of working in a virtual organization, giving an overview about some of the technical and social aspects that play a role in working with virtual communities.
On December 10th and 11th I will be attending the 4. Secure Linux Administration Conference 2009 (SLAC) in Berlin, Germany. I've been invited to talk about MySQL and will give two sessions about MySQL Backup & Security as well as MySQL High Availability Solutions. The Call for Papers for this event is still open, so if you have a technical, "best practice" talk that might be relevant for system administrators, consider submitting your proposal!
I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).
From the ChangeLog:
Deleted sample precleanup.pm hook as it has potential to cause harm and is too specialized on a particular use case (BUG#394668)
Fixed InnoDB recovery in case a relative path to the MySQL data directory is defined (BUG#38337), improved the documentation of relpath in the man page.
It's almost two weeks now since FrOSCon and the OpenSQL Camp subconference have taken place in Sankt Augustin, Germany — about time for a summary and update from my side!
First off, I would like to thank all of the participants and supporters, particularly my colleagues Regina Steyer and Iris Musiol for the perfect logistics and co-sponsoring as well as Uli Graef, Thorsten Frueauf, Matthias Schmidt, Alexander Rubin and Joerg Moellenkamp for manning the Sun booth and the help on site.
Another big Thank You goes out to my team mates Giuseppe and Colin as well as to Sheeri K. Cabral, who were a big help in keeping the OpenSQL Camp on track and by supporting the event by giving talks. In addition to that, Sheeri recorded most of the OpenSQL Camp sessions on video and published them in record time!
So here's a quick summary of both events from my side, starting with the main conference.
Sun was a Gold sponsor of the event and we had a booth right at the main entrance area; it could hardly be missed. It consisted of two large and two small desks as well as a divider behind them. For demos, we had a (slightly noisy) Sun Fire X2200 M2 Server and four SunRay 2 Thin Clients (which by themselves triggered a lot of questions and curiosity by many visitors). The booth was flanked by rollup-banners on both sides as well as various posters attached to the divider. Here's a picture of our booth before the event opened:
We demoed Open Solaris, Open HA Cluster, NetBeans/Java and MySQL. We also had a lot of brochures about various products, OpenSolaris 2009.06 Live-CDs incl. booklets as well as some MySQL-T-Shirts to hand out. We distributed over 300 CDs and received a lot of positive feedback about the distribution.
We also had a number of talks in the main conference track (both German and English):
The comments and ratings of these sessions were generally very positive. Our booth was well attended, especially during the session breaks. In total, there were over 1.400 visitors at the conference over the two days.
I personally did not attend many sessions in the main conference tracks, as I was too occupied with the OpenSQL Camp and the booth organization. However, I managed to listen to Uli Graef's talk, which was a very technical and interesting session about ZFS features and internals. Being a big fan of ZFS myself, this was a very worthwhile session to be at and my impression was that it encouraged others to take a closer look at this truly amazing file system.
The second talk I attended was Sunday's keynote by Dries Buytaert from the Drupal project about "The Secrets of Building and Participating in Open Source
Communities". Dries is a great speaker with visually stunning slides. He is funny, too — if you have a moment, you should watch the video recording of his keynote. An uncut "pre-release" version of his talk is already available as an OGG Video file.
As for previous FrOSCons (is that the proper plural?), there was a social event scheduled for Saturday evening, providing barbecue (Steaks and Sausages as well as vegetarian dishes) and drinks. This event usually takes place outside and is always an excellent opportunity for networking and talking with key people from other OSS communities and projects. And there was plenty of time for talking - the queues for the grilled food were long...
Here is a list of other blogs and articles about FrOSCon that are worth a read (in no particular order and both German and English):
In addition to the main conference tracks, FrOSCon also provided a number of so-called "Developer Rooms" to OSS projects, so that they could organize sub-conferences or hackfests of their own. We applied for a room to set up a conference dubbed "OpenSQL Camp", related to the topic of Open Source databases, which was approved.
We then sent out a call for papers and invited people from the many OSS database communities to join us and talk about their projects. Every session proposal was published on the OpenSQL Camp web site and people were able to vote on the sessions they were most interested in via email or twitter:
The organization and scheduling of the talks and speakers was done via the FrOSCon conference system (Pentabarf), which made it very easy to perform this task and also made sure that the OpenSQL Camp sessions were included in the main conference program. Below is a full list of sessions at our subconference (see the FrOSCon Program page for abstracts, speaker info, links and slides). We had two cancellations by speakers on short notice, but were able to cover the gaps with ad-hoc presentations. I'd like to send a special thanks to Geert Vanderkelen, who gave a great presentation about MySQL Cluster despite the very short notice and some technical difficulties at the beginning!
Most talks attracted between 20-50 attendees and we had a great mix of topics from several different database projects (with a slight majority of MySQL-related talks). The Panel Discussion (moderated by me), called the "OSS Toolshed Shootout" went quite well and the speakers had a good time answering questions on various topics about their projects. Thanks again to all OpenSQL Camp speakers for making this event a success!
All in all I think that both FrOSCon and OpenSQL Camp were well worth supporting and attending - we were able to provide insight and trigger some interesting discussions among the OSS enthusiasts and developers in the audience. It was also a good opportunity in get in touch with many people of other OSS communities, fostering the MySQL (and other Sun OSS projects) ecosystem.
Here is a Flickr slide show of my own pictures - more photos can be found in the FrOSCon Gallery and the links page on the Wiki.
I personally look forward to next year's FrOSCon - a Big Thanks to the organizers for another great event!
IPC 2009, the International PHP Conference will take place on November 15th-18th in Karlsruhe, Germany. While the deadline for the call for papers for the main conference program has already passed, there is still an opportunity to submit MySQL-related content: the organizers plan to have a special MySQL Day, which will take place on Tuesday, 17th of November.
Quoting for the Call for Papers web page:
We are looking for speakers joining the Architecture Day or the MySQL Day. "Architecture", in terms of project organization, business organization, tools & approaches etc. is becoming a key qualification to developers and teams. And MySQL still is one of the most common open source databases used in many of todays leading web applications. The International PHP Conference will provide attendees with professional and up to date information on both MySQL and Architecture in the world of PHP.
We would like to encourage you to support and contribute to this event by submitting interesting talks related to MySQL and PHP! If you have any questions or comments about this MySQL Day, feel free to get in touch with Robert Lippert or Björn Schotte directly. Thank you!
Some time ago I posted a compilation of applications and programming languages that provide an API to connect to the MySQL Server. As it turned out, I forgot a few that I would like to mention here:
Apache DBD API: a MySQL driver for mod_apr_dbd is not included in the official distribution, but can be obtained seperately from here. Some distributions (e.g. openSUSE) actually provide installable packages of this driver module.
GRASS MySQL driver: The MySQL database driver in GRASS enables GRASS applications to store vector attributes in a MySQL server. Alternatively, data can be stored in an embedded version of the MySQL Server.
HDBC: HDBC provides an abstraction layer between Haskell programs and SQL relational databases. This lets you write database code once, in Haskell, and have it work with any number of backend SQL databases.
libdrizzle: This is the client and protocol library for Drizzle and MySQL that provides both client and server interfaces. A number of APIs (e.g. Java, Lua, Perl and Ruby as well as Python) have been built on top of it, but I'm not quite sure if they all support the MySQL client/server protocol as well.
MySQL Connector/Python: MySQL Connector/Python is implementing the MySQL Client/Server protocol completely in Python. This means you don't have to compile anything - the MySQL client library doesn't even have to be installed on your system.
Do you know of any other MySQL Connectors/APIs? Let me know!
I'm happy to announce that the schedule for OpenSQL Camp 2009 (European Edition) has been published on the FrOSCon timetable now. We have a great selection of topics and speakers, so don't miss it! OpenSQL Camp is a subconference of FrOSCon, the Free and Open Source Conference, which will take place on August 22nd and 23rd in St. Augustin, Germany.
The admission fee for the entire conference (both days, incl. OpenSQL Camp) is 5 EUR, you can pre-register here until August 10th (and if you do so today, you will still get a free T-Shirt as well!). Of course, you can also just show up at the entrance and pay the entrance fee on site. The OpenSQL Camp will be located in Room C120 - see the instructions on the FrOSCon web page on how to get there and where to find accommodation.
In case you can't make it to Germany for the European Edition, Eric Day and Selena Deckelmann have started to organize another OpenSQL Camp in Portland, Oregon which will take place on November 14th and 15th. More details can be found on the OpenSQL Camp Wiki. Space is limited to 120 attendees, so sign up today and reserve your spot!
I'm happy to announce that my talk "Making MySQL administration a breeze - a look into a MySQL DBA's toolchest" has been accepted for this year's edition of the MySQL Conference & Expo in Santa Clara, which will take place on April 12-15, 2010. The session is currently scheduled for Wednesday 14th, 10:50 in Ballroom E.
On December 3-5, I'll be joining 
