Lenz Grimmer's blog

CMake is a cross-platform, open-source build system, maintained by Kitware, Inc.

From the CMake.org home page:

CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice.

It has been used for building the MySQL Server on Windows since MySQL 5.0 – the initial CMake build support was added in August 2006.

For building MySQL on all other platforms, the GNU autotools (autoconf, automake and libtool) are currently being used.

CMake is used in some other MySQL projects as well, e.g.

• MySQL Connector/C
• MySQL Connector/C++
• MySQL Proxy

On February 22nd, Vladislav Vaintroub pushed the changes required to implement WorkLog#5161 "CMake-based unified build system" into the "mysql-next-mr" branch (aka the "Celosia" mile stone).

From this version on, CMake can also be used to build MySQL on Linux and other Unix platforms. For the time being, the autoconf/automake files are still available as well, but will be phased out once the CMake build enviroment has reached the desired level of maturity. The change was announced on February 28th on our "internals" developer discussion list.

The purpose of WL#5161 is to simplify the MySQL build system. It is much easier and less error-prone to maintain a unified build system for all platforms than two separate ones.

CMake has been chosen because of several reasons; the worklog description lists a few pro-CMake arguments (slightly rephrased):

• CMake works on Windows. The GNU buildsystem does not really work and likely never will work natively on Windows (Using Cygwin is not really an option).
• Traditionally, new MySQL features that required changes in the build environment (e.g. the plugin system, unit tests, most recently googletest integration) were always implemented on Unix first, leaving Windows behind (sometimes for years). This would not happen with a unified build system.
• MySQL already uses CMake since 2006 on Windows, so we do not need to start from scratch, only port what we have to Unix.
• CMake runs on every OS and compiler we support.
• It is simple to obtain and install on a wide range of platforms. It is available in all major Linux package repositories (e.g. Ubuntu, Fedora, OpenSUSE). It is also in the OpenSolaris repository, known as SUNWCmake. It's in FreeBSD ports and available for Mac OS X. It is also very simple to compile it from source, the single prerequisite is a working C++ compiler and make utility.
• CMake has support for features we need and might need, e.g. system checks or cross-compiling.
• CMake provides integrated support for packaging. It can handle both simple packages (tar.gz or zip archives) and more complex things like DEB and RPM without much extra coding.
• Good integration with the popular IDEs (Visual Studio, Xcode, Eclipse CDT, KDevelop). Developing in an IDE makes the development process more enjoyable, and potentially it lowers the barrier for external contributors. Of course, CMake can generate traditional Unix Makefiles, which appear to be are superior to the ones generated by GNU autotools (for example, they have progress indicators, colored output and working dependencies).
• The scripting language used by CMake is simpler than m4 used by autotools.
• CMake is a single small tool, not a bunch of different tools as in GNU system (autoconf, autoheader, automake, libtool)

I'd like to mention a few additional reasons:

• Out-of-source builds – CMake can separate the build directory from the source directory. This is convenient, as your working source tree is not cluttered with object files and other fragments of the build process.
• Build configuration using a GUI. The cmake-gui package (based on Nokia/TrollTech's Qt library) provides a convenient way of enabling and configuring the various available build options. This is much better than having to memorize all the required defines and configuration flags.
• Integrated support for creating a wide range of package formats.

The CMake Wiki lists a number of other "nice to have" features.

From a developer perspective, I hope that it will make it much easier to finally implement two things that many developers working with MySQL have been waiting for (now that the build code has been cleaned up):

• Compiling the embedded MySQL Server (libmysqld) as a shared library
• Better support for cross-compilation

Building MySQL with CMake is quite simple and straighforward – the process is outlined on the MySQL Forge Wiki. The document is still work in progress and we'd like to encourage you to take a look at it, try to follow the steps and update/improve the Wiki page, if needed! Your feedback on the build process is appreciated. Feel free to join our internals mailing list to discuss your impressions and observations or submit a bug report via the Bug Database. It's likely that the build still has a few rough edges that we'd like to fix quickly (e.g. BUG#51502 – a fix for this one is already commited to the mysql-next-mr-bugfixing source tree and will be merged into the mysql-next-mr trunk soon).

If you're new to CMake, you might want to take a look at the "Getting Started With CMake (An End-User's Perspective) For Cross-Platform Building" screencast or the "Running CMake" article.

Happy hacking!

We've now published a new set of binary packages including the extended GIS functionality from the mysql-5.1-wl1326 source tree.

This release is based on the MySQL 5.1.35 code base and fixes the bugs mentioned below. It includes some improvements to the GIS functionality as well, so please use these packages for future testing of the MySQL GIS functionality.

The following GIS-related bugs were fixed in this version:

• Bug#31753: Buffer/area functions only return first row of set
• Bug#32032: Contains() does not work on MultiPolygons, may force a disconnect and/or result in extremely long query times.
• Bug#32100: contains, intersects functions never return. Query disconnects or times out.
• Bug#33035: Intersection function returns 'Function doesn't exist' error
• Bug#40874: Buffer function doesn't work with LINESTRING
• Bug#41481: Buffer function never returns, cpu 100%, system locks up
• Bug#43493: union function returns polygon not multipolygon for non-intersecting polygons
• Bug#44753: nan error in union function

I'd like to explicitly thank John Powell for his extensive testing and bug reporting so far, he helped us quite a lot to improve the quality of the GIS code.

Please help us by by testing this new release and reporting bugs at our public bug database. For convenience, please tag your Bug reports with "gis" and make sure to put them in the "Server: GIS" category!

Giuseppe stumbled over this website and recommended it to me as a potential useful service for organizing the registrations of the OpenSQL Camp: Eventbrite is a Python/MySQL-powered web site (judging from their job openings) that provides the following:

Eventbrite is the leading provider of online event management and ticketing services. Eventbrite makes it easy for anyone to hold a successful event of any type and size. Eventbrite is free if your event is free. If you sell tickets to your event, Eventbrite collects a small fee per ticket. So just like you, Eventbrite wants your event to be a big success.

The Eventbrite service includes many features and tools intended to let you perform three basic tasks really well:

1. Publish: Everything you need to create and easily personalize a custom web page for your event.
2. Promote: The tools you need to spread the word about your event and maximize attendance.
3. Sell: The immediate power to sell tickets and collect money online.

I won't use it for the OpenSQL Camp (as we will be able to piggyback on FrOSCon's event infrastructure), but it seems like a service worthwhile checking out, if you're looking for a way to organize your next MySQL meetup (and you're based in the US - currently Eventbrite seems to assume USD as the only available currency). I've added a note about it to the suggestions on the how to run a MySQL User Group page on the MySQL Forge Wiki. And if you do use it, please make sure to add your group to the list of MySQL User Groups! Thanks.

Searching the service for "MySQL" revealed that the MySQL Dublin Meetup actually uses it for organizing their upcoming meeting (June 24, but already sold out)!

As you may have heard, we're switching to a new release model with the upcoming MySQL 5.4 release.

If you are curious to learn more about what will change in the way in which future versions MySQL will be developed and released, make sure to attend our next MySQL University session about The New MySQL Release Model on Thursday, 11th of June, 14:00 UTC. Tomas Ulin, our director of MySQL server development will go through the planned changes and would also like to get your input and feedback on these changes.

We're using DimDim for broadcasting this session, which allows you to listen to the audio while watching the slides with your web browser. You can comment and discuss via a chat function, too! We're looking forward to your input. To attend, point your browser to this address (Adobe flash player required).

The session will be recorded and posted on the MySQL Forge Wiki, so you can watch the presentation later as well. You can also provide your feedback on the release model by posting on the MySQL Internals mailinglist.

The MySQL Conference & Expo 2009 in Santa Clara is now in full swing and Karen Padir just gave the opening keynote, talking about Sun's continued and improved commitment to Open Source and the upcoming MySQL products like the MySQL 5.4 performance release or MySQL Cluster 7.0. One of the activities that she mentioned in her keynote is our ongoing activity to improve the acceptance and incorporation of patches contributed by the community.

We've scheduled a BoF about this topic tonight (7:30pm in Ballroom A), where we would like to talk about the recent changes that we've made and discuss a new way in how to produce future releases of the MySQL Server on a shorter and more predictable schedule. We've invited Tomas Ulin (Director of MySQL Server) to join us and explain the proposed changes to the MySQL release model and how they will help us to incorporate patch contributions and make them available to the community at a faster rate.

Please join us and let us know what you think of these changes and what else we can do to make it easier and attractive to contribute patches to the MySQL Server! There will be free T-Shirts as well

I had a nice chat with Kieran from Acquia at DrupalCon last week - we discussed how people running local Drupal user groups could expand their outreach into other communities, in particular into the MySQL User Groups. Scott Mattoon captured our conversation on video, which is now available on blip.tv:

Click To Play

The gist of what we talked about: if you are organizing a local Drupal User Group Meetup, check out http://mysql.meetup.com to find out if there is a local MySQL user group nearby. Chances are high that there is! And if not, you may find at least people in the area that would be interested in meeting about this subject. We also maintain list of user groups on the MySQL Forge Wiki. Consider extending your invitation for your next meetup to these folks as well! It's very likely that someone would be interested to learn more about Drupal. The same applies to other user groups, e.g. from the PHP community.

 

I personally run a MySQL User Group here in Hamburg, and I usually extend my invitations to a number of channels and mailinglists, including the local PHP, Perl and Linux User Groups. Every once in a while, a new member from these communities shows up.

So this thing works the other way around, too: if you are the organizer of a MySQL Meetup, have you thought about looking at http://groups.drupal.org/ yet? Maybe you will find a Drupal User Group in your very own town that you could invite to learn more about MySQL and exchange contacts? If you are looking for more tips on how to run and expand your User Group, I've created a page with useful hints about this topic on the MySQL Forge Wiki. Your feedback and additions are very welcome!

In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.

However, Matt seems to reduce "hosting" to "providing downloads" only:

Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
[...]
It can tell you many things, but the key is to be able to glean insight from the earliest stage of your interaction with a potential customer, and that means you have to host your own downloads. Otherwise, you have no idea how or when a would-be customer downloads your code, which makes the "why" they download it less interesting, because it becomes less actionable.

I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.

Project hosting is not just about downloads

First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.

But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.

I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.

There is more to distribute than releases

At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.

Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.

Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.

I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.

I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:

Downloads stats are overrated

Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.

Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:

As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.

I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.

 Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).

So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.

 

I recently got a reminder mail to vote for the Sourceforge.Net 2008 Community Choice awards. Going through the list of finalists, I realized how many of these support MySQL as the database backend. It truly amazes me when I look at the wide range of available OSS applications today as well as how advanced many of these have become! More and more commercial applications can nowadays be replaced with Open Source alternatives/equivalents. And many times, MySQL is used to store the applications' data. This is a great trend!

After looking through the list, I spent a few minutes to add the relevant applications to the Project list on MySQL Forge. This section of the Forge is supposed to become a complete, "one-stop" directory of Open Source projects and applications that support or work with MySQL. You can tag, comment and vote on entries as well as just browse through the list. If you have not looked at it yet, make sure to stop by next time you are searching for a particular solution!

Do you have a favourite Open Source application that supports MySQL and is missing from the list? Consider adding it - it only take a few minutes! Thanks for your support - your contribution is appreciated!

 

A bit late in the game, but maybe somebody would be interested in working on this proposed project of mine:

PlanetMySQL currently is merely an aggregator of submitted RSS feeds, with some functionality for filtering content to keep the discussion on topic. Due to its high volume of posts, many gems get "lost in the noise" and are hard to retrieve.

We'd like to expand the functionality of PlanetMySQL significantly to provide more possibilities for community participation and interaction. For this project, we are looking for a talented PHP hacker to set up a site that provides the current functionality and more:

• Voting on articles/blog postings: it must be possible for logged in users to cast a vote on articles, similar to the Perlmonks.org voting system. This would allow providing different RSS feeds, e.g. only articles that have a certain rating and rankings for articles and authors. Positive votes accumulate and increase the "karma" value of the individual author as well as identifying the quality of a particular posting.
• Archiving/Tagging/Searching: it must be possible to add tags to the aggregated blog postings, to ease the searching of older articles and to facilitate the creation up of a searchable "Community Knowledgebase"
• Authentication: User logins must use the same username/password pair as MySQLForge and the rest of the mysql.com web site (to reduce duplication and allow better integration with other parts of MySQL Forge, e.g. the user profile page)
• Optionally, it should be possible to comment/discuss on the aggregated articles directly on the new PlanetMySQL site. These comments should be sent back to the original blog via Trackbacks (if applicable)
• It should be possible to group feeds from multiple, different authors as a "Team Feed", to allow rankings by Team/Group in addition to the ranking of individuals
• The site must provide Unicode support to allow the handling of postings in multiple languages (e.g. Japanese, Chinese)
• New feed submissions should not be subscribed automatically, but rather should be reviewed by a moderation team first (to avoid spam and off-topic feeds)

The implementor should first make an assessment if these new functions should be developed on top of the existing code base, or if it would make sense to rebuild the existing functionality plus the new features on top of an existing PHP Framework (e.g. Drupal, Silverstripe, Symfony or similar). Making use of Web2.0 techniques (AJAX) is encouraged, if it makes the site easier to use and visually appealing.

Please contact me directly (firstname at MySQL.com) or the mailing list, if you are interested in working on this task. Thanks!

 

While helping a user with some questions about the geospatial extensions of MySQL on the #mysql-dev IRC channel on Freenode, I stumbled over this blog: How to use MySQL Spatial Extensions. There currently is just one post, but it was exactly what we were looking for: "Using Circular Area Selection". Nice work! I hope the author will soon provide more examples of how to make use of this functionality.

By the way, there is work in progress to improve the GIS functionality in MySQL - if you are looking for new GIS functions that do not use MBRs, take a look at our GIS feature preview on the MySQL Forge. We plan to publish a new set of preview binaries once the 5.1.23 release has been published and merged into this code base. 

 

Just came off today's MySQL University session about How to Build MySQL on Windows - Reggie did a great job on explaining how to build the MySQL Server from Source on Windows using the Microsoft development toolchain and some additional required tools. I am glad to hear that we're making progress on making it easier for Windows developers to work with the source code and Reggie and the other members of our Windows Task Force (what a nice acronym this one makes!) have plenty of other ideas for improving that experience.

If you missed his session, the audio file and IRC log will be published from the MySQL University pages shortly.

And in case you haven't heard about MySQL University before, check out the pages on the MySQL Forge and take a look at the upcoming schedule! Next week, Jan is going to give an introduction to the MySQL Proxy, which will to be an interesting session for sure. Did I mention that attending MySQL University is free and everybody is welcome to join? All you need is an IRC client to post questions and discuss the topic and an application capable of playing OGG audio streams for listening to the presenter. The MySQL Forge Wiki has detailed instructions on how to get going. See you next week!

Today, I again spent about an hour to revert changes made by spambots on various MySQL Forge Wiki pages. As I was really sick of this, I now installed a new plugin: reCAPTCHA - this will hopefully raise the bar for spambots to create new user accounts automatically for spamming the Wiki. If you are a registered user already, you will probably not notice the change - by default, CAPTCHAs are only displayed on the following events:

• New user registration
• Anonymous edits that contain new external links
• Brute-force password cracking

Let's hope it helps! Please let me know if you still experience spam problems on the Wiki.

Yesterday I sent out the invitations for our third MySQL User Group Meeting in Hamburg, which will take place on Monday, 4th of September (which is the United MySQL Meetup Day). As usual, we will meet at 19:00 in the Chinese Restaurant "Ni Hao" - the food is excellent and we can use a separate room with a video projector there. If you happen to live around Hamburg, Germany and would like to join, please RSVP and join our mailing list!

Currently I am still pondering on the topic of the presentation - I suggested to invite Paul McCullagh to talk about PrimeBase XT, his transactional pluggable storage engine for MySQL. Would you think this is a suitable topic for a user group meeting? Let me know.

Speaking of MySQL User Groups: have you considered establishing one in your area yet? It's actually a lot of fun and MySQL AB will sponsor the meetup.com fees. In particular, I would like to encourage our MySQL Users in India to take initiative - there are more than 300 people interested in meeting other MySQL people in the major Indian cities like Hyderabad, Chennai, Bangalore and others. Step up and organize a meeting! We have collected some suggestions and hints on how to get you going on the MySQL Forge Wiki. Once you are up and running, make sure to add yourself to the growing list of MySQL User Groups!

We used to maintain a list of MySQL User Groups on the MySQL Developer Zone, which made it a bit inconvenient for new User Groups to get listed there. We've now moved the User Group list and some additional info to the Wiki on MySQL Forge (which also has received a facelift, thanks a lot to Colin and Ronald!). If you know of any other MySQL User Groups not listed there, feel free to add them! We are also looking forward to seeing more helpful hints on how to successfully start and maintain a user group. If you have any suggestions, make sure to include them - thanks!