Lenz Grimmer's blog

In case you are processing and working with geospatial data on MySQL, you may be interested in the following UDF (plugin) for MySQL 5.1: Koji Okumura from Oki Labs Japan has ported two functions from PostGIS into a MySQL UDF:

• distance_sphere(point, point): Returns linear distance in meters between two lat/lon points. Uses a spherical earth and radius of 6370986 meters. Faster than distance_spheroid(), but less accurate. Only implemented for points
• distance_spheroid(point, point, spheroid): Returns linear distance between two lat/lon points given a particular spheroid. Currently only implemented for points.

Since it's an UDF, it can be easily added to an already installed server. These two functions actually complement the improved precise GIS functions nicely (which provide a 2-dimensional DISTANCE() function).

You can download the UDF source tarball from here. Compiling it is pretty straightforward and requires a local copy of the MySQL 5.1 sources and the ususal build environment. For my testing, I used the 5.1 source tree that includes the additional precise geospatial functions. Below I just list the commands required to build and create the binary tarball and omit the lengthy output in between some of these commands:

$ bzr branch lp:~mysql/mysql-server/mysql-5.1-wl1326
$ cd mysql-5.1-wl1326
$ BUILD/compile-pentium-max
$ make bin-dist

You should now have a binary tarball mysql-5.1.26-rc-linux-i686.tar.gz that you can install or deploy by using the MySQL Sandbox, which I used for my further testing:

$ tar zxvf mysql-5.1.26-rc-linux-i686.tar.gz -C ~/opt/mysql
$ cd ~/opt/mysql
$ mv mysql-5.1.26-rc-linux-i686 5.1.26
$ make_sandbox 5.1.26
$ cd ~/sandboxes/msb_5_1_26
$ ./use
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.1.26-rc Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql [localhost] {msandbox} ((none)) >

Success! Let's see if the new GIS functions work:

mysql [localhost] {msandbox} ((none)) > SELECT DISTANCE (GEOMFROMTEXT('POINT(0 0)'),GEOMFROMTEXT('POINT(1000 1000)')) AS distance;
+-----------------+
| distance        |
+-----------------+
| 1414.2135623731 |
+-----------------+
1 row in set (0.00 sec)

Now let's compile and install the UDF that provides the new spherical distance functions. Download the sources and extract the tarball in the same directory in which you branched the MySQL server sources:

$ tar zxvf mysql-udf-distance_spheroid-1.0.tar.gz
$ cp distance_spheroid/*.{cc,h} mysql-5.1-wl1326/sql
$ cd !$
$ g++ -DMYSQL_SERVER -shared -o udf_distance_spheroid.so -I../regex -I../sql -I../include spatial.cc calc_distance_spheroid.cc udf_distance_spheroid.cc

This will build the shared object udf_distance_spheroid.so that you now have to copy into the plugins directory of your server:

$ install -D udf_distance_spheroid.so ~/opt/mysql/5.1.26/lib/mysql/plugin

Now we have to load the UDF and make the new functions known to the server. Get back into your sandbox or fire up the commandline client again:

mysql [localhost] {msandbox} ((none)) > CREATE FUNCTION distance_sphere RETURNS REAL SONAME "udf_distance_spheroid.so";
mysql [localhost] {msandbox} ((none)) > CREATE FUNCTION distance_spheroid RETURNS REAL SONAME "udf_distance_spheroid.so";

Now you're ready to experiment with these new functions! For my test case, I calculated the distance between Hamburg, Germany (where I currently live) and Heidelberg, Germany (where I grew up) by providing the Lat/Lon coordinates as points:

mysql [localhost] {msandbox} ((none)) > SELECT DISTANCE_SPHERE(GEOMFROMTEXT('POINT(53.583333 9.983333)'),GEOMFROMTEXT('POINT(49.412222 8.71)')) AS 'Distance (m)';
+-------------------+
| Distance (m)      |
+-------------------+
| 479037.4799112912 |
+-------------------+
1 row in set (0.00 sec)
mysql [localhost] {msandbox} ((none)) > SELECT DISTANCE_SPHEROID(GEOMFROMTEXT('POINT(53.583333 9.983333)'),GEOMFROMTEXT('POINT(49.412222 8.71)'),'SPHEROID["GRS_1980",6378137,298.257222101]') AS 'Distance (m)';
+-------------------+
| Distance (m)      |
+-------------------+
| 479345.1884839106 |
+-------------------+
1 row in set (0.00 sec)

Interestingly, using the virtual ruler on Google Earth gave me a distance of 471881.44 meters for the same coordinates. Not sure where the difference comes from, I assume they are using a slighly different projection system? It would be interesting to see, if these results match what you would get out of PostGIS using the same queries.

In any case, these two functions may come in handy, if your application needs to calculate distances between two points on a map. So give it a try! If you have tested this functionality and want to give feedback to Koji, please reply to his post on the MySQL GIS Forum.

Even though we had been preparing the migration to Bazaar for a while now, today's announcement kind of caught me by surprise. But I am very happy about this move!

While BitKeeper is an excellent tool and served us well the past eight (!) years, I was quite annoyed when BitMover decided to remove the fully functional free BitKeeper client, which effectively put our development back into a Cathedral: even though our source trees remained accessible via bkbits.net, the crippled bk client was only capable of cloning and pulling new revisions from there - it was not possible for an external developer to commit changes locally or to create patches. This turned out to become a severe roadblock for users that wanted to participate in the development of the server. While some of our other internal projects (like the documentation, connectors and GUI tools) moved to Subversion, this was not that easily doable for the MySQL Server source trees. Lots of internal processes and tools had been tightly coupled with the source code management and there never seemed to be the right time to make the switch, which potentially could have disrupted the ongoing development work.

By moving to a truly open, distributed revision control system, it will become much easier for external contributors to work on patches and maintain modifications or enhancements outside of the main MySQL Server source trees. In fact, work has already begun! When I look at the various repositories hosted on Launchpad.net, I already notice several source trees that are not maintained by MySQL Engineers, e.g. Marc Callaghan's Google patches, Percona's work on microsecond resolution or a branch of MySQL 5.0 that is maintained and used by the folks from Wikimedia. I too helped publishing a source tree today - Holyfoot's work on improving the GIS functionality is now also available from there! In addition to the MySQL Server, many other MySQL-related projects are hosted on LaunchPad. And by utilizing services like the fabolous OpenSUSE Build Service, it's actually quite easy to provide readily installable packages of these as well!

I hope this change will significantly lower the barriers for external contributors and make it easier for our own developers to merge and accept changes that have been created by others and received sufficient community testing. This is much more convenient than having to apply a patch that was posted a long time ago and may have already been abandoned by the contributor or suffered from bit-rot. Being able to maintain a patch within a live tree that can be kept in sync with the main repository is one of the great features of distributed revision control (and which I think is one of the technical reasons why the Linux Kernel development model works so well).

Kudos to everybody involved in making this happen!

While helping a user with some questions about the geospatial extensions of MySQL on the #mysql-dev IRC channel on Freenode, I stumbled over this blog: How to use MySQL Spatial Extensions. There currently is just one post, but it was exactly what we were looking for: "Using Circular Area Selection". Nice work! I hope the author will soon provide more examples of how to make use of this functionality.

By the way, there is work in progress to improve the GIS functionality in MySQL - if you are looking for new GIS functions that do not use MBRs, take a look at our GIS feature preview on the MySQL Forge. We plan to publish a new set of preview binaries once the 5.1.23 release has been published and merged into this code base. 

There is a lot of exciting stuff happening inside of MySQL AB. But due to the distributed nature of our company it's hardly possible to get a good overview about what the various teams of our development department are currently working on and what they have achieved since the last time we met.

So one cool new idea for our currently ongoing MySQL Developer Meeting in Heidelberg was to let developers show off their work to each other. They were encouraged to prepare demos, either in the form of slide shows or by running live demonstrations from their laptops. Last Thursday and Saturday we allocated time for these team exhibitions and the exhibitors set up tables in the meeting rooms for others to sit next to them, see the new and cool stuff and chat about it. The non-exhibiting attendees received a sheet of paper where they could collect signatures for each demo point they visited, the one that managed to see the most demos was eligible for winning a price. I managed to visit ~8 of them on Thursday - there was way more stuff to look at than what one could actually get done in the given time frame. Therefore we decided to repeat some of Thursday's exhibitions and included some new ones on Saturday.

Here is a recap of the exhibitions I saw on Thursday:

• Alexey 'Holyfoot' Botchkov demoed the new and improved GIS (geospacial information) functionality that I already blogged about some time ago. It provides a number of new GIS functions and most of the already existing ones no longer use MBRs. Holyfoot actually prepared a visual demo application that displayed a map of the earth by getting the polygons to draw out of the database. It was then highlighting the current country under the mouse cursor, displaying some additional information about it. Moving the mouse across country borders immediately switched the highlighting to the respective state. When using the old, MBR-based functions this would only happen when you crossed the borders of the bounding rectangle.
• My fellow colleague Giuseppe Maxia performed some MySQL Proxy Magic, misusing the MySQL Proxy for some fun stuff and also demonstrating some useful use cases (e.g. crosstabs). See his blog entry for some examples of what he was showing.
• Jay Pipes demoed his work on MySQL Forge 2.0, now with more Ajax magic, rating and commenting on projects and snippets as well as the obligatory rounded corners! There's still some work to be done, but I look forward to seeing it go live soon.
• Monty Tailor explained his work on NDB/Connectors, which provides bindings to many programming languages like Ruby, Perl, Lua, PHP and others. They make it possible to connect to a MySQL Cluster without having to go through the MySQL API, if you don't need to use SQL and just want to use MySQL Cluster as a highly available datastore (similar to memcached). It was interesting to compare the various test scripts for each language with each other - the Perl version looked quite ugly in comparison to the Ruby code...
• An alpha version of the MySQL Workbench was demoed by Johannes Taxacher from the GUI team. He demonstrated how one can visually create tables and set up the relations between them. The GUI is quite nice, but they support Windows only for now - Mac OS X and Linux are also planned, however.
• Kent Boortz from the build team showed off his work on creating a self-contained ODBC build environment. I was quite impressed by the complexity and steps involved in creating the binary packages of the ODBC driver we provide. As we support quite a number of platforms, Kent had to come up with all kinds of interesting tweaks for exotic architectures and strange compilers.
• Jess Balint from the Connectors team demoed the ODBC 5.1 driver by connecting to a MySQL Server using MS Access or with a MS Visual C++ application. It was nice to observe the functional UTF-8 support.
• Paul DuBois, long time member of our documentation team explained the build system they created in order to cope with the massive amount of build targets that are being created from the DocBook sources of the reference manual. In addition to various formats of the manual itself (e.g. PDF, HTML, Windows Help), they also build the manual pages that are included in our *nix distributions as well as some README files and the Changelogs. It is written in Ruby and is capable of distributing the build jobs to several machines.

That's all I was able to see on Thursday. I just wish I had some more time to see all of the other demos. On Saturday I saw the following demos:

• Marc Alff was demoing LTP-GCOV, a tool that visualizes the output of gcov as HTML pages and allows one to find out which code has been covered during the test suite run. It would be great if we could combine it with our internal pushbuild system, which automatically rebuilds the MySQL Server and runs the test suite on a number of platforms for every ChangeSet that is being pushed into a source tree. We already perform gcov build runs to make sure that new code is properly covered by the tests, but this tool would be a very helpful addition, allowing us to find code with low coverage much easier.
• Kristian Nielsen showed an early implementation of a multi-threaded ndbd, the process that runs on a MySQL Cluster data node. This should boost performance on multi-core systems (the current implementation is single-threaded). However, it's still in an early prototype phase, but looks interesting!
• Chad Miller demonstrated an alternative revision control system: bzr. Similar to BitKeeper, it is a fully distributed system which does not require a central server. Creating a local branch of an existing MySQL code tree was quite easy and it seems to be very similar in using it. What impressed me in particular was its capability of connecting to remote SVN repositories - it is actually possible to use bzr locally to work against a remote Subversion server! This should give us a good opportunity to start testing it against our existing SVN trees and gather some more practical experience with it.

All in all I was very impressed and excited to learn about all the cool things that are going on inside of the development team. Even though there is always a lot of work to be done and they're busy getting the MySQL 5.1 release out the door, they still find time to experiment and come up with great ideas and improvements in other areas as well. I got the impression that everybody really enjoyed being able to show off his work and also receiving good feedback from the others that were passing by. I hope we will repeat these team exhibitions at our next meeting!

While MySQL already provides some functionality to store and operate on geospatial data, the functionality leaves quite a lot to be desired and is far from providing full OpenGIS compatibility. Most notably is that all functions that query spatial data only operate on MBRs (minimum bounding rectangles), to simplify the operations.

Thanks to my colleague Alexey "Holyfoot" Botchkov from Izhevsk, Russia, some of the spatial relation functions like INTERSECTS and WITHIN now work in the way they are described by OpenGIS and not by using MBR's as it used to be. He has been working on improving the GIS functionality as a side project and the work has now reached a level at which he is ready to give it some public testing and solicit feedback about it.

Currently these new features have not been slated for inclusion in any upcoming MySQL major release. But the more feedback and testing this code receives, the faster it will reach a maturity level that makes it a potential feature candidate for new versions of MySQL.

In addition to improving already existing functionality, Holyfoot also implemented some new precise geospatial functions - the following functions are available now and use precise operations instead of MBRs:

• BUFFER(g1 geometry, d numeric) returns a Geometry defined by buffering a distance d around g1 where d is the distance units for the Spatial Reference of g1
• DIFFERENCE(g1 geometry, g2 geometry) returns a Geometry that is the closure of the set difference of g1 and g2
• DISTANCE(g1 geometry, g2 geometry) returns distance between g1 and g2
• INTERSECTION(g1 geometry, g2 geometry) returns a Geometry that is the set intersection of g1 and g2
• SYM_DIFFERENCE(g1 geometry, g2 geometry) returns a Geometry that is the closure of the set symmetric difference of g1 and g2 (logical XOR of space)
• UNION(g1 geometry, g2 geometry): returns a Geometry that is the set union of g1 and g2

The source tree (based on the MySQL 5.1 code base) is now available from our public BitKeeper trees at http://mysql.bkbits.net/ - please consult the reference manual for more information on how to build a MySQL binary from a source tree.

To discuss the new GIS functionality, please join our GIS Forum and make sure to report bugs at our public Bug Database!