Entries tagged as subversion
Monday, December 7. 2009
Last week, my colleagues Giuseppe, Kai and myself attended the SAPO Codebits event in Lisbon, Portugal. Codebits is an annual, invite-only hacking event, which went on for three days. The venue they chose this year was the "Cordoaria", a former rope factory located in the Belém district, close to the 25 de Abril Bridge (which is an impressive sight!). I have been told that the Cordoaria is the longest building in Portugal and I have no doubts about that! The building is so long that the crew used bicycles to get from one end to the other. I've taken a number of pictures from the event as well as from Lisbon itself, you can find them in this flickr set.
The organizers described this year's event as follows:
3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience.
I wholeheartedly agree, we had a great time! The conference started with sessions and presentations on a wide range of topics on the first two days. Afterwards, a 24-hour programming contest was held. I was invited to give two talks, one being my all-time favourite about "MySQL High Availability solutions" (slides, video), the other one was titled "Why you should be using a distributed version control system (DVCS) for your project" (video, slides). Both went quite well and the feedback I received was pretty positive. Giuseppe talked about "MySQL Schema Migration" (slides, video) and gave an "Introduction to Gearman" (video). Kai's talk was titled "Think before you develop" (video) and gave a nice roundup of tips and best practices for setting up and developing new web projects.
The Codebits session schedule was filled with amazing and interesting talks in four parallel tracks. Sometimes it was hard to choose – some other talks I attended and enjoyed:
Walter gave a lockpicking workshop after his presentation, which I attended as well. I was quite impressed (and a bit shocked) to find out how easy many locks can be opened this way! Later that evening there even was a live band named "Pornophonique" playing (one guy with a guitar, the other one using an Nintendo Game Boy for making music), but I missed that show as I was too busy opening more locks... Fortunately the concert and most of the sessions were recorded on video (in excellent quality) and are already available from the SAPO video pages. Kudos for this speedy service!
But this just matches my overall conclusion of this event: very well organized, great speakers and venue. Thanks to the organizers for having us, we really enjoyed our stay!
Thursday, November 5. 2009
This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!
Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.
Continue reading "Aspects and benefits of distributed version control systems (DVCS)"
Thursday, September 11. 2008
 While Colin beat me in blogging about Project Kenai, I think I can still provide some additional background information about this new project hosting service from Sun.
If you are a maintainer of an Open Source project, you currently have plenty of choice when it comes to getting your project hosted for free. One criterion could be your software configuration management system (SCM) of choice.
Some of the hosting services that I am currently aware of and the choice of SCM they offer include:
As disclosed by Tim Bray some days ago, there now is another option - Kenai is open for project hosting (currently by invitation only)! In his blog post, he interviews Nick Sieger, one of the developers behind this project about their motivation and intentions:
We need to demonstrate credibility in building on top of more traditional LAMP/SAMP web stacks (not just Java EE); and we need to show viability of Sun technologies and hardware for next-generation web applications.
In a nutshell, Kenai is a platform for:
- Developer collaboration
- Communities of connected developers
- Integrated collaboration services stack
Some of the features that are currently available include:
- SCM services using Subversion and Mercurial
- Bug Tracking (Bugzilla)
- Forums
- Wikis
- Mailing Lists (using Sympa)
Reading the interview with Nick and looking at some presentations slides for RailsConf from Fernando Castano (a jRuby and Database performance engineer at Sun and another member of the project team), I was able to gather a list of the tools and technologies they used to build Kenai:
I found it interesting that they decided to deploy and run the Rails application as a war file within the Glassfish application server (using Warbler). By the way, the fabolous OpenSUSE Build Service is a Rails application, too! So far, the entire site is powered by a single MySQL instance with query cache enabled.
The project is hosted on the following infrastructure:
You should check out Fernando's presentation for more technical details, tuning info and how they benchmarked the setup - it contains a number of useful tuning hints and performance graphs.
Last time I checked, 27 Projects have joined so far (e.g. jRuby, xVM Server). Kenai itself is developed on Kenai. It's going to be interesting what other projects will find their home there.
Nick also talked a bit about their future near term plans: to improve the usability and feature set, incrementally improve the site navigation and layout and adding support for hosting files/release downloads. They also consider offering Jira as an option to Bugzilla for bug tracking and Git as another SCM option.
There is an IRC channel #projectkenai on freenode.net, to get in touch with the developers directly. The mailing list for the Project Kenai site itself, is users@help.kenai.com - you can subscribe to this list here.
Tuesday, August 5. 2008
Today, the root file system on our public svn server nearly ran out of disk space. The reason? The /tmp directory was quickly filling up with temporary files created by websvn, which I set up parallel to the FishEye repository browser for testing purposes. A quick investigation of the apache log files revealed the culprit - a crawler from Microsoft was running haywire and decided to ignore the rules in the robots.txt file, even though it did actually looked at the file before!
Here is how robots.txt looked like (I now changed it to disallow everything):
User-agent: *
Disallow: /fisheye/
Disallow: /websvn/
If I am not mistaken, no crawler should actually consider going into the SVN browser directories. Some snippets from the apache log:
$ grep robots.txt /var/log/apache2/access_log | grep msn
65.55.208.178 - - [03/Aug/2008:16:58:35 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [03/Aug/2008:19:05:55 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [03/Aug/2008:22:14:47 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:00:31:32 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:00:57:38 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [04/Aug/2008:06:49:33 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:07:16:21 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:09:29:17 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.104.156 - - [04/Aug/2008:11:08:24 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [04/Aug/2008:11:29:34 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [05/Aug/2008:13:30:20 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.208.178 - - [05/Aug/2008:16:17:59 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
Good boy, it checks the robots.txt file. But what is this?
$ grep msnbot /var/log/apache2/access_log | tail -20
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/filedetails.php?repname=MySQL+Documentation&path=%2Fworkbench%2Fall-entities.ent&rev=9981&sc=1 HTTP/1.1" 200 6408 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/dl.php?repname=MySQL+Connector%2FJ&path=%2Fbranches%2Fbranch_5_0%2Fconnector-j%2F&rev=6600&isdir=1 HTTP/1.1" 200 40960 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:19 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Fproto-doc%2F&rev=9994&sc=1&isdir=1 HTTP/1.1" 200 36907 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8323&sc=0&isdir=1 HTTP/1.1" 200 15278 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Proxy&path=%2Ftrunk%2FDoxyfile&rev=365&sc=1&isdir=0 HTTP/1.1" 200 4162 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=Eventum&path=%2Feventum%2Freports%2F&rev=3542&sc=1&isdir=1 HTTP/1.1" 200 90591 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Fndbapi%2F&rev=9749&sc=0&isdir=1 HTTP/1.1" 200 21440 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8511&sc=0&isdir=1 HTTP/1.1" 200 18541 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
As you can see, it is happily crawling everything below /websvn/, which also includes links named "Tarball" - guess what they are good for? Yes, they create tarballs of a given SVN directory, using /tmp to build up the archive file... Within a very short amount of time, it used up more than 6 GB of disk space, as it seems as if websvn leaves these temporary directories behind, if the connection gets aborted or times out. We do have a cron job that wipes /tmp from files older than a certain amount of days, but it currently fills up much faster than what the cron job usually discards. I need to investigate if it is actually is a bug in websvn to leave these temporary dirs behind.
Hello Microsoft? Can you please fix your bots so they not only read but honor robots.txt files and stop DOSing our site? Thanks 
Friday, August 1. 2008
In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.
However, Matt seems to reduce "hosting" to "providing downloads" only:
Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
[...]
It can tell you many things, but the key is to be able to glean insight from the earliest stage of your interaction with a potential customer, and that means you have to host your own downloads. Otherwise, you have no idea how or when a would-be customer downloads your code, which makes the "why" they download it less interesting, because it becomes less actionable.
I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.
Project hosting is not just about downloads
First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.
But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.
I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.
There is more to distribute than releases
At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.
Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.
Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.
I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.
I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:
Downloads stats are overrated
Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.
Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:
As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.
I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.
Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).
So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.
Wednesday, June 25. 2008
JFYI: today I migrated the mylvmbackup source tree from my local Subversion repository on http://www.lenzg.net/ to a Bazaar repository on Launchpad.net.
This will hopefully make it easier for contributors to work on the code and share their modifications with others, removing me as the bottleneck for applying and testing patches for new releases. I chose Bazaar primarily because I wanted to get some more hands-on practice with it, now that the MySQL Server source trees have been transferred to it as well (see Kaj's announcement for details).
As mylvmbackup is closely related to the MySQL Server project, it made sense to choose the same platform and enjoy the cross-pollination effects and the infrastructure that Launchpad provides. Additionally, the distributed nature of Bazaar makes it more convenient to work with the code history and commiting changes locally without having to be online and connected to the SVN server.
I am sure that other DSCMSs like Git, Mercurial or darcs would have done the job equally well - nowadays it's very hard to choose
The "trunk" branch is now hosted on Launchpad. I assume that I will soon open up a development branch, that will receive heavier modifications first. I also plan to use the site for bug tracking and keeping track of feature requests (via Blueprints).
To create a local branch of the "trunk" repository, you can use the following command:
bzr branch lp:mylvmbackup
I also maintain a copy of that branch on my home server, just in case: http://www.lenzg.net/bzr/mylvmbackup/
To avoid confusion, I removed the Subversion repository on http://www.lenzg.net. Please use the Bazaar tree on Launchpad from now on. Thanks!
Thursday, June 19. 2008
 Even though we had been preparing the migration to Bazaar for a while now, today's announcement kind of caught me by surprise. But I am very happy about this move!
While BitKeeper is an excellent tool and served us well the past eight (!) years, I was quite annoyed when BitMover decided to remove the fully functional free BitKeeper client, which effectively put our development back into a Cathedral: even though our source trees remained accessible via bkbits.net, the crippled bk client was only capable of cloning and pulling new revisions from there - it was not possible for an external developer to commit changes locally or to create patches. This turned out to become a severe roadblock for users that wanted to participate in the development of the server. While some of our other internal projects (like the documentation, connectors and GUI tools) moved to Subversion, this was not that easily doable for the MySQL Server source trees. Lots of internal processes and tools had been tightly coupled with the source code management and there never seemed to be the right time to make the switch, which potentially could have disrupted the ongoing development work.
By moving to a truly open, distributed revision control system, it will become much easier for external contributors to work on patches and maintain modifications or enhancements outside of the main MySQL Server source trees. In fact, work has already begun! When I look at the various repositories hosted on Launchpad.net, I already notice several source trees that are not maintained by MySQL Engineers, e.g. Marc Callaghan's Google patches, Percona's work on microsecond resolution or a branch of MySQL 5.0 that is maintained and used by the folks from Wikimedia. I too helped publishing a source tree today - Holyfoot's work on improving the GIS functionality is now also available from there! In addition to the MySQL Server, many other MySQL-related projects are hosted on LaunchPad. And by utilizing services like the fabolous OpenSUSE Build Service, it's actually quite easy to provide readily installable packages of these as well!
I hope this change will significantly lower the barriers for external contributors and make it easier for our own developers to merge and accept changes that have been created by others and received sufficient community testing. This is much more convenient than having to apply a patch that was posted a long time ago and may have already been abandoned by the contributor or suffered from bit-rot. Being able to maintain a patch within a live tree that can be kept in sync with the main repository is one of the great features of distributed revision control (and which I think is one of the technical reasons why the Linux Kernel development model works so well).
Kudos to everybody involved in making this happen!
Monday, September 24. 2007
There is a lot of exciting stuff happening inside of MySQL AB. But due to the distributed nature of our company it's hardly possible to get a good overview about what the various teams of our development department are currently working on and what they have achieved since the last time we met.
So one cool new idea for our currently ongoing MySQL Developer Meeting in Heidelberg was to let developers show off their work to each other. They were encouraged to prepare demos, either in the form of slide shows or by running live demonstrations from their laptops. Last Thursday and Saturday we allocated time for these team exhibitions and the exhibitors set up tables in the meeting rooms for others to sit next to them, see the new and cool stuff and chat about it. The non-exhibiting attendees received a sheet of paper where they could collect signatures for each demo point they visited, the one that managed to see the most demos was eligible for winning a price. I managed to visit ~8 of them on Thursday - there was way more stuff to look at than what one could actually get done in the given time frame. Therefore we decided to repeat some of Thursday's exhibitions and included some new ones on Saturday.
Here is a recap of the exhibitions I saw on Thursday:
- Alexey 'Holyfoot' Botchkov demoed the new and improved GIS (geospacial information) functionality that I already blogged about some time ago. It provides a number of new GIS functions and most of the already existing ones no longer use MBRs. Holyfoot actually prepared a visual demo application that displayed a map of the earth by getting the polygons to draw out of the database. It was then highlighting the current country under the mouse cursor, displaying some additional information about it. Moving the mouse across country borders immediately switched the highlighting to the respective state. When using the old, MBR-based functions this would only happen when you crossed the borders of the bounding rectangle.
- My fellow colleague Giuseppe Maxia performed some MySQL Proxy Magic, misusing the MySQL Proxy for some fun stuff and also demonstrating some useful use cases (e.g. crosstabs). See his blog entry for some examples of what he was showing.
- Jay Pipes demoed his work on MySQL Forge 2.0, now with more Ajax magic, rating and commenting on projects and snippets as well as the obligatory rounded corners! There's still some work to be done, but I look forward to seeing it go live soon.
- Monty Tailor explained his work on NDB/Connectors, which provides bindings to many programming languages like Ruby, Perl, Lua, PHP and others. They make it possible to connect to a MySQL Cluster without having to go through the MySQL API, if you don't need to use SQL and just want to use MySQL Cluster as a highly available datastore (similar to memcached). It was interesting to compare the various test scripts for each language with each other - the Perl version looked quite ugly in comparison to the Ruby code...
- An alpha version of the MySQL Workbench was demoed by Johannes Taxacher from the GUI team. He demonstrated how one can visually create tables and set up the relations between them. The GUI is quite nice, but they support Windows only for now - Mac OS X and Linux are also planned, however.
- Kent Boortz from the build team showed off his work on creating a self-contained ODBC build environment. I was quite impressed by the complexity and steps involved in creating the binary packages of the ODBC driver we provide. As we support quite a number of platforms, Kent had to come up with all kinds of interesting tweaks for exotic architectures and strange compilers.
- Jess Balint from the Connectors team demoed the ODBC 5.1 driver by connecting to a MySQL Server using MS Access or with a MS Visual C++ application. It was nice to observe the functional UTF-8 support.
- Paul DuBois, long time member of our documentation team explained the build system they created in order to cope with the massive amount of build targets that are being created from the DocBook sources of the reference manual. In addition to various formats of the manual itself (e.g. PDF, HTML, Windows Help), they also build the manual pages that are included in our *nix distributions as well as some README files and the Changelogs. It is written in Ruby and is capable of distributing the build jobs to several machines.
That's all I was able to see on Thursday. I just wish I had some more time to see all of the other demos. On Saturday I saw the following demos:
- Marc Alff was demoing LTP-GCOV, a tool that visualizes the output of gcov as HTML pages and allows one to find out which code has been covered during the test suite run. It would be great if we could combine it with our internal pushbuild system, which automatically rebuilds the MySQL Server and runs the test suite on a number of platforms for every ChangeSet that is being pushed into a source tree. We already perform gcov build runs to make sure that new code is properly covered by the tests, but this tool would be a very helpful addition, allowing us to find code with low coverage much easier.
- Kristian Nielsen showed an early implementation of a multi-threaded ndbd, the process that runs on a MySQL Cluster data node. This should boost performance on multi-core systems (the current implementation is single-threaded). However, it's still in an early prototype phase, but looks interesting!
- Chad Miller demonstrated an alternative revision control system: bzr. Similar to BitKeeper, it is a fully distributed system which does not require a central server. Creating a local branch of an existing MySQL code tree was quite easy and it seems to be very similar in using it. What impressed me in particular was its capability of connecting to remote SVN repositories - it is actually possible to use bzr locally to work against a remote Subversion server! This should give us a good opportunity to start testing it against our existing SVN trees and gather some more practical experience with it.
All in all I was very impressed and excited to learn about all the cool things that are going on inside of the development team. Even though there is always a lot of work to be done and they're busy getting the MySQL 5.1 release out the door, they still find time to experiment and come up with great ideas and improvements in other areas as well. I got the impression that everybody really enjoyed being able to show off his work and also receiving good feedback from the others that were passing by. I hope we will repeat these team exhibitions at our next meeting!
Thursday, June 28. 2007
As I felt the itch to do some quick hacking yesterday, I decided to provide an RPM spec file for the MySQL proxy. The changes have been commited to the SVN trunk now and I added some hints to the INSTALL file on how to perform an RPM build.
Here is a quick summary of how to convert the current SVN code into an installable RPM. You build environment needs to fulfill a few additional prerequisites (a gcc compiler and the C library header files are taken for granted here), I added the versions I used on my openSUSE 10.2 system for reference:
- autoconf 2.56 or newer (autoconf-2.60)
- automake 1.9 or newer (automake-1.9.6)
- MySQL 5.0.x include files (MySQL-devel-5.0.41)
- pkg-config (pkgconfig-0.20)
- libevent 1.x or newer (libevent-1.1)
- lua 5.1 or newer (lua-5.1.1)
- glib2 2.4.0 or newer (glib2-2.12.4)
The following steps need to be performed:
# Check out the trunk of the SVN repository
$ svn co http://svn.mysql.com/svnpublic/mysql-proxy/trunk mysql-proxy
# Enter the directory and bootsrap autoconf/automake
$ cd mysql-proxy
$ sh ./autogen.sh
# Build the source distribution tarball
$ ./configure && make distcheck
# Convert the source tarball into a binary RPM
$ rpmbuild -tb --clean mysql-proxy-<version>.tar.gz
RPM will now perform the compiling and packaging. The end result will be an installable binary RPM package which will end up in the RPMS directory. Depending on your distribution this location will vary, on my openSUSE system the package ended up in in the following location from where I installed it directly:
$ sudo rpm -Uhv /usr/src/packages/RPMS/i586/mysql-proxy-0.5.1-0.i586.rpm
The mysql-proxy will be installed into /usr/sbin, the documentation and example scripts are being put into the default documentation directory (/usr/share/doc/packages/mysql-proxy in my case).
For now, you still have to start the proxy manually. Next thing is to create an init script that starts up the process at boot time.
Wednesday, November 15. 2006
I am happy to announce that version 0.2 of the mylvmbackup tool is now available!
mylvmbackup is a Perl script for quickly performing
backups of a MySQL server's databases using the Linux Logical Volume Manager (LVM). It creates a consistent LVM snapshot of the server's data directory which is then backed up without further blocking the server's operation. After version 0.1 was published in May this year, I did not really get much feedback about it. I had some ideas for improvements (see the TODO file included in the package), but never got around to actually start working on them. Thanks to Robin H. Johnson from the Gentoo project for contributing a number of new options and features as well as some code cleanups. His changes motivated me to make a few more modifications and improvements by myself, which have now been rolled into a new release. The new options provide some more flexibility in the way the script handles the logical volumes and how the backup files are being created. I also overhauled the building and packaging and added a Makefile to automate these procedures. For details, please refer to the ChangeLog and check the manual page and the README for additional info. A tarball and RPM of version 0.2 can now be downloaded from the project page. The SVN repository can now be browsed using WebSVN as well. Please give it a try! Your feedback is very welcome.
Wednesday, August 30. 2006
While the MySQL Server source trees are maintained using the BitKeeper revision control system, several other MySQL projects (Connectors, GUI-Tools and the Manual) use Subversion instead. To make it easier for external developers in getting familiar with the code base of the respective project, we now installed the FishEye SVN repository browser, which provides a very nice interface to the hosted repositories and boasts an impressive number of additional features like searching, diffing and RSS feeds. This will hopefully encourage more developers to participate and contribute to these projects. FishEye will also make it more convenient for our own developers to work with their SVN repositories and should soon become a valuable tool for us.
|
