The views or opinions expressed on this blog are my own and do not necessarily reflect the views or opinions of my current employer. The views or opinions expressed by visitors on this blog are theirs solely and may not reflect mine.
Entries tagged as development
Related tagsadministration article backup bios blog btrfs community conference configuration containers distribution domain event free froscon hint installation linux lvm meeting mysql opensolaris opensource oracle oss otn presentation RPM seminar site news snapshots social solaris storage sysadmin thinkpad tools tweak ubuntu update utility virtualization workshop zfs cluster cms code collaborating compiling contributing drupal eclipse email engine falcon fisl forge groupware hosting interview java licensing linuxtag logging mailinglist netbeans opengis openoffice OSS osx packaging rpm sun suse twitter virtualbox web wiki windows writings award cebit gui magazines php python survey baby connector gallery innodb life news Personal pictures travel bzr databases drizzle encryption hardware multimedia mylvmbackup perl programming slides snmp subversion university webinar betatest appliance cmake drivers gis internals plugins spatial studio xen bindings languages bof codebits concert fosdem git intellij mercurial opensqlcamp planetmysql scm vacation recording schwag errors doag ukoug video book contributions deutsch gsoc guug highavailability jobs lamp Linux MySQL patches personal porting proxy sfd streaming usergroup voting work amoocon brazil camera captcha documentation embedded ioug linuxcon ocfs2 openworld OS/2 oscon spam trademarks flightgear simulation gardening Hardware oow Site News trackball bdb sqlite boox ebook epub manual hotplug review security shell VoIP certification rss
Thursday, September 11. 2008
If you are a maintainer of an Open Source project, you currently have plenty of choice when it comes to getting your project hosted for free. One criterion could be your software configuration management system (SCM) of choice.
Some of the hosting services that I am currently aware of and the choice of SCM they offer include:
As disclosed by Tim Bray some days ago, there now is another option - Kenai is open for project hosting (currently by invitation only)! In his blog post, he interviews Nick Sieger, one of the developers behind this project about their motivation and intentions:
We need to demonstrate credibility in building on top of more traditional LAMP/SAMP web stacks (not just Java EE); and we need to show viability of Sun technologies and hardware for next-generation web applications.
In a nutshell, Kenai is a platform for:
Some of the features that are currently available include:
Reading the interview with Nick and looking at some presentations slides for RailsConf from Fernando Castano (a jRuby and Database performance engineer at Sun and another member of the project team), I was able to gather a list of the tools and technologies they used to build Kenai:
I found it interesting that they decided to deploy and run the Rails application as a war file within the Glassfish application server (using Warbler). By the way, the fabolous OpenSUSE Build Service is a Rails application, too! So far, the entire site is powered by a single MySQL instance with query cache enabled.
The project is hosted on the following infrastructure:
You should check out Fernando's presentation for more technical details, tuning info and how they benchmarked the setup - it contains a number of useful tuning hints and performance graphs.
Nick also talked a bit about their future near term plans: to improve the usability and feature set, incrementally improve the site navigation and layout and adding support for hosting files/release downloads. They also consider offering Jira as an option to Bugzilla for bug tracking and Git as another SCM option.
There is an IRC channel #projectkenai on freenode.net, to get in touch with the developers directly. The mailing list for the Project Kenai site itself, is firstname.lastname@example.org - you can subscribe to this list here.
Thursday, September 4. 2008
One of the sessions at DrupalCon I attended was Larry Garfield's talk about "Drupal Databases: The Next Generation", which gave me a good insight into the current state of the Drupal database layer and how they plan to overhaul it for Drupal 7. The key points that I took away:
Tuesday, August 19. 2008
Being able to use an Open Source DBMS to manage your data is nice, but what good would it be if you can't easily access it from your applications? One key factor to the popularity of MySQL is probably its wide range of available language bindings, which started with support for C, PHP and Perl from early on.
I've tried to gather a list of languages and their respective MySQL drivers/modules below. It's by no means complete or exhaustive, but I think I covered quite a lot of popular as well as exotic programming languages.
There is a number of connectors which are actually developed by the Sun Database Group (aka MySQL) itself and that are ready to use:
In addition to the above, there are several other connectors developed by Sun/MySQL, which are still under development:
But it's not only us who develop language bindings for the MySQL server. There is an abundance of drivers that are developed and maintained by the Community, independently from Sun/MySQL (but sometimes with support or guidance from MySQL engineers). The list below is not sorted in any particular order other than the sequence in how I found them over time:
I probably forgot some other drivers/bindings - if you have any more to add, please let me know!
Wednesday, August 13. 2008
The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!
Friday, August 1. 2008
In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.
However, Matt seems to reduce "hosting" to "providing downloads" only:
Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.
Project hosting is not just about downloads
First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.
But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.
I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.
There is more to distribute than releases
At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.
Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.
Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.
I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.
I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:
Downloads stats are overrated
Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.
Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:
As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.
I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.
Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).
So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.
Wednesday, June 25. 2008
This will hopefully make it easier for contributors to work on the code and share their modifications with others, removing me as the bottleneck for applying and testing patches for new releases. I chose Bazaar primarily because I wanted to get some more hands-on practice with it, now that the MySQL Server source trees have been transferred to it as well (see Kaj's announcement for details).
As mylvmbackup is closely related to the MySQL Server project, it made sense to choose the same platform and enjoy the cross-pollination effects and the infrastructure that Launchpad provides. Additionally, the distributed nature of Bazaar makes it more convenient to work with the code history and commiting changes locally without having to be online and connected to the SVN server.
The "trunk" branch is now hosted on Launchpad. I assume that I will soon open up a development branch, that will receive heavier modifications first. I also plan to use the site for bug tracking and keeping track of feature requests (via Blueprints).
To create a local branch of the "trunk" repository, you can use the following command:
bzr branch lp:mylvmbackup
I also maintain a copy of that branch on my home server, just in case: http://www.lenzg.net/bzr/mylvmbackup/
To avoid confusion, I removed the Subversion repository on http://www.lenzg.net. Please use the Bazaar tree on Launchpad from now on. Thanks!
Thursday, June 19. 2008
While BitKeeper is an excellent tool and served us well the past eight (!) years, I was quite annoyed when BitMover decided to remove the fully functional free BitKeeper client, which effectively put our development back into a Cathedral: even though our source trees remained accessible via bkbits.net, the crippled bk client was only capable of cloning and pulling new revisions from there - it was not possible for an external developer to commit changes locally or to create patches. This turned out to become a severe roadblock for users that wanted to participate in the development of the server. While some of our other internal projects (like the documentation, connectors and GUI tools) moved to Subversion, this was not that easily doable for the MySQL Server source trees. Lots of internal processes and tools had been tightly coupled with the source code management and there never seemed to be the right time to make the switch, which potentially could have disrupted the ongoing development work.
By moving to a truly open, distributed revision control system, it will become much easier for external contributors to work on patches and maintain modifications or enhancements outside of the main MySQL Server source trees. In fact, work has already begun! When I look at the various repositories hosted on Launchpad.net, I already notice several source trees that are not maintained by MySQL Engineers, e.g. Marc Callaghan's Google patches, Percona's work on microsecond resolution or a branch of MySQL 5.0 that is maintained and used by the folks from Wikimedia. I too helped publishing a source tree today - Holyfoot's work on improving the GIS functionality is now also available from there! In addition to the MySQL Server, many other MySQL-related projects are hosted on LaunchPad. And by utilizing services like the fabolous OpenSUSE Build Service, it's actually quite easy to provide readily installable packages of these as well!
I hope this change will significantly lower the barriers for external contributors and make it easier for our own developers to merge and accept changes that have been created by others and received sufficient community testing. This is much more convenient than having to apply a patch that was posted a long time ago and may have already been abandoned by the contributor or suffered from bit-rot. Being able to maintain a patch within a live tree that can be kept in sync with the main repository is one of the great features of distributed revision control (and which I think is one of the technical reasons why the Linux Kernel development model works so well).
Kudos to everybody involved in making this happen!
Friday, April 11. 2008
I am happy to announce the release of mylvmbackup version 0.8. mylvmbackup is a tool for quickly creating backups of a MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.
Below is the list of changes since version 0.6. You may wonder what happened to version 0.7 - it had a rather short life cycle as I was informed about a bug that I fixed quickly before I made a wider release announcement of 0.7.
Updated package are available from the home page and via the openSUSE Build Service as usual. Updated packages for Debian/Ubuntu and Gentoo Linux should also be available shortly. Enjoy!
Speaking of LVM snapshot backups: I will be giving a talk about this subject at our MySQL Conference 2008 in Santa Clara, CA next week. If you are curious about how MySQL can be backed up using this technology, please consider to stop by!
Friday, April 4. 2008
A bit late in the game, but maybe somebody would be interested in working on this proposed project of mine:
Please contact me directly (firstname at MySQL.com) or the mailing list, if you are interested in working on this task. Thanks!
Monday, March 10. 2008
Last week at CeBit I once again had a conversation with an application developer that was not aware that it's possible to embed the MySQL Server inside an application instead of running it as a separate process. I experienced something similar at FOSDEM recently, while talking with the Amarok developers - how come that this feature is so little known? In fact, there are two possibilities for embedding MySQL (and both of them are licensed under the GPL): the first one is using the libmysqld embedded library, which is suitable if you are developing C/C++ applications:
Especially consider the last sentence - it's actually fairly easy to create an application that supports connecting to both an embedded or a standalone server instance! Here are some related links to get you going:
If you're a Java developer, you should consider taking a look at our Connector/MXJ product - this allows you to embed a MySQL server in your Java application:
Some related links to further information/help:
Does your application make use of an embedded MySQL Server already? Please let me know! I'd love to learn more about the use cases and experiences you have made.
Back when I still worked at SuSE, I was in charge of maintaining a number or packages of the distribution (actually, you should still be able to find traces of my work in the RPM changelogs). Nowadays, I maintain a number of packages for openSUSE and other distributions on the openSUSE Build Service, which is just brilliant for this purpose.
If you happen to live in northern Germany and are interested to learn more about the RPM package manager and how to build packages, consider coming to the TU Harburg this coming Thursday (March 13th). At 19:00, I will give a presentation about this topic in building, D, room D1023 (in cooperation with the Hamburg branch of the German Unix User Group). More information (in German) can be obtained from here.
See you there!
Wednesday, February 20. 2008
Since almost a year now, we host a weekly training session for our engineers on Thursday (14:00 UTC winter time), coined the "MySQL University". While it's primary purpose is to share and distribute knowledge about a wide variety of topics relevant to our own developers, many of the sessions are of general interest for developers on other projects as well.
Therefore we hold this sessions in the public and everybody is welcome to attend! You can listen to the presentation via an OGG Audio stream, questions can be posted via IRC on the #mysql-university channel on freenode.net. The audio file and IRC log will be saved, so you can also listen to past university sessions at a later point in time again.
The next upcoming sessions include topics more closely related to MySQL Server development:
Thursday, February 14. 2008
We'd like to congratulate the Drupal Developer Team and Community for reaching this milestone and are happy that the MySQL Server continues to serve well as the database backend for this awesome content management platform!
I had the pleasure of evaluating and reviewing a previous release of Drupal for the Open Source Content Management System Award from Packt Publishing and it has been one of my favourites.
Keep up the good work!
Sunday, October 28. 2007
Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.
In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!
Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.
Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.
Monday, September 24. 2007
There is a lot of exciting stuff happening inside of MySQL AB. But due to the distributed nature of our company it's hardly possible to get a good overview about what the various teams of our development department are currently working on and what they have achieved since the last time we met.
So one cool new idea for our currently ongoing MySQL Developer Meeting in Heidelberg was to let developers show off their work to each other. They were encouraged to prepare demos, either in the form of slide shows or by running live demonstrations from their laptops. Last Thursday and Saturday we allocated time for these team exhibitions and the exhibitors set up tables in the meeting rooms for others to sit next to them, see the new and cool stuff and chat about it. The non-exhibiting attendees received a sheet of paper where they could collect signatures for each demo point they visited, the one that managed to see the most demos was eligible for winning a price. I managed to visit ~8 of them on Thursday - there was way more stuff to look at than what one could actually get done in the given time frame. Therefore we decided to repeat some of Thursday's exhibitions and included some new ones on Saturday.
Here is a recap of the exhibitions I saw on Thursday:
That's all I was able to see on Thursday. I just wish I had some more time to see all of the other demos. On Saturday I saw the following demos:
All in all I was very impressed and excited to learn about all the cool things that are going on inside of the development team. Even though there is always a lot of work to be done and they're busy getting the MySQL 5.1 release out the door, they still find time to experiment and come up with great ideas and improvements in other areas as well. I got the impression that everybody really enjoyed being able to show off his work and also receiving good feedback from the others that were passing by. I hope we will repeat these team exhibitions at our next meeting!
Show tagged entries
Original content in this work is licensed under a Creative Commons License