Lenz Grimmer's blog

Vorab ein Disclaimer: nein, ich bin nicht auf der Suche nach einen neuen Job

Ich habe auf der FrOSCon am letzten Wochenende Lukas Chaplin endlich mal persönlich getroffen, nachdem wir schon seit längerem in Online-Kontakt standen. Er betreibt das Job-Portal Linux Lancers (powered by MySQL), auf dem Stellenanzeigen mit dem Schwerpunkt auf Unix-, Linux- und Open Source-Jobs veröffentlicht werden. Dieses Jahr waren sie auf der FrOSCon als Gold-Sponsor und Aussteller vertreten und hatten wohl immer viel zu tun.

Ich finde diese Idee sehr gut – es zeigt, daß die Arbeit an und mit Open Source Software durchaus keine brotlose Kunst ist und Know-How in vielen Bereichen dringend gesucht wird. Mir ist bisher kein weiteres Portal mit diesem Fokus bekannt. Eine Suche nach "MySQL" liefert viele Hits, quer über die Republik verteilt. Kenntnisse und Erfahrung mit Opensource-Technologien zahlen sich aus! Die Website ist noch in einer frühen Betaphase, aber das machen die Inhalte wieder wett. Laut Lukas ist ein Relaunch des Portals in Arbeit, bei dem sowohl die Suchlogik als auch die Darstellung der Suchergebnisse überarbeitet wird.

Last week, my colleagues Giuseppe, Kai and myself attended the SAPO Codebits event in Lisbon, Portugal. Codebits is an annual, invite-only hacking event, which went on for three days. The venue they chose this year was the "Cordoaria", a former rope factory located in the Belém district, close to the 25 de Abril Bridge (which is an impressive sight!). I have been told that the Cordoaria is the longest building in Portugal and I have no doubts about that! The building is so long that the crew used bicycles to get from one end to the other. I've taken a number of pictures from the event as well as from Lisbon itself, you can find them in this flickr set.

The organizers described this year's event as follows:

3 days. 24 hours a day. 600 attendees. Talks. Workshops. Lots of food and beverages. 24 hour programming/hacking competition. Quizz Show. Rock Band Contest. Lots of gaming consoles. More food. More beverages. More coding. Sleeping areas. More fun. An unforgettable experience.

I wholeheartedly agree, we had a great time! The conference started with sessions and presentations on a wide range of topics on the first two days. Afterwards, a 24-hour programming contest was held. I was invited to give two talks, one being my all-time favourite about "MySQL High Availability solutions" (slides, video), the other one was titled "Why you should be using a distributed version control system (DVCS) for your project" (video, slides). Both went quite well and the feedback I received was pretty positive. Giuseppe talked about "MySQL Schema Migration" (slides, video) and gave an "Introduction to Gearman" (video). Kai's talk was titled "Think before you develop" (video) and gave a nice roundup of tips and best practices for setting up and developing new web projects.

The Codebits session schedule was filled with amazing and interesting talks in four parallel tracks. Sometimes it was hard to choose – some other talks I attended and enjoyed:

• Robert Nyman: JavaScript - From Birth To Closure (slides, video)
• Glenn Jones: Re-using social media data (slides)
• Brian LeRoux: PhoneGap: Mobile App Developer Zero to Hero (video)
• Brian LeRoux: The State of JavaScript on Mobile Devices
• Walter Belgers: Becoming a lockpicker

Walter gave a lockpicking workshop after his presentation, which I attended as well. I was quite impressed (and a bit shocked) to find out how easy many locks can be opened this way! Later that evening there even was a live band named "Pornophonique" playing (one guy with a guitar, the other one using an Nintendo Game Boy for making music), but I missed that show as I was too busy opening more locks... Fortunately the concert and most of the sessions were recorded on video (in excellent quality) and are already available from the SAPO video pages. Kudos for this speedy service!

But this just matches my overall conclusion of this event: very well organized, great speakers and venue. Thanks to the organizers for having us, we really enjoyed our stay!

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

Giuseppe stumbled over this website and recommended it to me as a potential useful service for organizing the registrations of the OpenSQL Camp: Eventbrite is a Python/MySQL-powered web site (judging from their job openings) that provides the following:

Eventbrite is the leading provider of online event management and ticketing services. Eventbrite makes it easy for anyone to hold a successful event of any type and size. Eventbrite is free if your event is free. If you sell tickets to your event, Eventbrite collects a small fee per ticket. So just like you, Eventbrite wants your event to be a big success.

The Eventbrite service includes many features and tools intended to let you perform three basic tasks really well:

1. Publish: Everything you need to create and easily personalize a custom web page for your event.
2. Promote: The tools you need to spread the word about your event and maximize attendance.
3. Sell: The immediate power to sell tickets and collect money online.

I won't use it for the OpenSQL Camp (as we will be able to piggyback on FrOSCon's event infrastructure), but it seems like a service worthwhile checking out, if you're looking for a way to organize your next MySQL meetup (and you're based in the US - currently Eventbrite seems to assume USD as the only available currency). I've added a note about it to the suggestions on the how to run a MySQL User Group page on the MySQL Forge Wiki. And if you do use it, please make sure to add your group to the list of MySQL User Groups! Thanks.

Searching the service for "MySQL" revealed that the MySQL Dublin Meetup actually uses it for organizing their upcoming meeting (June 24, but already sold out)!

Sorry for the downtime of this site - until around a week ago I hosted my home page on a trusty Genesi Pegasos II system (powered by a PowerPC G4 Processor clocked at 1GHz, using Debian 4.0 PPC with 512 MB of RAM), serving these pages from my home DSL connection. Unfortunately this system provided no means of redundancy - the hard disk drive died.

Luckily I perform frequent backups, so I moved most parts of the site to a shared hosting space now - the picture gallery is unfortunately too big to fit into the space that I have there. I'll try to move the pictures into my Flickr account instead, but this will take some time.

Note that the primary domain name of this site is now lenzg.net - lenzg.org, (the domain that I tried to promote as the official domain for my site) used to redirect to the home machine at lenz.homelinux.org. Both now redirect to the new address instead. I've initiated the move of the lenzg.org domain to the other provider as well, so soon this site will be available from both the .org and .net domain. Please don't link to lenz.homelinux.org anymore, as that site will eventually go out of service. Until then, a small openSUSE Linux box (Intel PIII, 500 MHz, 192 MB of RAM) running lighttpd will perform the URL redirection.

FYI, we've now added an Italian section on Planet MySQL: http://it.planetmysql.org

If you are a MySQL enthusiast from Italy and would like to start blogging about it in your native language, please consider submitting your feed for inclusion!

Giuseppe just recently started blogging in Italian as well and has already added his feed there.

It has not hit the MySQL DevZone Frontpage yet, but I just finished an interview with Adam Donnison, one of our web gurus behind the mysql.com web site.

Adam talks about what his work environment looks like, his experiences with MySQL 5.1 on the MySQL web sites and why he enjoys working for a virtual organization like ours. Enjoy!

This is more of an "behind the scenes" update and I hope that you won't see any (negative) changes on the PlanetMySQL front page or the RSS feeds: I just finished and commited the conversion of the backend script that performs the parsing and aggregation of feeds from requiring MagpieRSS to SimplePie.

This will provide better support for a wider range of feed types and should also fix a few quirks, e.g. that some postings (for example the one from Kevin Burton) only showed up as an "A" in the Planet's RSS feed. It hopefully also fixes a weirdness with time zones that some people were reporting, but this requires further investigation.

The size and complexity of the script was reduced significantly because of this change - SimplePie is a breeze to use in comparison to MagpieRSS and it's very well-documented. The developers provide some more reasons and a comparison on why you should also make this switch!

And in case you notice anything broken or weird on PlanetMySQL that might be related to the change, please let me know! This change was an important step for future improvements of the site.

Tomorrow (Thursday, 11th of September) at 9:00 PST/16:00 UTC/17:00 GMT/18:00 CET, there will be an new free MySQL University Session. MySQL University started as an internal training program for MySQL engineers, to share and spread knowledge about their areas of expertise and has been available to the public for quite some time now. It covers a wide range of technical topics around the MySQL Server and usually takes place once per week.

For the first time, the presentation will not be performed by (former) MySQL employees/developers, but by two of our "Sun Classic" colleagues: Jyri Virkki (OpenSolaris Web Stack community lead) and Murthy Chintalapati (Sr Engineering Manager, Web Stack development) will talk about the OpenSolaris Web Stack:

OpenSolaris Web Stack is an OpenSolaris project and community building an integrated stack of popular open source web tier infrastructure technologies such as Apache HTTP server, MySQL, memcached, PHP and Ruby On Rails optimized for Solaris platform. This session introduces OpenSolaris Web Stack, its status and future development including addition of newer technologies such as lighttpd, Varnish etc., as well as the ease of use features for developers and deployers. We will also be discussing an experimental web stack IPS package repository and it could be leveraged to build and make available popular end user applications such as Drupal.

MySQL University sessions are free to attend - all you need is an IRC client (to post your questions and comments) and an audio player capable of playing back an OGG audio stream, so you can listen to what is being said. See the Instructions for Attendees on the MySQL University pages for more information on how to log in and attend. The audio stream will be recorded and published on the MySQL University pages for later consumption, in case you can't make it or want to listen to a previous session.

 

Today, the root file system on our public svn server nearly ran out of disk space. The reason? The /tmp directory was quickly filling up with temporary files created by websvn, which I set up parallel to the FishEye repository browser for testing purposes. A quick investigation of the apache log files revealed the culprit - a crawler from Microsoft was running haywire and decided to ignore the rules in the robots.txt file, even though it did actually looked at the file before!

Here is how robots.txt looked like (I now changed it to disallow everything):

User-agent: *
Disallow: /fisheye/
Disallow: /websvn/

If I am not mistaken, no crawler should actually consider going into the SVN browser directories. Some snippets from the apache log:

$ grep robots.txt /var/log/apache2/access_log | grep msn
65.55.208.178 - - [03/Aug/2008:16:58:35 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [03/Aug/2008:19:05:55 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [03/Aug/2008:22:14:47 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:00:31:32 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:00:57:38 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.235.139 - - [04/Aug/2008:06:49:33 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [04/Aug/2008:07:16:21 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.25.136 - - [04/Aug/2008:09:29:17 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.104.156 - - [04/Aug/2008:11:08:24 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [04/Aug/2008:11:29:34 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.212.64 - - [05/Aug/2008:13:30:20 +0200] "GET /robots.txt HTTP/1.0" 200 53 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.208.178 - - [05/Aug/2008:16:17:59 +0200] "GET /robots.txt HTTP/1.1" 200 53 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

Good boy, it checks the robots.txt file. But what is this?

$ grep msnbot /var/log/apache2/access_log | tail -20
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/filedetails.php?repname=MySQL+Documentation&path=%2Fworkbench%2Fall-entities.ent&rev=9981&sc=1 HTTP/1.1" 200 6408 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:15 +0200] "GET /websvn/dl.php?repname=MySQL+Connector%2FJ&path=%2Fbranches%2Fbranch_5_0%2Fconnector-j%2F&rev=6600&isdir=1 HTTP/1.1" 200 40960 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:19 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Fproto-doc%2F&rev=9994&sc=1&isdir=1 HTTP/1.1" 200 36907 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8323&sc=0&isdir=1 HTTP/1.1" 200 15278 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=MySQL+Proxy&path=%2Ftrunk%2FDoxyfile&rev=365&sc=1&isdir=0 HTTP/1.1" 200 4162 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:21 +0200] "GET /websvn/rss.php?repname=Eventum&path=%2Feventum%2Freports%2F&rev=3542&sc=1&isdir=1 HTTP/1.1" 200 90591 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Fndbapi%2F&rev=9749&sc=0&isdir=1 HTTP/1.1" 200 21440 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.208.164 - - [05/Aug/2008:22:48:23 +0200] "GET /websvn/log.php?repname=MySQL+Documentation&path=%2Ffalcon%2F&rev=8511&sc=0&isdir=1 HTTP/1.1" 200 18541 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

As you can see, it is happily crawling everything below /websvn/, which also includes links named "Tarball" - guess what they are good for? Yes, they create tarballs of a given SVN directory, using /tmp to build up the archive file... Within a very short amount of time, it used up more than 6 GB of disk space, as it seems as if websvn leaves these temporary directories behind, if the connection gets aborted or times out. We do have a cron job that wipes /tmp from files older than a certain amount of days, but it currently fills up much faster than what the cron job usually discards. I need to investigate if it is actually is a bug in websvn to leave these temporary dirs behind.

Hello Microsoft? Can you please fix your bots so they not only read but honor robots.txt files and stop DOSing our site? Thanks

Did you know that many parts of the MySQL web sites provide news and updates via RSS Feeds? Markus Popp from our web team did a great job on making some of these more visible by adding RSS icons to the respective pages. If you want to keep up with what's happing at MySQL, consider adding the following feeds to your feed reader:

• New downloads on dev.mysql.com
  
• New DevZone articles
• PlanetMySQL (You probably knew that one already)
• MySQL Forge - All New Items
• MySQLForge Wiki - Recent changes
• MySQL Announcements on Launchpad.net
• Branch activity on Launchpad.net
• New MySQL Training Courses
• MySQL Live Web Seminars
• MySQL OnDemand Web Seminars
• Latest Quickpolls from MySQL AB
• Latest Updates from MySQL AB
• Latest Press Articles about MySQL AB
• The MySQL Bug database allows you to subscribe to the results of any search using the 'Feed' link that can be found on the search result page.
• The MySQL Forums provide an RSS feed for each forum, e.g. the Announcements Forum

Did I miss any? Do you know any other feed? Please let me know.

RSS feed readers exist for a wide range of platforms. I personally use Akregator, which is part of the KDE Desktop environment.

 

FYI, I changed two parameters on Planet MySQL to accomodate the current flood of postings coming from the MySQL Conference attendees: we now display the last 25 posts (instead of 10) on the front page, the RSS feed now includes the latest 100 posts (instead of 50). This should make sure that posts actually make it to the front page for at least some time, before they fall off again.

Please keep up the good blogging!

 

This thought has been floating around my head for quite some time now and I finally bit the bullet and released it from the draft state it had been sitting in for too long: there are quite many similarities between Open Source Software (OSS) projects and most of today's popular Web 2.0 sites, but there is also one odd difference that I wonder about.

For both worlds, the concept of collaboration, participation and giving more power to their users is a key component. OSS projects need contributors for patches and bug reports, but also for feedback, translations, artwork, advocacy in order to be popular and healthy. The project's developers need to be open for suggestions, listening to their user base on where the project should be heading. They also usually strive for open standards and APIs, allowing easy combination and interaction with other Open Source Software so that other projects can build their work on top of theirs or compliment it. A successful OSS application gains popularity by the snowball effect: users of the product find it useful and recommend it to other users seeking for similar functionality. These too start to contribute to the project in various ways, making it in turn more useful and attractive for even more users and so on.

The same is true for Web 2.0 sites: they depend on a user base that creates and provides content that makes their site useful and popular, e.g. blog postings, pictures, articles and other content, but also by providing feedback and bug reports about the site itself and recommending it to others. A successful Web2.0 site needs to be as open to user feedback as a regular OSS project in order to remain attractive and maintain a loyal user base. So it's a lot about participation and empowering the users to contribute and control the content - the part that actually makes a site useful and attractive. Many of these Web2.0 sites also provide public APIs (e.g. SOAP, REST, XML-RPC or RSS, just to name a few acronyms) that allow others to access their services and functionality and reuse or combine them with their own work. And by using technologies like AJAX, most of these sites actually look and feel like regular desktop applications already.

But there is a strikingly difference: even while most Web2.0 sites are based on Open Source technology (e.g. the LAMP stack or Ruby on Rails), only very few ones actually carry on the OSS philosophy and make their own code available as Open Source as well. Not that they would have to do so, as there is this so-called ASP loophole in the GPL and most other OSS licenses: web sites are not actually distributing their application code, so they are not required to disclose the source of their applications.

But why do they break the chain here? One could argue that the obvious reason for that is that most web sites are created and maintained by companies, that are doing this for business reasons and want to maintain their competitive edge over others. However, many sites started as a hobby project by an individual or a small group and were converted into commercial entitities once they became successful. Even in these early stages it's rare that sites provide their source code. But isn't Web2.0 mostly about the users and content and not so much about the technology behind it? Would it really make a difference if the source code of a site is freely available?

In the OSS world, users could just fork off a project, if they don't feel the project is going the right direction or it has been abandoned by the original author. This sometimes leads to much better applications than what the original project came up with. This also leads to a healthy competition and to much better software with less duplication of work in the long run. And this is being recognized by more and more software companies as well - there is a large and growing number of OSS applications that are not maintained by a group of volunteers, but by a commercial entity.

In the Web2.0 world, people start creating new sites from scratch again and again, resulting in a vast amount of sites that serve a similar purpose, all with their own warts and deficiencies, splitting up the user base and available content. This causes fragmentation and makes it much harder to create one useful resource. And they are all on their own in maintaining and improving their sites, to fulfill the requests of their user base and keeping them happy and loyal.

I wonder if and when Web2.0 developers will embrace and extend the concept of sharing their code as well, to let others help them to become even more powerful and popular. Once a site has gained a certain popularity, there is plenty of new features that their users will ask for, or bugs and deficiencies that are in need of fixing. Making the source code available would allow others to jump in here, opening yet another channel of possible contributions and community-building. The Open Source community provides numerous lessons to learn from. True, others may then be able to take the code and create a site that provides the same services and experience. But they still would have to gain traction by creating a community and content before they would become a serious competition.

One great example of how this could be done is Wikipedia - they maintain Mediawiki (the application that powers their site) as a regular OSS project, which has made it one of the most popular and powerful Wiki applications around and has allowed others to create Wiki sites about all kinds of topics on their own.

And while it's not really true Web2.0 company, I also welcome Linden Lab's decision to open up the Second Life client source code. I am confident that the experiences they make with this will encourage them to consider opening the sources of the server code as well at some point...

But they both have grasped that it's not so much about the technology and software, but much more about the users and the content created by the community that makes them popular and successful. This is something one can't simply take and create a competing offering, even though the source code is available.

It was long overdue, but now it has happened: Planet MySQL now provides a section to aggregate french blogs about MySQL! Thanks a lot to Jay for updating the code and Pascal Borghino for the localization. If you are a MySQL enthusiast located in France (or any other french-speaking country) and you enjoy writing about your passion in your native language, submit your feed now!

Sorry for the short outage, due to a small coding glitch the script that updates the planet feeds got stuck and it took us a bit to find the problem... But thanks to Jay we now spotted the bug and it was fixed quickly!