The views or opinions expressed on this blog are my own and do not necessarily reflect the views or opinions of my current employer. The views or opinions expressed by visitors on this blog are theirs solely and may not reflect mine.
Entries tagged as mylvmbackup
Related tagsbackup administration blog bzr cluster code collaborating community conference configuration contributing databases development distribution drizzle drupal encryption event froscon gallery hardware hint hosting innodb linux lvm mailinglist multimedia mysql news opensolaris opensource OSS packaging perl php pictures presentation programming python rpm site news slides snapshots solaris storage subversion suse sysadmin thinkpad tools travel university update utility web webinar zfs codebits concert eclipse forge fosdem git intellij java meeting mercurial netbeans opengis opensqlcamp oss planetmysql scm social sun vacation article betatest bindings cmake cms compiling connector drivers errors groupware gui installation internals osx virtualbox wiki windows writings doag ukoug video appliance award bof book contributions deutsch engine gsoc guug highavailability interview jobs lamp languages licensing life Linux linuxtag MySQL openoffice oracle patches personal porting proxy recording schwag sfd streaming studio survey twitter usergroup virtualization voting work xen amoocon baby brazil camera captcha cebit embedded falcon fisl free gis ioug linuxcon magazines ocfs2 openworld OS/2 oscon Personal plugins RPM spam spatial trademarks Hardware oow seminar Site News bios email logging trackball tweak ubuntu flightgear simulation bdb sqlite btrfs gardening otn workshop hotplug containers review security shell VoIP boox certification documentation ebook epub manual rss domain
Sunday, February 23. 2014
Probably the biggest highlight of 0.15 is the addition of zbackup as an additional backup type. I'd like to thank Ivan Korjavin for contributing this new feature.
Additionally, this release provides several improvements and bug fixes. For example, it's now possible to back up more than a single my.cnf configuration file by providing a directory name like /etc/mysql instead.
Monday, June 24. 2013
It's my great pleasure to announce the release of mylvmbackup version 0.14.
This release includes a large number of improvements, code cleanups, and new functionality.
I would like to thank Ask Bjørn Hansen, Ben Bonnel, Norbert Tretkowski, Neil Wilson, Klaus Ethgen and Alexandre Anriot for their feedback and contributions to this release.
Thursday, November 5. 2009
This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!
Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.
Continue reading "Aspects and benefits of distributed version control systems (DVCS)"
Sunday, September 6. 2009
I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).
From the ChangeLog:
Saturday, June 20. 2009
After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.
Some notable highlights from the ChangeLog:
Thursday, February 26. 2009
Concluded my first MySQL University Session about MySQL backups using file system snapshots - some questions remained unanswered...
Today I gave my first MySQL University session as a speaker, talking about Backing up MySQL using file system snapshots. The talk went quite well (at least that was my impression) and we had ~10 people attending. The slides (PDF) and a recording of the session are now available from the Wiki page. Unfortunately the recording lacks the audio track, which is a bit of a bummer. We've submitted a support request with the DimDim folks, so hopefully they can provide us with a complete recording.
There was one question during the session that I was not able to answer myself, so I'm asking for your insights here:
Consider we're using InnoDB and MyISAM tables on a file system that can be snapshotted (e.g. Linux LVM or ZFS) and we're performing the following operations:
The question that came up was if this actually still is a consistent backup, considering that InnoDB rolled back uncommited transactions. Does the state of the tables still match the binary log positions we noted before? I assume yes, as long as the transaction does not involve modifications non-transactional tables.
Another suggestion that came up was to change InnoDB's configuration variable innodb_max_dirty_pages_pct to "0" prior to performing the snapshot, to minimize the amount of dirty pages that have not been written to disk (and thus reducing the time required for recovering later). I wonder if this would make a difference...
What other InnoDB variables might have a noteworthy effect in the context of snapshot backups? I am looking forward to your comments.
Monday, December 1. 2008
Some days ago, I released version 0.11 of mylvmbackup a Perl script that performs consistent backups of a MySQL server by using LVM filesystem snapshots. The source archive as well as a generic RPM can be found on the project home page, packages for many Linux distributions are available on the openSUSE Build service.
This release includes some new functionality as well as numerous bug fixes and improvements, most notably:
I would like to thank all the contributors for their support! More details about the changes in this release, directly from the ChangeLog:
Saturday, September 20. 2008
I am happy to announce that mylvmbackup version 0.10 has been released.
You can download the updated package from the project home page or via the openSUSE Build Service.
This version fixes some bugs and includes new functionality:
Friday, August 1. 2008
In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.
However, Matt seems to reduce "hosting" to "providing downloads" only:
Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.
Project hosting is not just about downloads
First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.
But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.
I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.
There is more to distribute than releases
At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.
Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.
Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.
I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.
I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:
Downloads stats are overrated
Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.
Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:
As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.
I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.
Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).
So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.
Friday, July 11. 2008
I am happy to announce that a new version (0.9) of mylvmbackup has been released. This is the first release since the source code has been moved from Subversion to Bazaar and is now hosted on Launchpad.net. I would like to thank Robin H. Johnson and Patrick Hahn for providing the patches that contributed to this new release!
mylvmbackup is a tool for quickly creating backups of MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.
From the ChangeLog:
You can download a source tarball or RPM from the project home page. Additional packages for various Linux distributions can be obtained from the openSUSE Build Service. Packages for Gentoo and Debian should appear shortly, too.
Wednesday, June 25. 2008
This will hopefully make it easier for contributors to work on the code and share their modifications with others, removing me as the bottleneck for applying and testing patches for new releases. I chose Bazaar primarily because I wanted to get some more hands-on practice with it, now that the MySQL Server source trees have been transferred to it as well (see Kaj's announcement for details).
As mylvmbackup is closely related to the MySQL Server project, it made sense to choose the same platform and enjoy the cross-pollination effects and the infrastructure that Launchpad provides. Additionally, the distributed nature of Bazaar makes it more convenient to work with the code history and commiting changes locally without having to be online and connected to the SVN server.
The "trunk" branch is now hosted on Launchpad. I assume that I will soon open up a development branch, that will receive heavier modifications first. I also plan to use the site for bug tracking and keeping track of feature requests (via Blueprints).
To create a local branch of the "trunk" repository, you can use the following command:
bzr branch lp:mylvmbackup
I also maintain a copy of that branch on my home server, just in case: http://www.lenzg.net/bzr/mylvmbackup/
To avoid confusion, I removed the Subversion repository on http://www.lenzg.net. Please use the Bazaar tree on Launchpad from now on. Thanks!
Thursday, April 24. 2008
JFYI, I now placed a PDF of my MySQL Conference talk slides about "Performing MySQL backups using Linux LVM Snapshots" on my MySQL talks page. Enjoy!
Wednesday, April 16. 2008
While reading Colin's post about LugRadio Live, I stumbled over the Zumastor Linux Storage Project. Going through the project home page and their HOWTO got me curious - could this eventually become an alternative to using DRBD (for replicating data) and LVM snapshots (for performing backups)?
I assume it's not ready for production use yet, but it would sure be interesting to investigate on how to utilize it for the purpose of running MySQL on top of it...
I will keep an eye on this project, I wonder if I will have to add support for Zumastor snapshots to mylvmbackup at some point?
Friday, April 11. 2008
I am happy to announce the release of mylvmbackup version 0.8. mylvmbackup is a tool for quickly creating backups of a MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.
Below is the list of changes since version 0.6. You may wonder what happened to version 0.7 - it had a rather short life cycle as I was informed about a bug that I fixed quickly before I made a wider release announcement of 0.7.
Updated package are available from the home page and via the openSUSE Build Service as usual. Updated packages for Debian/Ubuntu and Gentoo Linux should also be available shortly. Enjoy!
Speaking of LVM snapshot backups: I will be giving a talk about this subject at our MySQL Conference 2008 in Santa Clara, CA next week. If you are curious about how MySQL can be backed up using this technology, please consider to stop by!
Saturday, January 19. 2008
While browsing the many blog entries on blogs.sun.com about the MySQL Acquisition (thanks a lot for the very warm welcome!), I stumbled over this (Python-based) utility: SnapBack, a tool that uses ZFS snapshots to perform physical backups of MySQL databases on Solaris. Very cool! This is actually something I was wanting to add to the mylvmbackup script, too - I have to take a closer look at how this is done (I tried to install OpenSolaris on a VirtualBox instance, but it caused it to crash the emulator).
(Page 1 of 2, totaling 22 entries) » next page
Show tagged entries
Original content in this work is licensed under a Creative Commons License