ITP: Percona XtraBackup - hot backup utility for MySQL
Hi! I'm Stewart and I work for Percona. One of the things I'm currently working on is ensuring all our free and open source software projects are packaged for all the major linux distributions - including my beloved Debian. We're wanting to have Percona XtraBackup be part of Debian. Percona XtraBackup is an online backup tool for MySQL, MariaDB, Percona Server and Percona XtraDB Cluster. In the near future, we will also wanting our packages for Percona Server and Percona XtraDB Cluster to be part of Debian. There is an open bug for Percona XtraBackup packaging: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620824 We are going to make "upload release to Debian" be part of our release process. We tend to make a new minor bugfix release (i.e. 2.1.x) every 2-3 months and a new major release every 12 (2.0 in April 2012, 2.1 in May 2013). We want to be an active and involved upstream. There have been a couple of issues with our source tarballs and build procedure that I've had to fix up before proposing that Percona XtraBackup be included. I've made changes on top of our 2.1.3 release that mean we now generate source tarballs that do not require an active internet connection to build. I've put up source tarball and packaging for unstable up at: http://www.percona.com/downloads/TESTING/XtraBackup/xtrabackup-2.1.3/release-2.1.3/610/source/ All future releases will produce tarballs like this, so they'll be in a more normal area of our downloads site - I'll also get one up in an official place for the current 2.1.3 release. I (and we, the rest of Percona) would really appreciate any review/comments/suggestions on this packaging - we're really keen to have Percona XtraBackup be in the Debian archives. Why is the tarball so big and why did it require an internet connection? Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts of InnoDB) to help it read InnoDB pages from disk and back them up safely. It also uses part of the code to replay REDO logs to prepare a backup before it can be restored. Due to various not-that-interesting reasons (all of which we do consider bugs) we currently produce different binaries for different MySQL versions (hence linking in code From different MySQL versions). It's important to note that we patch the InnoDB code inside MySQL to make it usable by xtrabackup and that we cannot just use the MySQL code already shipped in Debian for this without creating a nightmare for the Debian MySQL packaging team, the security team and ourselves (it's generally non-trivial to port this patch to new versions). There should never be any security implications of linking against this (not always up to date) MySQL code as we only use it to read (and write) files to local disk. Questions: 1) We have a transitional package called 'xtrabackup' that was (prior to 2.0) the package name was in our own APT repositories. Does it make sense to keep this in the Debian packaging? 2) We have trademarks. We've tried to come up with a trademark policy that makes everybody happy (well, as happy as you can be when you're dealing with trademark law). It's at: http://www.percona.com/doc/percona-xtrabackup/2.1/trademark-policy.html We believe this should satisfy any requirements of Linux distributions, but if there's any concerns, don't hesitate to ask. 3) Supported architectures. We see x86-64, x86 and very, very rarely, SPARC. I'd say that both people running XtraBackup on a non-x86 architecture are happy, but I'm pretty sure there's not even that many. While XtraBackup may compile (or even run) on other architectures, it currently makes approximately zero business sense for us to spend any time on it - although we're happy to take patches. Given this, how much "works and passes tests on all archs" is required to have packages accepted? 4) We plan on continuing to provide our own repositories as well and are wanting to not diverge in packaging from our repos and what's in Debian. Any tips/tricks are very welcome - we have "build debian packages" as part of or continuous integration efforts and plan to soon have "run the whole test suite on those packages" as part of it. Thanks in advance for the feedback! -- Stewart Smith pgpsxgLsSmZt7.pgp Description: PGP signature
Re: ITP: Percona XtraBackup - hot backup utility for MySQL
Paul Wise writes: > On Tue, Jul 2, 2013 at 2:41 PM, Stewart Smith wrote: > >> I'm Stewart and I work for Percona. One of the things I'm currently >> working on is ensuring all our free and open source software projects >> are packaged for all the major linux distributions - including my >> beloved Debian. > > Since you are part of upstream, please review our upstream guide and > the links in the external advice section: > > http://wiki.debian.org/UpstreamGuide Thanks for the pointer. It appears that we're pretty good on most things mentioned there and some things we've got plans to address. >> There is an open bug for Percona XtraBackup packaging: >> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620824 > > Since you intend to package it, you should retitle this bug to intent > to package and set yourself as the owner of it: > > http://www.debian.org/devel/wnpp/#l3 > http://www.debian.org/Bugs/server-control#retitle > http://www.debian.org/Bugs/server-control#owner Done. >> I've put up source tarball and packaging for unstable up at: >> http://www.percona.com/downloads/TESTING/XtraBackup/xtrabackup-2.1.3/release-2.1.3/610/source/ > > I'm unable to unpack the source package: > > $ dpkg-source -x percona-xtrabackup_2.1.3-610-1.dsc > dpkg-source: warning: extracting unsigned source package > (percona-xtrabackup_2.1.3-610-1.dsc) > dpkg-source: error: File ./percona-xtrabackup_2.1.3-610.orig.tar.gz > has size 39459 instead of expected 141074103 I'll investigate this first thing in the morning (it's getting late), although it looks as though the tar.gz didn't fully download (yes, it's really 135MB) > You may want to run some commands from the source tree after a build: > > http://wiki.debian.org/HowToPackageForDebian#Check_points_for_any_package I think there's some good ideas here that we can add to our CI infrastructure. I don't think there's anything that would be considered a blocker though. > Also, I encourage you to sign your Debian packages and also your > upstream tarballs using OpenPGP keys. Here are some best practices for > OpenPGP: > > https://we.riseup.net/riseuplabs+paow/openpgp-best-practices Yep, we typically do this. We use a key for the company rather than our individual ones for our current repositories. Is this an acceptable approach for packages going into Debian? Or should we just have the few individuals sign it if they're involved? >> Why is the tarball so big and why did it require an internet connection? >> Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts >> of InnoDB) > > An alternative to including the MySQL InnoDB code might be to > build-depend on mysql-source-5.5, unpack and patch it during the build > process and add Built-Using: mysql-5.5 (= $version) to the resulting > binary package. We'd need to depend on very specific 5.1, 5.5 and 5.6 versions of both MySQL and Percona Server (which isn't yet packaged) and we'd also have to patch it. Our short/medium term plan is to do away with having multiple server source trees and binaries. We instead plan to just make a modified MySQL branch where we just include the needed (modified) source. In an ideal world the needed code would be buildable as a library and we'd get the needed patches upstream. It is not an ideal world however :( >> 1) We have a transitional package called 'xtrabackup' that was (prior to >>2.0) the package name was in our own APT repositories. Does it make >>sense to keep this in the Debian packaging? > > Seems reasonable to include it if you really want to. There seem to be > more folks using percona-xtrabackup than xtrabackup though: I'm not really attached either way... It appears that it may not be worth having it. I'll defer to the experts on this one. >> 2) We have trademarks. We've tried to come up with a trademark policy >>that makes everybody happy (well, as happy as you can be when you're >>dealing with trademark law). >>It's at: >>http://www.percona.com/doc/percona-xtrabackup/2.1/trademark-policy.html >>We believe this should satisfy any requirements of Linux >>distributions, but if there's any concerns, don't hesitate to ask. > > I'd suggest mailing debian-legal about this question but based on the > trademark policy it appears that Debian would have to rename percona > if we wanted to add a security patch in stable? I'll bring it up there. We would see that as simply a patch that's needed for that version to be included in that Os and thus not an issue. > You may want to listen to this FOSDEM talk entitled 'How to Share a
Re: ITP: Percona XtraBackup - hot backup utility for MySQL
Vincent Bernat writes: > ❦ 2 juillet 2013 09:20 CEST, Paul Wise : > >>> Why is the tarball so big and why did it require an internet connection? >>> Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts >>> of InnoDB) >> >> An alternative to including the MySQL InnoDB code might be to >> build-depend on mysql-source-5.5, unpack and patch it during the build >> process and add Built-Using: mysql-5.5 (= $version) to the resulting >> binary package. > > The problem is that it should Build-Depends on Percona MySQL which is > not yet available in Debian. I think the first step would be to provide > Percona MySQL packages. Currently, it seems that there is no easy way to > provide packages as alternative to MySQL but maybe work has already > started with MariaDB. It would need to build-depend on specific MySQL and Percona Server versions (which end up being on-disk compatible with various MariaDB versions). We're working on getting Percona Server ready to be included too, but we'd be going for the current Percona Server versions not the ones we patch heavily to then build XtraBackup with. But, the main problem is that we need to patch the InnoDB source to build XtraBackup and the patch isn't trivial to port to the latest versions (and there's usually no benefit in doing so). It's better to think of XtraBackup patching and including the parts of InnoDB it needs and for reasons that are largely historical it's multiple binaries for multiple versions of InnoDB. Our short/medium term goal is to stop doing this and actually just have one set of InnoDB code from the most recent MySQL/Percona Server version and one XtraBackup binary that works on all server versions. In an ideal world there'd be some library we could link against and have the patches be accepted upstream.. but we're not in an ideal world. This would make it more like something like xfsprogs which does end up including some of the same code as the kernel (although we patch InnoDB while xfsprogs seems to be about at the point where that isn't necessary). So while we do understand how it'd be great to build against the source in mysql-source-5.5, it's just not really feasible to do so and maintain quality and maintainability. We're hoping that our short/medium term of just including the patched InnoDB bits we need in our source tree to mostly solve the problem. I hope this helps clarify. -- Stewart Smith pgpocLz_8VZAP.pgp Description: PGP signature
Re: ITP: Percona XtraBackup - hot backup utility for MySQL
Paul Wise writes: > On Tue, Jul 2, 2013 at 7:35 PM, Stewart Smith wrote: > >> I'll investigate this first thing in the morning (it's getting late), >> although it looks as though the tar.gz didn't fully download (yes, it's >> really 135MB) > > Looks like your tarball is named incorrectly (tar.gz instead of > orig.tar.gz) and your website returns a HTTP 302 code (redirect) and > some HTML for the orig.tar.gz URL rather than a 404. I'll ensure our process includes putting the tarball up named correctly. The tarball in that directory should be the right one when named correctly though (unless I brain-farted and got it wrong :) >> Yep, we typically do this. We use a key for the company rather than our >> individual ones for our current repositories. Is this an acceptable >> approach for packages going into Debian? Or should we just have the few >> individuals sign it if they're involved? > > It is a reasonable approach for upstream tarballs. Uploads to Debian > need to be signed by one of the keys in the developer keyring or a > subkey of one of those keys. My guess is the reasonable thing to do is ensure that a few of us have keys in the keyring then so that there's some redundancy (after all, we have to let people take vacation :) >> We'd need to depend on very specific 5.1, 5.5 and 5.6 versions of both >> MySQL and Percona Server (which isn't yet packaged) and we'd also have >> to patch it. >> >> Our short/medium term plan is to do away with having multiple server >> source trees and binaries. We instead plan to just make a modified MySQL >> branch where we just include the needed (modified) source. In an ideal >> world the needed code would be buildable as a library and we'd get the >> needed patches upstream. It is not an ideal world however :( > > I see, maybe you have chosen the way of least pain. We try to :) >> I'm not really attached either way... It appears that it may not be >> worth having it. I'll defer to the experts on this one. > > Maybe ask your userbase if they need it? I'll reach out through our support org. I suspect we'd be fine without it. >> I'll bring it up there. We would see that as simply a patch that's >> needed for that version to be included in that Os and thus not an issue. > > The danger is if that approach to interpretation if the policy were to > change. Okay. I'll bring it up on debian-devel and see if we need changes (it just means I probably get to talk trademark law more... which is generally something I try to avoid :) >> It's certainly something we may be interested at looking at... I guess >> we'll see what the buildds say :) >> >> We do have a test suite. It does (of course) depend on having server >> binaries around to run backup and restore with. Our CI infrastructure >> currently uses just binary tarballs of various server versions, but this >> probably isn't ideal for Debain so we'll have to come up with another >> solution. One thing to consider is that (like MySQL) running the test >> suite takes a non-trivial amount of time. > > Please do enable the test suite for Debian builds and run them with > the binaries built during the build process. The problem we'll have to solve is running against MySQL (and Percona Server) versions... and I'm not sure the best way to wrangle that (we probably have to fix a bug or two in the test suite to ensure it works against an installed binary - we currently download binary tarballs and run against them). -- Stewart Smith pgpftxrCZEIR9.pgp Description: PGP signature
Bug#718227: ITP: percona-playback -- A tool for replaying captured database server load
Package: wnpp Severity: wishlist Owner: stewart.sm...@percona.com Percona Playback is a tool for replaying the captured load of one database server against another in the most realistic way possible. Captured load can come in the form of MySQL slow query logs or tcpdump capture. It's multithreaded, modular and configurable to allow for flexibility and future extension. http://www.percona.com/doc/percona-playback/ http://www.percona.com/downloads/Percona-Playback/ Daily source packages for unstable: http://jenkins.percona.com/view/Playback/job/percona-playback-trunk-dpkg-source/debian_distribution=unstable,label_exp=sbuild/ Daily binary packages for unstable: http://jenkins.percona.com/view/Playback/job/percona-playback-trunk-dpkg/architecture=i386,debian_distribution=unstable,label_exp=sbuild/ http://jenkins.percona.com/view/Playback/job/percona-playback-trunk-dpkg/architecture=amd64,debian_distribution=unstable,label_exp=sbuild/ I'm intending to package the 0.7 release, which, since I'm in control of upstream, makes packaging for Debian easier and will incorporate any changes needed to be packaged. -- Stewart Smith pgpRyuykOArrx.pgp Description: PGP signature
Bug#718228: ITP: qpress -- Utility for manipulating quicklz archives
Package: wnpp Severity: wishlist Owner: stewart.sm...@percona.com From http://www.quicklz.com/ : QuickLZ is the world's fastest compression library, reaching 308 Mbyte/s per core. It can be used under a commercial license if such has been acquired or under GPL 1, 2 or 3 where anything released into public must be open source. - Simple to use and easy to integrate. Get done in minutes and continue developing! - Streaming mode for optimal compression ratio of small packets down to 200 - 300 bytes in size. - Auto-detection and fast treatment of incompressible data. Note: I'm working with upstream to fix a few issues with how they distribute the quicklz source that makes it a bit hard to package. -- Stewart Smith pgp4GOoYMhc1B.pgp Description: PGP signature
Re: Bits from the Release Team (Jessie freeze info)
Steven Chamberlain writes: > On 21/10/13 16:42, Niels Thykier wrote: >> I would love for us to have an automated system to give us a >> "weather-report" on the toolchain for each architecture. It would be >> nice both for us to see how ports are doing and for porters to spot and >> fix problems early. > > That sounds a lot like the purpose of Jenkins[0], but I'm not sure if > it's exactly suitable. It seems a little heavy, that someone could more > easily be able to script some cron jobs for a task than learn how to > use it. It's actually a pretty low barrier to entry, if you know what commands you need to run, it's pretty easy to get started with jenkins (create job, have it execute shell commands, write shell in box, hit build). I'd say that it's about 10 times more likely you'll get it right in Jenkins before you get it right in cron. > And Jenkins isn't available yet on all arches; some ports may not have > hardware powerful enough to run it. Maybe that doesn't matter - a > single Jenkins instance might be able to launch jobs via remote shells > to other boxes, running the actual test suite there, or maybe just to > fetch, analyse and report on the resulting log files. Jenkins can have slaves on remote hosts, via SSH. It runs a small java app there, so as long as the arch has a JVM then you're pretty right. -- Stewart Smith pgpK9r2igtQxx.pgp Description: PGP signature
Re: Bits from the Release Team (Jessie freeze info)
Geert Uytterhoeven writes: > On Wed, Oct 23, 2013 at 12:36 AM, Stewart Smith > wrote: >> Jenkins can have slaves on remote hosts, via SSH. It runs a small java >> app there, so as long as the arch has a JVM then you're pretty right. > > For whatever definition of small. I've seen it consuming 1 GiB of > memory... with 'm68k' in your email address your definition of small is likely much different than my "many years in large scale databases" small :) That being said... I haven't recently seen a slave jenkins java process more than one or two hundred mb. This is (of course) absolutely insane, as is the 4-6GB jenkins master process. However, dollars per GB of memory is suitably low that it's not worth me fixing it, instead it just sits there annoying me as it could undoubtedly be better -- Stewart Smith pgpYLYPIugD_P.pgp Description: PGP signature
Re: [debian-mysql] MySQL.. no.. _I_ need your help!
Clint Byrum writes: > I am asking you, the Debian developers, to step up and help. I am > basically unable to contribute more than an hour a month now. There is a > new round of secret CVE bugs to fix, and some old bugs that need to be > handled. I think my October hour is about to be available, so I might > be able to address those, but after that, if I don't get any more help, > I'm done. > > What can you do to help? > > - Raise your hand and say you'll help I'm currently putting effort into getting Percona Server packages able to be included and after that work is done, will be focusing on having less variation between them and the MySQL packages, hopefully improving everything as part of that effort. So this is a somewhat raised hand :) -- Stewart Smith pgphYjRKjv0J5.pgp Description: PGP signature
