Last week we wrote a blog post about introducting erasure codes into Swift. Today, I'm happy to share more technical details around this feature.
We've posted an overview of our design and some of the tradeoffs in the feature at http://swiftstack.com/blog/2013/07/17/erasure-codes-with-openstack-swift-digging-deeper/. I've also posted some slides with a high-level design at http://d.not.mn/EC_swift_proxy_design.pdf Here's the summary of the erasure code design: * Erasure codes (vs replicas) will be set on a per-container basis * Data will be erasure coded inline in the proxy server * API clients will control when data is replicated or erasure coded So what's next? How do we get this done? Of course, there's a lot of coding to be done. At a high level, and to tie everything together, I've updated Launchpad with several new blueprints. https://blueprints.launchpad.net/swift/+spec/swift-ec will keep track of the total work. More specifically, these are the areas that need to be worked on: * Proxy server (https://blueprints.launchpad.net/swift/+spec/ec-proxy-work) * EC Ring (https://blueprints.launchpad.net/swift/+spec/ec-ring) * EC Reconstructor (https://blueprints.launchpad.net/swift/+spec/ec-reconstructor) * EC Auditor (https://blueprints.launchpad.net/swift/+spec/ec-auditor) * EC Stripe Auditor (https://blueprints.launchpad.net/swift/+spec/ec-stripe-auditor) * EC library interface (https://blueprints.launchpad.net/swift/+spec/ec-library-interface) Of course, as work progresses, there may be other areas too. To facilitate the community dev work on this, I've done a couple of things. First, there is now an upstream "feature/ec" branch for this work. The purpose of having a separate ec branch is because this is a long-running feature that does not have a lot of independent features (eg it doesn't make sense to merge an EC Reconstructor into Swift without the rest of the EC work). We will frequently merge master back into the ec branch so that we don't get too divergent. Here's how to start using this branch locally: # go to the code and get the latest version cd swift && git fetch --all # checkout the upstream ec branch git checkout origin/feature/ec # create a local branch called ec to track the upstream ec branch git branch ec2 origin/feature/ec && git checkout ec # type, type, type # this will push the review to gerrit for review (normal rules apply). # on this ec branch, it will default to merge into the upstream feature/ec branch instead of master git review Second, I've also set up a trello board to keep track and discuss designs at https://trello.com/board/swift-erasure-codes/51e0814d4ee9022d2b002a2c. Why trello? Mostly because Launchpad isn't sufficient to have discussions around the design, and gerrit only supports discussion around a particular piece of code. Lastly, I would encourage anyone interested in this effort to participate in the #openstack-swift IRC channel on freenode and to attend the every-other-week swift meetings in #openstack-meeting (the next meeting is July 24 at 1900UTC). I'm really looking forward to this feature, and I'm excited to see memebers of the OpenStack community coming together to implement it. --John
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
