Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-24 Thread Pete Zaitcev
On Thu, 18 Jul 2013 12:31:02 -0500 Chuck Thier wrote: > I'm with Chmouel though. It seems to me that EC policy should be chosen by > the provider and not the client. For public storage clouds, I don't think > you can make the assumption that all users/clients will understand the > storage/laten

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-22 Thread John Dickinson
On Jul 22, 2013, at 9:34 AM, David Hadas wrote: > Hi, > > In Portland, we discussed a somewhat related issue of having multiple > replication levels in one Swift cluster. > It may be that a provider would not wish to expose the use of EC or the level > of replication used. For example a pro

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-22 Thread Luse, Paul E
allows for their efficient use and flexible definitions. Thx Paul From: David Hadas [mailto:david.ha...@gmail.com] Sent: Monday, July 22, 2013 9:35 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Swift] erasure codes, digging deeper Hi, In Portland, we discussed a somewh

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-22 Thread David Hadas
Hi, In Portland, we discussed a somewhat related issue of having multiple replication levels in one Swift cluster. It may be that a provider would not wish to expose the use of EC or the level of replication used. For example a provider may offer a predefined set of services such as "Gold", "Silve

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread John Dickinson
Yes, and I'd imagine that the normal default would be for replicated data. Moving the granularity from a container to account-based, as Chmouel and Chuck said, is interesting too. --John On Jul 18, 2013, at 11:32 AM, Christian Schwede wrote: > A solution to this might be to set the default po

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Christian Schwede
A solution to this might be to set the default policy as a configuration setting in the proxy. If you want a replicated swift cluster just allow this policy in the proxy and set it to default. The same for EC cluster, just set the allowed policy to EC. If you want both (and let your users decide wh

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Chuck Thier
I think you are missing the point. What I'm talking about is who chooses what data is EC and what is not. The point that I am trying to make is that the operators of swift clusters should decide what data is EC, not the clients/users. How the data is stored should be totally transparent to the u

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread John Dickinson
Are you talking about the parameters for EC or the fact that something is erasure coded vs replicated? For the first, that's exactly what we're thinking: a deployer sets up one (or more) policies and calls them Alice, Bob, or whatever, and then the API client can set that on a particular contai

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Clay Gerrard
I'm not sure if I agree with the giving justifications (harder to bill, confusing to users), but I *do* think it could simplify some of the implementation (since users can't create accounts withe the "wrong" storage policy willy nilly). I think eventually some use cases may want mixed accounts (pa

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Chuck Thier
I'm with Chmouel though. It seems to me that EC policy should be chosen by the provider and not the client. For public storage clouds, I don't think you can make the assumption that all users/clients will understand the storage/latency tradeoffs and benefits. On Thu, Jul 18, 2013 at 8:11 AM, Jo

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread John Dickinson
Check out the slides I linked. The plan is to enable an EC policy that is then set on a container. A cluster may have a replication policy and one or more EC policies. Then the user will be able to choose the policy for a particular container. --John On Jul 18, 2013, at 2:50 AM, Chmouel Bou

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Chmouel Boudjnah
On Thu, Jul 18, 2013 at 12:42 AM, John Dickinson wrote: > * Erasure codes (vs replicas) will be set on a per-container basis I was wondering if there was any reasons why it couldn't be as per-account basis as this would allow an operator to have different type of an account and different pric