Just a bump:) On Mon, Sep 1, 2014 at 1:49 PM, Marcus White <roastedseawee...@gmail.com> wrote: > Hello John > Thanks a lot for all your answers:) > > Some more follow up questios > > You mentioned you see both deployments.. > > 1. For the case where there are different IP addresses for different > regions, I am assuming a different keystone deployment for each site > with clients knowing the auth endpoints of each site, and getting the > swift endpoints from the auth endpoints service catalog. They know or > figure out which one is local themselves.. > > 2. For the multi geo case with multiple IPs, wanted to query here > again on the answer you mentioned, that the keystone database is > synced across sites..For the multi site case you had mentioned you see > in deployments with multiple IP addresses, how do you guys set up > keystone so that the user information is the same across all sites? > Probably related to Question 1. > > MW > > On Sun, Aug 31, 2014 at 11:41 PM, John Dickinson <m...@not.mn> wrote: >> >> On Aug 31, 2014, at 5:25 AM, Marcus White <roastedseawee...@gmail.com> wrote: >> >>> One last follow up question, hopefully! >>> >>> I agree, each proxy server can handle any requests..I wanted to see >>> what the typical deployment is for the multi geo case. >>> >>> For multi region global clusters, for example, each region would have >>> a separate IP address? Behind each regions IP address you can have >>> multiple proxy servers in each site, and a customer can connect to any >>> site? >> >> I've seen both. Really it depends on the goals of the global cluster and the >> existing infrastructure available. If the goal is a DR story, then often DNS >> is pointed at one region, and when the "R" of DR is needed, the DNS entry is >> updated to point to the other region. Other times, if the goal is to allow >> for lower-latency reads and writes, then then all regions are used for >> access all the time. The client either relies on some sort of >> geo-dns/anycast system to get to the closest endpoint, or the client chooses >> the preferred region from a list of available ones. >> >> Yes, I'd normally expect each region to have a separate public IP address >> with multiple proxies load balanced behind it. >> >>> >>> In a single site case, keystone would give you one end point, which is >>> the external load balancer Ip. For the multi site case, what happens >>> with keystone? Does it provide multiple end points? >>> >>> Another question is, does the keystone server itself have a different >>> external IP for each site? >> >> My understanding is that Keystone doesn't really work across wide geographic >> areas (I'd be happy to be wrong here--maybe a Keystone dev can jump in). I >> think the limitation is around the backing store Keystone uses to manage >> tokens and the service catalog. >> >> There are a few things to consider when using a central auth system with a >> globally distributed Swift cluster. First, note that if the auth token >> (identity info) isn't cached in the local region, Swift will need to ask the >> Keystone service for the information. Swift will cache it if possible, but >> know that the auth lookup will add latency (noticeable latency if the auth >> system is across an ocean). Second, make sure memcache and Swift's cache >> middleware are configured properly. For example, you don't want a global >> memcache pool to result in 50% of the internal "fetch from cache" requests >> to go across the WAN. You'd instead want a separate memcache pool for each >> region. >> >> As a side note, a centralized auth system actually makes the caching >> strategies easier, since there's less complexity to deal with when trying to >> get and cache and validate auth credentials and tokens. >> >> >>> >>> Just confused a bit on the deployment basics. >> >> Good luck! and feel free to ask more as needed. >> >>> >>> MW. >>> >>> >>> On Fri, Aug 29, 2014 at 7:51 PM, John Dickinson <m...@not.mn> wrote: >>>> >>>> On Aug 29, 2014, at 2:43 AM, Marcus White <roastedseawee...@gmail.com> >>>> wrote: >>>> >>>>> Thanks John:) >>>>> >>>>> Some more additional basic questions.. >>>>> >>>>> 1. For the multi site case, does Swift present a single end point, or >>>>> is it two separate regions with two separate IP addresses, and the >>>>> client can talk to either to get the data? I think it is the latter? >>>> >>>> Every proxy server in the cluster will be able to handle requests. There >>>> isn't a single "control node" that brokers everything. Often times, all of >>>> the proxy servers are put behind a load balancer or geo-dns or something >>>> to present a single domain for the cluster. >>>> >>>> >>>>> >>>>> 2. Are keystone databases synced across sites? I didnt know of an >>>>> option in keystone to do that, hence asking..If we need to >>>>> authenticate against keystone on different sites to access the same >>>>> object, it has to be that way? >>>> >>>> That is also my understanding, but I'm not an expert on Keystone options >>>> or deployment patterns. >>>> >>>>> >>>>> TIA, >>>>> >>>>> MW >>>>> >>>>> >>>>> On Wed, Aug 27, 2014 at 11:37 PM, John Dickinson <m...@not.mn> wrote: >>>>>> Yup, this is a totally possible failure scenario, and Swift will merge >>>>>> the data (using last-write-wins for overwrites) automatically when the >>>>>> partition is restored. But you'll still have full durability on writes, >>>>>> even with a partitioned global cluster. >>>>>> >>>>>> --John >>>>>> >>>>>> >>>>>> >>>>>> On Aug 27, 2014, at 10:49 AM, Marcus White <roastedseawee...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yup, thanks for the great explanation:) >>>>>>> >>>>>>> Another question, though related: If there are three regions, and two >>>>>>> get "split", there is now a partition. Both the split ones can talk to >>>>>>> the third, but not to each other. >>>>>>> >>>>>>> A PUT comes into one region, and it gets written to the local site. >>>>>>> Container information presumably gets updated here, including byte >>>>>>> count. >>>>>>> >>>>>>> Same thing happens on another site, where a PUT comes in and the >>>>>>> container information is updated with the byte count. >>>>>>> When the sites get back together, do the container servers make sure >>>>>>> its all correct in the end? >>>>>>> >>>>>>> If this is not a possible scenario, is there any case where the >>>>>>> container metadata can be different between two zones or regions >>>>>>> because of a partition, and independent PUTs can happen, and the data >>>>>>> has to be merged? Is that all done by the respective servers(container >>>>>>> or account?) >>>>>>> >>>>>>> I will be looking at the sources soon. >>>>>>> >>>>>>> Thanks again >>>>>>> MW. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 27, 2014 at 8:13 PM, Luse, Paul E <paul.e.l...@intel.com> >>>>>>> wrote: >>>>>>>> Marcus- >>>>>>>> >>>>>>>> Not sure how much nitty gritty detail you care to know as some of >>>>>>>> these answers will get into code specifics which you're better off >>>>>>>> exploring on your own so my explanation isn't potentially dated. At a >>>>>>>> high level though, the proxy looks up the nodes that are responsible >>>>>>>> for the storing of an object and its container via the rings. It >>>>>>>> passes that info to the storage nodes when it does the PUT request so >>>>>>>> when the storage node goes to update the container it's been told "and >>>>>>>> here are the nodes to send the container update to". It will send the >>>>>>>> updates to all of them. Similarly, once the container server has >>>>>>>> updated its database it goes and updates the appropriate account >>>>>>>> databases. >>>>>>>> >>>>>>>> Make sense? >>>>>>>> >>>>>>>> Thx >>>>>>>> Paul >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Marcus White [mailto:roastedseawee...@gmail.com] >>>>>>>> Sent: Wednesday, August 27, 2014 7:04 AM >>>>>>>> To: Luse, Paul E >>>>>>>> Cc: openstack >>>>>>>> Subject: Re: [Openstack] Swift questions >>>>>>>> >>>>>>>> Thanks Paul:) >>>>>>>> >>>>>>>> For the container part, you mentioned that node(meaning object >>>>>>>> server?) contacts the container server. Since you can have multiple >>>>>>>> container servers, how does the object server know which container >>>>>>>> server to contact? How and where the container gets updated is a bit >>>>>>>> confusing. With container rings and account rings being separate and >>>>>>>> in the proxy part, I am not sure I understand how that path works. >>>>>>>> >>>>>>>> MW >>>>>>>> >>>>>>>> On Wed, Aug 27, 2014 at 6:15 PM, Luse, Paul E <paul.e.l...@intel.com> >>>>>>>> wrote: >>>>>>>>> Hi Marcus, >>>>>>>>> >>>>>>>>> See answers below. Feel free to ask follow-ups, others may have more >>>>>>>>> to add as well. >>>>>>>>> >>>>>>>>> Thx >>>>>>>>> Paul >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Marcus White [mailto:roastedseawee...@gmail.com] >>>>>>>>> Sent: Wednesday, August 27, 2014 5:04 AM >>>>>>>>> To: openstack >>>>>>>>> Subject: [Openstack] Swift questions >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> Some questions on new and old features of Swift. Any help would be >>>>>>>>> great:) Some are very basic, sorry! >>>>>>>>> >>>>>>>>> 1. Does Swift write two copies and then return back to the client in >>>>>>>>> the 3 replica case, with third in the background? >>>>>>>>> >>>>>>>>> PL> Depends on the number of replicas, the formula for what we call >>>>>>>>> a quorum is n/2 + 1 which is the number of success responses we get >>>>>>>>> from the back end storage nodes before telling the client that all is >>>>>>>>> good. So, yes, with 3 replicas you need 2 good responses before >>>>>>>>> returning OK. >>>>>>>>> >>>>>>>>> 2. This again is a stupid question, but eventually consistent for an >>>>>>>>> object is a bit confusing, unless it is updated. If it is created, it >>>>>>>>> is either there or not and you cannot update the data within the >>>>>>>>> object. Maybe a POST can change the metadata? Or the container >>>>>>>>> listing shows its there but the actual object never got there? Those >>>>>>>>> are the only cases I can think of. >>>>>>>>> >>>>>>>>> PL> No, it's a good question because its asked a lot. The most >>>>>>>>> common scenario that we talk about for eventually consistent is the >>>>>>>>> consistency between the existence of an object and its presence in >>>>>>>>> the container listing so your thinking is pretty close. When an >>>>>>>>> object PUT is complete on a storage node (fully committed to disk), >>>>>>>>> that node will then send a message to the appropriate container >>>>>>>>> server to update the listing. It will attempt to do this >>>>>>>>> synchronously but if it can't, the update may be delayed w/o any >>>>>>>>> indication to the client. This is by design and means that it's >>>>>>>>> possible to get a successful PUT, be able to GET the object w/o any >>>>>>>>> problem however it may not yet show up in the container listing. >>>>>>>>> There are other scenarios that demonstrate the eventually consistent >>>>>>>>> nature of Swift, this is just a common and easy to explain one. >>>>>>>>> >>>>>>>>> 3. Once an object has been written, when and how is the container >>>>>>>>> listing, number of bytes, account listing (if new container created) >>>>>>>>> etc updated? Is there something done in the path of the PUT to >>>>>>>>> indicate this object belongs to a particular container and the number >>>>>>>>> of bytes etc is done in the background? A little clarification would >>>>>>>>> help:) >>>>>>>>> >>>>>>>>> PL> Covered as part of last question. >>>>>>>>> >>>>>>>>> 4. For the global clusters, is the object ring across regions or is >>>>>>>>> it the same with containers and accounts also? >>>>>>>>> >>>>>>>>> PL> Check out the SwiftStack blog if you haven't already at >>>>>>>>> https://swiftstack.com/blog/2013/07/02/swift-1-9-0-release/ and >>>>>>>>> there's also some other stuff (including a demo from the last summit) >>>>>>>>> that you can find googling around a bit too. The 'Region Tier' >>>>>>>>> element described in the blog addresses the makeup of a ring so can >>>>>>>>> be applied to both container and account rings also - I personally >>>>>>>>> didn't work on this feature so will leave it to one of the other guys >>>>>>>>> to comment more in this area. >>>>>>>>> >>>>>>>>> 5. For containers in global clusters, if a client queries the >>>>>>>>> container metadata from another site, is there a chance of it getting >>>>>>>>> the old metadata? With respect to the object itself, the eventually >>>>>>>>> consistent part is a bit confusing for me:) >>>>>>>>> >>>>>>>>> PL> There's always a chance of getting old "something" whether its >>>>>>>>> metadata or data, that's part of eventually consistent. In the face >>>>>>>>> of an outage (the P in the CAP theorem) Swift will always favor >>>>>>>>> availability which may mean older data or older metadata (object or >>>>>>>>> container listing) depending on the specific scenario. If deployed >>>>>>>>> correctly I don't believe use of global clusters increases the odds >>>>>>>>> of this happening though (again will count on someone else to say >>>>>>>>> more) and its worth emphasizing the getting "old stuff" is in the >>>>>>>>> face of some sort of failure (or big network congestion) so you >>>>>>>>> shouldn't think of eventually consistent as being a system where you >>>>>>>>> "get whatever you get". You'll get the latest greatest available >>>>>>>>> information. >>>>>>>>> >>>>>>>>> MW >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Mailing list: >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>>>> Post to : openstack@lists.openstack.org >>>>>>>>> Unsubscribe : >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mailing list: >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> Post to : openstack@lists.openstack.org >>>>>>> Unsubscribe : >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>> >>>> >>
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
