What is your expected behavior for when Client A writes to File B in Datacenter 1 and Client C writes to File B in Datacenter 2 at the exact same time?
I don't think you can perfectly achieve what you are requesting with Ceph or many other storage solutions. On Mon, May 21, 2018 at 9:33 AM, Up Safe <upands...@gmail.com> wrote: > I'll explain. > Right now we have 2 sites (racks) with several dozens of servers at each > accessing a NAS (let's call it a NAS, although it's an IBM v7000 Unified > that serves the files via NFS). > > The biggest problem is that it works active-passive, i.e. we always access > one of the storages for read/write > and the other one is replicated once every few hours, so it's more for > backup needs. > > In this setup once the power goes down in our main site - we're stuck with > a bit (several hours) outdated files > and we need to remount all of the servers and what not. > > The multi site ceph was supposed to solve this problem for us. This way we > would have only local mounts, i.e. > each server would only access the filesystem that is in the same site. And > if one of the sited go down - no pain. > > The files are rather small, pdfs and xml of 50-300KB mostly. > The total size is about 25 TB right now. > > We're a low budget company, so your advise about developing is not going > to happen as we have no such skills or resources for this. > Plus, I want to make this transparent for the devs and everyone - just an > infrastructure replacement that will buy me all of the ceph benefits and > allow the company to survive the power outages or storage crashes. > > > > On Mon, May 21, 2018 at 5:12 PM, David Turner <drakonst...@gmail.com> > wrote: > >> Not a lot of people use object storage multi-site. I doubt anyone is >> using this like you are. In theory it would work, but even if somebody has >> this setup running, it's almost impossible to tell if it would work for >> your needs and use case. You really should try it out for yourself to see >> if it works to your needs. And if you feel so inclined, report back here >> with how it worked. >> >> If you're asking for advice, why do you need a networked posix >> filesystem? Unless you are using proprietary software with this >> requirement, it's generally lazy coding that requires a mounted filesystem >> like this and you should aim towards using object storage instead without >> any sort of NFS layer. It's a little more work for the developers, but is >> drastically simpler to support and manage. >> >> On Mon, May 21, 2018 at 10:06 AM Up Safe <upands...@gmail.com> wrote: >> >>> guys, >>> please tell me if I'm in the right direction. >>> If ceph object storage can be set up in multi site configuration, >>> and I add ganesha (which to my understanding is an "adapter" >>> that serves s3 objects via nfs to clients) - >>> won't this work as active-active? >>> >>> >>> Thanks >>> >>> On Mon, May 21, 2018 at 11:48 AM, Up Safe <upands...@gmail.com> wrote: >>> >>>> ok, thanks. >>>> but it seems to me that having pool replicas spread over sites is a bit >>>> too risky performance wise. >>>> how about ganesha? will it work with cephfs and multi site setup? >>>> >>>> I was previously reading about rgw with ganesha and it was full of >>>> limitations. >>>> with cephfs - there is only one and one I can live with. >>>> >>>> Will it work? >>>> >>>> >>>> On Mon, May 21, 2018 at 10:57 AM, Adrian Saul < >>>> adrian.s...@tpgtelecom.com.au> wrote: >>>> >>>>> >>>>> >>>>> We run CephFS in a limited fashion in a stretched cluster of about >>>>> 40km with redundant 10G fibre between sites – link latency is in the order >>>>> of 1-2ms. Performance is reasonable for our usage but is noticeably >>>>> slower >>>>> than comparable local ceph based RBD shares. >>>>> >>>>> >>>>> >>>>> Essentially we just setup the ceph pools behind cephFS to have >>>>> replicas on each site. To export it we are simply using Linux kernel NFS >>>>> and it gets exported from 4 hosts that act as CephFS clients. Those 4 >>>>> hosts are then setup in an DNS record that resolves to all 4 IPs, and we >>>>> then use automount to do automatic mounting and host failover on the NFS >>>>> clients. Automount takes care of finding the quickest and available NFS >>>>> server. >>>>> >>>>> >>>>> >>>>> I stress this is a limited setup that we use for some fairly light >>>>> duty, but we are looking to move things like user home directories onto >>>>> this. YMMV. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On >>>>> Behalf Of *Up Safe >>>>> *Sent:* Monday, 21 May 2018 5:36 PM >>>>> *To:* David Turner <drakonst...@gmail.com> >>>>> *Cc:* ceph-users <ceph-users@lists.ceph.com> >>>>> *Subject:* Re: [ceph-users] multi site with cephfs >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> can you be a bit more specific? >>>>> >>>>> I need to understand whether this is doable at all. >>>>> >>>>> Other options would be using ganesha, but I understand it's very >>>>> limited on NFS; >>>>> >>>>> or start looking at gluster. >>>>> >>>>> >>>>> >>>>> Basically, I need the multi site option, i.e. active-active read-write. >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> On Wed, May 16, 2018 at 5:50 PM, David Turner <drakonst...@gmail.com> >>>>> wrote: >>>>> >>>>> Object storage multi-site is very specific to using object storage. >>>>> It uses the RGW API's to sync s3 uploads between each site. For CephFS >>>>> you >>>>> might be able to do a sync of the rados pools, but I don't think that's >>>>> actually a thing yet. RBD mirror is also a layer on top of things to sync >>>>> between sites. Basically I think you need to do something on top of the >>>>> Filesystem as opposed to within Ceph to sync it between sites. >>>>> >>>>> >>>>> >>>>> On Wed, May 16, 2018 at 9:51 AM Up Safe <upands...@gmail.com> wrote: >>>>> >>>>> But this is not the question here. >>>>> >>>>> The question is whether I can configure multi site for CephFS. >>>>> >>>>> Will I be able to do so by following the guide to set up the multi >>>>> site for object storage? >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> On Wed, May 16, 2018, 16:45 John Hearns <hear...@googlemail.com> >>>>> wrote: >>>>> >>>>> The answer given at the seminar yesterday was that a practical limit >>>>> was around 60km. >>>>> >>>>> I don't think 100km is that much longer. I defer to the experts here. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 16 May 2018 at 15:24, Up Safe <upands...@gmail.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> About a 100 km. >>>>> >>>>> I have a 2-4ms latency between them. >>>>> >>>>> >>>>> >>>>> Leon >>>>> >>>>> >>>>> >>>>> On Wed, May 16, 2018, 16:13 John Hearns <hear...@googlemail.com> >>>>> wrote: >>>>> >>>>> Leon, >>>>> >>>>> I was at a Lenovo/SuSE seminar yesterday and asked a similar question >>>>> regarding separated sites. >>>>> >>>>> How far apart are these two geographical locations? It does matter. >>>>> >>>>> >>>>> >>>>> On 16 May 2018 at 15:07, Up Safe <upands...@gmail.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm trying to build a multi site setup. >>>>> >>>>> But the only guides I've found on the net were about building it with >>>>> object storage or rbd. >>>>> >>>>> What I need is cephfs. >>>>> >>>>> I.e. I need to have 2 synced file storages at 2 geographical locations. >>>>> >>>>> Is this possible? >>>>> >>>>> Also, if I understand correctly - cephfs is just a component on top of >>>>> the object storage. >>>>> >>>>> Following this logic - it should be possible, right? >>>>> >>>>> Or am I totally off here? >>>>> >>>>> Thanks, >>>>> >>>>> Leon >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> Confidentiality: This email and any attachments are confidential and >>>>> may be subject to copyright, legal or some other professional privilege. >>>>> They are intended solely for the attention and use of the named >>>>> addressee(s). They may only be copied, distributed or disclosed with the >>>>> consent of the copyright owner. If you have received this email by mistake >>>>> or by breach of the confidentiality clause, please notify the sender >>>>> immediately by return email and delete or destroy all copies of the email. >>>>> Any confidentiality, privilege or copyright is not waived or lost because >>>>> this email has been sent to you by mistake. >>>>> >>>> >>>> >>> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
