Hi Chris and Craig, Thank you for sharing your experience with me about S3 API RADOS gateway!
Jerry Lam Senior Software Developer, Big Data Ontario Institute for Cancer Research MaRS Centre 661 University Avenue Suite 510 Toronto, Ontario Canada M5G 0A3 Email: jerry....@oicr.on.ca Toll-free: 1-866-678-6427 Twitter: @OICR_news www.oicr.on.ca<http://www.oicr.on.ca> This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization. From: Chris Jones <cjo...@cloudm2.com<mailto:cjo...@cloudm2.com>> Date: Friday, March 20, 2015 at 7:28 PM To: Craig Lewis <cle...@centraldesktop.com<mailto:cle...@centraldesktop.com>> Cc: Jerry Lam <jerry....@oicr.on.ca<mailto:jerry....@oicr.on.ca>>, "ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>" <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>> Subject: Re: [ceph-users] RADOS Gateway Maturity Hi Jerry, We using RGW and RBD in our OpenStack clusters and as stand alone clusters. We have six large clusters and adding more. Most of any issues we have faced have been self inflicted such as not currently supporting bucket names like host names. Some S3 tools only work that way which causes some of our developer customers grief. We are addressing that. We have built extensive testing frameworks around S3 RGW testing using a OpenStack or AWS EC2 or Google Cloud Platform to dynamically spin up worker nodes to distribute load for stressing and performance monitoring. I'm actually building a project called IQStack that will be released on github soon that does that plus other OpenStack testing for scalability. Anyway, there may be some incompatibilities depending on the feature set but most can be abstracted away and addressed. Just as a footnote, I finished running long load testing against AWS S3 and RGW. After tuning our load balancers, firewall rules and a few other tweaks I was able to get parity with AWS S3 up to 10GbE (max size of my load balancer I was testing with). We use several CDNs on video clips so my tests were with 2MB byte-range requests. Thanks, Chris On Fri, Mar 20, 2015 at 6:06 PM, Craig Lewis <cle...@centraldesktop.com<mailto:cle...@centraldesktop.com>> wrote: I have found a few incompatibilities, but so far they're all on the Ceph side. One example I remember was having to change the way we delete objects. The function we originally used fetches a list of object versions, and deletes all versions. Ceph is implementing objects versions now (I believe that'll ship with Hammer), so we had to call a different function to delete the object without iterating over the versions. AFAIK, that code should work fine if we point it at Amazon. I haven't tried it though. I've been using RGW (with replication) in production for 2 years now, although I'm not large. So far, all of my RGW issues have been Ceph issues. Most of my issues are caused by my under-powered hardware, or shooting myself in the foot with aggressive optimizations. Things are better with my journals on SSD, but the best thing I did was slow down with my changes. For example, I have 7 OSD nodes and 72 OSDs. When I add new OSDs, I add a couple at a time instead of adding all the disks in a node at once. Guess how I learned that lesson. :-) On Wed, Mar 18, 2015 at 10:03 AM, Jerry Lam <jerry....@oicr.on.ca<mailto:jerry....@oicr.on.ca>> wrote: Hi Chris, Thank you for your reply. We are also thinking about using the S3 API but we are concerned about how compatible it is with the real S3. For instance, we would like to design the system using pre-signed URL for storing some objects. I read the ceph documentation, it does not mention if it supports it or not. My question is do you guys find that the code using the RADOS S3 API can easily run in Amazon S3 without any change? If no, how much effort it is needed to make it compatible? Best Regards, Jerry From: Chris Jones <cjo...@cloudm2.com<mailto:cjo...@cloudm2.com>> Date: Tuesday, March 17, 2015 at 4:39 PM To: Jerry Lam <jerry....@oicr.on.ca<mailto:jerry....@oicr.on.ca>> Cc: "ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>" <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>> Subject: Re: [ceph-users] RADOS Gateway Maturity Hi Jerry, I currently work at Bloomberg and we currently have a very large Ceph installation in production and we use the S3 compatible API for rados gateway. We are also re-architecting our new RGW and evaluating a different Apache configuration for a little better performance. We only use replicas right now, no erasure coding yet. Actually, you can take a look at our current configuration at https://github.com/bloomberg/chef-bcpc. -Chris On Tue, Mar 17, 2015 at 10:40 AM, Jerry Lam <jerry....@oicr.on.ca<mailto:jerry....@oicr.on.ca>> wrote: Hi Ceph user, I’m new to Ceph but I need to use Ceph as the storage for the Cloud we are building in house. Did anyone use RADOS Gateway in production? How mature it is in terms of compatibility with S3 / Swift? Anyone can share their experience on it? Best Regards, Jerry _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Best Regards, Chris Jones [http://cloudcm2.appspot.com/img/cm2_horizontal.png]<http://www.cloudm2.com> cjo...@cloudm2.com<mailto:cjo...@cloudm2.com> (p) 770.655.0770<tel:770.655.0770> This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Best Regards, Chris Jones [http://cloudcm2.appspot.com/img/cm2_horizontal.png]<http://www.cloudm2.com> cjo...@cloudm2.com<mailto:cjo...@cloudm2.com> (p) 770.655.0770 This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
