Re: [ceph-users] Ceph and NFS
On Tue, Jan 19, 2016 at 12:36 PM, Gregory Farnum wrote: > > I've found that using knfsd does not preserve cephfs directory and file > > layouts, but using nfs-ganesha does. I'm currently using nfs-ganesha > 2.4dev5 > > and seems stable so far. > > Can you expand on that? In what manner is it not preserving directory > and file layouts? > -Greg > > My mistake - I wrongly assumed that the directories did not preserve layout - it was only subdirectories created under a directory that had a ceph.dir.layout xattr did not have ceph.dir.layout xattr. Files created under a subdir that does not have ceph.dir.layout still inherits the file layout. The same applies to kernel mounted as well as via knfsd, so it does preserve directory and file layouts. On a separate note, there's no way of knowing what the default ceph.dir.layout is for a subdirectory unless you traverse up the tree. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CentOS 7 iscsi gateway using lrbd
But interestingly enough, if you look down to where they run the targetcli ls, it shows a RBD backing store. Maybe it's using the krbd driver to actually do the Ceph side of the communication, but lio plugs into this rather than just talking to a dumb block device??? This needs further investigation. Could be very interesting. > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > ??? ??? > Sent: 18 January 2016 17:15 > To: Tyler Bishop > Cc: Dominik Zalewski ; ceph-users us...@lists.ceph.com> > Subject: Re: [ceph-users] CentOS 7 iscsi gateway using lrbd > > https://github.com/swiftgist/lrbd/wiki > According to lrbd wiki it still uses KRBD (see those /dev/rbd/... > devices in targetcli config). > I was thinking that Mike Christie developed a librbd module for LIO. > So what is it - KRBD or librbd? > > 2016-01-18 20:23 GMT+08:00 Tyler Bishop > : > > > > Well that's interesting. > > > > I've mounted block devices to the kernel and exported them to iscsi > > but the performance was horrible.. I wonder if this is any different? > > > > > > > > From: "Dominik Zalewski" > > To: ceph-users@lists.ceph.com > > Sent: Monday, January 18, 2016 6:35:20 AM > > Subject: [ceph-users] CentOS 7 iscsi gateway using lrbd > > > > Hi, > > I'm looking into implementing iscsi gateway with MPIO using lrbd - > > https://github.com/swiftgist/lrb > > > > > > > https://www.suse.com/docrep/documents/kgu61iyowz/suse_enterprise_st > ora > > ge_2_and_iscsi.pdf > > > > https://www.susecon.com/doc/2015/sessions/TUT16512.pdf > > > > From above examples: > > > > For iSCSI failover and load-balancing, > > > > these servers must run a kernel supporting the target_core_ > > > > rbd module. This also requires that the target servers run at > > > > least the version 3.12.48-52.27.1 of the kernel-default package. > > > > Updates packages are available from the SUSE Linux > > > > Enterprise Server maintenance channel. > > > > > > I understand that lrbd is basically a nice way to configure LIO and > > rbd across ceph osd nodes/iscsi gatways. Does CentOS 7 have same > > target_core_rbd module in the kernel or this is something Suse > > Enterprise Storage specific only? > > > > > > Basically will LIO+rbd work the same way on CentOS 7? Has anyone using > > it with CentOS? > > > > > > Thanks > > > > > > Dominik > > > > > > > > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CentOS 7 iscsi gateway using lrbd
On Tue, Jan 19, 2016 at 10:34 AM, Nick Fisk wrote: > But interestingly enough, if you look down to where they run the targetcli > ls, it shows a RBD backing store. > > Maybe it's using the krbd driver to actually do the Ceph side of the > communication, but lio plugs into this rather than just talking to a dumb > block device??? It does use krbd driver. Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS
Hi, I would like to know if CephFS will be marked as stable as soon as Ceph-1.0 will be released. Regards - Willi___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CentOS 7 iscsi gateway using lrbd
So is it a different approach that was used here by Mike Christie: http://www.spinics.net/lists/target-devel/msg10330.html ? It seems to be a confusion because it also implements target_core_rbd module. Or not? 2016-01-19 18:01 GMT+08:00 Ilya Dryomov : > On Tue, Jan 19, 2016 at 10:34 AM, Nick Fisk wrote: >> But interestingly enough, if you look down to where they run the targetcli >> ls, it shows a RBD backing store. >> >> Maybe it's using the krbd driver to actually do the Ceph side of the >> communication, but lio plugs into this rather than just talking to a dumb >> block device??? > > It does use krbd driver. > > Thanks, > > Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Again - state of Ceph NVMe and SSDs
It sounds like your just assuming these drives don't perform good... - Original Message - From: "Mark Nelson" To: ceph-users@lists.ceph.com Sent: Monday, January 18, 2016 2:17:19 PM Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs Take Greg's comments to heart, because he's absolutely correct here. Distributed storage systems almost as a rule love parallelism and if you have enough you can often hide other issues. Latency is probably the more interesting question, and frankly that's where you'll often start seeing the kernel, ceph code, drivers, random acts of god, etc, get in the way. It's very easy for any one of these things to destroy your performance, so you have to be *very* *very* careful to understand exactly what you are seeing. As such, don't trust any one benchmark. Wait until it's independently verified, possibly by multiple sources, before putting too much weight into it. Mark On 01/18/2016 01:02 PM, Tyler Bishop wrote: > One of the other guys on the list here benchmarked them. They spanked every > other ssd on the *recommended* tree.. > > - Original Message - > From: "Gregory Farnum" > To: "Tyler Bishop" > Cc: "David" , "Ceph Users" > Sent: Monday, January 18, 2016 2:01:44 PM > Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs > > On Sun, Jan 17, 2016 at 12:34 PM, Tyler Bishop > wrote: >> The changes you are looking for are coming from Sandisk in the ceph "Jewel" >> release coming up. >> >> Based on benchmarks and testing, sandisk has really contributed heavily on >> the tuning aspects and are promising 90%+ native iop of a drive in the >> cluster. > > Mmmm, they've gotten some very impressive numbers but most people > shouldn't be expecting 90% of an SSD's throughput out of their > workloads. These tests are *very* parallel and tend to run multiple > OSD processes on a single SSD, IIRC. > -Greg > >> >> The biggest changes will come from the memory allocation with writes. >> Latency is going to be a lot lower. >> >> >> - Original Message - >> From: "David" >> To: "Wido den Hollander" >> Cc: ceph-users@lists.ceph.com >> Sent: Sunday, January 17, 2016 6:49:25 AM >> Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs >> >> Thanks Wido, those are good pointers indeed :) >> So we just have to make sure the backend storage (SSD/NVMe journals) won’t >> be saturated (or the controllers) and then go with as many RBD per VM as >> possible. >> >> Kind Regards, >> David Majchrzak >> >> 16 jan 2016 kl. 22:26 skrev Wido den Hollander : >> >>> On 01/16/2016 07:06 PM, David wrote: Hi! We’re planning our third ceph cluster and been trying to find how to maximize IOPS on this one. Our needs: * Pool for MySQL, rbd (mounted as /var/lib/mysql or equivalent on KVM servers) * Pool for storage of many small files, rbd (probably dovecot maildir and dovecot index etc) >>> >>> Not completely NVMe related, but in this case, make sure you use >>> multiple disks. >>> >>> For MySQL for example: >>> >>> - Root disk for OS >>> - Disk for /var/lib/mysql (data) >>> - Disk for /var/log/mysql (binary log) >>> - Maybe even a InnoDB logfile disk >>> >>> With RBD you gain more performance by sending I/O into the cluster in >>> parallel. So when ever you can, do so! >>> >>> Regarding small files, it might be interesting to play with the stripe >>> count and stripe size there. By default this is 1 and 4MB. But maybe 16 >>> and 256k work better here. >>> >>> With Dovecot as well, use a different RBD disk for the indexes and a >>> different one for the Maildir itself. >>> >>> Ceph excels at parallel performance. That is what you want to aim for. >>> So I’ve been reading up on: https://communities.intel.com/community/itpeernetwork/blog/2015/11/20/the-future-ssd-is-here-pcienvme-boosts-ceph-performance and ceph-users from october 2015: http://www.spinics.net/lists/ceph-users/msg22494.html We’re planning something like 5 OSD servers, with: * 4x 1.2TB Intel S3510 * 8st 4TB HDD * 2x Intel P3700 Series HHHL PCIe 400GB (one for SSD Pool Journal and one for HDD pool journal) * 2x 80GB Intel S3510 raid1 for system * 256GB RAM * 2x 8 core CPU Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz or better This cluster will probably run Hammer LTS unless there are huge improvements in Infernalis when dealing 4k IOPS. The first link above hints at awesome performance. The second one from the list not so much yet.. Is anyone running Hammer or Infernalis with a setup like this? Is it a sane setup? Will we become CPU constrained or can we just throw more RAM on it? :D Kind Regards, David Majchrzak ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-cep
Re: [ceph-users] Again - state of Ceph NVMe and SSDs
Not at all! For all we know, the drives may be the fastest ones on the planet. My comment still stands though. Be skeptical of any one benchmark that shows something unexpected. 90% of native SSD/NVMe IOPS performance in a distributed storage system is just such a number. Look for test repeatability. Look for independent verification. The memory allocator tests that Sandisk initially performed and that were later independently verified by Intel, Red Hat, and others is a great example of this kind of process. Mark On 01/19/2016 07:01 AM, Tyler Bishop wrote: It sounds like your just assuming these drives don't perform good... - Original Message - From: "Mark Nelson" To: ceph-users@lists.ceph.com Sent: Monday, January 18, 2016 2:17:19 PM Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs Take Greg's comments to heart, because he's absolutely correct here. Distributed storage systems almost as a rule love parallelism and if you have enough you can often hide other issues. Latency is probably the more interesting question, and frankly that's where you'll often start seeing the kernel, ceph code, drivers, random acts of god, etc, get in the way. It's very easy for any one of these things to destroy your performance, so you have to be *very* *very* careful to understand exactly what you are seeing. As such, don't trust any one benchmark. Wait until it's independently verified, possibly by multiple sources, before putting too much weight into it. Mark On 01/18/2016 01:02 PM, Tyler Bishop wrote: One of the other guys on the list here benchmarked them. They spanked every other ssd on the *recommended* tree.. - Original Message - From: "Gregory Farnum" To: "Tyler Bishop" Cc: "David" , "Ceph Users" Sent: Monday, January 18, 2016 2:01:44 PM Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs On Sun, Jan 17, 2016 at 12:34 PM, Tyler Bishop wrote: The changes you are looking for are coming from Sandisk in the ceph "Jewel" release coming up. Based on benchmarks and testing, sandisk has really contributed heavily on the tuning aspects and are promising 90%+ native iop of a drive in the cluster. Mmmm, they've gotten some very impressive numbers but most people shouldn't be expecting 90% of an SSD's throughput out of their workloads. These tests are *very* parallel and tend to run multiple OSD processes on a single SSD, IIRC. -Greg The biggest changes will come from the memory allocation with writes. Latency is going to be a lot lower. - Original Message - From: "David" To: "Wido den Hollander" Cc: ceph-users@lists.ceph.com Sent: Sunday, January 17, 2016 6:49:25 AM Subject: Re: [ceph-users] Again - state of Ceph NVMe and SSDs Thanks Wido, those are good pointers indeed :) So we just have to make sure the backend storage (SSD/NVMe journals) won’t be saturated (or the controllers) and then go with as many RBD per VM as possible. Kind Regards, David Majchrzak 16 jan 2016 kl. 22:26 skrev Wido den Hollander : On 01/16/2016 07:06 PM, David wrote: Hi! We’re planning our third ceph cluster and been trying to find how to maximize IOPS on this one. Our needs: * Pool for MySQL, rbd (mounted as /var/lib/mysql or equivalent on KVM servers) * Pool for storage of many small files, rbd (probably dovecot maildir and dovecot index etc) Not completely NVMe related, but in this case, make sure you use multiple disks. For MySQL for example: - Root disk for OS - Disk for /var/lib/mysql (data) - Disk for /var/log/mysql (binary log) - Maybe even a InnoDB logfile disk With RBD you gain more performance by sending I/O into the cluster in parallel. So when ever you can, do so! Regarding small files, it might be interesting to play with the stripe count and stripe size there. By default this is 1 and 4MB. But maybe 16 and 256k work better here. With Dovecot as well, use a different RBD disk for the indexes and a different one for the Maildir itself. Ceph excels at parallel performance. That is what you want to aim for. So I’ve been reading up on: https://communities.intel.com/community/itpeernetwork/blog/2015/11/20/the-future-ssd-is-here-pcienvme-boosts-ceph-performance and ceph-users from october 2015: http://www.spinics.net/lists/ceph-users/msg22494.html We’re planning something like 5 OSD servers, with: * 4x 1.2TB Intel S3510 * 8st 4TB HDD * 2x Intel P3700 Series HHHL PCIe 400GB (one for SSD Pool Journal and one for HDD pool journal) * 2x 80GB Intel S3510 raid1 for system * 256GB RAM * 2x 8 core CPU Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz or better This cluster will probably run Hammer LTS unless there are huge improvements in Infernalis when dealing 4k IOPS. The first link above hints at awesome performance. The second one from the list not so much yet.. Is anyone running Hammer or Infernalis with a setup like this? Is it a sane setup? Will we become CPU constrained or can we just throw more RAM on it? :D Kind Regards, David
Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.
I was wondering if there is anything else I could provide outside of the
radosgw logs during the file upload? At this point I am uploading 7mb
files repeatedly to try to reproduce this error but so far I do not have
any missing keys in my test bucket. I can't see this happening if we
were to just use multipart uploads for files larger than say 1Gib but
then who is to say that we will not have corrupt multipart uploads.
On 1/15/16 4:36 PM, Yehuda Sadeh-Weinraub wrote:
The head object of a multipart object has 0 size, so it's expected.
What's missing is the tail of the object. I don't assume you have any
logs from when the object was uploaded?
Yehuda
On Fri, Jan 15, 2016 at 2:12 PM, seapasu...@uchicago.edu
wrote:
Sorry for the confusion::
When I grepped for the prefix of the missing object::
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD"
I am not able to find any chunks of the object::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
The only piece of the object that I can seem to find is the original one I
posted::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep
'NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959'
default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar
And when we stat this object is is 0 bytes as shown earlier::
lacadmin@kh28-10:~$ rados -p .rgw.buckets stat
'default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar'
.rgw.buckets/default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar
mtime 2015-11-04 15:29:30.00, size 0
Sorry again for the confusion.
On 1/15/16 3:58 PM, Yehuda Sadeh-Weinraub wrote:
Ah, I see. Misread that and the object names were very similar. No,
don't copy it. You can try to grep for the specific object name and
see if there are pieces of it lying around under a different upload
id.
Yehuda
On Fri, Jan 15, 2016 at 1:44 PM, seapasu...@uchicago.edu
wrote:
Sorry I am a bit confused. The successful list that I provided is from a
different object of the same size to show that I could indeed get a list.
Are you saying to copy the working object to the missing object? Sorry
for
the confusion.
On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote:
That's interesting, and might point at the underlying issue that
caused it. Could be a racing upload that somehow ended up with the
wrong object head. The 'multipart' object should be 4M in size, and
the 'shadow' one should have the remainder of the data. You can run
'rados stat -p .rgw.buckets ' to validate that. If that's the
case, you can copy these to the expected object names:
$ src_uploadid=wksHvto9gRgHUJbhm_TZPXJTZUPXLT2
$ dest_uploadid=pcu5Hz6foFXjlSxBat22D8YMcHlQOBD
$ rados -p .rgw.buckets cp
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~${src_uploadid}.1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~${dest_uploadid}.1
$ rados -p .rgw.buckets cp
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~${src_upload_id}.1_1
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~${dest_upload_id}.1_1
Yehuda
On Fri, Jan 15, 2016 at 1:02 PM, seapasu...@uchicago.edu
wrote:
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
Nothing was found. That said when I run the command with another prefix
snippet::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto'
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1_1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_2015010113_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1
On 1/15/16 12:05 PM, Yehuda Sadeh-Weinraub wrote:
On Fri, Jan 15, 2016 at 9:36 AM, seapasu...@uchicago.edu
wrote:
Hello Yehuda,
Here it is::
radosgw-admin object stat --bucket="noaa-nexrad-l2"
--object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar"
{
"name":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar",
"size": 7147520,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "b05f707271774dbd89674a0736c9406e",
"acl": 15
}
],
"acl_group_map": [
{
"group": 1,
"acl": 1
}
],
"grant_map": [
{
"id": "",
"grant": {
"type": {
"type": 2
Re: [ceph-users] CentOS 7 iscsi gateway using lrbd
Everyone is right - sort of :) It is that target_core_rbd module that I made that was rejected upstream, along with modifications from SUSE which added persistent reservations support. I also made some modifications to rbd so target_core_rbd and krbd could share code. target_core_rbd uses rbd like a lib. And it is also modifications to the targetcli related tool and libs, so you can use them to control the new rbd backend. SUSE's lrbd then handles setup/management of across multiple targets/gatways. I was going to modify targetcli more and have the user just pass in the rbd info there, but did not get finished. That is why in that suse stuff you still make the krbd device like normal. You then pass that to the target_core_rbd module with targetcli and that is how that module knows about the rbd device. The target_core_rbd module was rejected upstream, so I stopped development and am working on the approach suggested by those reviewers which instead of going from lio->target_core_rbd->krbd goes lio->target_core_iblock->linux block layer->krbd. With this approach you just use the normal old iblock driver and krbd and then I am modifying them to just work and do the right thing. On 01/19/2016 05:45 AM, Василий Ангапов wrote: > So is it a different approach that was used here by Mike Christie: > http://www.spinics.net/lists/target-devel/msg10330.html ? > It seems to be a confusion because it also implements target_core_rbd > module. Or not? > > 2016-01-19 18:01 GMT+08:00 Ilya Dryomov : >> On Tue, Jan 19, 2016 at 10:34 AM, Nick Fisk wrote: >>> But interestingly enough, if you look down to where they run the targetcli >>> ls, it shows a RBD backing store. >>> >>> Maybe it's using the krbd driver to actually do the Ceph side of the >>> communication, but lio plugs into this rather than just talking to a dumb >>> block device??? >> >> It does use krbd driver. >> >> Thanks, >> >> Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.
On Fri, Jan 15, 2016 at 5:04 PM, seapasu...@uchicago.edu
wrote:
> I have looked all over and I do not see any explicit mention of
> "NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959" in the logs nor do I
> see a timestamp from November 4th although I do see log rotations dating
> back to october 15th. I don't think it's possible it wasn't logged so I am
> going through the bucket logs from the 'radosgw-admin log show --object'
> side and I found the following::
>
> 4604932 {
> 4604933 "bucket": "noaa-nexrad-l2",
> 4604934 "time": "2015-11-04 21:29:27.346509Z",
> 4604935 "time_local": "2015-11-04 15:29:27.346509",
> 4604936 "remote_addr": "",
> 4604937 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 4604938 "user": "b05f707271774dbd89674a0736c9406e",
> 4604939 "operation": "PUT",
I'd expect a multipart upload completion to be done with a POST, not a PUT.
> 4604940 "uri":
> "\/noaa-nexrad-l2\/2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar",
> 4604941 "http_status": "200",
> 4604942 "error_code": "",
> 4604943 "bytes_sent": 19,
> 4604944 "bytes_received": 0,
> 4604945 "object_size": 0,
Do you see a zero object_size for other multipart uploads?
Yehuda
> 4604946 "total_time": 142640400,
> 4604947 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 4604948 "referrer": ""
> 4604949 }
>
> Does this help at all. The total time seems exceptionally high. Would it be
> possible that there is a timeout issue where the put request started a
> multipart upload with the correct header and then timed out but the radosgw
> took the data anyway?
>
> I am surprised the radosgw returned a 200 let alone placed the key in the
> bucket listing.
>
>
> That said here is another object (different object) that 404s:
> 1650873 {
> 1650874 "bucket": "noaa-nexrad-l2",
> 1650875 "time": "2015-11-05 04:50:42.606838Z",
> 1650876 "time_local": "2015-11-04 22:50:42.606838",
> 1650877 "remote_addr": "",
> 1650878 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 1650879 "user": "b05f707271774dbd89674a0736c9406e",
> 1650880 "operation": "PUT",
> 1650881 "uri":
> "\/noaa-nexrad-l2\/2015\/02\/25\/KVBX\/NWS_NEXRAD_NXL2DP_KVBX_2015022516_20150225165959.tar",
> 1650882 "http_status": "200",
> 1650883 "error_code": "",
> 1650884 "bytes_sent": 19,
> 1650885 "bytes_received": 0,
> 1650886 "object_size": 0,
> 1650887 "total_time": 0,
> 1650888 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 1650889 "referrer": ""
> 1650890 }
>
> And this one fails with a 404 as well. Does this help at all? Here is a
> successful object (different object) log entry as well just in case::
>
> 17462367 {
> 17462368 "bucket": "noaa-nexrad-l2",
> 17462369 "time": "2015-11-04 21:16:44.148603Z",
> 17462370 "time_local": "2015-11-04 15:16:44.148603",
> 17462371 "remote_addr": "",
> 17462372 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 17462373 "user": "b05f707271774dbd89674a0736c9406e",
> 17462374 "operation": "PUT",
> 17462375 "uri":
> "\/noaa-nexrad-l2\/2015\/01\/01\/KAKQ\/NWS_NEXRAD_NXL2DP_KAKQ_2015010108_20150101085959.tar",
> 17462376 "http_status": "200",
> 17462377 "error_code": "",
> 17462378 "bytes_sent": 19,
> 17462379 "bytes_received": 0,
> 17462380 "object_size": 0,
> 17462381 "total_time": 0,
> 17462382 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 17462383 "referrer": ""
> 17462384 }
>
> So I am guessing these are not pertinent as they look nearly identical.
> Unfortunately I do not have any client.radosgw.logs to show for the failed
> files for some reason. Is there anything else I can do to troubleshoot this
> issue. In the end the radosgw should have never list these files as they
> never completed successfully, right?
>
>
>
>
>
>
> On 1/15/16 4:36 PM, Yehuda Sadeh-Weinraub wrote:
>>
>> The head object of a multipart object has 0 size, so it's expected.
>> What's missing is the tail of the object. I don't assume you have any
>> logs from when the object was uploaded?
>>
>> Yehuda
>>
>> On Fri, Jan 15, 2016 at 2:12 PM, seapasu...@uchicago.edu
>> wrote:
>>>
>>> Sorry for the confusion::
>>>
>>> When I grepped for the prefix of the missing object::
>>>
>>> "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD"
>>>
>>> I am not a
[ceph-users] Repository with some internal utils
Hi, someone asked me if he could get access to the BTRFS defragmenter we used for our Ceph OSDs. I took a few minutes to put together a small github repository with : - the defragmenter I've been asked about (tested on 7200 rpm drives and designed to put low IO load on them), - the scrub scheduler we use to avoid load spikes on Firefly, - some basic documentation (this is still rough around the edges so you better like to read Ruby code if you want to peak at most of the logic, tune or hack these). Here it is: https://github.com/jtek/ceph-utils This is running in production for several months now and I didn't touch the code or the numerous internal tunables these scripts have for several weeks so it probably won't destroy your clusters. These scripts come without warranties though. Best regards, Lionel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] s3 upload to ceph slow after few chunks
Hey guys, I am having an s3 upload to ceph issue where in the upload seems to crawl after the first few chunks for a multipart upload. The test file is 38M in size and the upload was tried with s3 default chunk size at 15M and then tried again with chunk size set to 5M and then was tested again with multipart disabled. Each time the upload halted at seeming random amount and I cannot seem to figure out why. [10.0.144.2] $ /opt/utilities/s3cmd/s3cmd put ACCOUNT\ EXTRACT-GENERATED.PDF.test.2 's3://' -> s3:/// [part 1 of 3, 15MB] 15728640 of 15728640 100% in2s 7.43 MB/s done -> s3:/// [part 2 of 3, 15MB] 4096 of 15728640 0% in0s31.68 kB/s ERROR: Upload of 'ACCOUNT EXTRACT-GENERATED.PDF.test.2' part 2 failed. Use / 2~T2OI6bU-TYE_dOtm3tPIndtYTek-V0r to abort the upload, or ./s3cmd --upload-id 2~T2OI6bU-TYE_dOtm3tPIndtYTek-V0r put ... to continue the upload. See ya! Rishiraj Rana ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to observed civetweb.
Hey Kobi, You stated: > >> You can add: > >> *access_log_file=/var/log/civetweb/access.log > >> error_log_file=/var/log/civetweb/error.log* > >> > >> to *rgw frontends* in ceph.conf though these logs are thin on info > >> (Source IP, date, and request) How is this done exactly in the config file? I've tried various ways, nothing works. ie, neither of these work: [client.radosgw.] host = rgw socket path = /tmp/radosgw.sock keyring = /etc/ceph/ceph.client.radosgw.keyring log file = /var/log/radosgw/client.radosgw..log <-- this log gets output only access_log_file=/var/log/civetweb/access.log rgw frontends error log file = /var/log/radosgw/civetweb.error.log <-- nada rgw frontends access log file = /var/log/radosgw/civetweb.access.log <-- nada rgw print continue = True rgw enable ops log = False [rgw frontends] access_log_file=/var/log/civetweb/access.log <-- nada -Ben On Tue, Sep 8, 2015 at 11:21 AM, Kobi Laredo wrote: > Vickie, > > You can add: > *access_log_file=/var/log/civetweb/access.log > error_log_file=/var/log/civetweb/error.log* > > to *rgw frontends* in ceph.conf though these logs are thin on info > (Source IP, date, and request) > > Check out > https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md for > more civetweb configs you can inject through *rgw frontends* config > attribute in ceph.conf > > We are currently testing tuning civetweb's num_threads > and request_timeout_ms to improve radosgw performance > > *Kobi Laredo* > *Cloud Systems Engineer* | (*408) 409-KOBI* > > On Tue, Sep 8, 2015 at 8:20 AM, Yehuda Sadeh-Weinraub > wrote: > >> You can increase the civetweb logs by adding 'debug civetweb = 10' in >> your ceph.conf. The output will go into the rgw logs. >> >> Yehuda >> >> On Tue, Sep 8, 2015 at 2:24 AM, Vickie ch wrote: >> > Dear cephers, >> >Just upgrade radosgw from apache to civetweb. >> > It's really simple to installed and used. But I can't find any >> parameters or >> > logs to adjust(or observe) civetweb. (Like apache log). I'm really >> confuse. >> > Any ideas? >> > >> > >> > Best wishes, >> > Mika >> > >> > >> > ___ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to observed civetweb.
Of course, i figured this out. You meant just append it to the frontends setting. Very confusing as it's unlike every other ceph setting. rgw frontends = civetweb num_threads=150 error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log Any documentation, at all, on civetweb + radosgw on the Ceph site would be awesome.. Currently it all only references Apache+FastCGi. On Tue, Jan 19, 2016 at 8:42 PM, Ben Hines wrote: > Hey Kobi, > > You stated: > > > >> You can add: > > >> *access_log_file=/var/log/civetweb/access.log > > >> error_log_file=/var/log/civetweb/error.log* > > >> > > >> to *rgw frontends* in ceph.conf though these logs are thin on info > > >> (Source IP, date, and request) > > > > How is this done exactly in the config file? I've tried various ways, nothing > works. > > > ie, neither of these work: > > > [client.radosgw.] > host = > rgw socket path = /tmp/radosgw.sock > keyring = /etc/ceph/ceph.client.radosgw.keyring > log file = /var/log/radosgw/client.radosgw..log <-- this log gets > output only > access_log_file=/var/log/civetweb/access.log > > rgw frontends error log file = /var/log/radosgw/civetweb.error.log <-- nada > rgw frontends access log file = /var/log/radosgw/civetweb.access.log <-- nada > > rgw print continue = True > rgw enable ops log = False > > > [rgw frontends] > > access_log_file=/var/log/civetweb/access.log <-- nada > > > > -Ben > > > On Tue, Sep 8, 2015 at 11:21 AM, Kobi Laredo > wrote: > >> Vickie, >> >> You can add: >> *access_log_file=/var/log/civetweb/access.log >> error_log_file=/var/log/civetweb/error.log* >> >> to *rgw frontends* in ceph.conf though these logs are thin on info >> (Source IP, date, and request) >> >> Check out >> https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md for >> more civetweb configs you can inject through *rgw frontends* config >> attribute in ceph.conf >> >> We are currently testing tuning civetweb's num_threads >> and request_timeout_ms to improve radosgw performance >> >> *Kobi Laredo* >> *Cloud Systems Engineer* | (*408) 409-KOBI* >> >> On Tue, Sep 8, 2015 at 8:20 AM, Yehuda Sadeh-Weinraub >> wrote: >> >>> You can increase the civetweb logs by adding 'debug civetweb = 10' in >>> your ceph.conf. The output will go into the rgw logs. >>> >>> Yehuda >>> >>> On Tue, Sep 8, 2015 at 2:24 AM, Vickie ch >>> wrote: >>> > Dear cephers, >>> >Just upgrade radosgw from apache to civetweb. >>> > It's really simple to installed and used. But I can't find any >>> parameters or >>> > logs to adjust(or observe) civetweb. (Like apache log). I'm really >>> confuse. >>> > Any ideas? >>> > >>> > >>> > Best wishes, >>> > Mika >>> > >>> > >>> > ___ >>> > ceph-users mailing list >>> > ceph-users@lists.ceph.com >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> > >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
