[ceph-users] How to set correct pg_num?

[alan.zhang]

As the doc 
http://ceph.com/docs/master/rados/operations/placement-groups/


I get this formula:
(OSDs * 100)
Total PGs = 
  Replicas

In my ceph cluster have 1 pool, 5 osds,2 replicas, so I have set pg_num 
to 250.


one day , i ned to  add a pool(2 pools), what number shoud i set the 
pg_num?


another question, if i set the pg_num as 1024, it will be get slow 
request?


Thanks for help.


--
Talk is cheap,lead by example.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to set correct pg_num?

[xiao xianxia]

于 2014/3/19 15:11, alan.zhang 写道:

As the doc http://ceph.com/docs/master/rados/operations/placement-groups/

I get this formula:
(OSDs * 100)
Total PGs = 


This formula is recommended that pg nums assigned to each OSD controls 
below 100.



Replicas

In my ceph cluster have 1 pool, 5 osds,2 replicas, so I have set 
pg_num to 250.


one day , i ned to add a pool(2 pools), what number shoud i set the 
pg_num?

if you have two pools,each pool may set pg_num=125,


another question, if i set the pg_num as 1024, it will be get slow 
request?

yes,may be.


Thanks for help.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Broken bucket on upgrade

[Mike Bryant]

Hi,
I've just upgraded a test cluster to Emporer, and one of my S3 buckets
seems to have broken.

s3 access is returning a 500 code (UnknownError).

Running bucket stats, it's missing from the list.
Trying to do it explicitly:

radosgw-admin bucket stats --bucket=productimages
2014-03-19 10:06:17.829397 7ff0b81c7780  0 could not get bucket info for
bucket=productimages

I can see the header object in the .rgw pool:
rados --cluster=cit-external ls --pool .rgw
.pools.avail
productimages
test
tests3fs

Does anyone have any idea on what might have happened, or how I can get
this bucket back?

Cheers
Mike
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph mapreduce history server issue

[Gurvinder Singh]

Hi,

I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
hadoop. I have installed the ceph hadoop binding with hadoop 2.x
support. I am able to run the command such as

# hadoop fs -ls /
# hdfs dfs -touchz /test

But when I start the mapreduce job history server I am getting the error as

"Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
AbstractFileSystem for scheme: ceph"

Here is the link to mapreduce startup message listing the lib it linked
against while starting and it has both libcepfs and cephfs-hadoop jars.

http://pastebin.com/wED7nMMT

I have made the changes in to core-site.xml file, that's why I am able
to the normal hadoop commands. Let me know if you have any suggestions
how to resolve this.

Regards,
Gurvinder
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD as backend for iSCSI SAN Targets

[Jianing Yang]

Hi, Karol

Here is something that I can share. We are running Ceph as an Exchange
Backend via iSCSI. We currently host about 2000 mailboxes which is about
7 TB data overall. Our configuration is

- Proxy Node (with tgt daemon) x 2
- Ceph Monitor x 3 (virtual machines)
- Ceph OSD x 50 (SATA 7200rpm 2T), Replica = 2, Journal on OSD (I know it is
bad, but ...)

We tested RBD using fio and got a randwrite around 1500 iops. On the
living system, I saw the highest op/s around 3.1k.

I've benchmarked "tgt with librdb" vs "tgt with kernel rbd" using my
virtual machines. It seems that "tgt with librdb" doesn't perform
well. It has only 1/5 iops of kernel rbd.

We are new to Ceph and still finding ways to improve the performance. I
am really looking forward to your benchmark.

On Sun 16 Mar 2014 12:40:53 AM CST, Karol Kozubal wrote:

 > Hi Wido,

 > I will have some new hardware for running tests in the next two weeks or
 > so and will report my findings once I get a chance to run some tests. I
 > will disable writeback on the target side as I will be attempting to
 > configure an ssd caching pool of 24 ssd's with writeback for the main pool
 > with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be
 > running everything through 10Gig SFP+ Ethernet interfaces with a dedicated
 > cluster network interface, dedicated public ceph interface and a separate
 > iscsi network also with 10 gig interfaces for the target machines.

 > I am ideally looking for a 20,000 to 60,000 IOPS from this system if I can
 > get the caching pool configuration right. The application has a 30ms max
 > latency requirement for the storage.

 > In my current tests I have only spinners with SAS 10K disks, 4.2ms write
 > latency on the disks with separate journaling on SAS 15K disks with a
 > 3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned with
 > the overall operation apply latency that I have been seeing (1-6ms idle is
 > normal, but up to 60-170ms for a moderate workload using rbd bench-write)
 > however I am on a network where I am bound to 1500 mtu and I will get to
 > test jumbo frames with the next setup in addition to the ssd¹s. I suspect
 > the overall performance will be good in the new test setup and I am
 > curious to see what my tests will yield.

 > Thanks for the response!

 > Karol



 > On 2014-03-15, 12:18 PM, "Wido den Hollander"  wrote:

 > >On 03/15/2014 04:11 PM, Karol Kozubal wrote:
 > >> Hi Everyone,
 > >>
 > >> I am just wondering if any of you are running a ceph cluster with an
 > >> iSCSI target front end? I know this isn¹t available out of the box,
 > >> unfortunately in one particular use case we are looking at providing
 > >> iSCSI access and it's a necessity. I am liking the idea of having rbd
 > >> devices serving block level storage to the iSCSI Target servers while
 > >> providing a unified backed for native rbd access by openstack and
 > >> various application servers. On multiple levels this would reduce the
 > >> complexity of our SAN environment and move us away from expensive
 > >> proprietary solutions that don¹t scale out.
 > >>
 > >> If any of you have deployed any HA iSCSI Targets backed by rbd I would
 > >> really appreciate your feedback and any thoughts.
 > >>
 > >
 > >I haven't used it in production, but a couple of things which come to
 > >mind:
 > >
 > >- Use TGT so you can run it all in userspace backed by librbd
 > >- Do not use writeback caching on the targets
 > >
 > >You could use multipathing if you don't use writeback caching. Use
 > >writeback would also cause data loss/corruption in case of multiple
 > >targets.
 > >
 > >It will probably just work with TGT, but I don't know anything about the
 > >performance.
 > >
 > >> Karol
 > >>
 > >>
 > >> ___
 > >> ceph-users mailing list
 > >> ceph-users@lists.ceph.com
 > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 > >>
 > >
 > >
 > >--
 > >Wido den Hollander
 > >42on B.V.
 > >
 > >Phone: +31 (0)20 700 9902
 > >Skype: contact42on
 > >___
 > >ceph-users mailing list
 > >ceph-users@lists.ceph.com
 > >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 > ___
 > ceph-users mailing list
 > ceph-users@lists.ceph.com
 > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
 _
/ Save time starting to type a command or \
| file name, then press tab to complete   |
| Hit tab twice to bring up multiple  |
\ completion options. /
 -
   \
\
 _   ___
  ]OO|_n_n__][.  | |
   []_|__|)< | |
oooo  'oo -| oo\\_   ~~~|~~~
+--+--+--+--+--+--+--+--+--+--+--+--+--+
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo

[ceph-users] 答复: why objects are still in .rgw.buckets after deleted

[ljm李嘉敏]

You are right, but I still don’t know why the objects in .rgw.buckets are not 
overrided.
If the object name is produced through ino and ono, why the same file(bigger 
than 4M) have different result?

Thanks & Regards
Li JiaMin

发件人: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] 代表 Craig Lewis
发送时间: 2014年3月19日 9:00
收件人: ceph-users@lists.ceph.com
主题: Re: [ceph-users] why objects are still in .rgw.buckets after deleted

I recall hearing that RGW GC waits 2 hours before garbage collecting deleted 
chunks.

Take a look at https://ceph.com/docs/master/radosgw/config-ref/, the rgw gc * 
settings.  rgw gc obj min wait is 2 hours.





Craig Lewis
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com

Central Desktop. Work together in ways you never thought possible.
Connect with us   Website  |  
Twitter  |  
Facebook  |  
LinkedIn  |  
Blog
On 3/16/14 23:20 , ljm李嘉敏 wrote:
Hi all,

I have a question about the pool .rgw.buckets, when I upload a file(has been 
stripped because it is bigger than 4M) through swift API, it is stored in 
.rgw.buckets,
if I upload it again, why the objects in .rgw.buckets are not override? It is 
stored again and have different name. and when I delete the file, all of the 
objects in .rgw.buckets
aren’t delete even though I execute radosgw-admin gc process.

I also want to know something about the pool created for object gateway, why 
are they created and which role they will play? If anyone know about these,
please give me a guide, thanks.


Thanks & Regards
Li JiaMin

System Cloud Platform
3#4F108





___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph mapreduce history server issue

[Gurvinder Singh]

Hi,

I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
hadoop. I have installed the ceph hadoop binding with hadoop 2.x
support. I am able to run the command such as

# hadoop fs -ls /
# hdfs dfs -touchz /test

But when I start the mapreduce job history server I am getting the
error as

"Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
AbstractFileSystem for scheme: ceph"

Here is the link to mapreduce startup message listing the lib it linked
against while starting and it has both libcepfs and cephfs-hadoop jars.

http://pastebin.com/wED7nMMT

I have made the changes in to core-site.xml file, that's why I am able
to the normal hadoop commands. Let me know if you have any suggestions
how to resolve this.

Regards,
Gurvinder
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Broken bucket on upgrade

[Mike Bryant]

So I've done some more digging, and running the radosgw in debug mode I
found some messages from osd.3 saying IOError, when it was trying to get
.rgw:productimages.
I took that OSD down, and everything started working.

My question now is, why didn't that OSD suicide when it hit an IOError,
instead of causing the cluster to stop working?


On 19 March 2014 10:07, Mike Bryant  wrote:

> Hi,
> I've just upgraded a test cluster to Emporer, and one of my S3 buckets
> seems to have broken.
>
> s3 access is returning a 500 code (UnknownError).
>
> Running bucket stats, it's missing from the list.
> Trying to do it explicitly:
>
> radosgw-admin bucket stats --bucket=productimages
> 2014-03-19 10:06:17.829397 7ff0b81c7780  0 could not get bucket info for
> bucket=productimages
>
> I can see the header object in the .rgw pool:
> rados --cluster=cit-external ls --pool .rgw
> .pools.avail
> productimages
> test
> tests3fs
>
> Does anyone have any idea on what might have happened, or how I can get
> this bucket back?
>
> Cheers
> Mike
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph hadoop mapred history server issue

[Gurvinder Singh]

Hi,

I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
hadoop. I have installed the ceph hadoop binding with hadoop 2.x
support. I am able to run the command such as

# hadoop fs -ls /
# hdfs dfs -touchz /test

But when I start the mapreduce job history server I am getting the
error as

"Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
AbstractFileSystem for scheme: ceph"

Here is the link to mapreduce startup message listing the lib it linked
against while starting and it has both libcepfs and cephfs-hadoop jars.

"pastebin links seems to be causing message to go to spam"

I have made the changes in to core-site.xml file, that's why I am able
to the normal hadoop commands. Let me know if you have any suggestions
how to resolve this.

Regards,
Gurvinder
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Noah Watkins]

On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
 wrote:
> Hi,
>
> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
> support. I am able to run the command such as

>From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?

Can you paste your core-site.xml?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Contract position available at Cisco for a qualified Ceph/OpenStack engineer

[Don Talton (dotalton)]

Cisco is searching for an experienced DevOps engineer to work as part of a team 
characterizing the stability, scale and performance of a large distributed 
cloud architecture. This position focuses on locating the bottlenecks in the 
architecture and developing test suites to add to CI/CD efforts to ensure a 
base level of stability and performance is met with each iteration/build.  

Requirements:
  Automation and familiarity with jenkins/git/gerrit
  Experience with iSCSI, SAN, block, file, and object storage systems such as 
Swift,  Gluster, Ceph.
  Agile process experience

Puppet experience a plus

Please email me directly at dotal...@cisco.com if you are interested.

Donald Talton
Cloud Systems Development
Cisco Systems


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph mapreduce history server issue

[Gurvinder Singh]

On 03/19/2014 03:04 PM, Alejandro Bonilla wrote:
> Hi Gurvinder,
> 
> This setup sounds interesting. Which guide did you follow?
> 
There wasn't any specific guide to follow. But the documentation from
ceph wiki http://ceph.com/docs/master/cephfs/hadoop/ has helped. I can
write a step by step post once, I will get this issue fixed. To make it
easy to setup ceph and test with hadoop 2.x

Regards,
Gurvinder
> On Wed, Mar 19, 2014 at 6:44 AM, Gurvinder Singh
>  wrote:
>> Hi,
>>
>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
>> support. I am able to run the command such as
>>
>> # hadoop fs -ls /
>> # hdfs dfs -touchz /test
>>
>> But when I start the mapreduce job history server I am getting the error as
>>
>> "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
>> AbstractFileSystem for scheme: ceph"
>>
>> Here is the link to mapreduce startup message listing the lib it linked
>> against while starting and it has both libcepfs and cephfs-hadoop jars.
>>
>> http://pastebin.com/wED7nMMT
>>
>> I have made the changes in to core-site.xml file, that's why I am able
>> to the normal hadoop commands. Let me know if you have any suggestions
>> how to resolve this.
>>
>> Regards,
>> Gurvinder
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Gurvinder Singh]

On 03/19/2014 04:50 PM, Noah Watkins wrote:
> Since `hadoop -fs ls /` seems to work on your local node, can you
> verify that (1) it is in fact listing the contents of CephFS, and (2)
> that on your worker nodes where the error is occuring that the
> relevant dependencies (naming the Ceph hadoop bindings) are installed
> and in the classpath?
1) yes it is listing the cephfs contents.
2) Currently the setup is to start the yarn manager and mapreduce
history server on the same node and then setup the worker nodes. Yarn
starts up fine, but the mapreduce history server gives me the error. The
dependency lib are loaded in the mapreduce history server startup. Here
is the link to start up message which shows that the libcephfs and
cephfs-hadoop jar are loaded.

http://pastebin.com/wED7nMMT

I am wondering if this property value is correct


  fs.ceph.impl
  org.apache.hadoop.fs.ceph.CephFileSystem


or should I set it to something for hadoop 2.x I have tried setting it up as

org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as

# hadoop fs -ls /
-ls: Fatal internal error
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
at
org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at
org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
at java.lang.Class.getConstructor0(Class.java:2800)
at java.lang.Class.getDeclaredConstructor(Class.java:2043)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 17 more

Regards,
Gurvinder

> 
> The error itself looks like a missing dependency, but that exception
> being thrown might also be tirggered by other problems while loading
> the bindings.
> 
> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
>  wrote:
>> On 03/19/2014 03:51 PM, Noah Watkins wrote:
>>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>>>  wrote:
 Hi,

 I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
 hadoop. I have installed the ceph hadoop binding with hadoop 2.x
 support. I am able to run the command such as
>>>
>>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?
>>>
>> Yes.
>>> Can you paste your core-site.xml?
>>
>> Below is the relevant part for ceph conf
>>
>>  
>> fs.defaultFS
>> ceph://:6789/
>>   
>>
>>   
>> ceph.conf.file
>> /etc/ceph/ceph.conf
>>   
>>
>>   
>> ceph.auth.id
>> admin
>>   
>>
>>   
>> ceph.auth.keyfile
>> /etc/hadoop/conf/admin.secret
>>   
>>
>>   
>> fs.ceph.impl
>> org.apache.hadoop.fs.ceph.CephFileSystem
>>   
>>
>>
>>>
>>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Gurvinder Singh]

On 03/19/2014 05:18 PM, Noah Watkins wrote:
> Err, obviously switching things out for Ceph rather than Gluster.
> 
> On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins  
> wrote:
>> Looks like this is a configuration issue that has popped up with other
>> 3rd party file systems in Hadoop 2.x with YARN.
>>
>>   
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E
>>
>> Says use this:
>>
>>  
>>   fs.AbstractFileSystem.glusterfs.impl
>>   org.apache.hadoop.fs.glusterfs.GlusterFS
>>  
Interestingly this solves the mapred history server issue. It created
the history folder and started.

  
fs.AbstractFileSystem.ceph.impl
org.apache.hadoop.fs.ceph.CephHadoop2FileSystem


  

But now this fails as

hadoop fs -ls /
ls: No FileSystem for scheme: ceph

- Gurvinder
>>
>> Apparently rather than the `fs.ceph.impl` property in 2.x
>>
>>
>> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh
>>  wrote:
>>> On 03/19/2014 04:50 PM, Noah Watkins wrote:
 Since `hadoop -fs ls /` seems to work on your local node, can you
 verify that (1) it is in fact listing the contents of CephFS, and (2)
 that on your worker nodes where the error is occuring that the
 relevant dependencies (naming the Ceph hadoop bindings) are installed
 and in the classpath?
>>> 1) yes it is listing the cephfs contents.
>>> 2) Currently the setup is to start the yarn manager and mapreduce
>>> history server on the same node and then setup the worker nodes. Yarn
>>> starts up fine, but the mapreduce history server gives me the error. The
>>> dependency lib are loaded in the mapreduce history server startup. Here
>>> is the link to start up message which shows that the libcephfs and
>>> cephfs-hadoop jar are loaded.
>>>
>>> http://pastebin.com/wED7nMMT
>>>
>>> I am wondering if this property value is correct
>>>
>>> 
>>>   fs.ceph.impl
>>>   org.apache.hadoop.fs.ceph.CephFileSystem
>>> 
>>>
>>> or should I set it to something for hadoop 2.x I have tried setting it up as
>>>
>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as
>>>
>>> # hadoop fs -ls /
>>> -ls: Fatal internal error
>>> java.lang.RuntimeException: java.lang.NoSuchMethodException:
>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
>>> at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>>> at
>>> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
>>> at
>>> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
>>> at
>>> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
>>> at
>>> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
>>> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
>>> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
>>> Caused by: java.lang.NoSuchMethodException:
>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
>>> at java.lang.Class.getConstructor0(Class.java:2800)
>>> at java.lang.Class.getDeclaredConstructor(Class.java:2043)
>>> at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
>>> ... 17 more
>>>
>>> Regards,
>>> Gurvinder
>>>

 The error itself looks like a missing dependency, but that exception
 being thrown might also be tirggered by other problems while loading
 the bindings.

 On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
  wrote:
> On 03/19/2014 03:51 PM, Noah Watkins wrote:
>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>>  wrote:
>>> Hi,
>>>
>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
>>> support. I am able to run the command such as
>>
>> From github.com/noahdesu/cephfs-hadoop patched with the pull request 
>> PR#1?
>>
> Yes.
>> Can you paste your core-site.xml?
>
> Below is the relevant part for ceph conf
>
>  
> fs.defaultFS
> ceph://xx

Re: [ceph-users] Mounting with dmcrypt still fails

[Eric Eastman]

On Ubuntu 1310 with Ceph 0.72, after manually putting in the patch from 
http://tracker.ceph.com/issues/6966


I was able to create my dmcrypt OSD with:

ceph-deploy disk zap tca14:/dev/cciss/c0d1
ceph-deploy --verbose osd create --dmcrypt tca14:/dev/cciss/c0d1

Looking at the mount points with df showed:

df | grep ceph
/dev/dm-1   70568868  53976   70514892   1% 
/var/lib/ceph/osd/ceph-0
/dev/dm-3   70568868  52412   70516456   1% 
/var/lib/ceph/osd/ceph-3
/dev/dm-5   70568868  60852   70508016   1% 
/var/lib/ceph/osd/ceph-4
/dev/dm-7   70568868  35284   70533584   1% 
/var/lib/ceph/osd/ceph-5
/dev/dm-9   70568868  36736   70532132   1% 
/var/lib/ceph/osd/ceph-6


I tried to use whole new blank disk to create two separate partition 

(one for data and second for journal)
and use dmcrypt, but there is a problem with use this. It's looks like 

there is a problem with mounting or

formatting partitions.

OS is Ubuntu 13.04 with ceph v0.72 (emperor)

I used command:

ceph-deploy osd prepare ceph-node0:sdb --dmcrypt 

--dmcrypt-key-dir=/root --fs-type=xfs


[ceph-node0][WARNIN] INFO:ceph-disk:Will colocate journal with data on 

/dev/sdb

Eric
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [RGW] Setting cache control headers

[Steve Carter]

Is there a way to set cache control headers for objects served by the rados 
gateway?  In Apache one would modify the .htaccess file to set the required 
cache control headers, but I wonder how one would do this with rgw when using 
it as a CDN origin.

-Steve
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Gurvinder Singh]

On 03/19/2014 05:58 PM, Noah Watkins wrote:
> That certainly is odd. Does it work if you list both old and new
> properties (perhaps the CLI tools are looking at an older property..
> but that seems unlikely)? Sorry I don't have more answers, I haven't
> yet deploying Hadoop 2.x..
Another strange thing but.. putting the both the properties on
core-site.xml makes both tools works.

- Gurvinder
> 
> On Wed, Mar 19, 2014 at 9:30 AM, Gurvinder Singh
>  wrote:
>> On 03/19/2014 05:18 PM, Noah Watkins wrote:
>>> Err, obviously switching things out for Ceph rather than Gluster.
>>>
>>> On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins  
>>> wrote:
 Looks like this is a configuration issue that has popped up with other
 3rd party file systems in Hadoop 2.x with YARN.

   
 http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E

 Says use this:

  
   fs.AbstractFileSystem.glusterfs.impl
   org.apache.hadoop.fs.glusterfs.GlusterFS
  
>> Interestingly this solves the mapred history server issue. It created
>> the history folder and started.
>>
>>   
>> fs.AbstractFileSystem.ceph.impl
>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem
>> 
>> 
>>   
>>
>> But now this fails as
>>
>> hadoop fs -ls /
>> ls: No FileSystem for scheme: ceph
>>
>> - Gurvinder

 Apparently rather than the `fs.ceph.impl` property in 2.x


 On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh
  wrote:
> On 03/19/2014 04:50 PM, Noah Watkins wrote:
>> Since `hadoop -fs ls /` seems to work on your local node, can you
>> verify that (1) it is in fact listing the contents of CephFS, and (2)
>> that on your worker nodes where the error is occuring that the
>> relevant dependencies (naming the Ceph hadoop bindings) are installed
>> and in the classpath?
> 1) yes it is listing the cephfs contents.
> 2) Currently the setup is to start the yarn manager and mapreduce
> history server on the same node and then setup the worker nodes. Yarn
> starts up fine, but the mapreduce history server gives me the error. The
> dependency lib are loaded in the mapreduce history server startup. Here
> is the link to start up message which shows that the libcephfs and
> cephfs-hadoop jar are loaded.
>
> http://pastebin.com/wED7nMMT
>
> I am wondering if this property value is correct
>
> 
>   fs.ceph.impl
>   org.apache.hadoop.fs.ceph.CephFileSystem
> 
>
> or should I set it to something for hadoop 2.x I have tried setting it up 
> as
>
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as
>
> # hadoop fs -ls /
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at
> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at
> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
> at
> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
> at java.lang.Class.getConstructor0(Class.java:2800)
> at java.lang.Class.getDeclaredConstructor(Class.java:2043)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
> ... 17 more
>
> Regards,
> Gurvinder
>
>>
>> The error itself looks like a missing dependency, but that exception
>> being thrown might also be tirggered by other problems while loading
>> t

Re: [ceph-users] ceph hadoop mapred history server issue

[Gurvinder Singh]

On 03/19/2014 03:51 PM, Noah Watkins wrote:
> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>  wrote:
>> Hi,
>>
>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
>> support. I am able to run the command such as
> 
> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?
> 
Yes.
> Can you paste your core-site.xml?

Below is the relevant part for ceph conf

 
fs.defaultFS
ceph://:6789/
  

  
ceph.conf.file
/etc/ceph/ceph.conf
  

  
ceph.auth.id
admin
  

  
ceph.auth.keyfile
/etc/hadoop/conf/admin.secret
  

  
fs.ceph.impl
org.apache.hadoop.fs.ceph.CephFileSystem
  


> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Noah Watkins]

That certainly is odd. Does it work if you list both old and new
properties (perhaps the CLI tools are looking at an older property..
but that seems unlikely)? Sorry I don't have more answers, I haven't
yet deploying Hadoop 2.x..

On Wed, Mar 19, 2014 at 9:30 AM, Gurvinder Singh
 wrote:
> On 03/19/2014 05:18 PM, Noah Watkins wrote:
>> Err, obviously switching things out for Ceph rather than Gluster.
>>
>> On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins  
>> wrote:
>>> Looks like this is a configuration issue that has popped up with other
>>> 3rd party file systems in Hadoop 2.x with YARN.
>>>
>>>   
>>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E
>>>
>>> Says use this:
>>>
>>>  
>>>   fs.AbstractFileSystem.glusterfs.impl
>>>   org.apache.hadoop.fs.glusterfs.GlusterFS
>>>  
> Interestingly this solves the mapred history server issue. It created
> the history folder and started.
>
>   
> fs.AbstractFileSystem.ceph.impl
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem
> 
> 
>   
>
> But now this fails as
>
> hadoop fs -ls /
> ls: No FileSystem for scheme: ceph
>
> - Gurvinder
>>>
>>> Apparently rather than the `fs.ceph.impl` property in 2.x
>>>
>>>
>>> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh
>>>  wrote:
 On 03/19/2014 04:50 PM, Noah Watkins wrote:
> Since `hadoop -fs ls /` seems to work on your local node, can you
> verify that (1) it is in fact listing the contents of CephFS, and (2)
> that on your worker nodes where the error is occuring that the
> relevant dependencies (naming the Ceph hadoop bindings) are installed
> and in the classpath?
 1) yes it is listing the cephfs contents.
 2) Currently the setup is to start the yarn manager and mapreduce
 history server on the same node and then setup the worker nodes. Yarn
 starts up fine, but the mapreduce history server gives me the error. The
 dependency lib are loaded in the mapreduce history server startup. Here
 is the link to start up message which shows that the libcephfs and
 cephfs-hadoop jar are loaded.

 http://pastebin.com/wED7nMMT

 I am wondering if this property value is correct

 
   fs.ceph.impl
   org.apache.hadoop.fs.ceph.CephFileSystem
 

 or should I set it to something for hadoop 2.x I have tried setting it up 
 as

 org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as

 # hadoop fs -ls /
 -ls: Fatal internal error
 java.lang.RuntimeException: java.lang.NoSuchMethodException:
 org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
 at
 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
 at
 org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
 at
 org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
 at
 org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
 at
 org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
 at
 org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
 at
 org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
 at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
 at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
 at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
 Caused by: java.lang.NoSuchMethodException:
 org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
 at java.lang.Class.getConstructor0(Class.java:2800)
 at java.lang.Class.getDeclaredConstructor(Class.java:2043)
 at
 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
 ... 17 more

 Regards,
 Gurvinder

>
> The error itself looks like a missing dependency, but that exception
> being thrown might also be tirggered by other problems while loading
> the bindings.
>
> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
>  wrote:
>> On 03/19/2014 03:51 PM, Noah Watkins wrote:
>>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>>>  wrote:
 Hi,

 I have ceph 0.72.2 running on debian whe

[ceph-users] GSoC Deadline Approaching

[Patrick McGarry]

Calling all potential Google Summer of Code participants!

This is just a friendly reminder that there are only two days
remaining in the submission window to be considered for this year's
summer of code.  If you, or someone you know, is still planning on
submitting a proposal to work on the Ceph project, make sure you do so
by 19:00 UTC on 21 March.

If you have questions or need help putting the finishing touches on
your proposal I would be happy to help.  Keep in mind that proposals
are living documents and can be edited after the submission deadline.
However, Google is very strict about the submission deadline, and no
proposals will be accepted after it closes.

Feel free to contact me via email (patr...@inktank.com), irc
(scuttlemonkey on freenode/oftc), or twitter (@scuttlemonkey) if you
have questions.  If I can't answer your question, I'll be sure to get
the mentor to help give you the best possible chance at having your
proposal accepted.  Thanks, and good luck!


Best Regards,

Patrick McGarry
Director, Community || Inktank
http://ceph.com  ||  http://inktank.com
@scuttlemonkey || @ceph || @inktank
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Noah Watkins]

Err, obviously switching things out for Ceph rather than Gluster.

On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins  wrote:
> Looks like this is a configuration issue that has popped up with other
> 3rd party file systems in Hadoop 2.x with YARN.
>
>   
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E
>
> Says use this:
>
>  
>   fs.AbstractFileSystem.glusterfs.impl
>   org.apache.hadoop.fs.glusterfs.GlusterFS
>  
>
> Apparently rather than the `fs.ceph.impl` property in 2.x
>
>
> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh
>  wrote:
>> On 03/19/2014 04:50 PM, Noah Watkins wrote:
>>> Since `hadoop -fs ls /` seems to work on your local node, can you
>>> verify that (1) it is in fact listing the contents of CephFS, and (2)
>>> that on your worker nodes where the error is occuring that the
>>> relevant dependencies (naming the Ceph hadoop bindings) are installed
>>> and in the classpath?
>> 1) yes it is listing the cephfs contents.
>> 2) Currently the setup is to start the yarn manager and mapreduce
>> history server on the same node and then setup the worker nodes. Yarn
>> starts up fine, but the mapreduce history server gives me the error. The
>> dependency lib are loaded in the mapreduce history server startup. Here
>> is the link to start up message which shows that the libcephfs and
>> cephfs-hadoop jar are loaded.
>>
>> http://pastebin.com/wED7nMMT
>>
>> I am wondering if this property value is correct
>>
>> 
>>   fs.ceph.impl
>>   org.apache.hadoop.fs.ceph.CephFileSystem
>> 
>>
>> or should I set it to something for hadoop 2.x I have tried setting it up as
>>
>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as
>>
>> # hadoop fs -ls /
>> -ls: Fatal internal error
>> java.lang.RuntimeException: java.lang.NoSuchMethodException:
>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>> at
>> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
>> at
>> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
>> at
>> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
>> at
>> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
>> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
>> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
>> Caused by: java.lang.NoSuchMethodException:
>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
>> at java.lang.Class.getConstructor0(Class.java:2800)
>> at java.lang.Class.getDeclaredConstructor(Class.java:2043)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
>> ... 17 more
>>
>> Regards,
>> Gurvinder
>>
>>>
>>> The error itself looks like a missing dependency, but that exception
>>> being thrown might also be tirggered by other problems while loading
>>> the bindings.
>>>
>>> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
>>>  wrote:
 On 03/19/2014 03:51 PM, Noah Watkins wrote:
> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>  wrote:
>> Hi,
>>
>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
>> support. I am able to run the command such as
>
> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?
>
 Yes.
> Can you paste your core-site.xml?

 Below is the relevant part for ceph conf

  
 fs.defaultFS
 ceph://:6789/
   

   
 ceph.conf.file
 /etc/ceph/ceph.conf
   

   
 ceph.auth.id
 admin
   

   
 ceph.auth.keyfile
 /etc/hadoop/conf/admin.secret
   

   
 fs.ceph.impl
 org.apache.hadoop.fs.ceph.CephFileSystem
   


>

>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists

Re: [ceph-users] ceph hadoop mapred history server issue

[Noah Watkins]

Since `hadoop -fs ls /` seems to work on your local node, can you
verify that (1) it is in fact listing the contents of CephFS, and (2)
that on your worker nodes where the error is occuring that the
relevant dependencies (naming the Ceph hadoop bindings) are installed
and in the classpath?

The error itself looks like a missing dependency, but that exception
being thrown might also be tirggered by other problems while loading
the bindings.

On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
 wrote:
> On 03/19/2014 03:51 PM, Noah Watkins wrote:
>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
>>  wrote:
>>> Hi,
>>>
>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
>>> support. I am able to run the command such as
>>
>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?
>>
> Yes.
>> Can you paste your core-site.xml?
>
> Below is the relevant part for ceph conf
>
>  
> fs.defaultFS
> ceph://:6789/
>   
>
>   
> ceph.conf.file
> /etc/ceph/ceph.conf
>   
>
>   
> ceph.auth.id
> admin
>   
>
>   
> ceph.auth.keyfile
> /etc/hadoop/conf/admin.secret
>   
>
>   
> fs.ceph.impl
> org.apache.hadoop.fs.ceph.CephFileSystem
>   
>
>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph hadoop mapred history server issue

[Noah Watkins]

Looks like this is a configuration issue that has popped up with other
3rd party file systems in Hadoop 2.x with YARN.

  
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E

Says use this:

 
  fs.AbstractFileSystem.glusterfs.impl
  org.apache.hadoop.fs.glusterfs.GlusterFS
 

Apparently rather than the `fs.ceph.impl` property in 2.x


On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh
 wrote:
> On 03/19/2014 04:50 PM, Noah Watkins wrote:
>> Since `hadoop -fs ls /` seems to work on your local node, can you
>> verify that (1) it is in fact listing the contents of CephFS, and (2)
>> that on your worker nodes where the error is occuring that the
>> relevant dependencies (naming the Ceph hadoop bindings) are installed
>> and in the classpath?
> 1) yes it is listing the cephfs contents.
> 2) Currently the setup is to start the yarn manager and mapreduce
> history server on the same node and then setup the worker nodes. Yarn
> starts up fine, but the mapreduce history server gives me the error. The
> dependency lib are loaded in the mapreduce history server startup. Here
> is the link to start up message which shows that the libcephfs and
> cephfs-hadoop jar are loaded.
>
> http://pastebin.com/wED7nMMT
>
> I am wondering if this property value is correct
>
> 
>   fs.ceph.impl
>   org.apache.hadoop.fs.ceph.CephFileSystem
> 
>
> or should I set it to something for hadoop 2.x I have tried setting it up as
>
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as
>
> # hadoop fs -ls /
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at
> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at
> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
> at
> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.()
> at java.lang.Class.getConstructor0(Class.java:2800)
> at java.lang.Class.getDeclaredConstructor(Class.java:2043)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
> ... 17 more
>
> Regards,
> Gurvinder
>
>>
>> The error itself looks like a missing dependency, but that exception
>> being thrown might also be tirggered by other problems while loading
>> the bindings.
>>
>> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh
>>  wrote:
>>> On 03/19/2014 03:51 PM, Noah Watkins wrote:
 On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh
  wrote:
> Hi,
>
> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
> support. I am able to run the command such as

 From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1?

>>> Yes.
 Can you paste your core-site.xml?
>>>
>>> Below is the relevant part for ceph conf
>>>
>>>  
>>> fs.defaultFS
>>> ceph://:6789/
>>>   
>>>
>>>   
>>> ceph.conf.file
>>> /etc/ceph/ceph.conf
>>>   
>>>
>>>   
>>> ceph.auth.id
>>> admin
>>>   
>>>
>>>   
>>> ceph.auth.keyfile
>>> /etc/hadoop/conf/admin.secret
>>>   
>>>
>>>   
>>> fs.ceph.impl
>>> org.apache.hadoop.fs.ceph.CephFileSystem
>>>   
>>>
>>>

>>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 答复: why objects are still in .rgw.buckets after deleted

[Craig Lewis]

My understanding, from dealing with replication, is RadosGW is
copy-on-write. Overwriting an object is a delete and create, and the old
data gets garbage collected later.

I'm guessing a bit, but that's what I believe from Greg's comment about
RGW replication:
http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/7434

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website  | Twitter
 | Facebook
 | LinkedIn
 | Blog


On 3/19/14 03:58 , ljm李嘉敏 wrote:
>
> You are right, but I still don’t know why the objects in .rgw.buckets
> are not overrided.
>
> If the object name is produced through ino and ono, why the same
> file(bigger than 4M) have different result?
>
> Thanks & Regards
>
> Li JiaMin
>
> *发件人:*ceph-users-boun...@lists.ceph.com
> [mailto:ceph-users-boun...@lists.ceph.com] *代表 *Craig Lewis
> *发 送时间:*2014年3月19日9:00
> *收件人:*ceph-users@lists.ceph.com
> *主题:*Re: [ceph-users] why objects are still in .rgw.buckets after
> deleted
>
> I recall hearing that RGW GC waits 2 hours before garbage collecting
> deleted chunks.
>
> Take a look at https://ceph.com/docs/master/radosgw/config-ref/, the
> rgw gc * settings. rgw gc obj min wait is 2 hours.
>
>
>
>
> *Craig Lewis*
> Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com 
>
> *Central Desktop. Work together in ways you never thought possible.*
> Connect with us Website  | Twitter
>  | Facebook
>  | LinkedIn
>  | Blog
> 
>
> On 3/16/14 23:20 , ljm李嘉敏wrote:
>
> Hi all,
>
> I have a question about the pool .rgw.buckets, when I upload a
> file(has been stripped because it is bigger than 4M) through swift
> API, it is stored in .rgw.buckets,
>
> if I upload it again, why the objects in .rgw.buckets are not
> override? It is stored again and have different name. and when I
> delete the file, all of the objects in .rgw.buckets
>
> aren’t delete even though I execute radosgw-admin gc process.
>
> I also want to know something about the pool created for object
> gateway, why are they created and which role they will play? If
> anyone know about these,
>
> please give me a guide, thanks.
>
> Thanks & Regards
>
> Li JiaMin
>
> System Cloud Platform
>
> 3#4F108
>
>
>
>
> ___
>
> ceph-users mailing list
>
> ceph-users@lists.ceph.com 
>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW snapshots

[Anatol]

Craig Lewis  writes:

> 
> 
> I've looked into this a bit, and the
>   best I've come up with is to snapshot all of the RGW pools.  I
>   asked a similar question before: 
http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855
>   I am planning to have a 2nd cluster for disaster recovery, with
>   some in-house geo-replication.  
>   I haven't actually tried this yet.  I just setup my development
>   cluster, and this is on my list of things to test.  The basic
>   idea:
> Disable geo-replication
> Snapshot the Disaster Recovery cluster manually
> 
> Rollback all of the RGW pools to the snapshot I want to
>   restore from
> 
> Manually restore objects from the Disaster Recovery cluster
>   to the Production Cluster, probably using s3cmd
> 
> Return all of the RGW pools to the most recent snapshot
> Re-enable geo-replication
>   
> 

Hi,

I am facing the same issue (I want to create backups of buckets, and I would 
like to do this from snapshots so that I can get a consistent backup without 
stopping write operations).

Any news on how this worked out? Any new features that make this process 
simpler?

Thanks!


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW snapshots

[Craig Lewis]

I'm planning to add RGW Snapshots to Giant: 
https://wiki.ceph.com/Planning/Blueprints/Giant/rgw%3A_Snapshots .


I'm still getting my development environment setup, so I don't have 
anything on github yet.  If you're interested in testing, my repo is 
github.com/clewis/ceph


*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



On 3/19/14 11:26 , Anatol wrote:

Craig Lewis  writes:



 I've looked into this a bit, and the
   best I've come up with is to snapshot all of the RGW pools.  I
   asked a similar question before:

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

   I am planning to have a 2nd cluster for disaster recovery, with
   some in-house geo-replication.
   I haven't actually tried this yet.  I just setup my development
   cluster, and this is on my list of things to test.  The basic
   idea:
Disable geo-replication
 Snapshot the Disaster Recovery cluster manually

 Rollback all of the RGW pools to the snapshot I want to
   restore from

 Manually restore objects from the Disaster Recovery cluster
   to the Production Cluster, probably using s3cmd

 Return all of the RGW pools to the most recent snapshot
 Re-enable geo-replication
   


Hi,

I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).

Any news on how this worked out? Any new features that make this process
simpler?

Thanks!


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 答复: why objects are still in .rgw.buckets after deleted

[Yehuda Sadeh]

The RGW objects are [for most cases] immutable. Therefore in order to
provide read and write consistency we keep most of the data in
immutable rados objects, and do the operations on the mutable 'head'
part of the object atomically. So this allows us to do stuff like
having one user read data of the object, while a second user
overwrites the object and still the first user reads data that is
consistent. When an object is removed we remove the head part and the
rest is later on garbage collected. When it's rewritten, we write a
new immutable 'tail' for and just replace the head at the end (then
the old tail is garbage collected).

Yehuda

On Wed, Mar 19, 2014 at 11:31 AM, Craig Lewis  wrote:
> My understanding, from dealing with replication, is RadosGW is
> copy-on-write.  Overwriting an object is a delete and create, and the old
> data gets garbage collected later.
>
> I'm guessing a bit, but that's what I believe from Greg's comment about RGW
> replication:
> http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/7434
>
>
> Craig Lewis
> Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com
>
> Central Desktop. Work together in ways you never thought possible.
> Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog
>
> On 3/19/14 03:58 , ljm李嘉敏 wrote:
>
> You are right, but I still don’t know why the objects in .rgw.buckets are
> not overrided.
>
> If the object name is produced through ino and ono, why the same file(bigger
> than 4M) have different result?
>
>
>
> Thanks & Regards
>
> Li JiaMin
>
>
>
> 发件人: ceph-users-boun...@lists.ceph.com
> [mailto:ceph-users-boun...@lists.ceph.com] 代表 Craig Lewis
> 发 送时间: 2014年3月19日 9:00
> 收件人: ceph-users@lists.ceph.com
> 主题: Re: [ceph-users] why objects are still in .rgw.buckets after deleted
>
>
>
> I recall hearing that RGW GC waits 2 hours before garbage collecting deleted
> chunks.
>
> Take a look at https://ceph.com/docs/master/radosgw/config-ref/, the rgw gc
> * settings.  rgw gc obj min wait is 2 hours.
>
>
>
>
> Craig Lewis
> Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com
>
> Central Desktop. Work together in ways you never thought possible.
> Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog
>
> On 3/16/14 23:20 , ljm李嘉敏 wrote:
>
> Hi all,
>
>
>
> I have a question about the pool .rgw.buckets, when I upload a file(has been
> stripped because it is bigger than 4M) through swift API, it is stored in
> .rgw.buckets,
>
> if I upload it again, why the objects in .rgw.buckets are not override? It
> is stored again and have different name. and when I delete the file, all of
> the objects in .rgw.buckets
>
> aren’t delete even though I execute radosgw-admin gc process.
>
>
>
> I also want to know something about the pool created for object gateway, why
> are they created and which role they will play? If anyone know about these,
>
> please give me a guide, thanks.
>
>
>
>
>
> Thanks & Regards
>
> Li JiaMin
>
>
>
> System Cloud Platform
>
> 3#4F108
>
>
>
>
>
>
> ___
>
> ceph-users mailing list
>
> ceph-users@lists.ceph.com
>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Broken bucket on upgrade

[Gregory Farnum]

Exactly what errors did you see, from which log? In general the OSD
does suicide on filesystem errors.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Wed, Mar 19, 2014 at 4:06 AM, Mike Bryant  wrote:
> So I've done some more digging, and running the radosgw in debug mode I
> found some messages from osd.3 saying IOError, when it was trying to get
> .rgw:productimages.
> I took that OSD down, and everything started working.
>
> My question now is, why didn't that OSD suicide when it hit an IOError,
> instead of causing the cluster to stop working?
>
>
> On 19 March 2014 10:07, Mike Bryant  wrote:
>>
>> Hi,
>> I've just upgraded a test cluster to Emporer, and one of my S3 buckets
>> seems to have broken.
>>
>> s3 access is returning a 500 code (UnknownError).
>>
>> Running bucket stats, it's missing from the list.
>> Trying to do it explicitly:
>>
>> radosgw-admin bucket stats --bucket=productimages
>> 2014-03-19 10:06:17.829397 7ff0b81c7780  0 could not get bucket info for
>> bucket=productimages
>>
>> I can see the header object in the .rgw pool:
>> rados --cluster=cit-external ls --pool .rgw
>> .pools.avail
>> productimages
>> test
>> tests3fs
>>
>> Does anyone have any idea on what might have happened, or how I can get
>> this bucket back?
>>
>> Cheers
>> Mike
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Broken bucket on upgrade

[Mike Bryant]

I'm afraid the exact error got lost out of scrollback before I thought to
save it.
I was running radosgw in debug mode (-d --debug-rgw=20 --debug-ms=20), and
it was along the lines of this:
2014-03-19  7faeb7fff700 20 get_obj_state: rctx=0x7fae78002df0
obj=.rgw:productimages state=0x7fae7800a368 s->prefetch_data=0
2014-03-19 1 ... <== osd.3 ..  osd_op_reply(... ack = -5 (i/o error))
Not sure how much help that is though.


On 19 March 2014 18:55, Gregory Farnum  wrote:

> Exactly what errors did you see, from which log? In general the OSD
> does suicide on filesystem errors.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
> On Wed, Mar 19, 2014 at 4:06 AM, Mike Bryant 
> wrote:
> > So I've done some more digging, and running the radosgw in debug mode I
> > found some messages from osd.3 saying IOError, when it was trying to get
> > .rgw:productimages.
> > I took that OSD down, and everything started working.
> >
> > My question now is, why didn't that OSD suicide when it hit an IOError,
> > instead of causing the cluster to stop working?
> >
> >
> > On 19 March 2014 10:07, Mike Bryant  wrote:
> >>
> >> Hi,
> >> I've just upgraded a test cluster to Emporer, and one of my S3 buckets
> >> seems to have broken.
> >>
> >> s3 access is returning a 500 code (UnknownError).
> >>
> >> Running bucket stats, it's missing from the list.
> >> Trying to do it explicitly:
> >>
> >> radosgw-admin bucket stats --bucket=productimages
> >> 2014-03-19 10:06:17.829397 7ff0b81c7780  0 could not get bucket info for
> >> bucket=productimages
> >>
> >> I can see the header object in the .rgw pool:
> >> rados --cluster=cit-external ls --pool .rgw
> >> .pools.avail
> >> productimages
> >> test
> >> tests3fs
> >>
> >> Does anyone have any idea on what might have happened, or how I can get
> >> this bucket back?
> >>
> >> Cheers
> >> Mike
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph mapreduce history server issue

[Gregory Farnum]

I haven't worked with Hadoop in a while, but from the error it sounds
like the map reduce server needs another config option set specifying
which filesystem to work with. I don't think those instructions you
linked to are tested with hadoop 2.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Wed, Mar 19, 2014 at 3:44 AM, Gurvinder Singh
 wrote:
> Hi,
>
> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2
> hadoop. I have installed the ceph hadoop binding with hadoop 2.x
> support. I am able to run the command such as
>
> # hadoop fs -ls /
> # hdfs dfs -touchz /test
>
> But when I start the mapreduce job history server I am getting the error as
>
> "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
> AbstractFileSystem for scheme: ceph"
>
> Here is the link to mapreduce startup message listing the lib it linked
> against while starting and it has both libcepfs and cephfs-hadoop jars.
>
> http://pastebin.com/wED7nMMT
>
> I have made the changes in to core-site.xml file, that's why I am able
> to the normal hadoop commands. Let me know if you have any suggestions
> how to resolve this.
>
> Regards,
> Gurvinder
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] specific OSD hostnames

[Lluís Pàmies i Juárez]

Hello,

I want to bind an OSD to a virtual network interface (let's call it
eth0:0). The setting is the following. There are two net interfaces:
 eth0   --> 10.0.0.100
 eth0:0 --> 10.0.0.200
"/etc/hostname" contains "testnode" and in "/etc/hosts" I have:
10.0.0.100 testnode
10.0.0.200 testnode0

Both monitor and osd have the following rule in /etc/ceph/ceph.conf:

[osd.0]
host = testnode0
cluster addr = 10.0.0.200
public addr = 10.0.0.200

Then, I manually initialize the OSD (osd.0 in this example), and after
executing:

$ ceph osd crush add-bucket testnode0 host
$ ceph osd crush move testnode0 root=default
$ ceph osd crush add osd.0 1.0 host=testnode0

I can see the following crush map:

# buckets
host testnode0 {
 id -2 # do not change unnecessarily
 # weight 1.000
 alg straw
 hash 0 # rjenkins1
 item osd.0 weight 1.000
}
root default {
 id -1 # do not change unnecessarily
 # weight 1.000
 alg straw
 hash 0 # rjenkins1
 item testnode0 weight 1.000
}

However, after starting the OSD (start ceph-osd id=0), the OSD is
reallocated, changing the crushmap to:

# buckets
host testnode0 {
 id -2 # do not change unnecessarily
 # weight 0.000
 alg straw
 hash 0 # rjenkins1
}
host testnode {
 id -3 # do not change unnecessarily
 # weight 1.000
 alg straw
 hash 0 # rjenkins1
 item osd.0 weight 1.000
}
root default {
 id -1 # do not change unnecessarily
 # weight 1.000
 alg straw
 hash 0 # rjenkins1
 item testnode0 weight 0.000
 item testnode weight 1.000
}

Is there any way to avoid that and get the new OSD running under the
name "testnode0" ?

Thanks,

-- 
Lluís Pàmies i Juárez
http://lluis.pamies.cat
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Utilizing DAS on XEN or XCP hosts for Openstack Cinder

[Paul Mitchener]

Kyle,


Thanks for your prompt reply. I have been doing some further reading and 
planning after receiving your valuable input.

 

>> 1.   Is it possible to install Ceph and Ceph monitors on the the XCP

>> (XEN) Dom0 or would we need to install it on the DomU containing the 

>> Openstack components?

>I'm not a Xen guru but in the case of KVM I would run the OSDs on the 
>hypervisor to avoid virtualization overhead.

 

As you have suggested, our plan is to install Ceph at the hypervisor level, ie: 
Dom0

 

>> 2.   Is Ceph server aware, or Rack aware so that replicas are not stored

>> on the same server?

>Yes, placement is defined with your crush map and placement rules.

 

 

>> 3.   Are 4Tb OSD’s too large? We are attempting to restrict the qty of

>> OSD’s per server to minimise system overhead

>Nope!

 

 

>> Any other feedback regarding our plan would also be welcomed.

>I would probably run each disk as it's own OSD, which means you need a bit 
>more memory per host. Networking could certainly be a bottleneck with 8 to 16 
>spindle nodes. YMMV.

 

I had contemplated having 1 OSD per spindle, but my worry was both processing 
and RAM overhead as well as network bottlenecks. (ie: no budget for 10Gbe)

 

 

5.Will 2 x Bonded 1Gbe be sufficient for block storage for 7 – 10 
hypervisors with OSD’s on each made up of 4 x RAID0 7200RPM SAS drives & from 
user experience, what sort of data throughput would I expect to see?

Thanks

Paul

 

-Original Message-
From: Kyle Bader [mailto:kyle.ba...@gmail.com] 
Sent: Wednesday, 12 March 2014 7:56 AM
To: Paul Mitchener
Cc: ceph-users
Subject: Re: [ceph-users] Utilizing DAS on XEN or XCP hosts for Openstack Cinder

 

> 1.   Is it possible to install Ceph and Ceph monitors on the the XCP

> (XEN) Dom0 or would we need to install it on the DomU containing the 

> Openstack components?

 

I'm not a Xen guru but in the case of KVM I would run the OSDs on the 
hypervisor to avoid virtualization overhead.

 

> 2.   Is Ceph server aware, or Rack aware so that replicas are not stored

> on the same server?

 

Yes, placement is defined with your crush map and placement rules.

 

> 3.   Are 4Tb OSD’s too large? We are attempting to restrict the qty of

> OSD’s per server to minimise system overhead

 

Nope!

 

> Any other feedback regarding our plan would also be welcomed.

 

I would probably run each disk as it's own OSD, which means you need a bit more 
memory per host. Networking could certainly be a bottleneck with 8 to 16 
spindle nodes. YMMV.

 

-- 

 

Kyle

##

This e-mail message has been scanned for Viruses and Content and cleared by 
AUS-IP SecurMail (  
http://www.ausip.net.au/securmail) at 8:56:28 PM on 11 Mar 2014 
##

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd rebalance question

[xiao xianxia]

Hi,adding osds will bring data migration,but only a small part of all data.
you can view the crush algorithm to learn more.


于 2014/3/10 10:25, You, Rong 写道:

Hi guys,
  I need to add a extend server, which reside several osds, to a 
running ceph cluster. During add osds, ceph would not automatically modify the 
ceph.conf. So I manually modify the ceph.conf
And restart the whole ceph cluster with command: 'service ceph -a restart'. I 
just confused that if I restart the ceph cluster, ceph would rebalance the 
whole data(redistribution whole data) among osds? Or just move some
Data from existed osds to new osds? Anybody knows?
  Best regards!



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: 答复: why objects are still in .rgw.buckets after deleted

[ljm李嘉敏]

Hi Yehuda&Craig Lewis,

Thank you very much for your explanations, does the head object have some info 
about how to find the objects constituting the file, which is stripped?
I am interested in the construction of the head object, but I can't see content 
of it.

Thanks & Regards
Li JiaMin


-邮件原件-
发件人: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] 代表 Yehuda Sadeh
发送时间: 2014年3月20日 2:46
收件人: Craig Lewis
抄送: ceph-users@lists.ceph.com
主题: Re: [ceph-users] 答复: why objects are still in .rgw.buckets after deleted

The RGW objects are [for most cases] immutable. Therefore in order to provide 
read and write consistency we keep most of the data in immutable rados objects, 
and do the operations on the mutable 'head'
part of the object atomically. So this allows us to do stuff like having one 
user read data of the object, while a second user overwrites the object and 
still the first user reads data that is consistent. When an object is removed 
we remove the head part and the rest is later on garbage collected. When it's 
rewritten, we write a new immutable 'tail' for and just replace the head at the 
end (then the old tail is garbage collected).

Yehuda

On Wed, Mar 19, 2014 at 11:31 AM, Craig Lewis  wrote:
> My understanding, from dealing with replication, is RadosGW is 
> copy-on-write.  Overwriting an object is a delete and create, and the 
> old data gets garbage collected later.
>
> I'm guessing a bit, but that's what I believe from Greg's comment 
> about RGW
> replication:
> http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/7434
>
>
> Craig Lewis
> Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com
>
> Central Desktop. Work together in ways you never thought possible.
> Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog
>
> On 3/19/14 03:58 , ljm李嘉敏 wrote:
>
> You are right, but I still don’t know why the objects in .rgw.buckets 
> are not overrided.
>
> If the object name is produced through ino and ono, why the same 
> file(bigger than 4M) have different result?
>
>
>
> Thanks & Regards
>
> Li JiaMin
>
>
>
> 发件人: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] 代表 Craig Lewis
> 发 送时间: 2014年3月19日 9:00
> 收件人: ceph-users@lists.ceph.com
> 主题: Re: [ceph-users] why objects are still in .rgw.buckets after 
> deleted
>
>
>
> I recall hearing that RGW GC waits 2 hours before garbage collecting 
> deleted chunks.
>
> Take a look at https://ceph.com/docs/master/radosgw/config-ref/, the 
> rgw gc
> * settings.  rgw gc obj min wait is 2 hours.
>
>
>
>
> Craig Lewis
> Senior Systems Engineer
> Office +1.714.602.1309
> Email cle...@centraldesktop.com
>
> Central Desktop. Work together in ways you never thought possible.
> Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog
>
> On 3/16/14 23:20 , ljm李嘉敏 wrote:
>
> Hi all,
>
>
>
> I have a question about the pool .rgw.buckets, when I upload a 
> file(has been stripped because it is bigger than 4M) through swift 
> API, it is stored in .rgw.buckets,
>
> if I upload it again, why the objects in .rgw.buckets are not 
> override? It is stored again and have different name. and when I 
> delete the file, all of the objects in .rgw.buckets
>
> aren’t delete even though I execute radosgw-admin gc process.
>
>
>
> I also want to know something about the pool created for object 
> gateway, why are they created and which role they will play? If anyone 
> know about these,
>
> please give me a guide, thanks.
>
>
>
>
>
> Thanks & Regards
>
> Li JiaMin
>
>
>
> System Cloud Platform
>
> 3#4F108
>
>
>
>
>
>
> ___
>
> ceph-users mailing list
>
> ceph-users@lists.ceph.com
>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com