Running 0.94.5 as part of a Openstack enviroment, our ceph setup is 3x OSD
Nodes 3x MON Nodes, yesterday we had a aircon outage in our hosting
enviroment, 1 OSD node failed (offline with a the journal SSD dead) left
with 2 nodes running correctly, 2 hours later a second OSD node failed
complaining
tive, I cannot
> start vms I'm openstack and ceph -w states that rd is zero most of the time.
>
> I can restart the nodes and the osd reconnect and come in correctly..
>
> I was chasing a missing object but was able to correct that by setting the
> crush weight to 0.0 on the secon
ph -w states that rd is zero most of the time.
I can restart the nodes and the osd reconnect and come in correctly..
I was chasing a missing object but was able to correct that by setting the
crush weight to 0.0 on the secondary.
Kind regards
Lee
On Sun, 2 Sep 2018, 13:43 David C, wrote:
&
ms. What
> commands did you run to bring it back online?
>
> On Sun, Sep 2, 2018, 10:27 AM Lee wrote:
>
>> Ok I have a lot in the health detail...
>>
>> root@node31-a4:~# ceph health detail
>> HEALTH_ERR 64 pgs backfill; 27 pgs backfill_toofull; 39 pgs backfillin
near each other, there is a very real
> possibility that you will have some data loss from this.
>
> Regardless, your first step is to remove the OSDs that were on the failed
> journal. They are poison in your cluster.
>
> On Sun, Sep 2, 2018, 10:51 AM Lee wrote:
>
>> I fo
ec on osd.41
5 ops are blocked > 131.072 sec on osd.41
14 osds have slow requests
recovery 189851/6050948 objects degraded (3.138%)
recovery 3916293/6050948 objects misplaced (64.722%)
1 scrub errors
nobackfill,norecover,noscrub,nodeep-scrub flag(s) set
And still no Client IO..
On Sun, 2
Hi,
I follow and use your articles regularly to help with our Ceph environment,
I am looking for urgent help with our infrastructure after a series of
outages over the weekend has ground our ceph environment to its knees.
The system is .0.94.5 and deployed as part of open stack.
In the series of
tached storages.
Thanks in advance
&
Best regards,
Felix Lee ~
--
Felix Lee Academia Sinica Grid & Cloud.
Tel: +886-2-27898308
Office: Room P111, Institute of Physics, 128 Academia Road, Section 2,
Nankang, Taipei 115, Taiwan
__
orks.
In any case, thanks again for your reply.
Best regards,
Felix Lee ~
On 2014年05月30日 23:53, Craig Lewis wrote:
On 5/29/14 01:09 , Felix Lee wrote:
Dear experts,
Recently, a disk for one of our OSDs was failure and caused osd down,
after I recovered the disk and filesystem, I noticed two pro
raditional storage system we used to use, but it's unnecessary for CEPH
indeed, we still need to accommodate ourself to be real ceph user. :)
Best regards,
Felix Lee ~
On 2014年06月02日 13:04, Wido den Hollander wrote:
On 06/02/2014 12:41 PM, Felix Lee wrote:
Hi, Craig,
Many thanks for y
1 mons at {csA=192.168.122.106:6789/0}, election epoch 1,
quorum 0 csA
osdmap e37: 4 osds: 4 up, 4 in
pgmap v678: 192 pgs, 3 pools, 0 bytes data, 0 objects
20623 MB used, 352 GB / 372 GB avail
192 active+clean
#
Regards,
Jae
--
이재면 Jaemyoun Lee
15, 2014, Jaemyoun Lee wrote:
>
>> Hi All,
>>
>> I am using ceph 0.80.1 on Ubuntu 14.04 on KVM. However, I cannot connect
>> to the MON from a client using ceph-fuse.
>>
>> On the client, I installed the ceph-fuse 0.80.1 and added fuse. But, I
>> think it
client.4705 renew_caps()
2014-07-16 17:14:14.212277 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:14:34.214827 7f9a177fe700 10 client.4705 renew_caps()
Thx~
- Jae
On Wed, Jul 16, 2014 at 2:20 AM, Gregory Farnum wrote:
> On Tue, Jul 15, 2014 at 10:15 AM, Jaemyoun Lee wrote:
> > T
Thank you, Greg!
I solved it through creating MDS.
- Jae
On Wed, Jul 16, 2014 at 8:36 PM, Gregory Farnum wrote:
> Your MDS isn't running or isn't active.
> -Greg
>
>
> On Wednesday, July 16, 2014, Jaemyoun Lee wrote:
>
>>
>> The result is same.
>&
Hi all,
I wonder that OSDs use system calls of Virtual File System (i.e. open,
read, write, etc) when they access disks.
I mean ... Could I monitor I/O command requested by OSD to disks if I
monitor VFS?
- Jae
--
이재면 Jaemyoun Lee
CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang
Thanks for your rapid reply
- Jae
On Tue, Jul 22, 2014 at 1:29 AM, Gregory Farnum wrote:
> On Monday, July 21, 2014, Jaemyoun Lee wrote:
>
>> Hi all,
>>
>> I wonder that OSDs use system calls of Virtual File System (i.e. open,
>> read, write, etc) when t
uested by OSD to disks if I
> > monitor VFS?
>
> Ceph OSDs run on top of a traditional filesystem, so long as they
> support xattrs - xfs by default. As such you can use kernel
> instrumentation to view what is going on "under" the Ceph OSDs.
>
> --
>
> Kyle
>
>
29850 MB used, 27118 MB / 60088 MB avail
840 active+clean
Lee
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
I have a virtual test environment of an admin node and 3 mon + osd nodes,
built by just following the quick start guide. It seems to work OK but
ceph is constantly complaining about clock skew much greater than reality.
Clocksource on the virtuals is kvm-clock and they also run ntpd.
ceph-admin-n
I think I solved the problem. The clock skew only happens when restarting a
node to simulate hardware failure. The virtual comes up with a skewed clock
and ceph services start before ntp has time to adjust it, then there's a
delay before ceph rechecks the clock skew.
Lee
On Thu, Mar 26, 20
obals(), locals(),
generator_name)
ImportError: No module named generator.make
Lee
On Thu, Mar 26, 2015 at 1:14 PM, Quentin Hartman <
qhart...@direwolfdigital.com> wrote:
> I used this as a guide for building calamari packages w/o using vagrant.
> Worked great:
> http://bryanapperson.
I added the osd pool default min size = 1 to test the behavior when 2 of 3
OSDs are down, but the behavior is exactly the same as without it: when the
2nd OSD is killed, all client writes start to block and these
pipe.(stuff).fault messages begin:
2015-03-26 16:08:50.775848 7fce177fe700 0 monclie
8.122.111:0/1007741 >>
192.168.122.131:6789/0 pipe(0x7f4ec00251b0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7f4ec0025440).fault
And all writes block until I bring back an OSD.
Lee
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ah, thanks, got it, I wasn't thinking that mons and osds on the same node
isn't a likely real world thing.
You have to admit that pipe/fault log message is a bit cryptic.
Thanks,
Lee
___
ceph-users mailing list
ceph-users@lists.cep
27;t start
them either. The only thing that works is "ceph-deploy osd activate". Any
ideas?
Lee
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
hen the network
temporarily goes away - client had to be rebooted. Is this likely to be the
same underlying issue?
Lee
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
quick doc, before preparing
the OSDs?
--
Jaemyoun Lee
CPS Lab. ( Cyber-Physical Systems Laboratory in Hanyang University)
E-mail : jm...@cpslab.hanyang.ac.kr
Homepage : http://cpslab.hanyang.ac.kr
___
ceph-users mailing list
ceph-users@lists.cep
hello,
is there some methods to monitor osd nodes? for example the free size of
one osd node.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hi all,
I am new here and glad to see you guys.
Thanks for your hard work for providing a more stable, powerful,functional
ceph.
I was reading the source code of Ceph-0.72.2, and I've got a question:
what is the data packet format of Ceph?Or how are the packets packaged?
We know that typical tcp p
I'm new to Ceph, and looking for a new storage to replace legacy system.
My system has a lot of files accessing temporarily for 2 or 3 days.
Those files are uploaded from many clients everyday, and batch job deletes
unused files everyday.
In this case, can I use Ceph's pool to store daily uploade
Hi Lopez,
Thank you for recommendation! I will try to do it. :)
Regards,
Hyangtack
On Wed, Feb 12, 2014 at 2:28 PM, Jean-Charles Lopez wrote:
> Hi Lee
>
> You could use an Ceph RBD device on a server and export a directory that
> you would have created on this RBD though NFS.
>
uth
get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow
profile bootstrap-mds"]}]': finished
--- end ---
Best regards,
Jae
--
Jaemyoun Lee
CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University)
E-mail
d as it does not have the force option and then the old create
> failed because sda could not be prepared/formatted.
>
> Regards
> JC
>
> On Jun 5, 2016, at 10:22, Jaemyoun Lee wrote:
>
> Hi,
>
> When I run below script to install Ceph (10.2.0), I met an error "no o
Hi,
Did you check Admin API of Rados gateway?
http://docs.ceph.com/docs/master/radosgw/adminops/#get-usage
On Thu, Jun 23, 2016 at 5:07 PM, magicb...@hotmail.com <
magicb...@hotmail.com> wrote:
> Hi
>
> is there any possibility to make RadosGW to account stats for something
> similar to "rados
Hi,
You need to specify pool name.
rbd -p testpool info testvol11
On Thu, Jul 14, 2016 at 8:55 AM, EP Komarla
wrote:
> Hi,
>
>
>
> I am seeing an issue. I created 5 images testvol11-15 and I mapped them
> to /dev/rbd0-4. When I execute the command ‘rbd showmapped’, it shows
> correctly t
me how to install a specific version of Ceph by ceph-deploy?
or how to install Ceph on ARM processors?
Thanks,
Jae
--
Jaemyoun Lee
CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University)
E-mail: jaemy...@hanyang.ac.kr
Website: http://cpslab.hanyang.ac.kr
Personal
Hi,
I would like to create a rbd on client-node.
I created a cluster on admin-node successfully.
jae@admin-node$ ceph health
HEALTH_OK
A rbd was created on admin-node successfully.
jae@admin-node$ rbd created foo --size 1024
However, when I created a rbd on client-node, the error wa
Dear Laizer,
I did deploy the configuration and the key by '$ ceph-deploy admin
client-node' on admin-node.
jae@client-node$ ls /etc/ceph
ceph.client.admin.keyring ceph.conf rbdmap tmpoWLFTb
On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer
wrote:
Hello Lee,
Make sur
Dear Laizer,
Oh, I got it. the rbd was created successfully after I disabled the
firewall.
However, when the firewall is enabled and the 6789 port is allowed, the
authentication error is occurred.
Is there any other port?
On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer
wrote:
> Hello
ve the issue.
>
> Thanks.
>
>
> Daleep Singh Bais
>
>
> On 10/07/2016 10:14 AM, Jaemyoun Lee wrote:
>
> Dear Laizer,
>
> Oh, I got it. the rbd was created successfully after I disabled the
> firewall.
> However, when the firewall is enabled and the 6789 por
Dear all,
I have a trouble for using Ceph.
When I built Ceph by source code from the official repo with v10.2.3, I
couldn't create a Ceph cluster.
The stat of OSDs cannot be UP after they were activated.
I think the problem is Upstart because "make install" didn't copy conf
files to /etc/init/
#
foreground and
> debug) to make sure if it has anything to do with the system init
>
> Good Luck
>
> the footer sucks
>
>
> -- Original --
> *From: * "Jaemyoun Lee";
> *Date: * Fri, Nov 18, 2016 01:52 AM
> *To: * "
Hello,
I executed the RADOS tool to store an object as follows:
```
user@ClientA:~$ rados put -p=rbd objectA a.txt
```
I wonder how the client knows a completion of storing the object in some
OSDs.
Thanks,
Jae
--
Jaemyoun Lee
CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang
Dear Wido,
Is there a command to check the ACK? Or, may you tell me a source code
function for the received ACK?
Thanks,
Jae
On Thu, Dec 29, 2016 at 6:56 PM Wido den Hollander wrote:
>
> > Op 28 december 2016 om 12:58 schreef Jaemyoun Lee <
> jaemy...@hanyang.ac.kr>:
Thanks a lot!
On Fri, Dec 30, 2016 at 6:32 PM Shinobu Kinjo wrote:
> You can track activity of acting set by using:
>
> # ceph daemon osd.${osd id} dump_ops_in_flight
>
> On Fri, Dec 30, 2016 at 3:59 PM, Jaemyoun Lee
> wrote:
>
> Dear Wido,
> Is there a command to
Should we add big packet test in heartbeat? Right now the heartbeat
only test the little packet. If the MTU is mismatched, the heartbeat
can not find that.
2015-12-14 12:18 GMT+08:00 Chris Dunlop :
> On Sun, Dec 13, 2015 at 09:10:34PM -0700, Robert LeBlanc wrote:
>> I've had something similar to
count?
3) any other better ways...
Rgds
Lee
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
On 28/07/2019 12:11, Lee Norvall wrote:
Hi
I am using ceph-ansible to deploy and just looking for best way/tips on
how to export multiple pools/fs.
Ceph: nautilus (14.2.2)
NFS-Ganesha v 2.8
ceph-ansible
Hi Jeff
Thanks for the pointer on this. I found some details on this the other day and
your link is a big help. I will get this updated in my ansible playbook and
test.
Rgds
Lee
On 01/08/2019 17:03, Jeff Layton wrote:
On Sun, 2019-07-28 at 18:20 +, Lee Norvall wrote:
Update to this
Hi,
Recently, I'm evaluating the snpahsot feature of CephFS from kernel
client and everthing works like a charm. But, it seems that reverting
a snapshot is not available currently. Is there some reason or
technical limitation that the feature is not provided? Any insights
or ideas are appreciat
On Thu, 14 Nov 2019 at 07:07, Patrick Donnelly wrote:
>
> On Wed, Nov 13, 2019 at 2:30 AM Jerry Lee wrote:
> > Recently, I'm evaluating the snpahsot feature of CephFS from kernel
> > client and everthing works like a charm. But, it seems that reverting
> > a snapsh
I have three storage servers that provide NFS and iSCSI services to my
network, which serve data to four virtual machine compute hosts (two
ESXi, two libvirt/kvm) with several dozen virtual machines . I decided
to test out a Ceph deployment to see whether it could replace iSCSI as
the primary w
On 10/2/2013 2:24 PM, Gregory Farnum wrote:
There's a couple things here:
1) You aren't accounting for Ceph's journaling. Unlike a system such
as NFS, Ceph provides *very* strong data integrity guarantees under
failure conditions, and in order to do so it does full data
journaling. So, yes, cut y
On 10/2/2013 3:13 PM, Warren Wang wrote:
I agree with Greg that this isn't a great test. You'll need multiple
clients to push the Ceph cluster, and you have to use oflag=direct if
you're using dd.
I was not doing a test of overall performance but, rather, doing a
"smoke test" to see whether
On 10/2/2013 3:50 PM, Sage Weil wrote:
On Wed, 2 Oct 2013, Eric Lee Green wrote:
By contrast, that same dd to an iSCSI volume exported by one of the servers
wrote at 240 megabytes per second. Order of magnitude difference.
Can you see what 'rados -p rbd bench 60 write' tells you?
P
u might also just be able to enable
the CRUSH tunables (http://ceph.com/docs/master/rados/operations/crush-map/#tunables).
I experienced this (stuck active+remapped) frequently with the stock
0.41 apt-get/Ubuntu version of ceph. Less so with Bobtail.
Jeff Anderson-Lee
John, this is becoming a more
On 4/5/2013 10:32 AM, Gregory Farnum wrote:
On Fri, Apr 5, 2013 at 10:28 AM, Dimitri Maziuk wrote:
On 04/05/2013 10:12 AM, Wido den Hollander wrote:
Think about it this way. You have two racks and the network connection
between them fails. If both racks keep operating because they can still
r
Hi,
I’m playing with Jewel and discovered format 1 images have been deprecated.
Since the rbd kernel module in CentOS/RHEL 7 does not yet support format 2
images, how do I access RBD images created in Jewel from CentOS/RHEL 7 clients?
Thanks!
Steven
___
t 4:39 PM, Ilya Dryomov wrote:
>
> On Fri, May 13, 2016 at 10:11 PM, Steven Hsiao-Ting Lee
> wrote:
>> Hi,
>>
>> I’m playing with Jewel and discovered format 1 images have been deprecated.
>> Since the rbd kernel module in CentOS/RHEL 7 does not yet support form
59 matches
Mail list logo
