This worked.
However somebody should investigate why default is still jewel on Centos 7.4
Il 28/02/2018 00:53, jorpilo ha scritto:
Try using:
ceph-deploy --release luminous host1...
Mensaje original
De: Massimiliano Cuttini
Fecha: 28/2/18 12:42 a. m. (GMT+01:00)
Para: ceph-
Hi all,
I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and
OSD's updated fine.
When updating the MDS's (we have 2 active and 1 standby), I started
with the standby.
At the moment the standby MDS restarted into 12.2.4 [1], both active
MDSs (still running 12.2.2) suicided like thi
Quoting Dan van der Ster (d...@vanderster.com):
> Hi all,
>
> I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and
> OSD's updated fine.
12.2.4? Did you mean 12.2.3? Or did I miss something?
Gr. stefan
--
| BIT BV http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0
More:
here is the MDS_FEATURES map for a running 12.2.2 cluster:
compat: compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=file layout v2}
and here it is
(Sorry to spam)
I guess it's related to this fix to the layout v2 feature id:
https://github.com/ceph/ceph/pull/18782/files
-#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8,
"file layout v2")
+#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(9,
"file layout v2")
Is
On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote:
> (Sorry to spam)
>
> I guess it's related to this fix to the layout v2 feature id:
> https://github.com/ceph/ceph/pull/18782/files
>
> -#define MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 CompatSet::Feature(8,
> "file layout v2")
> +#define MDS_FEA
On Wed, Feb 28, 2018 at 11:38 AM, Patrick Donnelly wrote:
> On Wed, Feb 28, 2018 at 2:07 AM, Dan van der Ster wrote:
>> (Sorry to spam)
>>
>> I guess it's related to this fix to the layout v2 feature id:
>> https://github.com/ceph/ceph/pull/18782/files
>>
>> -#define MDS_FEATURE_INCOMPAT_FILE_LAY
Hello
I have a little ceph cluster with 3 nodes, each with 3x1TB HDD and
1x240GB SSD. I created this cluster after Luminous release, so all OSDs
are Bluestore. In my crush map I have two rules, one targeting the SSDs
and one targeting the HDDs. I have 4 pools, one using the SSD rule and
the o
Hi,
might be http://tracker.ceph.com/issues/22464
Can you check the OSD log file to see if the reported checksum is 0x6706be76?
Paul
> Am 28.02.2018 um 11:43 schrieb Marco Baldini - H.S. Amiata
> :
>
> Hello
>
> I have a little ceph cluster with 3 nodes, each with 3x1TB HDD and 1x240GB
> S
I am aware of monitor consensus requirement. It is taken care of (there is a
third room with only monitor node). My problem is about OSD redundancy, since I
can only use 2 server rooms for OSDs.
I could use EC-pools, lrc or any other ceph configuration. But I could not find
a configuration that
On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote:
> Hi all,
>
> I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and
> OSD's updated fine.
>
> When updating the MDS's (we have 2 active and 1 standby), I started
> with the standby.
>
> At the moment the standby MDS restarted i
On Wed, Feb 28, 2018 at 11:05 AM, John Spray wrote:
> On Wed, Feb 28, 2018 at 9:37 AM, Dan van der Ster wrote:
>> Hi all,
>>
>> I'm just updating our test cluster from 12.2.2 to 12.2.4. Mon's and
>> OSD's updated fine.
>>
>> When updating the MDS's (we have 2 active and 1 standby), I started
>> w
Hello,
with jewel we always used the python crush optimizer which gave us a
pretty good distribution fo the used space.
Since luminous we're using the included ceph mgr balancer but the
distribution is far from perfect and much worse than the old method.
Is there any way to tune the mgr balancer
I was building ceph in order to use with iSCSI.
But I just see from the docs that need:
*CentOS 7.5*
(which is not available yet, it's still at 7.4)
https://wiki.centos.org/Download
*Kernel 4.17*
(which is not available yet, it is still at 4.15.7)
https://www.kernel.org/
So I
Hi Stefan,
Which balancer mode are you using? crush-compat scores using a mix of
nobjects, npgs, and size. It's doing pretty well over here as long as
you have a relatively small number of empty PGs.
I believe that upmap uses nPGs only, and I haven't tested it enough
yet to know if it actually imp
On Wed, Feb 28, 2018 at 12:47 PM, Stefan Priebe - Profihost AG
wrote:
> Hello,
>
> with jewel we always used the python crush optimizer which gave us a
> pretty good distribution fo the used space.
>
> Since luminous we're using the included ceph mgr balancer but the
> distribution is far from per
Hi Massimiliano, have an espresso. You know the indians have a nice
saying
"Everything will be good at the end. If it is not good, it is still not
the end."
-Original Message-
From: Massimiliano Cuttini [mailto:m...@phoenixweb.it]
Sent: woensdag 28 februari 2018 13:53
To: ceph-user
Hi
I read the bugtracker issue and it seems a lot like my problem, even if
I can't check the reported checksum because I don't have it in my logs,
perhaps it's because of debug osd = 0/0 in ceph.conf
I just raised the OSD log level
ceph tell osd.* injectargs --debug-osd 5/5
I'll check OSD l
Am 28.02.2018 um 13:59 schrieb John Spray:
> On Wed, Feb 28, 2018 at 12:47 PM, Stefan Priebe - Profihost AG
> wrote:
>> Hello,
>>
>> with jewel we always used the python crush optimizer which gave us a
>> pretty good distribution fo the used space.
>>
>> Since luminous we're using the included ce
Am 28.02.2018 um 13:58 schrieb Dan van der Ster:
> Hi Stefan,
>
> Which balancer mode are you using? crush-compat scores using a mix of
> nobjects, npgs, and size. It's doing pretty well over here as long as
> you have a relatively small number of empty PGs.
>
> I believe that upmap uses nPGs only
Like John says, noout prevents an osd being marked out in the cluster. It
does not impede it from being marked down and back up which is the desired
behavior when restarting a server. What are you seeing with your osds
becoming unusable and needing to rebuild them?
When rebooting a server if it ta
On 27 Feb 2018 06:46, "Jan Pekař - Imatic" wrote:
I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue before
Luminous (i did the same tests before).
It is on my test 1 node cluster with lower memory then recommended (so
server is swapping) but it sho
Sorry for being rude Ross,
I follow Ceph since 2014 waiting for iSCSI support in order to use it
with Xen.
When finally it seemds it was implemented the OS requirements are
irrealistic.
Seems a bad prank. 4 year waiting for this... and still not true support
yet.
Il 28/02/2018 14:11, Marc
If you run your container in privileged mode you can mount ceph-fuse inside
of the VMs instead of from the shared resource on the host. I used a
configuration like this to test multi-tenency speed tests of CephFS using
ceph-fuse. The more mount points I used 1 per container, the more bandwidth
I wa
On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini
wrote:
> I was building ceph in order to use with iSCSI.
> But I just see from the docs that need:
>
> CentOS 7.5
> (which is not available yet, it's still at 7.4)
> https://wiki.centos.org/Download
>
> Kernel 4.17
> (which is not available ye
Am 28.02.2018 um 15:18 schrieb David Turner:
> If you run your container in privileged mode you can mount ceph-fuse inside
> of the VMs instead of from the shared resource on the host. I used a
> configuration like this to test multi-tenency speed tests of CephFS using
> ceph-fuse. The more moun
The rbd-nbd device no longer resizes online due to this bug:
https://lkml.org/lkml/2018/2/18/85
My tracker below, but this is not a Ceph issue.
https://tracker.ceph.com/issues/23137#change-108183
Has anyone heard any news and has any ability to inquire of the status
of the fix? I don't see any
Hi,
We recently learned on this list about the "rotational_journal = 1" for
some (all?) NVMe / SSD setups. We also hit this issue (see below). It
would eventually take a week to recover ... This was all "scratch data"
so didn't matter anyway. We recently had to do some reovery /
backfilling on our
Il 28/02/2018 15:19, Jason Dillaman ha scritto:
On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini
wrote:
I was building ceph in order to use with iSCSI.
But I just see from the docs that need:
CentOS 7.5
(which is not available yet, it's still at 7.4)
https://wiki.centos.org/Download
K
On Tue, Feb 27, 2018 at 3:27 PM, Oliver Freyermuth
wrote:
> As you can see:
> - Name collision for admin socket, since the helper is already running.
You can change the admin socket path using the `admin socket` config
variable. Use metavariables [1] to make the path unique.
> - A second helper
Hi,
TL;DR: we see "used" memory grows indefinitely on our OSD servers.
Until the point that either 1) a OSD process gets killed by OOMkiller,
or 2) OSD aborts (proably because malloc cannot provide more RAM). I
suspect a memory leak of the OSDs.
We were running 12.2.2. We are now running 12.2.3.
Hi Stefan,
can you disable compression and check if memory is still leaking.
If it stops then the issue is definitely somewhere along the "compress"
path.
Thanks,
Igor
On 2/28/2018 6:18 PM, Stefan Kooman wrote:
Hi,
TL;DR: we see "used" memory grows indefinitely on our OSD servers.
Until
On Feb 28, 2018 10:06 AM, "Max Cuttins" wrote:
Il 28/02/2018 15:19, Jason Dillaman ha scritto:
> On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini
> wrote:
>
>> I was building ceph in order to use with iSCSI.
>> But I just see from the docs that need:
>>
>> CentOS 7.5
>> (which is not ava
Thank you for your recommendation.
On Mon, Feb 26, 2018 at 5:03 PM, David Turner wrote:
> I would recommend continuing from where you are now and running `ceph osd
> reweight-by-utilization` again. Your weights might be a little more odd,
> but your data distribution should be the same. If you
On Wed, Feb 28, 2018 at 10:06 AM, Max Cuttins wrote:
>
>
> Il 28/02/2018 15:19, Jason Dillaman ha scritto:
>>
>> On Wed, Feb 28, 2018 at 7:53 AM, Massimiliano Cuttini
>> wrote:
>>>
>>> I was building ceph in order to use with iSCSI.
>>> But I just see from the docs that need:
>>>
>>> CentOS 7.5
>
This is the fourth bugfix release of Luminous v12.2.x long term stable
release series. This was primarily intended to fix a few build,
ceph-volume/ceph-disk issues from 12.2.3 and a few RGW issues. We
recommend all the users of 12.2.x series to update. A full changelog is
also published at the off
Max,
I understand your frustration.
However, last time I checked, ceph was open source.
Some of you might not remember, but one major reason why open source is
great is that YOU CAN DO your own modifications.
If you need a change like iSCSI support and it isn't there,
it is probably best, if yo
Hello,
I've created a Ceph cluster with 3 nodes and a FS to serve a webpage. The
webpage speed is good enough (near to NFS speed), and have HA if one FS die.
My problem comes when I deploy a git repository on that FS. The server
makes a lot of IOPS to check the files that have to update and then a
On Wed, Feb 28, 2018 at 9:17 AM, Max Cuttins wrote:
> Sorry for being rude Ross,
>
> I follow Ceph since 2014 waiting for iSCSI support in order to use it with
> Xen.
What OS are you using in Dom0 that you cannot just directly use krbd?
iSCSI is going to add an extra hop so it will never be able
Am 28.02.2018 um 16:09 schrieb Patrick Donnelly:
> On Tue, Feb 27, 2018 at 3:27 PM, Oliver Freyermuth
> wrote:
>> As you can see:
>> - Name collision for admin socket, since the helper is already running.
>
> You can change the admin socket path using the `admin socket` config
> variable. Use met
My thought is that in 4 years you could have migrated to a hypervisor that
will have better performance into ceph than an added iSCSI layer. I won't
deploy VMs for ceph on anything that won't allow librbd to work. Anything
else is added complexity and reduced performance.
On Wed, Feb 28, 2018, 11:
Hi Wido,
Are your mon's using rocksdb or still leveldb?
Are your mon stores trimming back to a small size after HEALTH_OK was restored?
One v12.2.2 cluster here just started showing the "is using a lot of
disk space" warning on one of our mons. In fact all three mons are now
using >16GB. I tried
All,
I have updated my test ceph cluster from Jewer (10.2.10) to Luminous
(12.2.4) using CentOS packages.
I have updated all packages, restarted all services with the proper
order but I get a warning that the Manager Daemon doesn't exist.
Here is the output:
# ceph -s
cluster:
id:
On Wed, Feb 28, 2018 at 3:02 AM Zoran Bošnjak <
zoran.bosn...@sloveniacontrol.si> wrote:
> I am aware of monitor consensus requirement. It is taken care of (there is
> a third room with only monitor node). My problem is about OSD redundancy,
> since I can only use 2 server rooms for OSDs.
>
> I co
On Wed, Feb 28, 2018 at 5:21 PM, Georgios Dimitrakakis
wrote:
> All,
>
> I have updated my test ceph cluster from Jewer (10.2.10) to Luminous
> (12.2.4) using CentOS packages.
>
> I have updated all packages, restarted all services with the proper order
> but I get a warning that the Manager Daemo
Indeed John,
you are right! I have updated "ceph-deploy" (which was installed via
"pip" that's why wasn't updated with the rest ceph packages) but now it
complaints that keys are missing
$ ceph-deploy mgr create controller
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/user/.
On 2018/02/27 4:23 pm, John Spray wrote:
On Tue, Feb 27, 2018 at 6:37 PM, Andre Goree wrote:
Is it still considered best practice to set 'noout' for OSDs that will
be
going under maintenance, e.g., rebooting an OSD ndoe for a kernel
update?
I ask, because I've set this twice now during times
I've looked and haven't found much information besides custom 3rd-party
plugins so I figured I'd ask here:
Is there a way to monitor a clusters 'health' via SNMP?
--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website - http://blog.drenet.net
PGP key - http://www.drenet.net/pubk
OK...now this is getting crazy...
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs:
Where has gone everything??
What's happening here?
G.
Indeed John,
you are right! I have updated "ceph-deploy" (which was installed vi
Does Xen still not support RBD? Ceph has been around for years now!
Met vriendelijke groeten,
--
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
Mark Schouten | Tuxis Internet Engineering
KvK: 61527076 | http://www.tuxis.nl/
T: 0318 200208 | i...@tuxis.nl
Van: Massimiliano Cut
I'm considering doing one-way rbd mirroring to a DR site. The documentation
states that my link to the DR site should have sufficient throughput to
support replication.
Our write activity is bursty. As such, we tend to see moments of high
throughput 4-6gbps followed by long bouts of basically no a
I am still trying to figure what is the problem here...
Initially the cluster was updated ok...
# ceph health detail
HEALTH_WARN noout flag(s) set; all OSDs are running luminous or later
but require_osd_release < luminous; no active mgr
noout flag(s) set
all OSDs are running luminous or later
On Wed, Feb 28, 2018 at 2:42 PM, Brady Deetz wrote:
> I'm considering doing one-way rbd mirroring to a DR site. The documentation
> states that my link to the DR site should have sufficient throughput to
> support replication.
>
> Our write activity is bursty. As such, we tend to see moments of hi
Great. We are read heavy. I assume the journals do not replicate reads. Is
that correct?
On Wed, Feb 28, 2018 at 1:50 PM, Jason Dillaman wrote:
> On Wed, Feb 28, 2018 at 2:42 PM, Brady Deetz wrote:
> > I'm considering doing one-way rbd mirroring to a DR site. The
> documentation
> > states that
On Wed, Feb 28, 2018 at 2:56 PM, Brady Deetz wrote:
> Great. We are read heavy. I assume the journals do not replicate reads. Is
> that correct?
Correct -- only writes (plus discards, snapshots, etc) are replicated.
> On Wed, Feb 28, 2018 at 1:50 PM, Jason Dillaman wrote:
>>
>> On Wed, Feb 28,
Hi,
looks like you haven’t run the ceph-deploy command with the same user name and
may be not the same current working directory. This could explain your problem.
Make sure the other daemons have a mgr cap authorisation. You can find on this
ML details about MGR caps being incorrect for OSDs an
You could probably write an SNMP module for the new ceph-mgr daemon. What
do you want to use to monitor Ceph that requires SNMP?
On Wed, Feb 28, 2018 at 1:13 PM Andre Goree wrote:
> I've looked and haven't found much information besides custom 3rd-party
> plugins so I figured I'd ask here:
>
> I
Could it be a problem that I have changed the hostname after the mon
creation?
What I mean is that
# hostname -s
ovhctrl
# ceph daemon mon.$(hostname -s) quorum_status
admin_socket: exception getting command descriptions: [Errno 2] No such
file or directory
But if I do it as "nefelus-cont
A more common search term for this might be Rack failure domain. The
premise is the same for room as it is for rack, both can hold hosts and be
set as the failure domain. There is a fair bit of discussion on how to
achieve multi-rack/room/datacenter setups. Datacenter setups are more
likely to h
I'm following up from awhile ago. I don't think this is the same bug. The
bug referenced shows "abort: Corruption: block checksum mismatch", and I'm
not seeing that on mine.
Now I've had 8 OSDs down on this one server for a couple of weeks, and I
just tried to start it back up. Here's a link to th
Which version of ceph-deploy are you using?
On Wed, Feb 28, 2018 at 4:37 AM Massimiliano Cuttini
wrote:
> This worked.
>
> However somebody should investigate why default is still jewel on Centos
> 7.4
>
> Il 28/02/2018 00:53, jorpilo ha scritto:
>
> Try using:
> ceph-deploy --release luminous h
Would anyone else appreciate a Google Calendar invitation for the CDMs?
Seems like a natural.
On 02/27/2018 09:37 PM, Leonardo Vaz wrote:
> Hey Cephers,
>
> This is just a friendly reminder that the next Ceph Developer Monthly
> meeting is coming up:
>
> http://wiki.ceph.com/Planning
>
> If yo
On Wed, 28 Feb 2018, Dan Mick wrote:
> Would anyone else appreciate a Google Calendar invitation for the CDMs?
> Seems like a natural.
Funny you should mention it! I was just talking to Leo this morning about
creating a public Ceph Events calendar that has all of the public events
(CDM, tech ta
Indeed that was the problem!
In case anyone else ever comes to the same condition please keep in
mind that no matter what you write at the "ceph-deploy" command it will
use at some point the the output from "hostname -s" and try to connect
to gather data from that monitor.
If you have changed
Didn't check at time.
I deployed everything from VM standalone.
The VM was just build up with fresh new centOS7.4 using minimal
installation ISO1708.
It's a completly new/fresh/empty system.
Then I run:
yum update -y
yum install wget zip unzip vim pciutils -y
yum install epel-release -y
yum up
Hi Max,
I had the same issue (under Ubuntu 16.04) but I have read the ceph-deploy 2.0.0
source code and saw a "—-release" flag for the install subcommand. You can
found the flag with the following command: ceph-deploy install --help
It looks like the culprit part of ceph-deploy can be found aro
On Wed, Feb 28, 2018 at 10:51:29PM +, Sage Weil wrote:
> On Wed, 28 Feb 2018, Dan Mick wrote:
> > Would anyone else appreciate a Google Calendar invitation for the CDMs?
> > Seems like a natural.
>
> Funny you should mention it! I was just talking to Leo this morning about
> creating a publi
Does anybody have some more input?
I keeped the balancer active for 24h now and it is rebalancing 1-3%
every 30 minutes but the distribution is still bad.
It seems to balance from left to right and than back from right to left...
Greets,
Stefan
Am 28.02.2018 um 13:47 schrieb Stefan Priebe - Pro
68 matches
Mail list logo
