Nice to hear this was resolved in the end.
Coming back to the beginning -- is it clear to anyone what was the
root cause and how other users can avoid this from happening? Maybe
some better default configs to warn users earlier about too-large
omaps?
Cheers, Dan
On Thu, Jun 13, 2019 at 7:36 PM H
Hi Maged,
min_alloc_size determines allocation granularity hence if object size
isn't aligned with its value allocation overhead still takes place.
E.g. with min_alloc_size = 16K and object size = 24K total allocation
(i.e. bluestore_allocated) would be 32K.
And yes, this overhead is perman
thank you very much for updating python-rtslib!!
could you maybe also do this for tcmu-runner (version 1.4.1)?
shaman repos are very convenient for installing and updating the ceph
iscsi stack, I would be very happy if I could continue using it
matthias
Am 14.06.19 um 18:08 schrieb Matthias Le
We received the large omap warning before, but for some reasons we could
not react quickly. We accepted the risk of the bucket becoming slow, but
had not thought of further risks ...
On 17.06.19 10:15, Dan van der Ster wrote:
Nice to hear this was resolved in the end.
Coming back to the begin
We have resharded a bucket with 60 million objects from 32 to 64
shards without any problem. (Though there were several slow ops at the
"stalls after counting the objects phase", so I set nodown as a
precaution).
We're now resharding that bucket from 64 to 1024.
In your case I wonder if it was the
Yes, but look into pgs array. It shouldn't be empty.
That should by addressed by this PR: https://tracker.ceph.com/issues/40377
Best Regards,
Rafał Wądołowski
On 16.06.2019 07:06, huang jun wrote:
> osd send osd beacons every 300s, and it's used to let mon know that
> osd is alive,
> for some
Earlier in bluestore's life, we couldn't handle a 4K min_alloc size on
NVMe without incurring pretty significant slowdowns (and also generally
higher amounts of metadata in the DB). Lately I've been seeing some
indications that we've improved the stack to the point where 4K
min_alloc no longer
Our resharding was directly from unsharded to 1024 shards. One could
imagine that an intermediate step would help, but I have no idea.
About bluefs size I would not know of something bad. Well maybe it was a
problem that we had a very small (20 GB) device (SSD LV) for the DB
which means spillo
I'll keep updating this until I find a solution so if anyone faces the same
problem he might have solution.
Atm : I install the new osd node with ceph-deploy and nothing change, node is
still not present in the cluster nor in the crushmap.
I decided to manually add it to the crush map :
ceph osd
I don’t have an answer for you, but it’s going to help others to have shown:
Versions of all nodes involved and multi-master configuration
Confirm forward and reverse DNS and SSH / remote sudo since you are using deploy
Specific steps that did not behave properly
> On Jun 17, 2019, at 6:29 AM, CUZA
There are customers asking for 500 million objects in a single object
storage bucket (i.e. 5000 shards), but also more. But we found some
places that say that there is a limit in the number of shards per
bucket, e.g.
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/obj
Hi Harald ,
We saw in our internal Veeam repo that only 4TB used created more than 10M
objects.
I don't know if Veeam need to list content inside the bucket, that will make
a 500 millions bucket not a good solution at least in our experience with
sharding.
I read someone in the IRC telling that
Hi Harry,
When dynamic resharding was introduced for luminous, this limit on the
number of bucket index shards was increased from 7877 to 65521. However,
you're likely to have problems with bucket listing performance before
you get to 7877 shards, because every listing request has to read from
Hey all,
At cephalocon some of us who work in scientific computing got together for a
BoF and had a good conversation. There was some interest in finding a way to
continue the conversation focused on ceph in scientific computing and htc/hpc
environments. We are considering putting together mont
Hi,
On 17/06/2019 16:00, Harald Staub wrote:
> There are customers asking for 500 million objects in a single object
> storage bucket (i.e. 5000 shards), but also more. But we found some
> places that say that there is a limit in the number of shards per
> bucket, e.g.
Our largest bucket was abou
On Wed, 5 Jun 2019, Sage Weil wrote:
> That brings us to an important decision: what time of year should we
> release? Once we pick the timing, we'll be releasing at that time *every
> year* for each release (barring another schedule shift, which we want to
> avoid), so let's choose carefully!
We have a two-site cluster with OSDs and pools localised in two locations. I
was now looking into setting up a rados gateway with the following properties:
- Pools should be EC pools whenever possible.
- Pools for specific buckets should be localised at OSDs on only one of the two
locations (mea
We observe the following on ceph fs clients with identical ceph fs mounts:
[frans@sophia1 ~]$ ls -l ../neda
ls: cannot access ../neda/NEWA_TEST: Permission denied
total 5
drwxrwxr-x 1 neda neda1 May 17 19:30 ffpy_test
-rw-rw-r-- 1 neda neda 135 May 17 21:06 mount_newa
drwxrwxr-x 1 neda neda
Please ignore the message below, it has nothing to do with ceph.
Sorry for the spam.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: ceph-users on behalf of Frank
Schilder
Sent: 17 June 2019 20:33
To: ceph-use
This was a little long to respond with on Twitter, so I thought I'd share
my thoughts here. I love the idea of a 12 month cadence. I like October
because admins aren't upgrading production within the first few months of a
new release. It gives it plenty of time to be stable for the OS distros as
we
On Mon, Jun 17, 2019 at 5:02 PM Harald Staub wrote:
> There are customers asking for 500 million objects in a single object
> storage bucket (i.e. 5000 shards), but also more. But we found some
> places that say that there is a limit in the number of shards per
> bucket, e.g.
>
We've got a clust
Hi.
I'm running a small single-host Ceph cluster on Proxmox (as my home NAS). I
want to encrypt my OSDs but I don't want the host's SSD to be a single
point of failure. What Ceph config/keyring/secret keys do I need to make
safe [encrypted] copies of to ensure that even if the host running the Cep
Is it possible to add and remove monitors in Mimic, using the new
centralized configuration method?
Regards
R
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hi Everyone,
Recently we moved a bunch of our servers from one rack to another. In
the late stages of this we hit a point when some requests were blocked
due to one pg being in "peered" state.
This was unexpected to us, but on discussion with Wido we understand
why this happened. However it's brou
The Script Reads a File, (Object Listing from the cache pool via the rados
ls -p), and starts flushing, and evicting to the Base Tier.
It respects any blocked requests on the Cluster, and toggles recovery
throttles based on the same (This is the case if the Flush + Evict) is
going in at the Same Ti
Usage is as follows:-
sh flush_recovery.sh filename.txt &
For Logging:-
sh flush_recovery.sh filename.txt >> /var/path/flush.log &
Thanks
Romit
On Tue, Jun 18, 2019 at 7:07 AM Romit Misra
wrote:
> The Script Reads a File, (Object Listing from the cache pool via the rados
> ls -p), and starts f
I am a student new to cephfs. I want see the ldout log in /src/client/Client.cc
(for example, ldout(cct, 20) << " no cap on " << dn->inode->vino() << dendl;).
Can anyone teach me? The /var/log/ceph dir on the client is empty.
___
ceph-users mailing list
you should add this to your ceph.conf
[client]
log file = /var/log/ceph/$name.$pid.log
debug client = 20
?? ?? 于2019年6月18日周二 上午11:18写道：
>
> I am a student new to cephfs. I want see the ldout log in
> /src/client/Client.cc (for example, ldout(cct, 20) << " no cap on " <<
> dn->inode->vino() << d
Can you open a tracker for this Dan and provide scrub logs with
debug_osd=20 and rados list-inconsistent-obj output?
On Mon, Jun 3, 2019 at 10:44 PM Dan van der Ster wrote:
>
> Hi Reed and Brad,
>
> Did you ever learn more about this problem?
> We currently have a few inconsistencies arriving wit
Hi,
What is the easiest and fastest way to upgrade Ceph clients which are
Openstack instances. We have a lot of Openstack Instances and rebooting
them will take a lot of time as we need to communicate with the end users.
Is there any other way like restarting a cinder service to get those
clients
Hi,
I didn't bother to create a twitter account just to be able to
participate in the poll.. so.. please count me in for October.
Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.co
31 matches
Mail list logo
