[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-02 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu) Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906496 Title: mgr can be very slow in a large c

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-03 Thread Ponnuvel Palaniyappan
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906496 Title: mgr can be very slow in a large ceph cluster To manage notifications about this bug go to: https://bugs.launchpad.net

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-02-22 Thread Ponnuvel Palaniyappan
focal packages are installed & verified for ceph 15.2.8. Attached the steps used. ** Attachment added: "sru_focal_ceph15.2.8.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1911900/+attachment/5466037/+files/sru_focal_ceph15.2.8.txt ** Tags removed: verification-needed-focal ** Tags

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-02-22 Thread Ponnuvel Palaniyappan
Ussuri packages are installed and verified (15.2.8-0ubuntu0.20.04.1~cloud0). In tests for focal and ussuri, rados bench was used to drive some I/O and randomly chosen PGs were repeated deeb-scrubbed in a loop (ceph pg deep-scrub ) to keep scrubbing process run at all times. ** Attachment added: "

[Bug 1909162] Re: cluster log slow request spam

2021-02-22 Thread Ponnuvel Palaniyappan
Ussuri packages are installed and verified (15.2.8-0ubuntu0.20.04.1~cloud0). In tests for focal and ussuri, rados bench was used to drive some I/O and randomly chosen PGs were repeated deeb-scrubbed in a loop (ceph pg deep-scrub ) to keep scrubbing process run at all times. ** Attachment added:

[Bug 1909162] Re: cluster log slow request spam

2021-02-22 Thread Ponnuvel Palaniyappan
Please ignore my comment #18. Posted accidentally in the wrong bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1909162 Title: cluster log slow request spam To manage notifications about this bug

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-02-23 Thread Ponnuvel Palaniyappan
Same steps followed for installing & test groovy ceph packages (15.2.8-0ubuntu0.20.10.1) ** Attachment added: "sru_groovy.txt" https://bugs.launchpad.net/cloud-archive/+bug/1911900/+attachment/5466340/+files/sru_groovy.txt ** Tags removed: verification-needed-groovy ** Tags added: verificatio

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-02-26 Thread Ponnuvel Palaniyappan
VERIFICAION DONE And FAILED for Focal. 1. Installed Focal 2. wipefs two devices: one cache and one backing 3. Setup a cache device on a nvme drive (/dev/nvme0n1p1) 4. Setup a backing device a hdd partition (/dev/sdb1) 5. Create a cache setup: sudo make-bcache -B /dev/sdb1 -C /dev/nvme0n1p1 6. Ena

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-02-26 Thread Ponnuvel Palaniyappan
The problem is that IndependentPlugin is not working in Focal's sosreport. The bcache plugin uses IndependentPlugin (as in, it's not tied to any specific Distro). IndependentPlugin was broken at some point since Bionic and has been fixed upstream in https://github.com/sosreport/sos/commit/a36e1b

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-03-01 Thread Ponnuvel Palaniyappan
FYI: upstream sosreport works as expected for bcache on both Bionic and Focal. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1913284 Title: [plugin] [bcache] add a new bcache plugin To manage notif

[Bug 1917288] Re: Missing to package ceph-kvstore-tool, ceph-monstore-tool, ceph-osdomap-tool in bionic-train UCA release

2021-03-01 Thread Ponnuvel Palaniyappan
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1917288 Title: Missing to package ceph-kvstore-tool, ceph-monstore-tool, ceph- osdomap-tool in bionic-train UCA release To manage

[Bug 1914584] Re: radosgw-admin user create error message confusing if user with email already exists

2021-03-01 Thread Ponnuvel Palaniyappan
Attaching the debdiff done for Luminous (based off Matthew Vernon's patch). ** Description changed: - Hi, + [Impact] - The error message if you try and create an S3 user with an email address - that is already associated with another S3 account is very confusing. + When creating a new S3 user

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-03-02 Thread Ponnuvel Palaniyappan
The third option is to cherry-pick the commit from upstream that fixed broken IndependentPlugin. It appears that Ubuntu sosreport doesn't get a "full sync" regularly; rather features/commits cherry-picked on a need basis. So this option might be the one that fits with the current process, iiuc. --

[Bug 1917494] [NEW] ceph-mgr hangs in large clusters

2021-03-02 Thread Ponnuvel Palaniyappan
Public bug reported: This is to track and SRU the upstream bug [0]. In large clusers, there's a GIL deadlock that happens (or observable) when ceph-mgr receive high number of requests from Prometheus module. This bug likely exists at least since the initial release of Nautilus. Upstream fixed th

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-03-03 Thread Ponnuvel Palaniyappan
Thanks, Eric. I'll update #1917651 with the debdiff. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1913284 Title: [plugin] [bcache] add a new bcache plugin To manage notifications about this bug g

[Bug 1917651] Re: IndependentPlugin is not working

2021-03-03 Thread Ponnuvel Palaniyappan
** Attachment added: "focal-debdiff.txt" https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1917651/+attachment/5472231/+files/focal-debdiff.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/

[Bug 1917651] Re: IndependentPlugin is not working

2021-03-03 Thread Ponnuvel Palaniyappan
Attaching debdiff for focal. ** Attachment added: "focal-debdiff.txt" https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1917651/+attachment/5472232/+files/focal-debdiff.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. ht

[Bug 1917651] Re: IndependentPlugin is not working

2021-03-03 Thread Ponnuvel Palaniyappan
Attached debdiff for groovy. ** Attachment added: "groovy-debdiff.txt" https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1917651/+attachment/5472233/+files/groovy-debdiff.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-03-03 Thread Ponnuvel Palaniyappan
Tested using the patch from [0] and it works (= generated sosreport contains the bcache stats). Attached the test sos report here. So once this is re-uploaded, I'll repeat the verification. [0] https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1917651/comments/3 ** Attachment added: "sos

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-05-28 Thread Ponnuvel Palaniyappan
Verified that log compactions occur in bluefs with read/write I/O. Attaching test notes. ** Attachment added: "queens_sru_1914911" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1914911/+attachment/5500895/+files/queens_sru_1914911 ** Tags removed: verification-queens-needed ** Tags adde

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-06-17 Thread Ponnuvel Palaniyappan
** Changed in: cloud-archive/queens Status: Triaged => Won't Fix ** Changed in: cloud-archive/rocky Status: Triaged => Won't Fix ** Changed in: cloud-archive/stein Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, wh

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-27 Thread Ponnuvel Palaniyappan
With SRU verification test, it doesn't quite work as intended. I believe the problem is with the upstream patch [0]. I've asked for clarification/confirmation [1]. I'll then have to mark the verifications failed, I am afraid. This obviously affects 15.2.11 release as well. I'll try to get this fix

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-28 Thread Ponnuvel Palaniyappan
Submitted patch upstream: https://github.com/ceph/ceph/pull/41065 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914584 Title: [SRU] radosgw-admin user create error message confusing if user with

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-28 Thread Ponnuvel Palaniyappan
** Tags removed: verification-needed verification-needed-focal verification-needed-groovy ** Tags added: verification-failed verification-failed-focal verification-failed-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https:/

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-28 Thread Ponnuvel Palaniyappan
Confirmed that the patch needs re-work. So marking all verification failed. I've opened an issue in upstream tracker [0]. Submitted the patch upstream [1]. Once that gets approved, backports to Nautilus, Octopus, and Pacific to follow. I am not sure how 15.2.11 release can be worked around whethe

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-29 Thread Ponnuvel Palaniyappan
** Tags removed: verification-ussuri-needed ** Tags added: verification-ussuri-failed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914584 Title: [SRU] radosgw-admin user create error message confu

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-04-30 Thread Ponnuvel Palaniyappan
This (incorrect) patch is in upstream master and backported to Nautilus, Octopus, and Pacific. This isn't critical and doesn't break functionality - except for an incorrect error message if user attributes modification fails. However, the reported problem still exists. So I've marked all verificati

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-04-12 Thread Ponnuvel Palaniyappan
The regression that this patch fixes wasn't introduced (or backported to) Luminous by upstream. So this doesn't affect Bionic (confirmed by checking Ubuntu's latest Bionic source too). ** Changed in: ceph (Ubuntu Bionic) Status: New => Invalid -- You received this bug notification because

[Bug 1936136] Re: ceph on bcache performance regression

2021-07-15 Thread Ponnuvel Palaniyappan
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1936136 Title: ceph on bcache performance regression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu

[Bug 1914584] Re: [SRU] radosgw-admin user create error message confusing if user with email already exists

2021-07-29 Thread Ponnuvel Palaniyappan
** Changed in: cloud-archive/victoria Status: Triaged => New ** Changed in: cloud-archive/train Status: Triaged => Won't Fix ** Changed in: cloud-archive/ussuri Status: Fix Committed => New ** Changed in: cloud-archive Status: Fix Released => New ** Changed in: ceph

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-09 Thread Ponnuvel Palaniyappan
** Changed in: cloud-archive/ussuri Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages w

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-09 Thread Ponnuvel Palaniyappan
** Attachment added: "out__ussuri_before.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5432708/+files/out__ussuri_before.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-09 Thread Ponnuvel Palaniyappan
SRU tests are oood for ussuri (observed perf numbers are better). Did a short rados benchmark and attached the results (before and after upgrade). The steps taken are: - Upgrade the packages on monitor, and OSDs in order. - Restart monitor, manager and OSDs in order. - Repeat the same benchmark te

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-09 Thread Ponnuvel Palaniyappan
** Attachment added: "out__ussuri_after.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5432709/+files/out__ussuri_after.txt ** Tags removed: verification-ussuri-needed ** Tags added: verification-ussuri-done -- You received this bug notification because you are

[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64

2020-11-10 Thread Ponnuvel Palaniyappan
I've encountered this failure, too: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac /autopkgtest- bionic/bionic/amd64/libv/libvirt/20201109_214647_148b5@/log.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subsc

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-10 Thread Ponnuvel Palaniyappan
The smoke-lxc test failure (reported in comment #37) could be due to https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180. LP#1899180 suggests this could be treated as flaky. If it's easy, coud try re-running the regression tests for 12.2.13-0ubuntu0.18.04.5 to see if it's indeed a Heise

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-11 Thread Ponnuvel Palaniyappan
** Changed in: cloud-archive/stein Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages w

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-16 Thread Ponnuvel Palaniyappan
^^ The above is for Stein verification. ** Tags removed: verification-stein-needed ** Tags added: verification-stein-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-16 Thread Ponnuvel Palaniyappan
Performed same as for Ussuri (comment #34) and attached results here. ** Attachment added: "Stein_verification.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5434835/+files/Stein_verification.txt -- You received this bug notification because you are a member of

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-16 Thread Ponnuvel Palaniyappan
** Changed in: cloud-archive/train Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1894453 Title: Building Ceph packages w

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-16 Thread Ponnuvel Palaniyappan
Test results for Train. ** Attachment added: "train_verification.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5434889/+files/train_verification.txt ** Tags removed: verification-train-needed ** Tags added: verification-train-done -- You received this bug noti

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-17 Thread Ponnuvel Palaniyappan
Re-run of the regression tests is OK for 12.2.13-0ubuntu0.18.04.5 [0]. Test results for 12.2.13-0ubuntu0.18.04.5 is good. Attached before & after test results. [0] https://people.canonical.com/~ubuntu-archive/proposed- migration/bionic/update_excuses.html#ceph ** Attachment added: "bionic_verifi

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-11-17 Thread Ponnuvel Palaniyappan
James Page has done focal-proposed verification already (comment #43) - with that, all tests are done and verified. Please let me know if you have any questions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.ne

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-12-04 Thread Ponnuvel Palaniyappan
Tested queens-proposed following the same steps noted in #34. Like others results previously, this shows better results as well. Attached the test notes. ** Changed in: cloud-archive/queens Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) ** Changed in: ceph (Ubuntu Bio

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-12-04 Thread Ponnuvel Palaniyappan
Tested queens-proposed following the same steps noted in #34. Like others results previously, this shows better results as well. Attached the test notes. ** Attachment added: "queens_out.beore" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5440913/+files/queens_out.be

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-12-04 Thread Ponnuvel Palaniyappan
** Attachment added: "queens_out.after" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5440914/+files/queens_out.after ** Tags removed: verification-queens-needed ** Tags added: verification-queens-done -- You received this bug notification because you are a member o

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-05 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu) Status: New => Incomplete ** Changed in: ceph (Ubuntu) Status: Incomplete => New ** Changed in: ceph (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-07 Thread Ponnuvel Palaniyappan
Attaching diff for Bionic (Ceph 12.2.13) - same as before but easier to read. ** Attachment added: "bug1906496.patch-bionic-12.2.13" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5441740/+files/bug1906496.patch-bionic-12.2.13 -- You received this bug notification be

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-07 Thread Ponnuvel Palaniyappan
Attaching debdiff for Bionic (Ceph 12.2.13). ** Attachment added: "Bionic-Ceph-12.2.13-debdiff" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5441739/+files/debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-08 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu) Status: Confirmed => Won't Fix ** Changed in: ceph (Ubuntu) Status: Won't Fix => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906496 Title:

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-09 Thread Ponnuvel Palaniyappan
debdiff for 13.2.9 ** Attachment added: "debdiff-ceph-13.2.9" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5442308/+files/debdiff-ceph-13.2.9 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.

[Bug 1906496] Re: mgr can be very slow in a large ceph cluster

2020-12-09 Thread Ponnuvel Palaniyappan
Patch file for 13.2.9 ** Attachment added: "bug1906496.patch-13.2.9" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5442309/+files/bug1906496.patch-13.2.9 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. ht

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2020-12-10 Thread Ponnuvel Palaniyappan
** Summary changed: - mgr can be very slow in a large ceph cluster + [SRU] mgr can be very slow in a large ceph cluster ** Description changed: - upstream implemented a new feature [1] that will check/report those long - network ping times between osds, but it introduced an issue that ceph- - mg

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2020-12-13 Thread Ponnuvel Palaniyappan
@Corey, yes, I am happy to do the SRU verification when the packages are available. I've updated the [Test case] section to note a simplified, functional test. ** Description changed: - [Impact] + [Impact] Ceph upstream implemented a new feature [1] that will check/report those long network p

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2020-12-14 Thread Ponnuvel Palaniyappan
** Description changed: [Impact] - Ceph upstream implemented a new feature [1] that will check/report those long network ping times between osds, but it introduced an issue that ceph-mgr might be very slow because it needs to dump all the new osd network ping stats [2] for some tasks, this ca

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-10-07 Thread Ponnuvel Palaniyappan
@Corey, Had a discussion with Daniel Hill in SEG, and we think the risk is low. So, yes, we'd like to backport this to all releases. I am happy to do SRU (and/or tests) once this is ready. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubunt

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-07 Thread Ponnuvel Palaniyappan
SRU tests: Deployed a cluster with many OSDs with the new packages; I/O was driven from a VM (both reads & writes). Enabled a number mgr modules, too. And under load, the cluster was functioning and mgr was still responding. Attaching some relevant info on the tests here. -- You received this

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-07 Thread Ponnuvel Palaniyappan
Stein UCA 13.2.9-0ubuntu0.19.04.1~cloud3 ** Attachment added: "stein.sru" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5450096/+files/stein.sru -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bug

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-07 Thread Ponnuvel Palaniyappan
Bionic 12.2.13-0ubuntu0.18.04.6 ** Attachment added: "bionic.sru" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1906496/+attachment/5450097/+files/bionic.sru ** Tags removed: verification-needed-bionic verification-stein-needed ** Tags added: verification-needed-done verification-stein-

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-01-07 Thread Ponnuvel Palaniyappan
bionic debdiff ** Changed in: sosreport (Ubuntu Bionic) Status: New => In Progress ** Changed in: sosreport (Ubuntu Focal) Status: New => In Progress ** Changed in: sosreport (Ubuntu Groovy) Status: New => Incomplete ** Changed in: sosreport (Ubuntu Groovy) Status:

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-01-07 Thread Ponnuvel Palaniyappan
focal debdiff ** Patch added: "focal.debdiff" https://bugs.launchpad.net/ubuntu/focal/+source/sosreport/+bug/1910264/+attachment/5450256/+files/focal.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.n

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-01-07 Thread Ponnuvel Palaniyappan
groovy debdiff ** Patch added: "groovy.debdiff" https://bugs.launchpad.net/ubuntu/focal/+source/sosreport/+bug/1910264/+attachment/5450257/+files/groovy.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchp

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-01-07 Thread Ponnuvel Palaniyappan
Hi Eric, Attached debdiff's for bionic, focal, and groovy. Please let me know if they're OK. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910264 Title: [plugin][ceph] include time-sync-status for

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-07 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu Bionic) Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) ** Changed in: cloud-archive/stein Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) ** Changed in: cloud-archive/queens Assignee: (unassigned) => Ponnuvel Pal

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-07 Thread Ponnuvel Palaniyappan
** Description changed: [Impact] Ceph upstream implemented a new feature [1] that will check/report those long network ping times between OSDs, but it introduced an issue that ceph-mgr might be very slow because it needs to dump all the new OSD network ping stats [2] for some tasks, this ca

[Bug 1906496] Re: [SRU] mgr can be very slow in a large ceph cluster

2021-01-08 Thread Ponnuvel Palaniyappan
Queens verification of 12.2.13-0ubuntu0.18.04.6~cloud0 Ran same tests on Queens - ceph mgr was functional and responsive under cluster load. ** Attachment added: "queens.sru" https://bugs.launchpad.net/ubuntu/bionic/+source/ceph/+bug/1906496/+attachment/5450803/+files/queens.sru ** Tags rem

[Bug 1917651] Re: IndependentPlugin is not working

2021-03-04 Thread Ponnuvel Palaniyappan
Attached debdiff for hirsute. ** Description changed: [IMPACT] Regression found while doing the verification testing of (LP: #1913284) As described by my colleague Ponnuvel: " The problem is that IndependentPlugin is not working in Focal's sosreport. The bcache plugin uses Indepe

[Bug 1917651] Re: IndependentPlugin is not working

2021-03-04 Thread Ponnuvel Palaniyappan
Done both - thanks, Eric! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1917651 Title: IndependentPlugin is not working To manage notifications about this bug go to: https://bugs.launchpad.net/ubun

[Bug 1915705] Re: Ceph block device permission gets wrongly reverted by udev

2021-03-09 Thread Ponnuvel Palaniyappan
James Page suggested this might need to be done in the packaging as well. Once the fix is finalized in charm-ceph-osd, I'll update the udev rules in debian/udev/95-ceph-osd-lvm.rules. ** Also affects: ceph (Ubuntu) Importance: Undecided Status: New -- You received this bug notificatio

[Bug 1852441] Re: In bionic, one of the ceph packages installed causes chrony to auto-install even on lxd

2021-03-11 Thread Ponnuvel Palaniyappan
In case of ceph-mon, now it goes into error state because chrony is not there. So ntp charm should update the status before it checks if chrony is installed." ** Also affects: ntp-charm Importance: Undecided Status: New ** Changed in: ntp-charm Assignee: (unassigned) => Ponn

[Bug 1917494] Re: ceph-mgr hangs in large clusters

2021-03-16 Thread Ponnuvel Palaniyappan
igh ** Changed in: ceph (Ubuntu) Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1917494 Title: ceph-mgr hangs in

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-03-16 Thread Ponnuvel Palaniyappan
** Description changed: [Impact] For a certain type of workload, the bluefs might never compact the log file, which would cause the bluefs log file slowly grows to a huge size (some bigger than 1TB for a 1.5T device). There are more details in the bluefs perf counters when this iss

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-03-16 Thread Ponnuvel Palaniyappan
Attached debdiff for bionic (fixed the previous patch which had additional unnecessary changes). ** Attachment added: "bionic.debdiff.fixed" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1914911/+attachment/5477289/+files/bionic.debdiff.fixed -- You received this bug notification becau

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-21 Thread Ponnuvel Palaniyappan
An update: I have built rocksdb with & without RelWithDebInfo. Some differences are: - the exact set of additional flags used with RelWithDebInfo are: -O2 -NDEBUG -fno-omit-frame-pointer -momit-leaf-frame-pointer. - the size of rocksdb shared library (librocksdb.so): default-build = 156M, with

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-24 Thread Ponnuvel Palaniyappan
@Corey > The individual building of rocksdb files doesn't give off much information in the package builds to see compiler flags set. VERBOSE flag would show full flags - however, that'd need changing rules & rebuilding again. That we can RelWithDebInfo is sufficient to confirm that it does pass t

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-28 Thread Ponnuvel Palaniyappan
Hi @Corey, (Ignore my comments as I've managed to setup a cluster without needing ceph-deploy). I have installed the packages and setup a cluster. The cluster was functional and found no issues with the Groovy packages in my tests. I have done some of the Ceph's benchmarks [0] using the Ceph clu

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-28 Thread Ponnuvel Palaniyappan
** Attachment added: "Ceph tests on Octopus - rocksdb optimized.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5415022/+files/Ceph%20tests%20on%20Octopus%20-%20rocksdb%20optimized.txt -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-28 Thread Ponnuvel Palaniyappan
** Attachment added: "Ceph tests on Octopus - rocksdb UNoptimized.txt" https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1894453/+attachment/5415021/+files/Ceph%20tests%20on%20Octopus%20-%20rocksdb%20UNoptimized.txt -- You received this bug notification because you are a member of Ubuntu Bu

[Bug 1894453] Re: Building Ceph packages with RelWithDebInfo

2020-09-29 Thread Ponnuvel Palaniyappan
@Tyler, Indeed it does. My cluster tests were done in a "cloud-on-cloud" environment, so I didn't want to stress too much on the actual numbers (whereas rocksdb tests were done locally in a tightly controlled environment). As a whole, we can definitely say that RelWithDebInfo is the way to go for r

[Bug 1914584] Re: radosgw-admin user create error message confusing if user with email already exists

2021-02-05 Thread Ponnuvel Palaniyappan
** Also affects: ceph (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Bionic) Importanc

[Bug 1914584] Re: radosgw-admin user create error message confusing if user with email already exists

2021-02-05 Thread Ponnuvel Palaniyappan
** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Also affects: cloud-archive/stein Importance: Undecided Status: New ** Also affects: cloud-archive/queens Importance: Undecided Status: New ** Also affects: cloud-archive/train Importance:

[Bug 1913284] Re: [plugin] [bcache] add a new bcache plugin

2021-02-06 Thread Ponnuvel Palaniyappan
Hi @Eric, Updated the description with SRU. Please let me know if anything needs change/unclear. Thanks. ** Description changed: - A new plugin for bcache stats are proposed upstream [0] and there's a PR - currently under review [1]. + [Impact] + + Gather bcache stats as part of sos report. +

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-08 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu) Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: bluefs doesn't compact log file T

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-10 Thread Ponnuvel Palaniyappan
** Also affects: ceph (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: bluefs doesn't compact log file To manage notification

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-10 Thread Ponnuvel Palaniyappan
** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/queens Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bu

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-10 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu Bionic) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: bluefs doesn't compact log file To manage notifications about this

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-10 Thread Ponnuvel Palaniyappan
** Changed in: ceph (Ubuntu Bionic) Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: bluefs doesn't compact log

[Bug 1914911] Re: bluefs doesn't compact log file

2021-02-11 Thread Ponnuvel Palaniyappan
Attached debdiff for Bionic. ** Patch added: "bionic.debdiff" https://bugs.launchpad.net/cloud-archive/+bug/1914911/+attachment/5462589/+files/bionic.debdiff ** Tags added: sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to U

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-02-11 Thread Ponnuvel Palaniyappan
** Summary changed: - bluefs doesn't compact log file + [SRU] bluefs doesn't compact log file -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: [SRU] bluefs doesn't compact log file To

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-02-11 Thread Ponnuvel Palaniyappan
** Description changed: - For a certain type of workload, the bluefs might never compact the log file, - which would cause the bluefs log file slowly grows to a huge size + [Impact] + + For a certain type of workload, the bluefs might never compact the log + file, which would cause the bluefs l

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-02-11 Thread Ponnuvel Palaniyappan
Attaching a gitdiff based off branch: applied/12.2.13-0ubuntu0.18.04.6 ** Attachment added: "ceph12.2.13-lp1914911.gitdiff" https://bugs.launchpad.net/cloud-archive/+bug/1914911/+attachment/5462713/+files/ceph12.2.13-lp1914911.gitdiff -- You received this bug notification because you are a

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-02-19 Thread Ponnuvel Palaniyappan
Verified that time-sync-status is generated. Attached file shows sosreport and ceph time-sync-status output from a test run. ** Attachment added: "time-sync-verification-focal.txt" https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1910264/+attachment/5465390/+files/time-sync-verificati

[Bug 1910264] Re: [plugin][ceph] include time-sync-status for ceph mon

2021-02-19 Thread Ponnuvel Palaniyappan
Verified that time-sync-status is generated. Attached file shows sosreport and ceph time-sync-status output from a test run on Groovy. ** Attachment added: "time-sync-verification-groovy.txt" https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1910264/+attachment/5465394/+files/time-syn

[Bug 1914911] Re: [SRU] bluefs doesn't compact log file

2021-02-21 Thread Ponnuvel Palaniyappan
** Tags added: seg -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914911 Title: [SRU] bluefs doesn't compact log file To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-

[Bug 1840043] Re: bcache: Performance degradation when querying priority_stats

2021-01-25 Thread Ponnuvel Palaniyappan
A question: Does stat(2) calls (lstat, fstat and the like) on priority_stats cause the same issue? Context: https://github.com/sosreport/sos/pull/2384#discussion_r563975063 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-01-25 Thread Ponnuvel Palaniyappan
** Also affects: ceph (Ubuntu) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Hirsute) Importance: Unde

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-01-25 Thread Ponnuvel Palaniyappan
** No longer affects: ceph (Ubuntu Hirsute) ** No longer affects: ceph (Ubuntu Groovy) ** No longer affects: ceph (Ubuntu Focal) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1911900 Title: [SRU]

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-01-25 Thread Ponnuvel Palaniyappan
** Also affects: cloud-archive/ussuri Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1911900 Title: [SRU] Active scrub blocks upmap balancer To manage not

[Bug 1913284] [NEW] [plugin] [bcache] add a new bcache plugin

2021-01-26 Thread Ponnuvel Palaniyappan
Public bug reported: A new plugin for bcache stats are proposed upstream [0] and there's a PR currently under review [1]. [0] https://github.com/sosreport/sos/issues/2378 [1] https://github.com/sosreport/sos/pull/2384 ** Affects: sosreport (Ubuntu) Importance: Undecided Status: Ne

[Bug 1911900] Re: [SRU] Active scrub blocks upmap balancer

2021-01-26 Thread Ponnuvel Palaniyappan
** Also affects: ceph (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: ceph (Ubuntu Bionic) Importanc

  1   2   3   >