[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
[ Impact ]
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core) as a
`ValueError` is raised when attempting to parse the output.
[ Test Plan ]
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) check the logs:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`.
- * In a fresh instance, install an LSB module, and the proposed
- Landscape Client:
+ * In a fresh instance, install an LSB module and the proposed Landscape
+ Client:
- * sudo apt install -y lsb-core landscape-client # should pull new
- version
+ * sudo apt install -y lsb-core
- * Now register the container with Landscape Server, wait for package-
+ * Now register the client with Landscape Server, wait for package-
reporter to run again and verify no errors are raised.
* Additionally, make sure that package-reporter can report package
information to the server properly.
[ Where problems could occur ]
* Because the fix also changed the approach to read the os-release file
instead of the lsb_release command output, this could introduce
unforeseen behavior with code that was written with the original
implementation in mind.
* However, without these fixes, Landscape Client cannot report package
data with lsb modules installed so it is hard to imagine how this would
be worse.
[ Other Info ]
* GH PR for the fix: https://github.com/canonical/landscape-
client/pull/170
Original bug report below:
Package reporter from landscape client is failing to report to server
because seemingly of a lsb parse issue.
/var/log/landscape/package-reporter.log
2023-08-10 13:42:00,414 ERROR[MainThread] None
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653,
in _runCallbacks
current.result = callback(current.result, *args, **kw)
File
"/usr/lib/python3/dist-packages/landscape/client/package/taskhandler.py", line
217, in got_server_uuid
lsb_release_info = parse_lsb_release(self.lsb_release_filename)
File "/usr/lib/python3/dist-packages/landscape/lib/lsb_release.py", line
31, in parse_lsb_release
dist, desc, release, code_name, _ = lsb_info.decode().split("\n")
ValueError: too many values to unpack (expected 5)
Multiple LSB Packages are installed
/usr/bin/lsb_release
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
lsb_release -a
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description:Ubuntu 18.04.3 LTS
Release:18.04
Codename: bionic
** Description changed:
[ Impact ]
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core) as a
`ValueError` is raised when attempting to parse the output.
[ Test Plan ]
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) check the logs:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`.
- * In a fresh instance, install an LSB module and the proposed Landscape
- Client:
-
- * sudo apt install -y lsb-core
+ * In a fresh instance, install an LSB module (ex. `lsb-core`) and the
+ proposed version of Landscape Client.
* Now register the client with Landscape Server, wait for package-
reporter to run again and verify no errors are raised.
* Additionally, make sure that package-reporter can report package
information to the server properly.
[ Where problems could occur ]
* Because the fix also changed the approach to read the os-re
[Bug 2103463] Re: [SRU] Repository profiles only act on .list files and do not affect .source files Edit
** Also affects: landscape-client (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2103463 Title: [SRU] Repository profiles only act on .list files and do not affect .source files Edit To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2103463/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage of landscape-package-reporter
** Description changed:
[ Impact ]
* The current implementation of `landscape-package-reporter` will use a
lot more CPU% than needed by creating `Origin` objects from the Python
`apt` package when computing changes in packages. This has been a
complaint from users as Landscape Client can use 100% of the CPU when
package-reporter is computing the changes to report back to Landscape
Server.
[ Test Plan ]
* Install `python3-psutil` and the proposed version of Landscape
Client.
* Replace this line in `/usr/lib/python3/dist-
packages/landscape/client/package/reporter.py`...:
* def _compute_packages_changes(self): # noqa: max-complexity: 13
* ...with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
* Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
* Make sure that package-reporter can report package information to the
server properly:
* Install a package on the client machine manually and ensure it
shows up under the instance packages in the Landscape Server UI.
* Search for the package in Landscape Server and mark it for
removal. Wait for the activity to run and ensure the package was
removed.
* Search for a package that should be available to install for the
instance (ex. devscripts) in the UI and install it to the client
instance via Landscape Server.
- * Update all possible packages on the client instance using the old
+ * Update all possible packages on the client instance using the old
Landscape Server UI ("Advanced options" - "You can request that all
possible upgrades be applied to the selected computers."). You can also
manually upgrade all packages that have an update.
* In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
+
+ * Repeat the steps under "Make sure that package-reporter can report
+ package information to the server properly" for the unmodified client.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the results from the current version (control) of
Landscape Client on Focal: https://pastebin.com/y
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage of landscape-package-reporter
Hi Andreas, thanks for your feedback. I've updated the testing instructions, but let me know if you'd like further improvements. I think it would be a good idea to also apply the fix to Noble, Oracular, and Plucky. I can create new MPs for each now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble to reduce CPU usage of landscape-package-reporter
** Summary changed: - Update Focal, Jammy to reduce CPU usage of landscape-package-reporter + Update Focal, Jammy, Noble to reduce CPU usage of landscape-package-reporter ** Merge proposal linked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/483650 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble to reduce CPU usage of landscape-package- reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble to reduce CPU usage of landscape-package-reporter
** Merge proposal linked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/483649 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble to reduce CPU usage of landscape-package- reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter
** Changed in: landscape-client (Ubuntu) Status: Incomplete => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular to reduce CPU usage of landscape-package-reporter
** Summary changed: - Update Focal, Jammy, Noble to reduce CPU usage of landscape-package-reporter + Update Focal, Jammy, Noble, Oracular to reduce CPU usage of landscape-package-reporter -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular to reduce CPU usage of landscape- package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter
I've created the MPs for all the series :) However as a note, the unit tests for Landscape Client need to be fixed by https://code.launchpad.net/~mitchburton/ubuntu/+source/landscape- client/+git/landscape-client/+merge/483505 so the Plucky MP for this fix must wait for that to be merged. ** Summary changed: - Update Focal, Jammy, Noble, Oracular to reduce CPU usage of landscape-package-reporter + Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter ** Merge proposal linked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/483695 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter
** Description changed:
[ Impact ]
* The current implementation of `landscape-package-reporter` will use a
lot more CPU% than needed by creating `Origin` objects from the Python
`apt` package when computing changes in packages. This has been a
complaint from users as Landscape Client can use 100% of the CPU when
package-reporter is computing the changes to report back to Landscape
Server.
[ Test Plan ]
* Install `python3-psutil` and the proposed version of Landscape
Client.
* Replace this line in `/usr/lib/python3/dist-
packages/landscape/client/package/reporter.py`...:
* def _compute_packages_changes(self): # noqa: max-complexity: 13
* ...with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
* Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
* Make sure that package-reporter can report package information to the
server properly:
* Install a package on the client machine manually and ensure it
shows up under the instance packages in the Landscape Server UI.
* Search for the package in Landscape Server and mark it for
removal. Wait for the activity to run and ensure the package was
removed.
* Search for a package that should be available to install for the
- instance (ex. devscripts) in the UI and install it to the client
- instance via Landscape Server.
+ instance (ex. hello) in the UI and install it to the client instance via
+ Landscape Server.
* Update all possible packages on the client instance using the old
Landscape Server UI ("Advanced options" - "You can request that all
possible upgrades be applied to the selected computers."). You can also
manually upgrade all packages that have an update.
* In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
- * Repeat the steps under "Make sure that package-reporter can report
+ * Repeat the steps under "Make sure that package-reporter can report
package information to the server properly" for the unmodified client.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the resul
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
** Summary changed: - Update Focal, Jammy, Noble, Oracular, Plucky to reduce CPU usage of landscape-package-reporter + Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
** Merge proposal linked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/485551 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
** Merge proposal linked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/485553 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
- Package reporter from landscape client is failing to report to server
- because seemingly of a lsb parse issue.
+ [ Impact ]
+
+ * Error when parsing lsb_release, landscape-package-reporter unable to
+ report package list to server (LP: #2031036):
+
+ * The version of Landscape Client in the Ubuntu main repository for
+ Focal and Jammy fails to report package information Landscape Server
+ when the client machine has lsb modules installed (ex. lsb-core).
+
+ [ Test Plan ]
+
+ * Perform the following for both Focal and Jammy.
+
+ * `lxc launch ubuntu:focal # or ubuntu:jammy`
+
+ * `lxc exec -- su -l ubuntu # shell into
+ container`
+
+ * `sudo apt update && sudo apt install -y landscape-client`
+
+ * `lsb_release -a # Check no lsb modules`
+
+ * `sudo apt-get install -y lsb-core # install lsb module`
+
+ * `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
+ system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
+ $ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
+ $HOSTNAME`
+
+ * Accept the machine's registration if needed and wait for package
+ reporting to (try and) run:
+
+ * tail /var/log/landscape/package-reporter.log
+
+ * You should see `ValueError: too many values to unpack`. Now remove
+ this container from Landscape and delete it.
+
+ * Create a fresh container of the series and add my ppa:
+
+ * sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
+ sudo apt update
+
+ * Now, install an LSB module, `psutil`, and the proposed Landscape
+ Client:
+
+ * sudo apt install -y lsb-core python3-psutil landscape-client #
+ should pull new version
+
+ * Replace line 601 (602 on Jammy, 596 if not proposed version) of
+ `/usr/lib/python3/dist-packages/landscape/client/package/reporter.py`
+ with the following to apply the benchmarking to the function which
+ computes changes in packages to report back to Landscape Server:
+
+ def _compute_packages_changes(self): # noqa: max-complexity: 13
+ import cProfile
+ import pstats
+ from datetime import datetime
+ import psutil
+
+ profile = cProfile.Profile()
+ process = psutil.Process()
+ start_cpu_times = process.cpu_times()
+ profile.enable()
+
+ result = self.compute_packages_change_inner()
+
+ end_cpu_times = process.cpu_times()
+
+ profile.disable()
+
+ user_time = end_cpu_times.user - start_cpu_times.user
+ system_time = end_cpu_times.system - start_cpu_times.system
+ total_cpu_time = user_time + system_time
+
+ output_path = "/var/lib/landscape/client/result.txt"
+ with open(output_path, "a") as fp:
+ now = datetime.now()
+ fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
+ stats = pstats.Stats(profile, stream=fp)
+ stats.strip_dirs().sort_stats("cumulative").print_stats(10)
+ fp.write(f"CPU Time: {total_cpu_time}s\n")
+ return result
+
+ def compute_packages_change_inner(self):
+
+ * Now register the container with Landscape Server, wait for package-
+ reporter to run again and verify no errors are raised:
+
+ * sudo landscape-config --silent
+ --url="$LANDSCAPE_ROOT_URL/message-system" --ping-
+ url="$LANDSCAPE_ROOT_URL/ping" --account-name $ACCOUNT_NAME
+ --registration-key $REGISTRATION_KEY --computer-title $HOSTNAME
+
+ * sudo watch tail -n 30 /var/log/landscape/package-reporter.log
+
+ * Wait for package reporter to run and check the benchmarking results
+ for the proposed version:
+
+ * less /var/lib/landscape/client/result.txt
+
+ * Additionally, make sure that package-reporter can report package
+ information to the server properly.
+
+ * Create a new, fresh container of the same series, install the current
+ version of Landscape Client from the main Ubuntu repo for that series
+ and add the benchmarking like before. Do not install lsb-core as it will
+ crash package-reporter before the benchmark runs. Make sure the
+ benchmarking is in place before continuing.
+
+ * sudo apt update && sudo apt install -y landscape-client
+ python3-psutil
+
+ * Connect this container to the same server and wait for package
+ reporter to run, and compare the benchmarking results with the proposed
+ version:
+
+ * less /var/lib/landscape/client/result.txt
+
+ * The cpu times reported by the proposed version should be
+ significantly less than those of the current version. With these
+ changes, this benchmark ran about 11x faster on Jammy and 15x faster on
+ Focal in my testing. See my full observations in the Other Info section.
+
+ [ Where problems could occur ]
+
+ * Because the fix also changed the approach to read the os-release file
+ instead of the lsb_release command output, this could introduce
+ unforeseen behavior with code that was wri
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage for package reporter
** Summary changed: - Update Focal, Jammy to reduce CPU usage for package reporter and fix lsb output parsing to recognize modules + Update Focal, Jammy to reduce CPU usage for package reporter -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy to reduce CPU usage for package reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core).
[ Test Plan ]
* `lxc launch ubuntu:focal # or ubuntu:jammy`
* `lxc exec -- su -l ubuntu # shell into
container`
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) run:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`.
* Reset the process (ex. delete container, remove from Landscape)
* In a fresh instance, install an LSB module, and the proposed
Landscape Client. You can add my ppa to get the proposed version:
* sudo apt install -y lsb-core landscape-client # should pull new
version
* Now register the container with Landscape Server, wait for package-
- reporter to run again and verify no errors are raised:
-
- * sudo landscape-config --silent
- --url="$LANDSCAPE_ROOT_URL/message-system" --ping-
- url="$LANDSCAPE_ROOT_URL/ping" --account-name $ACCOUNT_NAME
- --registration-key $REGISTRATION_KEY --computer-title $HOSTNAME
-
- * sudo watch tail -n 30 /var/log/landscape/package-reporter.log
+ reporter to run again and verify no errors are raised.
* Additionally, make sure that package-reporter can report package
information to the server properly.
[ Where problems could occur ]
* Because the fix also changed the approach to read the os-release file
instead of the lsb_release command output, this could introduce
unforeseen behavior with code that was written with the original
implementation in mind.
* However, without these fixes, Landscape Client cannot report package
data with lsb modules installed so it is hard to imagine how this would
be worse.
[ Other Info ]
* GH PR for the fix: https://github.com/canonical/landscape-
client/pull/170
Original bug report below:
Package reporter from landscape client is failing to report to server
because seemingly of a lsb parse issue.
/var/log/landscape/package-reporter.log
2023-08-10 13:42:00,414 ERROR[MainThread] None
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653,
in _runCallbacks
current.result = callback(current.result, *args, **kw)
File
"/usr/lib/python3/dist-packages/landscape/client/package/taskhandler.py", line
217, in got_server_uuid
lsb_release_info = parse_lsb_release(self.lsb_release_filename)
File "/usr/lib/python3/dist-packages/landscape/lib/lsb_release.py", line
31, in parse_lsb_release
dist, desc, release, code_name, _ = lsb_info.decode().split("\n")
ValueError: too many values to unpack (expected 5)
Multiple LSB Packages are installed
/usr/bin/lsb_release
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
lsb_release -a
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description:Ubuntu 18.04.3 LTS
Release:18.04
Codename: bionic
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2031036
Title:
Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server
To manage notifications about this bug go to:
https://bugs.launchpad.net/landscape-client/+bug/2031036/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage of landscape-package-reporter
** Description changed:
[ Impact ]
* The current implementation of `landscape-package-reporter` will use a
lot more CPU% than needed by creating `Origin` objects from the Python
`apt` package when computing changes in packages. This has been a
complaint from users as Landscape Client can use 100% of the CPU when
package-reporter is computing the changes to report back to Landscape
Server.
[ Test Plan ]
- * Install `psutil` and the proposed Landscape Client:
-
- * sudo apt install -y python3-psutil landscape-client # should pull
- new version
+ * Install `python3-psutil` and the proposed Landscape Client.
* Replace this line in `/usr/lib/python3/dist-
packages/landscape/client/package/reporter.py`...:
* def _compute_packages_changes(self): # noqa: max-complexity: 13
* ...with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
* Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
* Additionally, make sure that package-reporter can report package
information to the server properly.
* In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the results from the current version (control) of
Landscape Client on Focal: https://pastebin.com/y04PE9rw
* Raw paste for the results from the modified (proposed) of Landscape
Client on Focal: https://pastebin.com/1cbHVGQR
* Raw paste for the results from the current version (control) of
Landscape Client on Jammy: https://pastebin.com/X3d1M8BV
* Raw paste for the results from the modified (proposed) of Landscape
Client on Jammy: https://pastebin.com/TtXpqJDE
* Original LP question/discussion for "High CPU usage of
/usr/bin/landscape-package-reporter":
https://answers.launchpad.net/landscape-client/+question/403745
* GH PR for the improvement to the performance of package-reporter:
https://github.com/canonical/landscape-client/pull/303
** Description changed:
[ Impact ]
* The current implementation of `landscape-package-reporter` will use a
lot more
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage for package reporter
** Description changed:
[ Impact ]
-
- * Error when parsing lsb_release, landscape-package-reporter unable to
- report package list to server (LP: #2031036):
-
- * The version of Landscape Client in the Ubuntu main repository for
- Focal and Jammy fails to report package information Landscape Server
- when the client machine has lsb modules installed (ex. lsb-core).
* High CPU usage of package-reporter:
* package-reporter will use a lot more CPU% than needed by creating
`Origin` objects from the Python `apt` package. This has been a complain
for customers as Landscape Client can use 100% of the CPU when package-
reporter is computing package changes to report back to Landscape
Server.
[ Test Plan ]
- * Perform the following for both Focal and Jammy.
-
- * `lxc launch ubuntu:focal # or ubuntu:jammy`
-
- * `lxc exec -- su -l ubuntu # shell into
- container`
-
- * `sudo apt update && sudo apt install -y landscape-client`
-
- * `lsb_release -a # Check no lsb modules`
-
- * `sudo apt-get install -y lsb-core # install lsb module`
-
- * `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
- system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
- $ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
- $HOSTNAME`
-
- * Accept the machine's registration if needed and wait for package
- reporting to (try and) run:
-
- * tail /var/log/landscape/package-reporter.log
-
- * You should see `ValueError: too many values to unpack`. Now remove
- this container from Landscape and delete it.
-
- * Create a fresh container of the series and add my ppa:
+ * You can add my ppa to get the proposed version:
* sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
sudo apt update
- * Now, install an LSB module, `psutil`, and the proposed Landscape
- Client:
+ * Install the proposed Landscape Client:
- * sudo apt install -y lsb-core python3-psutil landscape-client #
- should pull new version
+ * sudo apt install -y python3-psutil landscape-client # should pull
+ new version
* Replace line 601 (602 on Jammy, 596 if not proposed version) of
`/usr/lib/python3/dist-packages/landscape/client/package/reporter.py`
with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
- * Now register the container with Landscape Server, wait for package-
- reporter to run again and verify no errors are raised:
-
- * sudo landscape-config --silent
- --url="$LANDSCAPE_ROOT_URL/message-system" --ping-
- url="$LANDSCAPE_ROOT_URL/ping" --account-name $ACCOUNT_NAME
- --registration-key $REGISTRATION_KEY --computer-title $HOSTNAME
-
- * sudo watch tail -n 30 /var/log/landscape/package-reporter.log
-
- * Wait for package reporter to run and check the benchmarking results
- for the proposed version:
+ * Now register the container with Landscape Server and wait for package
+ reporter to run and check the benchmarking results for the proposed
+ version:
* less /var/lib/landscape/client/result.txt
* Additionally, make sure that package-reporter can report package
information to the server properly.
- * Create a new, fresh container of the same series, install the current
- version of Landscape Client from the main Ubuntu repo for that series
- and add the benchmarking like before. Do not install lsb-core as it will
- crash package-reporter before the benchmark runs. Make sure the
- benchmarking is in place before continuing.
+ * In a fresh instance, install the current version of Landscape Client
+ from the main Ubuntu repo for that series and add the benchmarking like
+ before.
* sudo apt update && sudo apt install -y landscape-client
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core).
[ Test Plan ]
-
- * Perform the following for both Focal and Jammy.
* `lxc launch ubuntu:focal # or ubuntu:jammy`
* `lxc exec -- su -l ubuntu # shell into
container`
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) run:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`. Now remove
this container from Landscape and delete it.
* Create a fresh container of the series and add my ppa:
* sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
sudo apt update
- * Now, install an LSB module, `psutil`, and the proposed Landscape
- Client:
+ * Now, install an LSB module, and the proposed Landscape Client:
- * sudo apt install -y lsb-core python3-psutil landscape-client #
- should pull new version
-
- * Replace line 601 (602 on Jammy, 596 if not proposed version) of
- `/usr/lib/python3/dist-packages/landscape/client/package/reporter.py`
- with the following to apply the benchmarking to the function which
- computes changes in packages to report back to Landscape Server:
-
- def _compute_packages_changes(self): # noqa: max-complexity: 13
- import cProfile
- import pstats
- from datetime import datetime
- import psutil
-
- profile = cProfile.Profile()
- process = psutil.Process()
- start_cpu_times = process.cpu_times()
- profile.enable()
-
- result = self.compute_packages_change_inner()
-
- end_cpu_times = process.cpu_times()
-
- profile.disable()
-
- user_time = end_cpu_times.user - start_cpu_times.user
- system_time = end_cpu_times.system - start_cpu_times.system
- total_cpu_time = user_time + system_time
-
- output_path = "/var/lib/landscape/client/result.txt"
- with open(output_path, "a") as fp:
- now = datetime.now()
- fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
- stats = pstats.Stats(profile, stream=fp)
- stats.strip_dirs().sort_stats("cumulative").print_stats(10)
- fp.write(f"CPU Time: {total_cpu_time}s\n")
- return result
-
- def compute_packages_change_inner(self):
+ * sudo apt install -y lsb-core landscape-client # should pull new
+ version
* Now register the container with Landscape Server, wait for package-
reporter to run again and verify no errors are raised:
* sudo landscape-config --silent
--url="$LANDSCAPE_ROOT_URL/message-system" --ping-
url="$LANDSCAPE_ROOT_URL/ping" --account-name $ACCOUNT_NAME
--registration-key $REGISTRATION_KEY --computer-title $HOSTNAME
* sudo watch tail -n 30 /var/log/landscape/package-reporter.log
- * Wait for package reporter to run and check the benchmarking results
- for the proposed version:
-
- * less /var/lib/landscape/client/result.txt
-
* Additionally, make sure that package-reporter can report package
information to the server properly.
- * Create a new, fresh container of the same series, install the current
- version of Landscape Client from the main Ubuntu repo for that series
- and add the benchmarking like before. Do not install lsb-core as it will
- crash package-reporter before the benchmark runs. Make sure the
- benchmarking is in place before continuing.
-
- * sudo apt update && sudo apt install -y landscape-client
- python3-psutil
-
- * Connect this container to the same server and wait for package
- reporter to run, and compare the benchmarking results with the proposed
- version:
-
- * less /var/lib/landscape/client/result.txt
-
- * The cpu times reported by the proposed version should be
- significantly less than those of the current version. With these
- changes, this benchmark ran about 11x faster on Jammy and 15x faster on
- Focal in my testing. See my full observations in the Other Info section.
-
[ Where problems could occur ]
- * Because the fix also changed the approach to read the os-release file
- instead of the lsb_release command output, this could introduce
- unfor
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage for package reporter and fix lsb output parsing to recognize modules
** Also affects: landscape-client (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy to reduce CPU usage for package reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage for package reporter
** Description changed:
[ Impact ]
* High CPU usage of package-reporter:
* package-reporter will use a lot more CPU% than needed by creating
`Origin` objects from the Python `apt` package. This has been a complain
for customers as Landscape Client can use 100% of the CPU when package-
reporter is computing package changes to report back to Landscape
Server.
[ Test Plan ]
- * You can add my ppa to get the proposed version:
-
- * sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
- sudo apt update
-
- * Install the proposed Landscape Client:
+ * Install `psutil` and the proposed Landscape Client:
* sudo apt install -y python3-psutil landscape-client # should pull
new version
- * Replace line 601 (602 on Jammy, 596 if not proposed version) of
- `/usr/lib/python3/dist-packages/landscape/client/package/reporter.py`
- with the following to apply the benchmarking to the function which
+ * Replace this line in `/usr/lib/python3/dist-
+ packages/landscape/client/package/reporter.py`:
+
+ * def _compute_packages_changes(self): # noqa: max-complexity: 13
+
+ * with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
- * Now register the container with Landscape Server and wait for package
+ * Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
* Additionally, make sure that package-reporter can report package
information to the server properly.
- * In a fresh instance, install the current version of Landscape Client
+ * In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the results from the current version (control) of
Landscape Client on Focal: https://pastebin.com/y04PE9rw
* Raw paste for the results from the modified (proposed) of Landscape
Client on Focal: https://pastebin.com/1cbHVGQR
* Raw paste for the results from the current version (control) of
Landscape Client on Jammy: https://pastebin.com/X3d1M8BV
* Raw paste for the results from the modified (proposed) of Landscape
Client on Jammy:
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage for package reporter
** Description changed:
[ Impact ]
- * High CPU usage of package-reporter:
-
- * package-reporter will use a lot more CPU% than needed by creating
- `Origin` objects from the Python `apt` package. This has been a complain
- for customers as Landscape Client can use 100% of the CPU when package-
- reporter is computing package changes to report back to Landscape
+ * The current implementation of `landscape-package-reporter` will use a
+ lot more CPU% than needed by creating `Origin` objects from the Python
+ `apt` package when computing changes in packages. This has been a
+ complaint from users as Landscape Client can use 100% of the CPU when
+ package-reporter is computing the changes to report back to Landscape
Server.
[ Test Plan ]
* Install `psutil` and the proposed Landscape Client:
* sudo apt install -y python3-psutil landscape-client # should pull
new version
* Replace this line in `/usr/lib/python3/dist-
packages/landscape/client/package/reporter.py`:
- * def _compute_packages_changes(self): # noqa: max-complexity: 13
+ * def _compute_packages_changes(self): # noqa: max-complexity: 13
- * with the following to apply the benchmarking to the function which
+ * with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
* Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
* Additionally, make sure that package-reporter can report package
information to the server properly.
* In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the results from the current version (control) of
Landscape Client on Focal: https://pastebin.com/y04PE9rw
* Raw paste for the results from the modified (proposed) of Landscape
Client on Focal: https://pastebin.com/1cbHVGQR
* Raw paste for the results from the current version (control) of
Landscape Client on Jammy: https://pastebin.com/X3d1M8BV
* Raw paste for the results from the modified (proposed) of Landscape
Client on Jammy: https://pastebin.com/TtXpqJDE
* Original
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core).
[ Test Plan ]
-
- * `lxc launch ubuntu:focal # or ubuntu:jammy`
-
- * `lxc exec -- su -l ubuntu # shell into
- container`
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) run:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`.
* Reset the process (ex. delete container, remove from Landscape)
* In a fresh instance, install an LSB module, and the proposed
- Landscape Client. You can add my ppa to get the proposed version:
+ Landscape Client:
* sudo apt install -y lsb-core landscape-client # should pull new
version
* Now register the container with Landscape Server, wait for package-
reporter to run again and verify no errors are raised.
* Additionally, make sure that package-reporter can report package
information to the server properly.
[ Where problems could occur ]
* Because the fix also changed the approach to read the os-release file
instead of the lsb_release command output, this could introduce
unforeseen behavior with code that was written with the original
implementation in mind.
* However, without these fixes, Landscape Client cannot report package
data with lsb modules installed so it is hard to imagine how this would
be worse.
[ Other Info ]
* GH PR for the fix: https://github.com/canonical/landscape-
client/pull/170
Original bug report below:
Package reporter from landscape client is failing to report to server
because seemingly of a lsb parse issue.
/var/log/landscape/package-reporter.log
2023-08-10 13:42:00,414 ERROR[MainThread] None
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653,
in _runCallbacks
current.result = callback(current.result, *args, **kw)
File
"/usr/lib/python3/dist-packages/landscape/client/package/taskhandler.py", line
217, in got_server_uuid
lsb_release_info = parse_lsb_release(self.lsb_release_filename)
File "/usr/lib/python3/dist-packages/landscape/lib/lsb_release.py", line
31, in parse_lsb_release
dist, desc, release, code_name, _ = lsb_info.decode().split("\n")
ValueError: too many values to unpack (expected 5)
Multiple LSB Packages are installed
/usr/bin/lsb_release
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
lsb_release -a
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description:Ubuntu 18.04.3 LTS
Release:18.04
Codename: bionic
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core).
[ Test Plan ]
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
- reporting to (try and) run:
+ reporting to (try and) check the logs:
* tail /var/log/landscape/package-reporter.log
* You should see `ValueError: too many values to unpack`.
* Reset the process (ex. delete container, remove from Landscape)
* In a fresh instance, install an LSB module, and the proposed
Landscape Client:
* sudo apt install -y lsb-core landscape-client # should pull new
version
* Now register the container with Landscape Server, wait for package-
reporter to run again and verify no errors are raised.
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fails to report package information Landscape Server
when the client machine has lsb modules installed (ex. lsb-core).
[ Test Plan ]
* `lxc launch ubuntu:focal # or ubuntu:jammy`
* `lxc exec -- su -l ubuntu # shell into
container`
* `sudo apt update && sudo apt install -y landscape-client`
* `lsb_release -a # Check no lsb modules`
* `sudo apt-get install -y lsb-core # install lsb module`
* `sudo landscape-config --silent --url="$LANDSCAPE_ROOT_URL/message-
system" --ping-url="$LANDSCAPE_ROOT_URL/ping" --account-name
$ACCOUNT_NAME --registration-key $REGISTRATION_KEY --computer-title
$HOSTNAME`
* Accept the machine's registration if needed and wait for package
reporting to (try and) run:
* tail /var/log/landscape/package-reporter.log
- * You should see `ValueError: too many values to unpack`. Now remove
- this container from Landscape and delete it.
+ * You should see `ValueError: too many values to unpack`.
- * Create a fresh container of the series and add my ppa:
+ * Reset the process (ex. delete container, remove from Landscape)
- * sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
+ * In a fresh instance, install an LSB module, and the proposed
+ Landscape Client. You can add my ppa to get the proposed version:
+
+ * sudo add-apt-repository ppa:jansdhillon/landscape-client-dev &&
sudo apt update
-
- * Now, install an LSB module, and the proposed Landscape Client:
* sudo apt install -y lsb-core landscape-client # should pull new
version
* Now register the container with Landscape Server, wait for package-
reporter to run again and verify no errors are raised:
* sudo landscape-config --silent
--url="$LANDSCAPE_ROOT_URL/message-system" --ping-
url="$LANDSCAPE_ROOT_URL/ping" --account-name $ACCOUNT_NAME
--registration-key $REGISTRATION_KEY --computer-title $HOSTNAME
* sudo watch tail -n 30 /var/log/landscape/package-reporter.log
* Additionally, make sure that package-reporter can report package
information to the server properly.
[ Where problems could occur ]
- * Because the fix also changed the approach to read the os-release file
instead of the lsb_release command output, this could introduce unforeseen
behavior with code that was written with the original implementation in mind.
-
- * However, without these fixes, Landscape Client cannot report package data
with lsb modules installed so it is hard to imagine how this would be worse.
+ * Because the fix also changed the approach to read the os-release file
+ instead of the lsb_release command output, this could introduce
+ unforeseen behavior with code that was written with the original
+ implementation in mind.
+
+ * However, without these fixes, Landscape Client cannot report package
+ data with lsb modules installed so it is hard to imagine how this would
+ be worse.
[ Other Info ]
- * GH PR for the fix for LP#2031036:
- https://github.com/canonical/landscape-client/pull/170
+ * GH PR for the fix: https://github.com/canonical/landscape-
+ client/pull/170
+
+
Original bug report below:
Package reporter from landscape client is failing to report to server
because seemingly of a lsb parse issue.
/var/log/landscape/package-reporter.log
2023-08-10 13:42:00,414 ERROR[MainThread] None
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653,
in _runCallbacks
current.result = callback(current.result, *args, **kw)
File
"/usr/lib/python3/dist-packages/landscape/client/package/taskhandler.py", line
217, in got_server_uuid
lsb_release_info = parse_lsb_release(self.lsb_release_filename)
File "/usr/lib/python3/dist-packages/landscape/lib/lsb_release.py", line
31, in parse_lsb_release
dist, desc, release, code_name, _ = lsb_info.decode().split("\n")
ValueError: too many values to unpack (expected 5)
Multiple LSB Packages are installed
/usr/bin/lsb_release
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
lsb_release -a
LSB Version:
core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description:Ubuntu 18.04.3 LTS
Release:18.04
Codename: bionic
** Description changed:
[ Impact ]
* Error when parsing lsb_release, landscape-package-reporter unable to
report package list to server (LP: #2031036):
* The version of Landscape Client in the Ubuntu main repository for
Focal and Jammy fail
[Bug 2099283] Re: Update Focal, Jammy to reduce CPU usage of landscape-package-reporter
** Description changed:
[ Impact ]
* The current implementation of `landscape-package-reporter` will use a
lot more CPU% than needed by creating `Origin` objects from the Python
`apt` package when computing changes in packages. This has been a
complaint from users as Landscape Client can use 100% of the CPU when
package-reporter is computing the changes to report back to Landscape
Server.
[ Test Plan ]
* Install `python3-psutil` and the proposed version of Landscape
Client.
* Replace this line in `/usr/lib/python3/dist-
packages/landscape/client/package/reporter.py`...:
* def _compute_packages_changes(self): # noqa: max-complexity: 13
* ...with the following to apply the benchmarking to the function which
computes changes in packages to report back to Landscape Server:
def _compute_packages_changes(self): # noqa: max-complexity: 13
import cProfile
import pstats
from datetime import datetime
import psutil
profile = cProfile.Profile()
process = psutil.Process()
start_cpu_times = process.cpu_times()
profile.enable()
result = self.compute_packages_change_inner()
end_cpu_times = process.cpu_times()
profile.disable()
user_time = end_cpu_times.user - start_cpu_times.user
system_time = end_cpu_times.system - start_cpu_times.system
total_cpu_time = user_time + system_time
output_path = "/var/lib/landscape/client/result.txt"
with open(output_path, "a") as fp:
now = datetime.now()
fp.write(f"\n- Run on: {now.strftime('%Y-%m-%d
%H:%M:%S')} -\n\n")
stats = pstats.Stats(profile, stream=fp)
stats.strip_dirs().sort_stats("cumulative").print_stats(10)
fp.write(f"CPU Time: {total_cpu_time}s\n")
return result
def compute_packages_change_inner(self):
* Now register the client with Landscape Server and wait for package
reporter to run and check the benchmarking results for the proposed
version:
* less /var/lib/landscape/client/result.txt
- * Additionally, make sure that package-reporter can report package
- information to the server properly.
+ * Make sure that package-reporter can report package information to the
+ server properly:
+
+ * Install a package on the client machine manually and ensure it
+ shows up under the instance packages in the Landscape Server UI.
+
+ * Search for the package in Landscape Server and mark it for
+ removal. Wait for the activity to run and ensure the package was
+ removed.
+
+ * Search for a package that should be available to install for the
+ instance (ex. devscripts) in the UI and install it to the client
+ instance via Landscape Server.
+
+ * Update all possible packages on the client instance using the old
+ Landscape Server UI ("Advanced options" - "You can request that all
+ possible upgrades be applied to the selected computers."). You can also
+ manually upgrade all packages that have an update.
+
+ * Remove a package from the client instance using Landscape Server
+ and make sure it is removed cleanly.
* In a fresh instance, install the current version of Landscape Client
from the main Ubuntu repo for that series and add the benchmarking like
before.
* sudo apt update && sudo apt install -y landscape-client
python3-psutil
* Connect this client to the same server and wait for package reporter
to run, and check the benchmarking results for the current (unmodified)
version of Landscape Client:
* less /var/lib/landscape/client/result.txt
* The cpu times reported by the proposed version should be
significantly less than those of the current version. With these
changes, this benchmark ran about 11x faster on Jammy and 15x faster on
Focal in my testing. See my full observations in the Other Info section.
[ Where problems could occur ]
* This approach tweaks how package changes are computed, the
accuracy/reliability could be compromised. Further, this could lead to
issues with Landscape Client reporting package information too
frequently or at all.
* This proposed fix gets the archives directly from each package
versions rather than getting thing information from the Origin class of
the Python apt package. However, the point of this is to check if the
given package version matches the format of the security or backport
archives, so it's possible that this change could lead to package
version candidates being incorrectly skipped/not skipped.
[ Other Info ]
* Benchmarking observations:
https://docs.google.com/spreadsheets/d/1AOGhskitsj5XbgJGg0Ab-
EcfG4GxcB8wjRnWX0wMefE
* Raw paste for the results from the current version (control) of
Landscape Client on Focal: https://pastebin.com/y0
[Bug 2031036] Re: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server
** Merge proposal unlinked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/482537 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2031036 Title: Error when parsing lsb_release, landscape-package-reporter unable to report package list to server To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2031036/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
** Merge proposal unlinked: https://code.launchpad.net/~jansdhillon/ubuntu/+source/landscape-client/+git/landscape-client/+merge/482537 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
Hey Dave, I've created a branch for Questing. Please let me know if there are further changes you would like to see :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2099283 Title: Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2099283] Re: [SRU] Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce CPU usage of landscape-package-reporter
I’ve been testing the proposed package against the current version for
several days now and I would say this passes validation and fixes the
bug.
---
Testing steps taken:
* Install current version of LS Client from archive, as well as
python3-psutil for benchmarking the CPU performance.
* Add 'plucky-proposed' pocket to ubuntu.sources
* Enable pinning for Landscape Client to allow the proposed package to
be discovered as upgradable by Landscape Server.
* Wait for package reporter to run and verify the package is marked by
Landscape as upgradable, then upgrade it through Landscape and again
verify the proposed package is now listed as installed on the Client. I
also tested manually upgrading the package through apt.
* Complete testing plan as described in the bug report to verify
package management still works in all cases.
* Complete (relevant) Landscape Client-specific SRU QA verification
steps described in this public document
https://wiki.ubuntu.com/LandscapeUpdates under Packaging QA, and this
internal doc https://canonical-landscape-internal.readthedocs-
hosted.com/en/latest/wiki_migration/landscape_client_sru_tests/.
In this example, we can see the benchmark runtime was reduced by 60.39% after
upgrading the package to the proposed version:
- Run on: 2025-06-03 22:38:09 -
3077064 function calls (3077060 primitive calls) in 10.689
seconds
Ordered by: cumulative time
List reduced from 207 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
10.2410.241 10.689 10.689
reporter.py:686(compute_packages_change_inner)
1516780.1900.0007.3340.000 package.py:675(origins)
1542260.1290.0007.1190.000 package.py:299(__init__)
1542266.9900.0006.9900.000 {method 'find_index' of
'apt_pkg.SourceList' objects}
758390.0600.0002.1720.000 store.py:151(get_hash_id)
758450.1160.0002.1240.000 store.py:20(inner)
758390.0840.0001.9180.000 store.py:52(get_hash_id)
758451.6780.0001.6780.000 {method 'execute' of
'sqlite3.Cursor' objects}
1516780.0710.0000.6240.000
facade.py:490(is_package_installed)
1542330.0500.0000.4840.000 package.py:429(__eq__)
CPU Time: 10.25s
(upgrade Landscape Client to proposed, reinstate psutil benchmarking
from bug report)
- Run on: 2025-06-04 00:36:07 -
2493282 function calls (2493279 primitive calls) in 4.359
seconds
Ordered by: cumulative time
List reduced from 205 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
10.3530.3534.3594.359
reporter.py:686(compute_packages_change_inner)
766630.0700.0002.8470.000 store.py:151(get_hash_id)
766690.1290.0002.8140.000 store.py:20(inner)
766630.1070.0002.5530.000 store.py:52(get_hash_id)
766692.2440.0002.2440.000 {method 'execute' of
'sqlite3.Cursor' objects}
1533260.0750.0000.7210.000
facade.py:490(is_package_installed)
1561900.0510.0000.5490.000 package.py:429(__eq__)
1561900.2310.0000.4980.000 package.py:400(_cmp)
766630.0470.0000.2250.000 facade.py:470(get_package_hash)
10.0170.0170.2150.215 facade.py:184(get_locked_packages)
CPU Time: 4.06s
** Tags removed: verification-needed verification-needed-plucky
** Tags added: verification-done-plucky
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2099283
Title:
[SRU] Update Focal, Jammy, Noble, Oracular, Plucky, Questing to reduce
CPU usage of landscape-package-reporter
To manage notifications about this bug go to:
https://bugs.launchpad.net/landscape-client/+bug/2099283/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
