jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971606 )
Change subject: [IMPR] use process_cpu_count to select the default number of
worker
......................................................................
[IMPR] use process_cpu_count to select the default number of worker
os.cpu_count() returns the total number of CPUs that the current
machinevhas. On Unix, sched_setaffinity() can reduce the number of CPU
"usable" by a process. With Python 3.13 os.process_cpu_count() can be
used to get the number of logical CPUs usable by the calling thread of
the current process. This patch uses os.process_cpu_count() like in
Python 3.13 to select the default number of worker for
concurrent.futures.
Change-Id: Id66f9bb32cec80206434f39277db7b4c62867ec3
---
M pywikibot/scripts/preload_sites.py
1 file changed, 28 insertions(+), 5 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/scripts/preload_sites.py
b/pywikibot/scripts/preload_sites.py
index afda243..4d4799e 100755
--- a/pywikibot/scripts/preload_sites.py
+++ b/pywikibot/scripts/preload_sites.py
@@ -22,15 +22,19 @@
#
# Distributed under the terms of the MIT license.
#
-import os
from concurrent.futures import ThreadPoolExecutor, wait
from datetime import datetime
from typing import Optional, Union
import pywikibot
-from pywikibot.backports import List, Set, removeprefix
+from pywikibot.backports import Dict, List, Set, removeprefix
from pywikibot.family import Family
+try: # Python 3.13
+ from os import process_cpu_count # type: ignore[attr-defined]
+except ImportError:
+ from os import cpu_count as process_cpu_count
+
#: supported families by this script
families_list = [
@@ -44,7 +48,9 @@
'wiktionary',
]
-exceptions = {
+# Ignore sites from preloading
+# example: {'wikiversity': ['beta'], }
+exceptions: Dict[str, List[str]] = {
}
@@ -80,8 +86,8 @@
"""
start = datetime.now()
if worker is None:
- # Python 3.8 default
- worker = min(32, (os.cpu_count() or 1) + 4)
+ # Python 3.13 default
+ worker = min(32, (process_cpu_count() or 1) + 4)
# to allow adding futures in preload_family the workers must be one
# more than families are handled
worker = max(len(families) * 2, worker)
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971606
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Id66f9bb32cec80206434f39277db7b4c62867ec3
Gerrit-Change-Number: 971606
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <[email protected]>
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
