jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1325864?usp=email )
Change subject: data_ingestion: Hash cached photo buffer
......................................................................
data_ingestion: Hash cached photo buffer
Keep the cached download typed as BytesIO and hash its buffer directly.
Avoid copying the complete photo and mark SHA-1 as non-security use.
Change-Id: I26858c7d5abe04d97a9d956cc9f38a3a3823af96
---
M scripts/data_ingestion.py
1 file changed, 6 insertions(+), 6 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/data_ingestion.py b/scripts/data_ingestion.py
index 53d0a9b..cd050e9 100755
--- a/scripts/data_ingestion.py
+++ b/scripts/data_ingestion.py
@@ -101,7 +101,7 @@
import io
import os
import posixpath
-from typing import Any, BinaryIO
+from typing import Any
from urllib.parse import urlparse
import pywikibot
@@ -131,7 +131,7 @@
urlparse(url)[2])[1]
ext = filename.split('.')[-1]
self.metadata['_ext'] = None if ext == filename else ext
- self.contents = None
+ self.contents: io.BytesIO | None = None
if not site:
site = pywikibot.Site('commons')
@@ -139,12 +139,12 @@
# default title
super().__init__(site, self.get_title('%(_filename)s.%(_ext)s'))
- def download_photo(self) -> BinaryIO:
+ def download_photo(self) -> io.BytesIO:
"""Download the photo and store it in an io.BytesIO object.
TODO: Add exception handling
"""
- if not self.contents:
+ if self.contents is None:
image_file = fetch(self.URL).content
self.contents = io.BytesIO(image_file)
return self.contents
@@ -157,8 +157,8 @@
TODO: Add exception handling, fix site thing
"""
- hash_object = hashlib.sha1()
- hash_object.update(self.download_photo().getvalue())
+ hash_object = hashlib.sha1(
+ self.download_photo().getbuffer(), usedforsecurity=False)
return [page.title(with_ns=False)
for page in self.site.allimages(
sha1=base64.b16encode(hash_object.digest()))]
--
To view, visit
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1325864?usp=email
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings?usp=email
Gerrit-MessageType: merged
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I26858c7d5abe04d97a9d956cc9f38a3a3823af96
Gerrit-Change-Number: 1325864
Gerrit-PatchSet: 2
Gerrit-Owner: Mahveotm <[email protected]>
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: jenkins-bot
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
