Germandrummer92 commented on code in PR #1847:
URL: https://github.com/apache/libcloud/pull/1847#discussion_r1072754536
##########
libcloud/utils/files.py:
##########
@@ -75,7 +75,13 @@ def read_in_chunks(iterator, chunk_size=None,
fill_size=False, yield_empty=False
return
if fill_size:
- if empty or len(data) >= chunk_size:
+ chunk_start = 0
Review Comment:
not sure if it makes it better just as an idea, but to encapsulate the
comment also you could move it into a method now?
```python
def _optimized_chunked_generator(data: bytes, chunk_size: int) ->
Generator[bytes, None, bytes]:
# We want to emit chunk_size large chunks, but chunk_size can be
larger or smaller than the chunks returned
# by get_data. We need to yield in a loop to avoid large amounts of
data piling up.
# The loop also avoids copying all data #chunks amount of times by
keeping the original data as is.
chunk_start = 0
while chunk_start + chunk_size < len(data):
yield data[chunk_start : chunk_start + chunk_size]
chunk_start += chunk_size
data = data[chunk_start:]
return data
...
data = yield from _optimized_chunk_generator(data=data,
chunk_size=chunk_size)
```
might make this method itself a tiny bit more readable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
