On 11/2/2017 9:18 AM, ROGER GRAYDON CHRISTMAN wrote:
I have a partial answer to my own question:
This seems to work for me:
---
link = urllib.request.urlopen(urlpath)
data = link.read().decode('utf-8').split('\n')
reader = csv.DictReader(data)
for row in reader:
---
I think here my concern is that now 'data' is now a variable
in my program's memory (all of the data),
instead of streamlike. I suppose I did read
somewhere about setting a stream option.
csv.reader and csv.DictReader are transformation iterators whose first
argument must be an iterable of string lines. Given an iterator of
bytes lines, you just need to interpose a bytes to string iterator --
something like (untested)
def strgen(bytesource, encoding):
for line in bytesource:
yield line.decode(encoding)
with urllib.request.urlopen(urlpath) as link:
lines = strgen(link, 'utf-8')
reader = csv.DictReader(lines) # plus any other args
for row in reader:
process(row)
Iterators are intended to be chained together like this.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
