On Tue, 06 Oct 2015 20:20:40 +0200, Peter Otten wrote:
> Jaydip Chakrabarty wrote:
>
>> On Tue, 06 Oct 2015 14:33:51 +0200, Peter Otten wrote:
>>
>>> Jaydip Chakrabarty wrote:
>>>
>>>> On Tue, 06 Oct 2015 01:34:17 +1100, Chris Angelico wrote:
>>>>
>>>>> On Tue, Oct 6, 2015 at 1:06 AM, Tim Chase
>>>>> <python.l...@tim.thechases.com> wrote:
>>>>>> That way, if you determine by line 3 that your million-row CSV file
>>>>>> has no blank columns, you can get away with not processing all
>>>>>> million rows.
>>>>>
>>>>> Sure, although that effectively means the entire job is moot. I
>>>>> kinda assume that the OP knows that there are some blank columns
>>>>> (maybe lots of them). The extra check is unnecessary unless it's
>>>>> actually plausible that there'll be no blanks whatsoever.
>>>>>
>>>>> Incidentally, you have an ordered_headers list which is the blank
>>>>> columns in order; I think the OP was looking for a list of the
>>>>> _non_blank columns. But that's a trivial difference, easy to tweak.
>>>>>
>>>>> ChrisA
>>>>
>>>> Thanks to you all. I got it this far. But while writing back to
>>>> another csv file, I got this error - "ValueError: dict contains
>>>> fields not in fieldnames: None". Here is my code.
>>>>
>>>> rdr = csv.DictReader(fin, delimiter=',')
>>>> header_set = set(rdr.fieldnames)
>>>> for r in rdr:
>>>> header_set = set(h for h in header_set if not r[h])
>>>> if not header_set:
>>>> break
>>>>
>>>> for r in rdr:
>>>> data = list(r[i] for i in header_set)
>>>>
>>>> dw = csv.DictWriter(fout, header_set)
>>>> dw.writeheader()
>>>> dw.writerows(data)
>>>
>>> Sorry, this is not the code you ran. I could guess what the missing
>>> parts might be, but it is easier for both sides if you provide a small
>>> script that actually can be executed and a small dataset that shows
>>> the behaviour you describe. Then post the session and especially the
>>> traceback. Example:
>>>
>>> $ cat my_data.csv 0 $ cat my_code.py print
>>> 1/int(open("my_data.csv").read())
>>> $ python my_code.py Traceback (most recent call last):
>>> File "my_code.py", line 1, in <module>
>>> print 1/int(open("my_data.csv").read())
>>> ZeroDivisionError: integer division or modulo by zero
>>>
>>> Don't retype, use cut and paste. Thank you.
>>
>> I downloaded gmail contacts in google csv format. There are so many
>> columns. So I was trying to create another csv with the required
>> columns.
>> Now when I tried to open the gmail csv file with csv DictReader, it
>> said the file contained NULL characters.
>> So first I did -
>>
>> data = open(fn, 'rb').read()
>> fout = open(ofn, 'wb')
>> fout.write(data.replace('\x00', ''))
>> fout.close()
>> shutil.move(ofn, fn)
>>
>> Then I found, there were some special characters in the file. So, once
>> again I opened the file and did -
>>
>> data = open(fn, 'rb').read()
>> fout = open(ofn, 'wb')
>> fout.write(data.replace('\xff\xfe', ''))
>> fout.close()
>> shutil.move(ofn, fn)
>
> Uh this looks like the file is in UTF-16. Use
>
> import codecs fn = ...
> ofn = ...
> with codecs.open(fn, encoding="utf-16") as f:
> with codecs.open(ofn, "w", encoding="utf-8") as g:
> g.writelines(f)
> ...
>
> to convert it to UTF-8 which is compatible with the csv module of Python
> 2.
>
>> Now it seemed right.
>
> Only if all characters are encodable as iso-8859-1.
>
>> So I started to remove empty columns.
>>
>> fin = open(fn, 'rb')
>> fout = open(ofn, 'wb')
>>
>> rdr = csv.DictReader(fin, delimiter=',')
>> flds = rdr.fieldnames header_set = set(rdr.fieldnames)
>> for r in rdr:
>> header_set = set(h for h in header_set if not r[h])
>> if not header_set:
>> break
>> for r in rdr:
>> data = list(r[i] for i in header_set)
>>
>> dw = csv.DictWriter(fout, data[0].keys())
>> dw.writeheader()
>> dw.writerows(data)
>>
>> fin.close()
>> fout.close()
>>
>> But, I am getting error at dw.writerows(data). I put the whole code
>> here.
>> Please help.
>
> I really meant it when I asked you to post the code you actually ran,
> and the traceback it produces.
>
> When I fill in the blanks by guessing
>
> $ cat in.csv one,two,three foo,,
> bar,,baz $ cat remove_empty_colums.py import csv fn = "in.csv"
> ofn = "out.csv"
>
> fin = open(fn, 'rb')
> fout = open(ofn, 'wb')
>
> rdr = csv.DictReader(fin, delimiter=',')
> flds = rdr.fieldnames header_set = set(rdr.fieldnames)
> for r in rdr:
> header_set = set(h for h in header_set if not r[h])
> if not header_set:
> break
> for r in rdr:
> data = list(r[i] for i in header_set)
>
> dw = csv.DictWriter(fout, data[0].keys())
> dw.writeheader()
> dw.writerows(data)
>
> fin.close()
> fout.close()
>
> and then run the resulting script I get
>
> $ python remove_empty_colums.py Traceback (most recent call last):
> File "remove_empty_colums.py", line 18, in <module>
> dw = csv.DictWriter(fout, data[0].keys())
> NameError: name 'data' is not defined
>
> So this is my traceback, and while the NameError is trivial to fix
> (reopen the file or do a seek) but not sufficient to make the script do
> what you want it doesn't seem to be the problem you ran into.
> So you have a different script. I'd really like to see it, and the
> traceback it produces.
Yes, that is so.
Thank you all for the help.
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
