[issue16606] hashlib memory leak

2012-12-04 Thread Eric Snow
Changes by Eric Snow : -- status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue16606] hashlib memory leak

2012-12-04 Thread Thorsten Simons
Thorsten Simons added the comment: OK, learned something again - should have known this :-( Thank you! Thorsten -- ___ Python tracker ___ __

[issue16606] hashlib memory leak

2012-12-04 Thread Lukas Lueg
Lukas Lueg added the comment: Thorsten, the problem is that you are using line-based syntax. The code 'for buffer in f:' will read one line per iteration and put it to 'buffer'; for a file opened in binary mode, the iterator will always seek to the next b'\n'. Depending on the content of the f

[issue16606] hashlib memory leak

2012-12-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: Well, it's not immediately obvious what the exact problem could be. Are you reading a regular text file? Or is it a binary file where maybe the '\n' character appears very rarely? If it can be reproduced with a smaller file, perhaps you can attach it somewher

[issue16606] hashlib memory leak

2012-12-04 Thread Thorsten Simons
Thorsten Simons added the comment: Antoine, this was of great help - no memory leaking anymore... So, I asume that somewhere in the iteration the read file is buffered? Does that make sense or - was it the developers intention? Thank you, Regards, Thorsten -- resolution: -> works for

[issue16606] hashlib memory leak

2012-12-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: What happens if you replace iteration with something like: with open(file, "rb") as f: while True: data = f.read(16384) if not data: break myhash.update(data) -- nosy: +pitrou

[issue16606] hashlib memory leak

2012-12-04 Thread Thorsten Simons
Thorsten Simons added the comment: forgot to say that this is about huge files (tested w/ a 10GB file) -- ___ Python tracker ___ ___ P

[issue16606] hashlib memory leak

2012-12-04 Thread Thorsten Simons
New submission from Thorsten Simons: hashlib seems to leak memory when used on a Linux box (whereas the same works fine when run under Windows 7) (tested w/ Python 3.2.1 and 3.2.3) import hashlib #file = 'B:\\video\\TEST\\01_file_10G' file = '/video/TEST/01_file_10G' myhash = hashlib.sha256(