Bugs item #1478529, was opened at 2006-04-28 17:46 Message generated for change (Comment added) made by markshep You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Windows Group: 3rd Party Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: Mark Sheppard (markshep) Assigned to: Nobody/Anonymous (nobody) Summary: size limit exceeded for read() from network drive Initial Comment: If you've got a network share mounted as a local drive then Windows has a limit of 67,076,095 (0x03ff7fff) bytes for a read from an open file on that drive. Running the python read() method on an open file larger than this size throws an "IOError: [Errno 22] Invalid argument" exception. A fix would be for python to internally use multiple reads so as to not exceed this limit. ---------------------------------------------------------------------- >Comment By: Mark Sheppard (markshep) Date: 2006-05-03 11:38 Message: Logged In: YES user_id=1512331 Thanks for closing this bug without giving me a chance to follow up! The problem isn't caused by a limitation of my machine - it's got 3 GiB of RAM. I've done some more testing on this and the problem only appears when connected to a server running certain SMB implementations: The local Windows XP machine A remote Windows XP machine Samba 3.0.22 on Linux When connected to servers running the following SMB implementations the problem isn't present: Windows NT 4.0 Server Windows Server 2000 Windows Server 2003 Standard Edition As this error is being returned by the underlying fread() call the proper place for it to be fixed is there, but the chances of Microsoft doing so for Windows XP are negligible. As you're trying to provide a cross-platform language then having to put up with OS's undocumented warts is just part of the job. As it's entirely possible for you to implement a work-around for this problem Python I think you should. One of reasons for using a high level language like Python is to be insulated from system quirks likes this. If you're refusing to smooth over these quirks where possible then you're undermining that reason. The documentation for Python's read() method on a file handle already says "Note that this method may call the underlying C function fread() more than once", so this possibility is already catered for in the documentation. As this problem only affects remotely mounted filesystems the workaround need only be used for such filesystems. You can determine whether or not a drive is a network one by using the GetDriveType() Windows call. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2006-05-03 02:45 Message: Logged In: YES user_id=31435 Sorry, I'm closing this as "3rd Party, Won't Fix". It's certainly not Python's doing that Microsoft's stdio implementation has peculiar undocumented warts (Python just calls the platform C's fread() here), so at best this is a request for enhancement rather than a Python bug. If there is a bug here, it's Microsoft's bug, and then the proper source for a fix is also Microsoft. This is especially true since the two people who have tried this here don't see the same behavior -- we don't even know what "the bug" is. ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-05-02 20:00 Message: Logged In: YES user_id=21627 I could reproduce the write problem on XPSP2; I get the Win32 error ERROR_NO_SYSTEM_RESOURCES after fwrite returns (from GetLastError). I can't reproduce the fread problem, though: in Python, f.read(90*2**20) just returns with a 90MiB string. So it could be a limitation of your machine (e.g. it might not have enough memory), or of the server machine. I'm hesitant to add a work-around for that into Python if this isn't a system limitation. Performing multiple reads is also bad: what happens if the first read succeeds, and the second one fails? It might be that the system *really* is out of resources. ---------------------------------------------------------------------- Comment By: Mark Sheppard (markshep) Date: 2006-05-02 11:48 Message: Logged In: YES user_id=1512331 I'm running Windows XP. I've been unable to find any documentation about this exact problem - only that fwrite thing. But my testing shows that it works if I do file.read(67076095), but throws an exception with file.read(67076096). I'm not suggesting limiting all reads from Python. All I'm suggesting is that under the hood the Windows implementation of Python's read() call actually uses multiple fread() (or whatever) calls if more than 67076095 bytes need to be read. That's all. No interface changes. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2006-04-30 17:23 Message: Logged In: YES user_id=31435 Martin, here's an MS article seemingly related to this: http://support.microsoft.com/default.aspx?scid=kb;en-us;899149 However, it's about writing to a file on a network drive, not reading from it. It says that opening the file in 'w+b' mode, instead of 'wb' mode, is a workaround. I couldn't find anything documenting the same kind of problem for reading. ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-04-30 11:10 Message: Logged In: YES user_id=21627 What version of Windows are you using? Do you know of any documentation of this limit? (without actually testing, I find it hard to believe that this limit exists in Windows) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-04-29 14:23 Message: Logged In: YES user_id=849994 How can it be determined whether exactly this restriction caused the "invalid argument" error? If it can't, there's nothing that can be done -- restricting all reads just because of a Windows limitation doesn't seem right. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
