[web2py] upload bug in filename
Hi , I tried to send a file with a form containing a upload field( actually it is a SQLFORM). When i try to add a new record with the administrative db interface , it breaks if the filename if very long ( not sure where it comes from ). Any idea? Thx
[web2py] Re: upload bug in filename
don't really know but i tried with that filename : aaa.pdf (which containt 111 a if you trust me :) and get the error ticket , showing this last call File "C:\Users\M\Desktop\web2py_src\web2py\gluon\sql.py", line 2699, in store dest_file = open(pathfilename, 'wb') IOError: [Errno 2] No such file or directory: 'C:\\Users\\M\\Desktop\ \web2py_src\\web2py\\applications\\Library/databases\\..\\uploads\ \books.file.acf7b28c26fb29c3.6161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161.pdf' (the filename is 205 characters long ,but pathfilename is proably bigger). Do you know if the 255 limit is python limit ? Where does it come from ? any idea how to sort it out without changing the pathfilename to a shorter version(actually the generated filename could be changed to something shorter but i don t know where it is generated )? M BTW , i use 1.79.2 On 15 juin, 05:32, Thadeus Burgess wrote: > how long is the filename? > > Isn't there like a 255 char limit to filenames? > -- > Thadeus > > > > On Mon, Jun 14, 2010 at 4:58 PM, Swell wrote: > > Hi , > > I tried to send a file with a form containing a upload > > field( actually it is a SQLFORM). When i try to add a new record with > > the administrative db interface , it breaks if the filename if very > > long ( not sure where it comes from ). > > > Any idea? > > > Thx
[web2py] Re: upload bug in filename
There is nothing suspicious here about the filename ( it was an example here, but it has the same effect with a file containing the description oof the paper + all the authors ). I am almost sure that it is related to size of the path + filename that is not correctly handled . But i can't spot where is it in the source code . Am i the only one to see that issue? ( one thing also iis that i am running web2py on windows , but i dont think that it is the issue here ) M
[web2py] Re: upload bug in filename
thx yarko for your detailed answer.
I have not written code so far, I have been using the administrative
interface provided by web2py to manage the records in the DB.
I am going to generate another test showing this kind of behaviour
with a more regular filename.
On Jun 17, 11:51 pm, Yarko Tymciurak
wrote:
> On Jun 17, 4:59 pm, Swell wrote:
>
> > There is nothing suspicious here about the filename ( it was an
> > example here, but it has the same effect with a file containing the
> > description oof the paper + all the authors ). I am almost sure that
> > it is related to size of the path + filename that is not correctly
> > handled . But i can't spot where is it in the source code .
>
> OK - let's go back to your error report:
>
> /-
> File "C:\Users\M\Desktop\web2py_src\web2py\gluon\sql.py", line 2699,
> in store
> dest_file = open(pathfilename, 'wb')
> IOError: [Errno 2] No such file or directory: 'C:\\Users\\M\\Desktop\
> \web2py_src\\web2py\\applications\\Library/databases\\..\\uploads\
> \books.file.acf7b28c26fb29c3.6161616161616161616161616161616161616161616161
> 616161616161616161616161616161616161616161616161616161616161616161616161616
> 161616161616161616161616161616161616161616161616161.pdf'
> \--
>
> To begin with - this is a REALLY suspicious filename; 61 repeated is
> not what I would expect from the store() code (below).
> In fact, have a look atwww.asciitable.com: 61 is the character code
> for "="; I suspect you have a bug in your controller.
>
> Why that would generate this filename is yet another question.
>
> Now let's look at it in the context of the code the error quotes:
>
> /--
> def store(self, file, filename=None, path=None):
>
> # have you defined a custom_store?
> # -- if you have, then you are doing this!
> if hasattr(self,'custom_store'):
> return self.custom_store(file,filename,path)
> if not filename:
> filename = file.name
>
> # the fact that your path has NOT had these replaced, I think
> # you must NOT be traversing this code:
> filename = os.path.basename(filename.replace('/', os.sep)\
> .replace('\\', os.sep))
>
>
> # the pertinent naming stuff is here: the filename is encoded
>
> uuid_key = web2py_uuid().replace('-', '')[-16:]
>
> # this just obfuscates the name (and makes it longer - can make
> it considerably longer).
> # for example, if your passed in filename is 235 characters,
> the
> # encoded_filename version is 940 characters long!!!
>
> encoded_filename = base64.b16encode(filename).lower()
>
> # this makes the stored name composed of:
> # table name, field name, key, and that really long encoded
> name;
> newfilename = '%s.%s.%s.%s' % \
> (self._tablename, self.name, uuid_key, encoded_filename)
>
> # this limits the filename length, in this case truncating
> # the encoded_filename part:
> newfilename = newfilename[:200] + '.' + extension
> if self.uploadfield == True:
> if path:
> pass
> elif self.uploadfolder:
> path = self.uploadfolder
> else:
> path = os.path.join(self._db._folder, '..', 'uploads')
> if self.uploadseparate:
> path = os.path.join(path,"%s.%s" % (self._tablename,
> self.name),uuid_key[:2])
> if not os.path.exists(path):
> os.makedirs(path)
> pathfilename = os.path.join(path, newfilename)
>
> # because this is attempting to open a file for write/
> binary,
> # and since you are experiencing an uncaught exception
> # which _could_ be triggered by filename size, other O/S
> related things,
> # this open() call should really be in a try / except -
> to catch
> # the error message from "normal users" --- but the only
> reason
> # a write open would fail: permissions, file path
> errors (e.g. code)
> dest_file = open(pathfilename, 'wb')
> shutil.copyfileobj(file, dest_file)
> dest_file.close()
> return newfilename
> \---
>
> In sum, after working through this I suspect it is your code
> (somewhere) which is causing your problems; I think you should debug
> it, and give us enough information to help you without asking us to go
> on a blind hunting expedition.
>
> Regards,
> - Yarko
>
>
>
>
>
> > Am i the only one to see that issue? ( one thing also iis that i am
> > running web2py on windows , but i dont think that it is the issue
> > here )
>
> > M
[web2py] Re: upload bug in filename
Ok
I will post some code here to show as much as possible
here is the db definition:
db.define_table('papers',
Field('title','string'),
Field('author','string'),
Field('file','upload'),
)
then if i try to use the database administration tool available in
Models section and try to insert a file like
"The Pricing of Options and Corporate Liabilities_Journal of Political
Economy.pdf" i then get a ticket complaining about
IOError: [Errno 2] No such file or directory:
C:\\Users\\M\\Desktop\\web2py_src\\web2py\\applications\\Library/
databases\\..\\uploads\
\papers.file.a33b16f5af29f6d8.5468652050726963696e67206f66204f7074696f6e7320616e6420436f72706f72617465204c696162696c69746965735f4a6f75726e616c206f6620506f6c69746963616c2045636f6e6f6d792e706466.pdf'
i also checked the file actually has never been properly uploaded ( it
is not present in the upload directory). Nothing special about size of
the file ( if i just change manually the filename it works ). So it
seems that the bug is when we receiving the file .
hope we can sort this out
Thx all
M
[web2py] Re: upload bug in filename
Thx you for your help, i am thinking now that it is maybe not the filename itself that is wrong but maybe filepathname. I mean that it is the concat of the basename and filename is maybe too long . i am going to look at this (by changing the web2py src tree to c:\ so that a shorter basename is going to be generated ) i ll let u know Emmanuel
[web2py] Re: upload bug in filename
Ok i have been able to reproduce the bug , it works perfectly if the web2py tree is in c:\ it doesnot if it is copied on the desktop(C:\Users\M\Desktop\web2py) so it seems that it is related the filenamepath(basename+filename)
[web2py] Re: upload bug in filename
Yes i am using web2py 1.79.2 on windows 7 64b Can someone try to save the web2py files in a very long directory name ( something like 200 characters ) and use the kind of filename i provided ( long ) and see what is happening on mac/linux? I am doing local tests only , it means that the file are on my local drive with these names so the fs can handle them properly. Python may have a pb or the admin interface code don t expect such a long FILEPATHNAME (base dir + filename ) .
[web2py] Re: upload bug in filename
Hi I may hae found something if i look at line 2688 in gluon/sql.py i can see in the function store: newfilename = newfilename[:200] + '.' + extension but so far so good it doesn t break anything to truncate the filename. later in the function ( line 2699) i see pathfilename = os.path.join(path, newfilename) dest_file = open(pathfilename, 'wb') and then an exception is thrown here so it seems it is an open issue i think that if we can t fix this bug , maybe adding some exception handling code here would be intersting. don t know yet if it is coming from python(2.6.4),windows 7 64, but at least we know the web2py call that breaks it.
[web2py] Re: upload bug in filename
ok i hae done that and it complains about IOError: [Errno 2] No such file or directory: 'a a' it seems that there is a max length for the filename ( as described here http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx#maximum%5Fpath%5Flength ). It maybe related to the fact that filename are encoded in base64 which eentually be longer thant the original filename ( + all the decoration about table, fields , uid ) So it seems to be not a direct web2py issue , but something that we need to address. How about changing the scheme creation to something shorter ( like table.field.uuid and that is it ) That should be more portable . What are your thoughts ?
