module import search path strangeness
I have a python script (part of a django application, if it makes any
difference) which is exhibiting the following behaviour:
import my_module # succeeds
imp.find_module("my_module") # fails, raising ImportError
which is completely baffling me. According to sys.path, both should
fail; the directory containing my_module is not in sys.path (though
the my_module directory itself is). More puzzlingly, printing out
my_module.__file__ gives:
/home/tow/test/my_module/../my_module/__init__.pyc
I don't really understand what the ".." is doing in there.
Can someone explain what I'm missing here, it's got me stumped.
Toby
--
http://mail.python.org/mailman/listinfo/python-list
Re: module import search path strangeness
On Aug 12, 4:59 am, "Gabriel Genellina" <[EMAIL PROTECTED]>
wrote:
> En Mon, 11 Aug 2008 19:19:19 -0300, tow <[EMAIL PROTECTED]>
> escribi :
>
> > I have a python script (part of a django application, if it makes any
> > difference) which is exhibiting the following behaviour:
>
> > import my_module # succeeds
> > imp.find_module("my_module") # fails, raising ImportError
>
> > which is completely baffling me. According to sys.path, both should
> > fail; the directory containing my_module is not in sys.path (though
> > the my_module directory itself is).
>
> my_module is not a module but a package, right? Else I don't understand
> the above statement.
Sorry, sloppy terminology on my part, yes, my_module is a package.
> > More puzzlingly, printing out
> > my_module.__file__ gives:
>
> > /home/tow/test/my_module/../my_module/__init__.pyc
>
> > I don't really understand what the ".." is doing in there.
>
> > Can someone explain what I'm missing here, it's got me stumped.
>
> Perhaps you have ".." in sys.path? And the current directory happens to be
> /home/tow/test/my_module?
The current directory is actually a subdirectory of /home/tow/test/
my_module,
but ".." is not in sys.path (nor is anything containing "..") In any
case, if
it were a matter of sys.path, surely the imp.find_module call above
should succeed?
Basically, I had thought that import and imp.find_module used exactly
the same
search path, but the above example shows that at least in this
circumstance they
don't; import is picking up additional search paths from somewhere -
what am I missing?
Toby
--
http://mail.python.org/mailman/listinfo/python-list
Re: module import search path strangeness
On Aug 12, 9:56 am, Peter Otten <[EMAIL PROTECTED]> wrote:
> tow wrote:
> > Basically, I had thought that import and imp.find_module used exactly
> > the same
> > search path, but the above example shows that at least in this
> > circumstance they
> > don't; import is picking up additional search paths from somewhere -
> > what am I missing?
>
> Grepping through the django source finds
>
> ./trunk/django/core/management/__init__.py:
> sys.path.append(os.path.join(project_directory, os.pardir))
Hmm. It turns out that that is indeed the issue, but in a way that
wasn't immediately obvious to me. Looking at it in more context:
sys.path.append(os.path.join(project_directory, os.pardir))
project_module = __import__(project_name, {}, {}, [''])
sys.path.pop()
sys.path is extended, the project module is imported, then the
additional path is dropped from sys.path.
And if I comment out those sys.path manipulations, I get the result I
expect later on.
What I think is happening is that in that stanza, project_module
(which is my_module in my case) is
imported from the altered sys.path. sys.path is then put back, but
python now knows about my_module.
As a result when my_module is imported again elsewhere, python already
knows about it, so no searching
of sys.path is done (which is good, because it wouldn't be found on
the current sys.path), and the
import apparently succeeds, though in fact it's effectively
(actually?) a no-op.
However, imp.find_module("my_module") forces a search of the sys.path,
and thus fails.
Or at least, that is what I surmise, given that I see
import my_module # succeeds
imp.find_module("my_module") # fails, raising ImportError
So, to answer my original question, the difference in search behaviour
between "import" and "imp.find_module" is that the former might not
look at sys.path at all if the module has already been loaded, while
the latter will only search on the current sys.path.
Am I right?
Toby
--
http://mail.python.org/mailman/listinfo/python-list
Re: module import search path strangeness
On Aug 12, 4:59 pm, Peter Otten <[EMAIL PROTECTED]> wrote:
> tow wrote:
> > On Aug 12, 9:56 am, Peter Otten <[EMAIL PROTECTED]> wrote:
> >> tow wrote:
>
> >> > Basically, I had thought that import and imp.find_module used exactly
> >> > the same
> >> > search path, but the above example shows that at least in this
> >> > circumstance they
> >> > don't; import is picking up additional search paths from somewhere -
> >> > what am I missing?
>
> >> Grepping through the django source finds
>
> >> ./trunk/django/core/management/__init__.py:
> >> sys.path.append(os.path.join(project_directory, os.pardir))
>
> > Hmm. It turns out that that is indeed the issue, but in a way that
> > wasn't immediately obvious to me. Looking at it in more context:
>
> > sys.path.append(os.path.join(project_directory, os.pardir))
> > project_module = __import__(project_name, {}, {}, [''])
> > sys.path.pop()
>
> Ouch.
>
> > So, to answer my original question, the difference in search behaviour
> > between "import" and "imp.find_module" is that the former might not
> > look at sys.path at all if the module has already been loaded, while
> > the latter will only search on the current sys.path.
>
> > Am I right?
>
> Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
> before it falls back to the more costly alternatives.
>
> Peter
Thanks, at least I understand what's going on now.
Toby
--
http://mail.python.org/mailman/listinfo/python-list
module imports and st_mtime
I'm seeing a very strange effect which is confusing me - in brief, one
python process appears to temporarily affect the os.stat results of
another - perhaps someone can enlighten me.
This is on Mac OS X Leopard, using the system python (2.5)
The issue arises using Django. The default Django http server runs a
watcher thread, which checks if any code is changing, and reloads
itself. It does this by iterating over all loaded modules, and
checking the mtime of each __file__. This was behaving oddly, and
finding out why exposed this strangeness. (The relevant code is in
django/utils/autoreload.py)
Some of the code running under this django server imports simplejson,
the C-implemented module of which has been put at /Users/tow/.python-
eggs/simplejson-2.0.9-py2.5-macosx-10.5-i386.egg-tmp/simplejson/
_speedups.so
This hasn't been touched since it was installed:
ls -l ~/.python-eggs/simplejson-2.0.9-py2.5-macosx-10.5-i386.egg-tmp/
simplejson/_speedups.so
-rwxr-xr-x 1 tow staff 77596 12 Aug 17:56 /Users/tow/.python-eggs/
simplejson-2.0.9-py2.5-macosx-10.5-i386.egg-tmp/simplejson/
_speedups.so
If I check the mtime of that file from within django, it finds it
correctly:
print datetime.datetime.utcfromtimestamp(os.stat("/Users/tow/.python-
eggs/simplejson-2.0.9-py2.5-macosx-10.5-i386.egg-tmp/simplejson/
_speedups.so").st_mtime)
2009-08-12 17:56:02
The strange effect occurs when I open another python process, and
import simplejson there as well. As soon as I've done that, the mtime
that Django sees slips by an hour:
print datetime.datetime.utcfromtimestamp(os.stat("/Users/tow/.python-
eggs/simplejson-2.0.9-py2.5-macosx-10.5-i386.egg-tmp/simplejson/
_speedups.so").st_mtime)
2009-08-12 16:56:02
In fact, to be precise, this happens as soon as the
simplejson._speedups module *finishes* being imported. (Tested by
stepping through every line with pdb)
The second Python process still sees the correct mtime, though, both
before and after it imports simplejson.
Restarting the Django process resets its view of the world, and it
sees the correct mtime again.
The current time as seen by the Django process is correct both before
and after the mtime slippage.
This seems to be 100% reproducible here, except for the time offset.
Usually it loses one hour, sometimes it gains 5 hours. (For what it's
worth, I'm currently on GMT, but the file was created during daylight
savings time).
I haven't managed to replicate it when the first process is something
other than Django. I've seen the effect on other Mac OS machines, but
haven't tested it on Linux so far. I've only seen the effect with
simplejson's C module, but I think this is the only C module which
might be being imported twice in this way for me.
Does anyone have any ideas what might be going on, or where further to
look? I'm at a bit of a loss.
Toby
--
http://mail.python.org/mailman/listinfo/python-list
