On Aug 24, 4:32 pm, Thomas Jollans <tho...@jollybox.de> wrote: > On Tuesday 24 August 2010, it occurred to Darren Dale to exclaim: > > > > > > > On Aug 23, 9:58 am, Darren Dale <dsdal...@gmail.com> wrote: > > > The following script runs without problems on Ubuntu and Windows 7. > > > h5py is a package wrapping the hdf5 library (http://code.google.com/p/ > > > h5py/): > > > > from multiprocessing import Pool > > > import h5py > > > > def update(i): > > > print i > > > > def f(i): > > > "hello foo" > > > return i*i > > > > if __name__ == '__main__': > > > pool = Pool() > > > for i in range(10): > > > pool.apply_async(f, [i], callback=update) > > > pool.close() > > > pool.join() > > > > On OS X 10.6 (tested using python-2.6.5 from MacPorts), I have to > > > comment out the as-yet unused h5py import, otherwise I get a > > > traceback: > > What on earth is h5py doing there? If what you're telling us is actually > happening, and the code works 1:1 on Linux and Windows, but fails on OSX, and > you're using the same versions of h5py and Python, then the h5py > initialization code is not only enticing multiprocessing to try to pickle > something other than usual, but it is also doing that due to some platform- > dependent witchcraft, and I doubt there's very much separating the OSX > versions from the Linux versions of anything involved.
Your analysis was spot on. About a year ago, I contributed a patch to h5py which checks to see if h5py is being imported into an active IPython session. If so, then a custom tab completer is loaded to make it easier to navigate hdf5 files. In the development version of IPython, a function that used to return None if there was no instance of an IPython interactive shell now creates and returns a new instance. This was the cause of the error I was reporting. If one were to install ipython from the master branch at github or from http://ipython.scipy.org/dist/testing/ipython-dev-nightly.tgz, then the following script will reproduce the problem. I'm not sure why this causes an error, but I'll discuss it with the IPython devs. Thank you Thomas and Benjamin for helping me understand the problem. Darren from multiprocessing import Pool import IPython.core.ipapi as ip ip.get() def update(i): print i def f(i): return i*i if __name__ == '__main__': pool = Pool() for i in range(10): pool.apply_async(f, [i], callback=update) pool.close() pool.join() -- http://mail.python.org/mailman/listinfo/python-list
