Optimizing if statement check over a numpy value
Dear all,
I have the following piece of code. I am reading a numpy dataset from an hdf5
file and I am changing values to a new value if they equal 1.
There is 90 percent chance that (if id not in myList:) is true and in 10
percent of time is false.
with h5py.File(inputFile, 'r') as f1:
with h5py.File(inputFile2, 'w') as f2:
ds=f1["MyDataset"].value
myList=[list of Indices that must not be given the new_value]
new_value=1e-20
for index,val in np.ndenumerate(ds):
if val==1.0 :
id=index[0]+1
if id not in myList:
ds[index]=new_value
dset1 = f2.create_dataset("Cell Ids", data=cellID_ds)
dset2 = f2.create_dataset("Porosity", data=poros_ds)
My numpy array has 16M data and it takes 9 hrs to run. If I comment my if
statement (if id not in myList:) it only takes 5 minutes to run.
Is there any way that I can optimize this if statement.
Thank you very much in Advance for your help.
Best Regards,
--
https://mail.python.org/mailman/listinfo/python-list
Re: Optimizing if statement check over a numpy value
On Thursday, July 23, 2015 at 1:43:00 PM UTC+2, Jeremy Sanders wrote: > Heli Nix wrote: > > > Is there any way that I can optimize this if statement. > > Array processing is much faster in numpy. Maybe this is close to what you > want > > import numpy as N > # input data > vals = N.array([42, 1, 5, 3.14, 53, 1, 12, 11, 1]) > # list of items to exclude > exclude = [1] > # convert to a boolean array > exclbool = N.zeros(vals.shape, dtype=bool) > exclbool[exclude] = True > # do replacement > ones = vals==1.0 > # Note: ~ is numpy.logical_not > vals[ones & (~exclbool)] = 1e-20 > > I think you'll have to convert your HDF array into a numpy array first, > using numpy.array(). > > Jeremy Dear all, I tried the sorted python list, but this did not really help the runtime. I haven´t had time to check the sorted collections. I solved my runtime problem by using the script from Jeremy up here. It was a life saviour and it is amazing how powerful numpy is. Thanks a lot Jeremy for this. By the way, I did not have to do any array conversion. The array read from hdf5 file using h5py is already a numpy array. The runtime over an array of around 16M reduced from around 12 hours (previous script) to 3 seconds using numpy on the same machine. Thanks alot for your help, -- https://mail.python.org/mailman/listinfo/python-list
Porting Python Application to a new linux machine
Dear all, I have my python scripts that use several python libraries such as h5py, pyside, numpy In Windows I have an installer that will install python locally on user machine and so my program gets access to this local python and runs successfully. How can I do this in Linux ? ( I want to install python plus my program on the user machine.) I do not want to use the user´s python or to install python on the user´s machine on root. Thanks in Advance for your help, -- https://mail.python.org/mailman/listinfo/python-list
PyInstaller+ Python3.5 (h5py import error)
Dear all, Thanks a lot for your replies. Very helpful. I have already done some trials with Virtualenv, but PyInstaller is much closer to the idea of an installer you can pass to someone. I have been using development version of PyInstaller in order to be able to use it with my script written with Python versin 3.3.5. I started with a very simple script just to test. I use the following command to create the distribution folder. pyinstaller test.py my script contains the following few lines and it runs ok on my own machine. import numpy as np import h5py a=np.arange(10) print(a) inputFiles="test.h5" with h5py.File(inputFiles, 'w') as inputFileOpen: pass I am getting the following error related to importing h5py. test returned -1 Traceback (most recent call last): File "", line 2, in File "/usr/lib/python3.3/site-packages/PyInstaller-3.0.dev2-py3.3.egg/PyInstaller/loader/pyimod03_importers.py", line 311, in load_module File "/usr/lib64/python3.3/site-packages/h5py/__init__.py", line 23, in File "/usr/lib/python3.3/site-packages/PyInstaller-3.0.dev2-py3.3.egg/PyInstaller/loader/pyimod03_importers.py", line 493, in load_module File "h5r.pxd", line 21, in init h5py._conv (/tmp/pip_build_root/h5py/h5py/_conv.c:6563) File "/usr/lib/python3.3/site-packages/PyInstaller-3.0.dev2-py3.3.egg/PyInstaller/loader/pyimod03_importers.py", line 493, in load_module File "_objects.pxd", line 12, in init h5py.h5r (/tmp/pip_build_root/h5py/h5py/h5r.c:2708) File "/usr/lib/python3.3/site-packages/PyInstaller-3.0.dev2-py3.3.egg/PyInstaller/loader/pyimod03_importers.py", line 493, in load_module File "_objects.pyx", line 1, in init h5py._objects (/tmp/pip_build_root/h5py/h5py/_objects.c:6407) ImportError: No module named 'h5py.defs' If I modify my script to import numpy as np import h5py a=np.arange(10) print(a) then, the created exectuable will run successfully on other linux machines. Does anybody have any idea why I am getting the following h5py import error? My spec file also looks like this: # -*- mode: python -*- block_cipher = None a = Analysis(['test.py'], pathex=['/home/albert/test'], binaries=None, datas=None, hiddenimports=[], hookspath=None, runtime_hooks=None, excludes=None, win_no_prefer_redirects=None, win_private_assemblies=None, cipher=block_cipher) pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher) exe = EXE(pyz, a.scripts, exclude_binaries=True, name='test', debug=False, strip=None, upx=True, console=True ) coll = COLLECT(exe, a.binaries, a.zipfiles, a.datas, strip=None, upx=True, name='test') Thank you very much in Advance for your help, -- https://mail.python.org/mailman/listinfo/python-list
Re: PyInstaller+ Python3.5 (h5py import error)
Thanks Christian, It turned out that h5py.defs was not the only hidden import that I needed to add. I managed to get it working with the follwoing command adding 4 hidden imports. pyinstaller --hidden-import=h5py.defs --hidden-import=h5py.utils --hidden-import=h5py.h5ac --hidden-import=h5py._proxy test.py is there anyway that you can use to add all h5py submodules all together? Thanks, -- https://mail.python.org/mailman/listinfo/python-list
