[PySpark] Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Fri, 06 Jan 2023 07:00:43 -0800 

     Hello,

  I'm trying to install SciPy using a bootstrap script and then use it to
calculate a new field in a dataframe, running on AWS EMR.

  Although the SciPy website states that only NumPy is needed, when I tried
to install SciPy using pip, pip kept failing, complaining about missing
software, until I ended up with this bootstrap script:






*sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
numpysudo pip3 install -U scipy*

  At this point, the bootstrap seems to be successful, but then at this
line:

*from scipy.stats import norm*

  I get the following error:

*ValueError: numpy.ndarray size changed, may indicate binary
incompatibility. Expected 88 from C header, got 80 from PyObject*

  Any advice on how to proceed? Thanks!

     Best, Oliver

-- 
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>

Reply via email to