Howdy Kenneth,
On 07/11/2018 03:59 PM, Kenneth Hoste wrote:
Hi Jack,
On 11/07/2018 20:04, Jack Perdue wrote:
Howdy Kenneth,
For example, I'm trying to build IPython and as
a Bundle with extensions trying to fetch files from:
https://files.pythonhosted.org/packages/source/i/ipython/
which doesn't exist.
That's not what I'm seeing with the just released EasyBuild v3.6.2
(this behavior hasn't changed compared to EasyBuild v3.6.1).
Here's a relevant bit from the EasyBuild log that shows it's
downloading directly from https://pypi.python.org/packages (not via
https://files.pythonhosted.org/packages).
== 2018-07-11 22:53:46,457 filetools.py:471 INFO Downloaded file
ipython_urls.html from url https://pypi.python.org/simple/ipython to
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:46,457 filetools.py:491 INFO Successful download
of file ipython_urls.html from url
https://pypi.python.org/simple/ipython to path
/Users/kehoste/work/TMP/eb-_E1vS6/tmppmy5pk/ipython_urls.html
== 2018-07-11 22:53:51,753 filetools.py:471 INFO Downloaded file
ipython-6.2.1.tar.gz from url
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
to /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 filetools.py:491 INFO Successful download
of file ipython-6.2.1.tar.gz from url
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
to path /tmp/i/IPython/extensions/ipython-6.2.1.tar.gz
== 2018-07-11 22:53:51,753 easyblock.py:730 INFO Successfully
downloaded source file ipython-6.2.1.tar.gz from
https://pypi.python.org/packages/fa/50/974211502bd72873728d44c3013fe79875c819c8fb69f778bcfd67bc7d38/ipython-6.2.1.tar.gz
That's not what I am seeing here. I have:
('ipython', version, {
'source_urls': [PYPI_LOWER_SOURCE],
'modulename': 'IPython',
'checksums': [
'eca537aa61592aca2fef4adea12af8e42f5c335004dfa80c78caf80e8b525e5c', #
ipython-6.4.tar.gz
],
}),
and this is in the log:
== 2018-07-12 13:22:45,782 filetools.py:444 DEBUG Trying to download
%(namelower)s_urls.html from
https://pypi.python.org/simple/%(namelower)s to
/tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22/%(namelower)s_urls.html
== 2018-07-12 13:22:45,782 filetools.py:451 DEBUG Using timeout of 10
seconds for initiating download
== 2018-07-12 13:22:45,782 filetools.py:1136 DEBUG Not creating existing
path /tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22
== 2018-07-12 13:22:45,917 filetools.py:476 WARNING URL
https://pypi.python.org/simple/%(namelower)s was not found (HTTP
response code 400), not trying again
== 2018-07-12 13:22:45,917 filetools.py:494 WARNING Download of
https://pypi.python.org/simple/%(namelower)s to
/tmp/j-perdue/tmp/eb-3gbeni/tmpHM8L22/%(namelower)s_urls.html failed,
done trying
== 2018-07-12 13:22:45,918 filetools.py:400 DEBUG Failed to download
https://pypi.python.org/simple/%(namelower)s to determine available PyPI
URLs for %(namelower)s
== 2018-07-12 13:22:45,918 filetools.py:436 DEBUG Failed to extract hash
using pattern
'.*/ipython-6\.4\.tar\.gz(?:#md5=[a-f0-9]{32}|#sha256=[a-f0-9]{64})$'
from list of URLs: []
== 2018-07-12 13:22:45,918 easyblock.py:707 DEBUG Failed to derive
alternate PyPI URL for
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz,
so retaining the original
== 2018-07-12 13:22:45,918 easyblock.py:717 DEBUG Trying to download
file ipython-6.4.tar.gz from
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz ...
== 2018-07-12 13:22:45,918 filetools.py:444 DEBUG Trying to download
ipython-6.4.tar.gz from
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz
== 2018-07-12 13:22:45,918 filetools.py:451 DEBUG Using timeout of 10
seconds for initiating download
== 2018-07-12 13:22:45,919 filetools.py:1136 DEBUG Not creating existing
path /sw/eb/sources/i/IPython/extensions
== 2018-07-12 13:22:46,050 filetools.py:476 WARNING URL
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz
was not found (HTTP response code 400), not trying again
== 2018-07-12 13:22:46,051 filetools.py:494 WARNING Download of
https://pypi.python.org/packages/source/%(nameletterlower)s/%(namelower)s/ipython-6.4.tar.gz
to /sw/eb/sources/i/IPython/extensions/ipython-6.4.tar.gz failed, done
trying
Any ideas why that isn't getting expanded properly? Your's seems
to have worked.
jack
See:
IPython-6.2.1-foss-2017a-Python-3.6.4.eb
(which is IPython-6.4-foss-2017b-Python-3.6.3.eb here).
I saw some issues in framework covering problems
with the templates in extensions. Should
I change all the hard-coded dirs for extensions
as just PYPI_SOURCE and let EB handle the rest?
PYPI_SOURCE probably isn't going to work with extensions...
But if you use source URLs like
https://pypi.python.org/packages/source/i/ipython/, EasyBuild will
automagically translate them to the correct hash URL via
https://pypi.python.org/simple ...
regards,
Kenneth
Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
[email protected] http://hprc.tamu.edu
HPRC Helpdesk: [email protected]
On 07/11/2018 12:50 PM, Kenneth Hoste wrote:
Hi Jack,
On 11/07/2018 01:38, Jack Perdue wrote:
As recently as a week or two ago, the "legacy"
PyPi site was still serving files from old source_urls.
That no longer seems to be the case.
And the new URLs seem to be a random mess.
Has anyone pondered how to deail with this?
EasyBuild has dealt with this for a while (since v2.8.0), by
auto-translating PyPI URLs, see
https://github.com/easybuilders/easybuild-framework/pull/1749 .
Are you saying that functionality is broken?
Seems to be working fine (we also check this in the framework test
suite, of course):
$ python -c "from test.framework.utilities import init_config; from
easybuild.tools.filetools import derive_alt_pypi_url; init_config();
print
derive_alt_pypi_url('https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz')"
INFO: This is (based on) vsc.install.shared_setup 0.11.2
https://pypi.python.org/packages/5b/03/e135b19fadeb9b1ccb45eac9f60ca2dc3afe72d099f6bd84e03cb131f9bf/easybuild-2.7.0.tar.gz
The source tarball can be downloaded using "curl -OL" just fine...
regards,
Kenneth
