I did try streamparse, but it failed to deploy the topology to the remote Storm cluster because of some internal errors which I wasn't able to track down. I am not surprised though, because even manual install of python 2.7 or virtual environment is painful on CentOS 6.5. So, I took the topology JAR produced by Streamparse and tried to manually deploy on Storm, but ran into the Python issues that I am trying to resolve in this thread.
Thanks, Alec On Mon, May 2, 2016 at 8:40 PM, Jiaming Lin <[email protected]> wrote: > You can try streamparse, I haven't used it, but it looks charming. > > > On Tuesday, May 3, 2016, Alec Swan <[email protected]> wrote: > >> Thanks, Joaquin, but I've seen all these articles since I spent the >> entire weekend trying to get my topology to run on CentOS/Python2.7. >> Getting it to work on Mac OSX was a breeze because I can easily install >> Python 2.7 on it. >> >> It'd be great if somebody with the actually CentOS/Python2.7 experience >> could share their thoughts. >> >> Thanks, >> >> Alec >> >> On Mon, May 2, 2016 at 7:59 PM, Joaquin Menchaca <[email protected]> >> wrote: >> >>> virtualenvwrappers >>> - http://virtualenvwrapper.readthedocs.io/en/latest/ >>> - https://pypi.python.org/pypi/virtualenvwrapper/ >>> Pyenv >>> - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial >>> - https://github.com/yyuu/pyenv >>> Docker >>> >>> You can have a Python in another directory, then put that directory in >>> the path. Maybe linuxbrew can help with that. >>> >>> Linuxbrew - http://linuxbrew.sh >>> >>> Disclaimer, I haven't tested any of these with Storm. I personally >>> don't use CentOS (or RHEL) any more, because they are dinosaurs when it >>> comes to scripting languages and web development. But many love the slow >>> churn and stability. >>> >>> I also found these: >>> - >>> https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only >>> - >>> https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4 >>> - http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/ >>> On May 2, 2016 3:35 PM, "Alec Swan" <[email protected]> wrote: >>> >>>> Hello, >>>> >>>> I am having a real hard time configuring Storm to use python2.7 virtual >>>> environment that I installed on my CentOS 6.5 host. >>>> >>>> CentOS 6.5 ships with python2.6 but storm requires at least python2.7. >>>> Switching to python2.7 breaks some of CentOS functionality, so I had to >>>> install python2.7 as a separate environment. >>>> >>>> I also read that if I set BASH_ENV env var to point to the script that >>>> activates the virtual environment, then it will be loaded before scripts >>>> are executed. I also changed the first line in storm.py to: >>>> #!/usr/bin/env python >>>> so that the right python executable is being used. However, I still >>>> have no luck and keep getting the error shown below after topology is >>>> successfully submitted. >>>> >>>> Has anyone figured out how to use Storm with python virtual >>>> environments? >>>> >>>> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died! >>>> java.lang.RuntimeException: Error when launching multilang subprocess >>>> python: error while loading shared libraries: libpython2.7.so.1.0: >>>> cannot open shared object file: No such file or directory >>>> >>>> at >>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at >>>> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) >>>> [storm-core-0.10.0.jar:0.10.0] >>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] >>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73] >>>> Caused by: java.io.IOException: Broken pipe >>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>> ~[?:1.8.0_73] >>>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>>> ~[?:1.8.0_73] >>>> at >>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >>>> ~[?:1.8.0_73] >>>> at >>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >>>> ~[?:1.8.0_73] >>>> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >>>> ~[?:1.8.0_73] >>>> at >>>> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at >>>> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at >>>> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> at >>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64) >>>> ~[storm-core-0.10.0.jar:0.10.0] >>>> ... 5 more >>>> >>>> >>>> Thanks, >>>> >>>> Alec >>>> >>> >>
