On Dataflow you should be able to use /opt/userowned On Fri, Apr 17, 2020 at 9:01 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote:
> See inline > > > — > Eila > www.orielesearch.com > https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/> > p.co <https://www.meetup.com/Deep-Learning-In-Production/> > m/Deep-Learning-In-Production > <https://www.meetup.com/Deep-Learning-In-Production/> > > Sent from my iPhone > > On Apr 17, 2020, at 11:32 AM, Luke Cwik <lc...@google.com> wrote: > > > When you said you checked '/usr/local/', did you check inside the docker > container or on the VM itself? > > Yes. Checked as part of the DoFn function that runs on the Docker > > Have you tried adding an echo command before and after your script runs > and looked for them in the stackdriver logs? > > No. I see the name of the setup.py that I have in the logs > I can add the echo command as well > > It should help you locate any errors that might have happened when > executing the custom commands. > > Will try that. Is it correct to target to this folder. Any other folder is > ‘dedicated’ for custom downloads? > > Thanks > Eila > > > On Thu, Apr 16, 2020 at 6:30 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi all, >> >> This is a question that I have post on user-h...@apache.org. in case, >> the other one is not a valid address. posting here again. if you have >> already received it, apologies for the spam. >> >> I hope that you are all well. I would like to copy a tools library into >> the worker machines and uses the setup.py file. I have update the >> CUSTOM_COMMANDS: >> >> CUSTOM_COMMANDS = [ >> ["wget", "-O", >> "/usr/local/sratoolkit.tar.gz","http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-centos_linux64.tar.gz";], >> ["tar", "-xzf", "/usr/local/sratoolkit.tar.gz","-C","/usr/local/"]] >> >> When I am looking for the execution files at the /user/local folder, I >> can not find the tools that I have copied to the worker. What is the right >> & easiest way to copy tools libraries into the worker machines? I was able >> to get the expected behavior of the local runner and now it's a matter of >> finding the right way to reproduce it for the dataflow runner. >> >> I am using python 3.5 with the latest apache-beam 2.20 and latest >> dataflow Thanks a lot, eilalan >> -- >> Eila >> <http://www.orielresearch.com> >> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >> >
