When you said you checked '/usr/local/', did you check inside the docker container or on the VM itself?
Have you tried adding an echo command before and after your script runs and looked for them in the stackdriver logs? It should help you locate any errors that might have happened when executing the custom commands. On Thu, Apr 16, 2020 at 6:30 PM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi all, > > This is a question that I have post on user-h...@apache.org. in case, the > other one is not a valid address. posting here again. if you have > already received it, apologies for the spam. > > I hope that you are all well. I would like to copy a tools library into > the worker machines and uses the setup.py file. I have update the > CUSTOM_COMMANDS: > > CUSTOM_COMMANDS = [ > ["wget", "-O", > "/usr/local/sratoolkit.tar.gz","http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-centos_linux64.tar.gz";], > ["tar", "-xzf", "/usr/local/sratoolkit.tar.gz","-C","/usr/local/"]] > > When I am looking for the execution files at the /user/local folder, I can > not find the tools that I have copied to the worker. What is the right & > easiest way to copy tools libraries into the worker machines? I was able to > get the expected behavior of the local runner and now it's a matter of > finding the right way to reproduce it for the dataflow runner. > > I am using python 3.5 with the latest apache-beam 2.20 and latest dataflow > Thanks a lot, eilalan > -- > Eila > <http://www.orielresearch.com> > Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >
