It looks like anaconda assumes that the directory doesn't exist before installation. I would recommend using a new directory such as ["/opt/userowned/anaconda.sh", "-b","-p","/opt/userowned/anaconda/"] but if you really want it to be in "/opt/userowned", you should add "-f": ["/opt/userowned/anaconda.sh","-f","-b","-p","/opt/userowned/"]
See the Anaconda silent install[1] instructions for more details. 1: https://docs.anaconda.com/anaconda/install/silent-mode/#linux-macos On Fri, Apr 17, 2020 at 9:28 PM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Thank you. I was able to tar my libraries at the /opt/userowned fodler. > I am using setup.py from this url (recommended your apache-beam > documentation) > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py > > Last, I want to install anaconda. I have a library with dependencies that > I rather have anaconda take care of. using the custom commands, I am > executing the following commands. the first two are running fine, the last > one, fires and error: > ["wget", "-O", "/opt/userowned/anaconda.sh"," > https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh";], > ['echo', 'Change conda permissions'], > ["chmod", "777", "/opt/userowned/anaconda.sh"], > ["/opt/userowned/anaconda.sh", "-b","-p","/opt/userowned/"] > > the last command failed with the following error: > Command output: b"ERROR: File or directory already exists: > '/opt/userowned/'\nIf you want to update an existing installation, use the > -u option.\n" > > Please let me know if you have any recommendations on how to install > anaconda > > Many thanks, > Eila > > On Fri, Apr 17, 2020 at 12:12 PM Luke Cwik <lc...@google.com> wrote: > >> On Dataflow you should be able to use /opt/userowned >> >> On Fri, Apr 17, 2020 at 9:01 AM OrielResearch Eila Arich-Landkof < >> e...@orielresearch.org> wrote: >> >>> See inline >>> >>> >>> — >>> Eila >>> www.orielesearch.com >>> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/> >>> p.co <https://www.meetup.com/Deep-Learning-In-Production/> >>> m/Deep-Learning-In-Production >>> <https://www.meetup.com/Deep-Learning-In-Production/> >>> >>> Sent from my iPhone >>> >>> On Apr 17, 2020, at 11:32 AM, Luke Cwik <lc...@google.com> wrote: >>> >>> >>> When you said you checked '/usr/local/', did you check inside the docker >>> container or on the VM itself? >>> >>> Yes. Checked as part of the DoFn function that runs on the Docker >>> >>> Have you tried adding an echo command before and after your script runs >>> and looked for them in the stackdriver logs? >>> >>> No. I see the name of the setup.py that I have in the logs >>> I can add the echo command as well >>> >>> It should help you locate any errors that might have happened when >>> executing the custom commands. >>> >>> Will try that. Is it correct to target to this folder. Any other folder >>> is ‘dedicated’ for custom downloads? >>> >>> Thanks >>> Eila >>> >>> >>> On Thu, Apr 16, 2020 at 6:30 PM OrielResearch Eila Arich-Landkof < >>> e...@orielresearch.org> wrote: >>> >>>> Hi all, >>>> >>>> This is a question that I have post on user-h...@apache.org. in case, >>>> the other one is not a valid address. posting here again. if you have >>>> already received it, apologies for the spam. >>>> >>>> I hope that you are all well. I would like to copy a tools library into >>>> the worker machines and uses the setup.py file. I have update the >>>> CUSTOM_COMMANDS: >>>> >>>> CUSTOM_COMMANDS = [ >>>> ["wget", "-O", >>>> "/usr/local/sratoolkit.tar.gz","http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-centos_linux64.tar.gz";], >>>> ["tar", "-xzf", "/usr/local/sratoolkit.tar.gz","-C","/usr/local/"]] >>>> >>>> When I am looking for the execution files at the /user/local folder, I >>>> can not find the tools that I have copied to the worker. What is the right >>>> & easiest way to copy tools libraries into the worker machines? I was able >>>> to get the expected behavior of the local runner and now it's a matter of >>>> finding the right way to reproduce it for the dataflow runner. >>>> >>>> I am using python 3.5 with the latest apache-beam 2.20 and latest >>>> dataflow Thanks a lot, eilalan >>>> -- >>>> Eila >>>> <http://www.orielresearch.com> >>>> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >>>> >>> > > -- > Eila > <http://www.orielresearch.com> > Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >
