Replying here just for documentation purposes (so anyone finding the thread knows it).
This was sorted out on toolforge (for the internal tricks used see https://phabricator.wikimedia.org/T363417), though it's an upstream issue so the "final" fix will come in some time. On 04/30 13:52, Bryan Davis wrote: > On Tue, Apr 30, 2024 at 12:26 PM Sascha Brawer via Cloud > <cloud@lists.wikimedia.org> wrote: > > > > Dear all, > > > > Has anyone figured out how to use Toolforge Build Service for a tool (job + > > webservice) written in golang? > > I got <https://gitlab.wikimedia.org/toolforge-repos/bridgebot> working > last week, and in doing so I struggled with the same things you are > reporting today. > > > I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but > > I'm struggling with it. > > To debug this, I wrote a minimal job that just prints a hello message to > > stdout: > > https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go > > > > Because Toolforge is emulating Heroku, it appears necessary to tell > > buildpack which binaries should get packaged into the container. The "// > > +heroku install" comment on line 14 of the following go.mod file seems to > > do the trick, together with listing the binaries again in a Heroku Procfile: > > https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 > > https://github.com/brawer/wikidata-qrank/blob/main/Procfile > > > > When running the following command on login.toolforge.org, the project > > seems to be built just fine: > > $ toolforge build start https://github.com/brawer/wikidata-qrank > > > > Some interesting log message from build service: > > [step-build] 2024-04-30T18:13:31.170521374Z Building packages: > > [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello > > [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder > > [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver > > [step-build] 2024-04-30T18:13:44.563334902Z > > [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] > > [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: > > [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello > > [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder > > [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver > > [step-build] 2024-04-30T18:13:44.563488376Z - web: hello > > [step-build] 2024-04-30T18:13:44.577068596Z > > [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] > > [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, > > qrank-builder, hello > > > > The "web: hello" in the launch table looks very suspicious; it's not what > > my Procfile states (see link above). > > I noticed similar behavior when building my project. My sensemaking is > that the golang buildpack is configured to always generate a "web" > process because of Heroku's internal needs. In our stack Profile > processing is done by a separate buildpack that the golang buildpack > is not aware of. > > > But at least the build seems to have been successful: > > > > $ toolforge build show > > Build ID: qrank-buildpacks-pipelinerun-vhcrt > > Start Time: 2024-04-30T18:13:03Z > > End Time: 2024-04-30T18:14:00Z > > Status: ok > > Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 > > Parameters: > > Source URL: https://github.com/brawer/wikidata-qrank > > Ref: N/A > > Envvars: N/A > > Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest > > > > Now, I'd be happy to just run the "hello" job and see its output log. > > However, when I do this: > > $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest > > --mount=all --filelog > > > > The job runs for about five minutes (?!?!), and then: > > $ cat hello.out > > ERROR: failed to launch: bash exec: argument list too long > > > > So clearly something is off... but what? How to run a golang tool on > > Toolforge when using Build Service? > > I ran into this behavior as well and reported it as > <https://phabricator.wikimedia.org/T363417>. The workaround that I was > able to find so I could move forward was to create a very slim shell > script wrapper to launch my process from the Procfile: > <https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/commit/f05aa13d2309a55d5db286e7a38da45facb53b8a> > > In your case I think that you might be able to just put > `/layers/heroku_go/go_target/bin/hello` as the command you want to run > in either the Procfile or as the `--command` argument to `toolforge > jobs`. > > You can use `webservice --backend=kubernetes buildservice shell` to > get an interactive shell inside of a container running the image you > have built. This can be useful in debugging things. We have another > task in the backlog about making this shell functionality a bit more > discoverable now that we also have need for it with the jobs framework > (<https://phabricator.wikimedia.org/T311917>). > > I hope that all helps a bit. It is frustrating to be using new tools > and run into such troubles. Thank you for taking the time to look for > help on the mailing list though! Sharing our troubles and wins with > each other helps build our community. > > Bryan > -- > Bryan Davis Wikimedia Foundation > Principal Software Engineer Boise, ID USA > [[m:User:BDavis_(WMF)]] irc: bd808 > _______________________________________________ > Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
signature.asc
Description: PGP signature
_______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
