On Thu, Jul 28, 2022 at 5:37 PM Chamikara Jayalath via dev <dev@beam.apache.org> wrote: > > On Thu, Jul 28, 2022 at 4:51 PM Lina Mårtensson <lina@camus.energy> wrote: >> >> Thanks for the detailed answers! >> >> I totally get the points about development & maintenance cost, and, >> from a user perspective, about getting the performance right. >> >> I decided to try out the Spanner connector to get a sense of how well >> the x-language approach works in our world, since that's an existing >> x-language connector. >> Overall, it works and with minimal intervention as you say - it is >> very slow, though. >> I'm a little confused about "portable runners" - if I understand this >> correctly, this means we couldn't run with the DirectRunner anymore if >> using an x-language connector? (At least it didn't work when I tried >> it.) > > > You'll have to use the portable DirectRunner - > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/portability > > Job service for this can be started using following command: > python apache_beam/runners/portability/local_job_service_main.py -p <port>
Note that the Python direct runner is already a portable runner, so you shouldn't have to do anything special (like start up a separate job service and pass extra options) to run locally. Just use the cross-language transforms as you would any normal Python transform. The goal is to make this as smooth and transparent as possible; please keep coming back to us if you find rough edges.
