Re: Passing "conf" arguments using a portable runner in Java (spark job runner)

Hi Jon, sorry for the late replay. A while ago I was struggling with as well. Unfortunately, there’s no direct way to do this per pipeline. However, you can set default arguments by passing them to the job service container using the environment variable _JAVA_OPTIONS. I hope this still helps!

Re: Re:

Hi Jon, I just want to check in here briefly, are you still looking for support on this? Sadly yes, this totally lacks documentation and isn’t straight forward to set up. /Moritz On 21.06.23, 23:47, "Jon Molle via user" wrote: Hi Pavel, Thanks for your response! I took a look at running Beam

Re: Re:

Hi Moritz, Yes, yes I am. I've gotten to the point where I've got a job service runner set up and am trying to get the spark and archive storage volume set up properly. I can only find vague references, but it seems like there needs to be an accessible shared drive between the spark cluster worke

Fetch Truststore File Inside a Flex Template image for Confluent Kafka

Hi, We are trying to store the truststore.jks file inside the Flex Template Docker but while using it in the pipeline we are unable to locate it. we tried pulling the image and we can see the file is present in the docker at \tmp\trust.jks but while using the same we get the following error. *E

Re: Growing checkpoint size with Python SDF for reading from Redis streams

Your SDF looks fine. I wonder if there is an issue with how Flink is implementing SDFs (e.g. not garbage collecting previous remainders). On Tue, Jul 18, 2023 at 5:43 PM Nimalan Mahendran wrote: > > Hello, > > I am running a pipeline built in the Python SDK that reads from a Redis > stream via a

EFO KinesisIO watermarking doubt

Hi, We are implementing EFO Kinesis IO reader provided by apache beam. I see that in code that for implementation of getCurrentTimestamp we always return getApproximateArrivalTimestamp and not the event time which we may have set for that record using withCustomWatermarkPolicy. Please refer: https

Can we use RedisIO to write records from an unbounded collection

Hi, I was planning to use the RedisIO write/writeStreams function in a streaming pipeline. https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/redis/RedisIO.html The pipeline would read an unbounded collection from Kinesis and update redis. It will update data for which key ex