I'm submitting the job via an operator provided by Google Cloud Platform. Here's a rough sample showing how I do it - though this is just with the wordcount sample: https://github.com/THEjoezack/beam-on-spark-on-kubernetes
The driver shows as "Completed" before it starts again, but I'll dig deeper since it sounds fishy to you. :) On Mon, Jun 8, 2020 at 4:53 PM Kyle Weaver <kcwea...@google.com> wrote: > > There is no error > > Are you sure? That sounds like a crash loop to me. It might take some > digging through various Kubernetes logs to find the cause. > > Can you provide more information about how you're running the job? > > On Mon, Jun 8, 2020 at 1:50 PM Joseph Zack <joseph.z...@broadcom.com> > wrote: > >> Anybody out there running Beam on Spark? >> >> I am pulling data from a Kafka topic with KafkaIO, but the job keeps >> restarting. There is no error, it just.... >> >> 1. creates the driver >> 2. creates the executors >> 3. runs for a few seconds >> 4. terminates the executors >> 5. terminates the driver >> 6. goto #1 >> >> I'm new to Beam, and completely new to Spark so I'm not sure how it's >> supposed to work. Is this expected behavior? I expected the Beam job to run >> continuously. Either I'm missing a setting, or I'm misunderstanding how >> things are supposed to work. >> >> Thanks for your consideration! >> >> -- >> >> Joseph Zack >> Software Engineer | Information Security Group | Symantec Enterprise >> Division >> Broadcom >> >> mobile: 407.920.4930 <(407)%20920-4930> >> >> joseph.z...@broadcom.com | broadcom.com >> > -- Joseph Zack Software Engineer | Information Security Group | Symantec Enterprise Division Broadcom mobile: 407.920.4930 joseph.z...@broadcom.com | broadcom.com
smime.p7s
Description: S/MIME Cryptographic Signature
