Getting back to this, as far as I can tell the job is exiting without error, even though it sees an unbounded dataset.
Below is a link to the full logs for the job run: https://pastebin.com/Rh6vTqWU The rough steps: * I submit the job via the GCP spark operator <https://github.com/GoogleCloudPlatform/spark-on-k8s-operator> * The driver spins up and creates executors * It connects to a Kafka topic * Job completes and shuts down Am I missing something? Is this what is supposed to happen? [image: image.png] On Mon, Jun 8, 2020 at 4:56 PM Joseph Zack <joseph.z...@broadcom.com> wrote: > I'm submitting the job via an operator provided by Google Cloud Platform. > Here's a rough sample showing how I do it - though this is just with the > wordcount sample: > https://github.com/THEjoezack/beam-on-spark-on-kubernetes > > The driver shows as "Completed" before it starts again, but I'll dig > deeper since it sounds fishy to you. :) > > On Mon, Jun 8, 2020 at 4:53 PM Kyle Weaver <kcwea...@google.com> wrote: > >> > There is no error >> >> Are you sure? That sounds like a crash loop to me. It might take some >> digging through various Kubernetes logs to find the cause. >> >> Can you provide more information about how you're running the job? >> >> On Mon, Jun 8, 2020 at 1:50 PM Joseph Zack <joseph.z...@broadcom.com> >> wrote: >> >>> Anybody out there running Beam on Spark? >>> >>> I am pulling data from a Kafka topic with KafkaIO, but the job keeps >>> restarting. There is no error, it just.... >>> >>> 1. creates the driver >>> 2. creates the executors >>> 3. runs for a few seconds >>> 4. terminates the executors >>> 5. terminates the driver >>> 6. goto #1 >>> >>> I'm new to Beam, and completely new to Spark so I'm not sure how it's >>> supposed to work. Is this expected behavior? I expected the Beam job to run >>> continuously. Either I'm missing a setting, or I'm misunderstanding how >>> things are supposed to work. >>> >>> Thanks for your consideration! >>> >>> -- >>> >>> Joseph Zack >>> Software Engineer | Information Security Group | Symantec >>> Enterprise Division >>> Broadcom >>> >>> mobile: 407.920.4930 <(407)%20920-4930> >>> >>> joseph.z...@broadcom.com | broadcom.com >>> >> > > -- > > Joseph Zack > Software Engineer | Information Security Group | Symantec Enterprise > Division > Broadcom > > mobile: 407.920.4930 > > joseph.z...@broadcom.com | broadcom.com > -- Joseph Zack Software Engineer | Information Security Group | Symantec Enterprise Division Broadcom mobile: 407.920.4930 joseph.z...@broadcom.com | broadcom.com
smime.p7s
Description: S/MIME Cryptographic Signature
