Spark Structured Streaming - unable to change max.poll.records (showing as 1)

Hello All, i've a Spark structured streaming job which reads from Kafka, does processing and puts data into Mongo/Kafka/GCP Buckets (i.e. it is processing heavy) I'm consistently seeing the following warnings: ``` 22/09/06 16:55:03 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordi

Re: Jupyter notebook on Dataproc versus GKE

I’ve used Argo for K8s scheduling, for awhile it’s also what Kubeflow used underneath for scheduling. On Tue, Sep 6, 2022 at 10:01 AM Mich Talebzadeh wrote: > Thank you all. > > Has anyone used Argo for k8s scheduler by any chance? > > On Tue, 6 Sep 2022 at 13:41, Bjørn Jørgensen > wrote: > >>

Re: Jupyter notebook on Dataproc versus GKE

Thank you all. Has anyone used Argo for k8s scheduler by any chance? On Tue, 6 Sep 2022 at 13:41, Bjørn Jørgensen wrote: > "*JupyterLab is the next-generation user interface for Project Jupyter > offering all the familiar building blocks of the classic Jupyter Notebook > (notebook, terminal, te

[ANNOUNCE] Apache Kyuubi (Incubating) released 1.6.0-incubating

Hi all, The Apache Kyuubi (Incubating) community is pleased to announce that Apache Kyuubi (Incubating) 1.6.0-incubating has been released! Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for large-scale data processing and analytics, build on top of multiple compule e

Re: Jupyter notebook on Dataproc versus GKE

"*JupyterLab is the next-generation user interface for Project Jupyter offering all the familiar building blocks of the classic Jupyter Notebook (notebook, terminal, text editor, file browser, rich outputs, etc.) in a flexible and powerful user interface.*" https://github.com/jupyterlab/jupyterlab

Re: Error in Spark in Jupyter Notebook

That just says a task failed - no real info there. YOu have to look at Spark logs from the UI to see why. On Tue, Sep 6, 2022 at 7:07 AM Mamata Shee wrote: > Hello, > > I'm using spark in Jupyter Notebook, but when performing some queries > getting the below error, can you please tell me what is

Error in Spark in Jupyter Notebook

Hello, I'm using spark in Jupyter Notebook, but when performing some queries getting the below error, can you please tell me what is the actual reason for this or any suggestions to make it work? *Error:* [image: image.png] Thank you -- CONFIDENTIALITY NOTICE: