Re: Data source v2 streaming sinks does not support Update mode

Will do, thanks! On Tue, Jan 19, 2021 at 1:39 PM Gabor Somogyi wrote: > Thanks for double checking the version. Please report back with 3.1 > version whether it works or not. > > G > > > On Tue, 19 Jan 2021, 07:41 Eric Beabes, wrote: > >> Confirmed. The cluster Admin said his team installed the

Re: Data source v2 streaming sinks does not support Update mode

Thanks for double checking the version. Please report back with 3.1 version whether it works or not. G On Tue, 19 Jan 2021, 07:41 Eric Beabes, wrote: > Confirmed. The cluster Admin said his team installed the latest version > from Cloudera which comes with Spark 3.0.0-preview2. They are going

Re: Data source v2 streaming sinks does not support Update mode

Confirmed. The cluster Admin said his team installed the latest version from Cloudera which comes with Spark 3.0.0-preview2. They are going to try to upgrade it with the Community edition Spark 3.1.0. Thanks Jungtaek for the tip. Greatly appreciate it. On Tue, Jan 19, 2021 at 8:45 AM Eric Beabes

Re: Data source v2 streaming sinks does not support Update mode

>> "Could you please make sure you're not using "3.0.0-preview". This could be the reason. I will check with our Hadoop cluster administrator. It's quite possible that they installed the "Preview" mode. Yes, the code works in the Local dev environment. On Tue, Jan 19, 2021 at 5:29 AM Jungtaek Li

Re: Data source v2 streaming sinks does not support Update mode

I see no issue from running this code in local dev. (changed the scope of Spark artifacts to "compile" of course) Could you please make sure you're not using "3.0.0-preview"? In 3.0.0-preview update mode was restricted (as the error message says) and it was reverted before releasing Spark 3.0.0. C

Re: Data source v2 streaming sinks does not support Update mode

And also include some test data as well. I quickly looked through the code and the code may require a specific format of the record. On Tue, Jan 19, 2021 at 12:10 AM German Schiavon wrote: > Hi, > > This is the jira and > regarding the repo

Re: Data source v2 streaming sinks does not support Update mode

Hi, This is the jira and regarding the repo, I believe just commit it to your personal repo and that should be it. Regards On Mon, 18 Jan 2021 at 15:46, Eric Beabes wrote: > Sorry. Can you please tell me where to create the JIRA? Also is

Re: Data source v2 streaming sinks does not support Update mode

Sorry. Can you please tell me where to create the JIRA? Also is there any specific Github repository I need to commit code into - OR - just in our own? Please let me know. Thanks. On Mon, Jan 18, 2021 at 7:07 PM Gabor Somogyi wrote: > Thanks you, as we've asked could you please create a jira and

Re: Data source v2 streaming sinks does not support Update mode

Thanks you, as we've asked could you please create a jira and commit the code into github? It would speed things up a lot. G On Mon, Jan 18, 2021 at 2:14 PM Eric Beabes wrote: > Here's a very simple reproducer app. I've attached 3 files: > SparkTest.scala, QueryListener.scala & pom.xml. Copyin

Re: Data source v2 streaming sinks does not support Update mode

Here's a very simple reproducer app. I've attached 3 files: SparkTest.scala, QueryListener.scala & pom.xml. Copying contents in the email as well: package com.myorg import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.hadoop.security.UserGro

Re: Data source v2 streaming sinks does not support Update mode

Ok. I will work on creating a reproducible app. Thanks. On Wed, Jan 13, 2021 at 3:57 PM Gabor Somogyi wrote: > Just reached this thread. +1 on to create a simple reproducer app and I > suggest to create a jira attaching the full driver and executor logs. > Ping me on the jira and I'll pick this

Re: Data source v2 streaming sinks does not support Update mode

Just reached this thread. +1 on to create a simple reproducer app and I suggest to create a jira attaching the full driver and executor logs. Ping me on the jira and I'll pick this up right away... Thanks! G On Wed, Jan 13, 2021 at 8:54 AM Jungtaek Lim wrote: > Would you mind if I ask for a s

Re: Data source v2 streaming sinks does not support Update mode

Would you mind if I ask for a simple reproducer? Would be nice if you could create a repository in Github and push the code including the build script. Thanks in advance! On Wed, Jan 13, 2021 at 3:46 PM Eric Beabes wrote: > I tried both. First tried 3.0.0. That didn't work so I tried 3.1.0. > >

Re: Data source v2 streaming sinks does not support Update mode

I tried both. First tried 3.0.0. That didn't work so I tried 3.1.0. On Wed, Jan 13, 2021 at 11:35 AM Jungtaek Lim wrote: > Which exact Spark version did you use? Did you make sure the version for > Spark and the version for spark-sql-kafka artifact are the same? (I asked > this because you've sa

Re: Data source v2 streaming sinks does not support Update mode

Which exact Spark version did you use? Did you make sure the version for Spark and the version for spark-sql-kafka artifact are the same? (I asked this because you've said you've used Spark 3.0 but spark-sql-kafka dependency pointed to 3.1.0.) On Tue, Jan 12, 2021 at 11:04 PM Eric Beabes wrote:

Re: Data source v2 streaming sinks does not support Update mode

Hi, Can you post the whole message? I'm trying to find what might be causing it. A small reproducible example would be of help too. Thank you. Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.c