Re: Pyspark: Issue using sql in foreachBatch sink

2020-08-03 Thread muru
Thanks Jungtaek for your help. On Fri, Jul 31, 2020 at 6:31 PM Jungtaek Lim wrote: > Python doesn't allow abbreviating () with no param, whereas Scala does. > Use `write()`, not `write`. > > On Wed, Jul 29, 2020 at 9:09 AM muru wrote: > >> In a pyspark SS job, trying to use sql instead of sql f

Re: Pyspark: Issue using sql in foreachBatch sink

2020-07-31 Thread Jungtaek Lim
Python doesn't allow abbreviating () with no param, whereas Scala does. Use `write()`, not `write`. On Wed, Jul 29, 2020 at 9:09 AM muru wrote: > In a pyspark SS job, trying to use sql instead of sql functions in > foreachBatch sink > throws AttributeError: 'JavaMember' object has no attribute '

Pyspark: Issue using sql in foreachBatch sink

2020-07-28 Thread muru
In a pyspark SS job, trying to use sql instead of sql functions in foreachBatch sink throws AttributeError: 'JavaMember' object has no attribute 'format' exception. However, the same thing works in Scala API. Please note, I tested in spark 2.4.5/2.4.6 and 3.0.0 and got the same exception. Is it a