DataSourceV2 community sync #3

Hi everyone, I just sent out an invite for the next DSv2 community sync for Wednesday, 28 Nov at 5PM PST. We have a few topics left over from last time to cover. A few people wanted to cover catalog APIs, so I put two items on the agenda: - The TableCatalog proposal (and other catalog APIs)

Re: Automated formatting

That seems like a good first step. Opened a PR / jira ticket with that approach at https://github.com/apache/spark/pull/23148 If anyone tests this and finds a file that doesn't format well (e.g. fails scalastyle afterwards) just let me know, happy to tweak scalafmt config options. On Thu, Nov 22

Re: [Spark SQL]: Does Spark SQL 2.3+ suppor UDT?

Thanks and Regards, Suny Tyagi Phone No : 9885027192 On Mon, Nov 26, 2018 at 10:31 PM Suny Tyagi wrote: > Hi Team, > > > I was going through this ticket > https://issues.apache.org/jira/browse/SPARK-7768?jql=text%20~%20%22public%20udt%22 > and > could not understand that if spark support UDT i

[SPARK-26160] Make assertNotBucketed call in DataFrameWriter::save optional

Hello, I have a proposal for a small improvement in the Datasource API and I'd like to know if it sounds like a change the Spark project would accept. Currently, the `.save` method in DataFrameWriter will fail if the dataframe is bucketed and/or sorted. This makes sense, since there is no way o

Re: [SS] FlatMapGroupsWithStateExec with no commitTimeMs metric?

Thanks Jungtaek Lim! Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twit