Expected Release date for Spark 4?

2024-08-23 Thread Vikash Lavaniya
Hi, We've been working with Spark 4 preview for a while and would like to know about the release date of Spark 4 release. Will you be able to provide a planned release date for Spark 4 to helps us plan better? Regards, Vikash

Need support for vulnerability CVE-2023-39410

2024-08-23 Thread Ravichandran, Abhirami
Hi Team, We are using spark-sql 3.5.1 and hadoop-client-runtime 3.4.0 . There is a vulnerability(CVE-2023-39410) that has been reported in org.apache.avro_avro (1.9.2) version. Can you please let us know by when you are planning to release an updated version of spark-sql jar with the upgraded v

Re: Batch to Kafka

2024-08-23 Thread Rommel Holmes
Hi, community Right now i am using the batch to kafka in pyspark to send dataframe into kafka https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#writing-the-output-of-batch-queries-to-kafka The solution works with not so big dataframe. While the dataframe is big, si