ShuffleDataIO: where is the reading part of the API?

[Enrico Minack]

Hi devs,

the docs of org.apache.spark.shuffle.api.ShuffleDataIO read:

    An interface for plugging in modules for storing and reading
temporary shuffle data.

but the API does only provide interface for writing shuffle data:
- ShuffleExecutorComponents.createMapOutputWriter
- ShuffleExecutorComponents.createSingleFileMapOutputWriter

Where is the reading part of ShuffleDataIO?
How can I plug in implementations for reading shuffle data via
ShuffleDataIO?

I am aware of org.apache.spark.shuffle.ShuffleManager.

Cheers,
Enrico


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[Spark SQL] KafkaWriteTask: allow customising timestamp - PR

[Peter Fischer]

Hi!

We wrote a wrapper around the kafka writer to add client-side schema
validation. In the process we noticed that there was no way to change a
kafka record's timestamp when writing. So we extended spark-sql-kafka to
support it and would love to hear your feedback.

JIRA: https://issues.apache.org/jira/browse/SPARK-50160
GitHub: https://github.com/apache/spark/pull/48695

-- 
Peter Fischer (he/him)
Senior Software Engineer, Search Platform
Wikimedia Foundation

Re: [DISCUSS] Apache Spark 3.5 LTS Period (~ 2026 April)

[Dongjoon Hyun]

Here is a concrete PR for this.

- https://github.com/apache/spark-website/pull/565

Dongjoon

On 2024/10/29 17:49:12 Dongjoon Hyun wrote:
> Hi, All.
> 
> Thank you again everyone for making together
> Apache Spark 4.0.0-preview2 and Apache Spark 3.4.4 EOL releases.
> 
> I believe it's time to decide on the Apache Spark 3.5 LTS period.
> (Actually, there were several questions about this already.)
> 
> Apache Spark community has the written LTS versioning policy.
> 
> https://spark.apache.org/versioning-policy.html
> > The last minor release within a major release
> > will typically be maintained for longer as an “LTS” release.
> > For example, 2.4.0 was released in November 2nd 2018 and
> > had been maintained for 31 months
> 
> In line with our tradition, I would like to propose to update
> our website like the following to improve the LTS awareness.
> 
> > For example, 3.5.0 was released on September 13th 2023 and
> > will be maintained for 31 months until April 12th 2026.
> 
> FYI, the following is a summarized release timeline since Spark 1.
> 
> Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
> Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
> Spark 3: 2020.06 (3.0.0) ~ 2026.04 (3.5.x)
> Spark 4: 2025.02 (4.0.0 ~ )
> - January 15th 2025 Code freeze.
> - February 1st 2025 QA period.
> - February 15th 2025Release candidates (RC)...
> 
> WDYT?
> 
> Best Regards,
> Dongjoon Hyun
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org