Re: Spark 3.2.4 pom NOT FOUND on maven

2023-04-18 Thread Dongjoon Hyun
Thank you for reporting, Enrico. I verified your issue report and also double-checked that both the original official Apache repository and Google Maven Mirror works correctly. Given that, it could be due to some transient issues because the artifacts are copied from Apache repository to Maven

Re: hadoop-2 profile to be removed in 3.5.0

2023-04-18 Thread Steve Loughran
This is truly wonderful. 1. I have an internal patch related to committer stuff I could submit now 2. if someone wants to look at it where FileSystem.open() is used *and you have the file length, file path, or simply know whether you plan to do random or sequential IO*, switch to openFile(). on s3

Re: Slack for Spark Community: Merging various threads

2023-04-18 Thread Mich Talebzadeh
Hi, After all the fireworks and quote "this super duper topic" has a decision being made on what to use as the official or endorsed slack type platform? This I think would be most useful for the monolithic product release votes a least (yep the one with +1 etc). To be fair, there has been a lot

Re: Parametrisable output metadata path

2023-04-18 Thread Wojciech Indyk
Thank you for your response! I misread "data lake" as "delta lake", my bad. Anyway I need to write output to file system. I see your point about data lakes, however migrations take time, so at least from this perspective I wouldn't deprecate FileStreamSink. I hope FileStreamSink will be still maint