+1 since its a correctness issue, I think its ok to change the behavior to
make sure the user is aware of it and let them decide.
Tom
On Saturday, November 7, 2020, 01:00:11 AM CST, Liang-Chi Hsieh
wrote:
Hi devs,
In Spark structured streaming, chained stateful operators possibly prod
Hi all,
It’s time to send in our quarterly ASF board report on Nov 11, so I wanted to
include anything notable going on that we want to appear in the board archive.
Here is my draft; let me know if you have suggested changes.
===
Apache Spark is a fa
I'm staring at https://issues.apache.org/jira/browse/HADOOP-17372 and a
stack trace which claims that a com.amazonaws class doesn't implement an
interface which it very much does
2020-11-10 05:27:33,517 [ScalaTest-main-running-S3DataFrameExampleSuite]
WARN fs.FileSystem (FileSystem.java:createFil
+1 for Apache Spark 3.1.0.
Bests,
Dongjoon.
On Tue, Nov 10, 2020 at 6:17 AM Tom Graves
wrote:
> +1 since its a correctness issue, I think its ok to change the behavior to
> make sure the user is aware of it and let them decide.
>
> Tom
>
> On Saturday, November 7, 2020, 01:00:11 AM CST, Liang-C
+1, I agree with Tom.
On Tue, Nov 10, 2020 at 3:00 PM Dongjoon Hyun
wrote:
> +1 for Apache Spark 3.1.0.
>
> Bests,
> Dongjoon.
>
> On Tue, Nov 10, 2020 at 6:17 AM Tom Graves
> wrote:
>
>> +1 since its a correctness issue, I think its ok to change the behavior
>> to make sure the user is aware o
An extra RPC call is a concern for the catalog implementation. It is simple
to cache the result of a call to avoid a second one if the catalog chooses.
I don't think that an extra RPC that can be easily avoided is a reasonable
justification to add caches in Spark. For one thing, it doesn't solve t
