Re: [Spark Core] Adaptive dynamic partition pruning

Which version are you using? I test it in spark 3.2.1 and sure that dynamic pruning works in queries with multi joins. BTW, could you execute ‘explain extended your sql’? > 2022年11月10日 02:10，hajyoussef amine 写道： > > Hello everyone, > > Let me take the following spark sql example to demonstrate

Re: [Spark Core] Adaptive dynamic partition pruning

FYI, https://medium.com/@prabhakaran.electric/spark-3-0-feature-dynamic-partition-pruning-dpp-to-avoid-scanning-irrelevant-data-1a7bbd006a89 This blog may be

Re: [Spark Core] Adaptive dynamic partition pruning

able2). > In the case where (dimension_table JOIN fact_table1) is small enough, the > result ideally can be treated as another dimension table and thus used to > prune the fact_table2. I don't find an easy way to implement it though. > > > On Fri, Nov 11, 2022 at 4:32 PM Jie

Re: sequence file write

It seems that the name is “sequencefile”. > 2022年11月14日 20:59，Shrikant Prasad 写道： > > Hi, > > I have an application which writes a dataframe into sequence file using > df.write.format("sequence").insertInto("hivetable1") > > This was working fine with Spark 2.7. > Now I am trying to migrate

Re: sequence file write

efaultSource > > Regards, > Shrikant > > On Mon, 14 Nov 2022 at 6:35 PM, Jie Han wrote: > It seems that the name is “sequencefile”. > > > 2022年11月14日 20:59，Shrikant Prasad 写道： > > > > Hi, > > > > I have an application which writes a dataframe into s