Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

Hi Ximo, sorry for a super late reply. Just FYI https://github.com/apache/spark/pull/32198 is merged in master, and you can try it out if building with latest master. Let me know if it works or not for your case, thanks. Cheng Su From: Cheng Su Date: Monday, September 21, 2020 at 8:54 AM To: X

Re: [DISCUSS] Add RocksDB StateStore

Hi all, Following the latest comments in SPARK-34198 , Databricks decided to donate the commercial implementation of the RocksDBStateStore. Compared with the original decision, there’s only one topic we want to raise again for discussion: can we d

Re: [DISCUSS] Add RocksDB StateStore

I think adding RocksDB state store to sql/core directly would be OK. Personally I also voted "either way is fine with me" against RocksDB state store implementation in Spark ecosystem. The overall stance hasn't changed, but I'd like to point out that the risk becomes quite lower than before, given