Re: Async RDD saves

This looks interesting.. anyways, it will be good if you can elaborate more on the expectations and the various other ways you had tried before deciding to do it this way... Regards, Kalyan. On Fri, Aug 7, 2020, 11:24 PM Edward Mitchell wrote: > I will agree that the side effects of using Futur

Re: Async RDD saves

Hi both, Thanks for your replies! Sean, your proposal to use a driver-side future wrapping the blocking call sounds a lot easier indeed. But I want to ensure that canceling the future in the driver code kills the corresponding tasks on all executors. If I wrap the driver-side call in a standard

Why is V2SessionCatalog not a CatalogExtension?

Hi, Just started exploring Catalog Plugin API and noticed these two classes: CatalogExtension and V2SessionCatalog. Why is V2SessionCatalog not a CatalogExtension? - V2SessionCatalog extends TableCatalog with SupportsNamespaces [1] - CatalogExtension extends TableCatalog, SupportsNamespaces [2]

Re: [VOTE] Release Spark 2.4.7 (RC1)

I'm going to go ahead and vote -0 then based on that then. On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun wrote: > Hi, All. > > Unfortunately, there is an on-going discussion about the new decimal > correctness. > > Although we fixed one correctness issue at master and backported it > partially t

Re: Why is V2SessionCatalog not a CatalogExtension?

A v2catalog can stand alone, it doesn't have to extend the built in catalog. On Sat, Aug 8, 2020, 7:36 AM Jacek Laskowski wrote: > Hi, > > Just started exploring Catalog Plugin API and noticed these two classes: > CatalogExtension and V2SessionCatalog. > > Why is V2SessionCatalog not a CatalogEx

Re: [VOTE] Release Spark 2.4.7 (RC1)

Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago. [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms https://github.com/apache/spark/pull/29383 It seems that the patch is already ready in this case. I raised the

Re: [VOTE] Release Spark 2.4.7 (RC1)

Thanks for letting us know about the two issues above, Dongjoon. I've checked the release materials (signatures, tag, ...) and it looks fine, too. Also, I run the tests on my local Mac (java 1.8.0) with the options `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`