Spark 3.1 first RC date

2020-10-01 Thread Igor Dvorzhak
I know that Spark 3.1 release is anticipated in December. Taking this into account is there any planned date for the first Spark 3.1 RC cut? Is it expected to be in October or maybe November? smime.p7s Description: S/MIME Cryptographic Signature

Re: My I report a special comparaison of executions leading on issues on Spark JIRA ?

2020-10-01 Thread Sean Owen
Yes indeed, in fact you seem to be describing Spark 2->3 changes that are already documented in the spark 3 migration guide. On Thu, Oct 1, 2020 at 7:08 AM Russell Spitzer wrote: > You are always welcome to create a jira or jiras, but you may find you get > a faster response by asking about your

Re: My I report a special comparaison of executions leading on issues on Spark JIRA ?

2020-10-01 Thread Russell Spitzer
You are always welcome to create a jira or jiras, but you may find you get a faster response by asking about your issues on the mailing list first. That may help in identifying whether your issues are already logged or not, or whether there is a solution that can be applied right away. On Thu, O

My I report a special comparaison of executions leading on issues on Spark JIRA ?

2020-10-01 Thread Marc Le Bihan
Hello, I currently run a Spark project based on cities, local authorities, enterprises, local communities, etc. Ten Datasets written in Java are doing operations going from simple join to elaborate ones. Language used is Java. 20 integrations tests with the whole data (20 GB) takes seven hour. *A