Sort-merge join improvement

without changing the sort-merge join algorithm? 2. We believe there is a more general pattern here and that this could help in other similar situations where secondary sorting is available. Would you agree? 3. Would you like us to open a JIRA ticket and create a pull request

Re: Sort-merge join improvement

As instructed offline, I opened a JIRA for this: https://issues.apache.org/jira/browse/SPARK-24020 I will create a pull request soon. Le 4/17/2018 à 6:21 PM, Petar Zecevic a écrit : Hello everybody We (at University of Zagreb and University of Washington) have implemented an optimization of

Re: Sort-merge join improvement

Hi, the PR tests completed successfully (https://github.com/apache/spark/pull/21109). Can you please review the patch and merge it upstream if you think it's OK? Thanks, Petar Le 4/18/2018 à 4:52 PM, Petar Zecevic a écrit : As instructed offline, I opened a JIRA for this:

Re: Sort-merge join improvement

-optimized SMJ. Merging this would help us tremendously and I believe this can be useful in other applications, too. Can you please review (https://github.com/apache/spark/pull/21109) and merge the patch? Thank you, Petar Zecevic Le 4/23/2018 à 6:28 PM, Petar Zecevic a écrit : Hi, the PR

Re: Sort-merge join improvement

sides, all the new code is well contained in separate classes (unless it was necessary to change existing ones). So I believe this is ready to be merged. Can some of the committers please take another look at this and accept the PR? Thank you, Petar Zecevic Le 5/15/2018 à 10:55 AM, Petar Zece

Jenkins build errors

Hi, Jenkins build for my PR (https://github.com/apache/spark/pull/21109 ; https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92023/testReport/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/) keeps failing. First it

Re: Jenkins build errors

n-tests --parallelism=4 ; received return code 255 Attempting to post to Github... > Post successful. Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab

Re: Jenkins build errors

ns. On Tue, Jun 19, 2018, 2:53 AM Petar Zecevic <mailto:petar.zece...@gmail.com>> wrote: Thanks, but unfortunately, it died again. Now at pyspark tests: ==

Re: Jenkins build errors

be a mirror > problem, throttling, etc. But there again haven't spotted another failing > Hive test. > > On Wed, Jun 20, 2018 at 1:55 AM Petar Zecevic wrote: > > It's still dying. Back to this error (it used to be spark-2.2.0 before): > > java.io.IOException: Cannot

Re: Spark development with IntelliJ

This helped me: http://stackoverflow.com/questions/26995023/errorscalac-bad-option-p-intellij-idea On 8.1.2015. 11:00, Jakub Dubovsky wrote: Hi devs, I'd like to ask if anybody has experience with using intellij 14 to step into spark code. Whatever I try I get compilation error: Error:sc

Re: Jar for Spark developement

You can check out the Spark in Action book. In my (not so humble) opinion, it's very good for beginners. Petar (author) On 21.6.2016. 18:01, tesm...@gmail.com wrote: Hi, Beginner in Spark development. Took time to configure Eclipse + Scala. Is there any tutorial that can help beginners.