Re: spark.sql.codegen.comments not in SQLConf?

It's probably because it is annoying to propagate that using SQL conf. On Wed, May 10, 2017 at 3:38 AM Jacek Laskowski wrote: > Hi, > > It seems that spark.sql.codegen.comments property [1] didn't find its > place in SQLConf [2] that appears to be the place for all Spark > SQL-related properties

Re: How about the fetch the shuffle data in one same machine?

I don't think it is Loopback only localhost or 127.0.0.1 will go. And the benchmarks test code should be simple don't involve calculate. Just make two test codes one just read the file from local the other just read the file from netty Read the different file size(small -> big), should have differ

spark.sql.codegen.comments not in SQLConf?

Hi, It seems that spark.sql.codegen.comments property [1] didn't find its place in SQLConf [2] that appears to be the place for all Spark SQL-related properties (for codegen surely). Don't think it merits a JIRA issue so just asking here. If agreed, I'd like to propose a PR. Thanks. [1] https:

Re: How about the fetch the shuffle data in one same machine?

There is a JIRA about this thing ( https://issues.apache.org/jira/browse/SPARK-6521). In the current Spark shuffle fetch still leverages Netty even two executors are on the same node, but according to the test on the JIRA, the performance is close whether to bypass network or not. From my understan

How about the fetch the shuffle data in one same machine?

Hi all, Now Spark only think the executorId same that fetch local file, but for same IP different ExecutorId will fetch using network that actually it can be fetch in the local Or Loopback. Apparently fetch the local file that it is fast that can use the LVS cache. How do you think? Regards -Ra