Thanks for responding, Reynold, Marcelo and Marcin. >And I think that's really what Mark is proposing. Basically, "don't >intentionally break backwards compatibility unless it's really >required" (e.g. SPARK-12130). That would allow option B to work.
Yeah, that's exactly what Option B is proposing. I also don't think it'd make a huge difference to go back to full class name but I have explicitly added Lianhui to this thread, who worked on SPARK-12130, so he can correct me if I am blantantly wrong. And, even then, we could keep the Spark1 and Spark2 shuffle services compatible by doing mapping of short-long names or Abstract getBlockData implementation, if we decide it's necessary. Mark On Mon, Apr 18, 2016 at 3:23 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <r...@databricks.com> wrote: > > IIUC, the reason for that PR is that they found the string comparison to > > increase the size in large shuffles. Maybe we should add the ability to > > support the short name to Spark 1.6.2? > > Is that something that really yields noticeable gains in performance? > > If it is, it seems like it would be simple to allow executors register > with the full class name, and map the long names to short names in the > shuffle service itself. > > You could even get fancy and have different ExecutorShuffleInfo > implementations for each shuffle service, with an abstract > "getBlockData" method that gets called instead of the current if/else > in ExternalShuffleBlockResolver.java. > > -- > Marcelo >
