Github user codedeft commented on the pull request:
https://github.com/apache/spark/pull/2868#issuecomment-60713165
Here's one number. But this requires constant re-caching new node Id caches
and unpersisting old node Id caches that is not reflected in the code yet. I'm
not sure if frequent persisting of a new RDD from a previously persisted RDD is
a cheap operation, but at least in this data set, it seems fast. Let me know if
you guys know more about persistence mechanism.
mnist dataset, 750 columns with 60000 rows (only two partitions). 8
executors. 10-class classification. 100 trees trained, 30 max depth. Gini. With
the default fraction testing.
Without node-id caching, it took 24 mins 34 seconds.
With node-id caching with persisting the cache every two iteration, it took
16 minutes 42 seconds.
So we see noticeable benefits, as long as we frequently recache the node Id
cache.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
