Hi all, Does anyone happen to know what tests Databricks uses for the Spark distribution certification suite? Is it simply the tests that run as CI on Spark pull requests, or is there something more involved?
The web site ( https://databricks.com/spark/certification/certified-spark-distribution) says that "the certification process is fully transparent with open-source tests (developed and maintained by the community and used to test each release of Apache Spark), lightweight, and 100% free." But that's all the information I can find about the process, so I was hoping someone could elucidate it for me. It seems like community-based downstream distributions (e.g., BigTop, Fedora, etc) could benefit from running a similar test suite to the official Spark distribution certification suite in their regular CI, even if the results were (necessarily) not official. best, wb
