Yes that's a great option when the modeling process itself doesn't really need Spark. You can use any old modeling tool you want and get the parallelism in tuning via hyperopt's Spark integration.
On Thu, Apr 1, 2021 at 10:50 AM Williams, David (Risk Value Stream) <david.willi...@lloydsbanking.com.invalid> wrote: > Classification: Public > > > > Many thanks for the info. So you wouldn’t use sklearn with Spark for > large datasets but use it with smaller datasets and using hyperopt to build > models in parallel for hypertuning on Spark? > > >
