Re: ideas for MLlib development

2015-03-04 Thread Robert Dodier
Thanks for your reply, Evan. > It may make sense to have a more general Gibbs sampling > framework, but it might be good to have a few desired applications > in mind (e.g. higher level models that rely on Gibbs) to help API > design, parallelization strategy, etc. I think I'm more interested in a

Re: ideas for MLlib development

2015-03-03 Thread Evan R. Sparks
Hi Robert, There's some work to do LDA via Gibbs sampling in this JIRA: https://issues.apache.org/jira/browse/SPARK-1405 as well as this one: https://issues.apache.org/jira/browse/SPARK-5556 It may make sense to have a more general Gibbs sampling framework, but it might be good to have a few desi

ideas for MLlib development

2015-03-03 Thread Robert Dodier
Hi, I have some ideas for MLlib that I think might be of general interest so I'd like to see what people think and maybe find some collaborators. (1) Some form of Markov chain Monte Carlo such as Gibbs sampling or Metropolis-Hastings. Any kind of Monte Carlo method is readily parallelized so Spar