Re: Contributing to MLlib: Proposal for Clustering Algorithms

Thanks, Jeremy. I'm abandoning my initial approach, and I'll work on optimizing your example (so it doesn't do the breeze-vector conversions every time KMeans is called). I need to finish a few other projects first, though, so it may be a couple weeks. In the mean time, Yu also created a JIRA fo

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hey RJ, Sorry for the delay, I'd be happy to take a look at this if you can post the code! I think splitting the largest cluster in each round is fairly common, but ideally it would be an option to do it one way or the other. -- Jeremy - jeremy freeman, phd neuroscientist

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hi Yu, A standardized API has not been implemented yet. I think it would be better to implement the other clustering algorithms then extract a common API. Others may feel differently. :) Just a note, there was a pre-existing JIRA for hierarchical KMeans SPARK-2429

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hi all, I am also interested in specifying a common framework. And I am trying to implement a hierarchical k-means and a hierarchical clustering like single-link method with LSH. https://issues.apache.org/jira/browse/SPARK-2966 If you have designed the standardized clustering algorithms API, plea

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hi all, I wanted to follow up. I have a prototype for an optimized version of hierarchical k-means. I wanted to get some feedback on my apporach. Jeremy's implementation splits the largest cluster in each round. Is it better to do it that way or to split each cluster in half? Are there are an

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hi RJ, that sounds like a great idea. I'd be happy to look over what you put together. -- Jeremy -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-Proposal-for-Clustering-Algorithms-tp7212p7418.html Sent from the Apache Spark Devel

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Nice to meet you, Jeremy! This is great! Hierarchical clustering was next on my list -- currently trying to get my PR for MiniBatch KMeans accepted. If it's cool with you, I'll try converting your code to fit in with the existing MLLib code as you suggest. I also need to review the Decision Tree

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hi all, Cool discussion! I agree that a more standardized API for clustering, and easy access to underlying routines, would be useful (we've also been discussing this when trying to develop streaming clustering algorithms, similar to https://github.com/apache/spark/pull/1361) For divisive, hier

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Might be worth checking out scikit-learn and mahout to get some broad ideas— Sent from Mailbox On Thu, Jul 10, 2014 at 4:25 PM, RJ Nowling wrote: > I went ahead and created JIRAs. > JIRA for Hierarchical Clustering: > https://issues.apache.org/jira/browse/SPARK-2429 > JIRA for Standarized Cluste

Re: Contributing to MLlib: Proposal for Clustering Algorithms

I went ahead and created JIRAs. JIRA for Hierarchical Clustering: https://issues.apache.org/jira/browse/SPARK-2429 JIRA for Standarized Clustering APIs: https://issues.apache.org/jira/browse/SPARK-2430 Before submitting a PR for the standardized API, I want to implement a few clustering algorith

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Cool seems like a god initiative. Adding a couple extra high quality clustering implantations will be great. I'd say it would make most sense to submit a PR for the Standardised API first, agree that with everyone and then build on it for the specific implementations. — Sent from Mailbox On We

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Thanks everyone for the input. So it seems what people want is: * Implement MiniBatch KMeans and Hierarchical KMeans (Divide and conquer approach, look at DecisionTree implementation as a reference) * Restructure 3 Kmeans clustering algorithm implementations to prevent code duplication and confor

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Yeah if one were to replace the objective function in decision tree with minimizing the variance of the leaf nodes it would be a hierarchical clusterer. On Tue, Jul 8, 2014 at 2:12 PM, Evan R. Sparks wrote: > If you're thinking along these lines, have a look at the DecisionTree > implementation

Re: Contributing to MLlib: Proposal for Clustering Algorithms

If you're thinking along these lines, have a look at the DecisionTree implementation in MLlib. It uses the same idea and is optimized to prevent multiple passes over the data by computing several splits at each level of tree building. The tradeoff is increased model state and computation per pass o

Re: Contributing to MLlib: Proposal for Clustering Algorithms

No was thinking more top-down: assuming a distributed kmeans system already existing, recursively apply the kmeans algorithm on data already partitioned by the previous level of kmeans. I haven't been much of a fan of bottom up approaches like HAC mainly because they assume there is already a dis

Re: Contributing to MLlib: Proposal for Clustering Algorithms

K doesn't matter much I've tried anything from 2^10 to 10^3 and the performance doesn't change much as measured by precision @ K. (see table 1 http://machinelearning.wustl.edu/mlpapers/papers/weston13). Although 10^3 kmeans did outperform 2^10 hierarchical SVD slightly in terms of the metrics, 2^10

Re: Contributing to MLlib: Proposal for Clustering Algorithms

The scikit-learn implementation may be of interest: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.Ward.html#sklearn.cluster.Ward It's a bottom up approach. The pair of clusters for merging are chosen to minimize variance. Their code is under a BSD license so it can be used as

Re: Contributing to MLlib: Proposal for Clustering Algorithms

sure. more interesting problem here is choosing k at each level. Kernel methods seem to be most promising. On Tue, Jul 8, 2014 at 1:31 PM, Hector Yee wrote: > No idea, never looked it up. Always just implemented it as doing k-means > again on each cluster. > > FWIW standard k-means with euclide

Re: Contributing to MLlib: Proposal for Clustering Algorithms

No idea, never looked it up. Always just implemented it as doing k-means again on each cluster. FWIW standard k-means with euclidean distance has problems too with some dimensionality reduction methods. Swapping out the distance metric with negative dot or cosine may help. Other more useful clust

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Hector, could you share the references for hierarchical K-means? thanks. On Tue, Jul 8, 2014 at 1:01 PM, Hector Yee wrote: > I would say for bigdata applications the most useful would be hierarchical > k-means with back tracking and the ability to support k nearest centroids. > > > On Tue, Jul

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Having a common framework for clustering makes sense to me. While we should be careful about what algorithms we include, having solid implementations of minibatch clustering and hierarchical clustering seems like a worthwhile goal, and we should reuse as much code and APIs as reasonable. On Tue,

Re: Contributing to MLlib: Proposal for Clustering Algorithms

Thanks, Hector! Your feedback is useful. On Tuesday, July 8, 2014, Hector Yee wrote: > I would say for bigdata applications the most useful would be hierarchical > k-means with back tracking and the ability to support k nearest centroids. > > > On Tue, Jul 8, 2014 at 10:54 AM, RJ Nowling > wrot

Re: Contributing to MLlib: Proposal for Clustering Algorithms

I would say for bigdata applications the most useful would be hierarchical k-means with back tracking and the ability to support k nearest centroids. On Tue, Jul 8, 2014 at 10:54 AM, RJ Nowling wrote: > Hi all, > > MLlib currently has one clustering algorithm implementation, KMeans. > It would

Re: Contributing to MLlib on GLM

Poisson and Gamma regressions for modeling count data are definitely important in spark.mllib.regression. So don’t worry. Let’s change the updater to SquaredL2Updater as we discussed in the PR. Then we can ask Jenkins to run the test. On Jul 8, 2014, at 3:00 AM, xwei wrote: > Hi Gang, > > No

Re: Contributing to MLlib on GLM

Hi Gang, No admin is looking at our patch:( do you have some suggestions so that our patch can get noticed by the admin? Best regards, Xiaokai On Mon, Jun 30, 2014 at 8:18 PM, Gang Bai [via Apache Spark Developers List] wrote: > Thanks Xiaokai, > > I’ve created a pull request to merge featur

Re: Contributing to MLlib

thanks for the input. at the moment , I don't have any code commits yet. I wanted to discuss the algorithm implementation prior to the code submission. (never work with Git\ GutHub - so I hope this isn't very basic stuff) -- View this message in context: http://apache-spark-developer

Re: Contributing to MLlib

Alex, please send the pull request to apache/spark instead of your own repo, following the instructions in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark Thanks, Xiangrui On Wed, Jul 2, 2014 at 12:41 PM, RJ Nowling wrote: > Hey Alex, > > I'm also a new contributor. I c

Re: Contributing to MLlib

Hey Alex, I'm also a new contributor. I created a pull request for the KMeans MiniBatch implementation here: https://github.com/apache/spark/pull/1248 I also created a JIRA here: https://issues.apache.org/jira/browse/SPARK-2308 As part of my work, I started to refactor the common code to crea

Re: Contributing to MLlib

I opened a JIRA (https://issues.apache.org/jira/browse/SPARK-2344) and a pull request for this (https://github.com/salexln/spark/pull/1) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7158.html Sent from the Apache Spark

Re: Contributing to MLlib

thanks for the response ! that's is exactly the way i wanted to implement it :) I will create JIRA ticket and a request. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7157.html Sent from the Apache Spark Developers List

Re: Contributing to MLlib

Hi there, Generally we try to avoid duplicating logic if possible, particularly for algorithms that share a great deal of algorithmic similarity. See, for example, the way we implement Logistic regression vs. Linear regression vs. Linear SVM with different gradient functions all on top of SGD or L

Re: Contributing to MLlib

guys??? anyone??? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7155.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Contributing to MLlib on GLM

Thanks Xiaokai, I’ve created a pull request to merge features in my PR to your repo. Please take a review here https://github.com/xwei-datageek/spark/pull/2 . As for GLMs, here at Sina, we are solving the problem of predicting the num of visitors who read a particular news article or watch an o

Re: Contributing to MLlib on GLM

Hi Gang, No worries! I agree LBFGS would converge faster and your test suite is more comprehensive. I'd like to merge my branch with yours. I also agree with your viewpoint on the redundancy issue. For different GLMs, usually they only differ in gradient calculation but the regression.sca

Re: Contributing to MLlib on GLM

Hi Xiaokai, My bad. I didn't notice this before I created another PR for Poisson regression. The mails were buried in junk by the corp mail master. Also, thanks for considering my comments and advice in your PR. Adding my two cents here: * PoissonRegressionModel and GammaRegressionModel have t

Re: Contributing to MLlib on GLM

Yes, that's what we did: adding two gradient functions to Gradient.scala and create PoissonRegression and GammaRegression using these gradients. We made a PR on this. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-on-GLM-tp7033p7

Re: Contributing to MLlib on GLM

Well, as you said, MLLib already supports GLM in a sense. Except they only support two link functions - identity (linear regression) and logit (logistic regression). It should not be too hard to add other link functions, as all you have to do is add a different gradient function for Poisson/Gamma,

Re: Contributing to MLlib on GLM

Hi Xiaokai, Also take a look through Xiangrui's slides from HadoopSummit a few weeks back: http://www.slideshare.net/xrmeng/m-llib-hadoopsummit The roadmap starting at slide 51 will probably be interesting to you. Andrew On Tue, Jun 17, 2014 at 7:37 PM, Sandy Ryza wrote: > Hi Xiaokai, > > I

Re: Contributing to MLlib on GLM

Hi Xiaokai, I think MLLib is definitely interested in supporting additional GLMs. I'm not aware of anybody working on this at the moment. -Sandy On Tue, Jun 17, 2014 at 5:00 PM, Xiaokai Wei wrote: > Hi, > > I am an intern at PalantirTech and we are building some stuff on top of > MLlib. In P