Hi -
I am running a canopy cluster on a document corpus (about 15,000 articles) with Tanimoto. I am having a problem with achieving an approximate target cluster number of around 70-100 articles per cluster. I have cranked T1, T2 up to the maximum but can't get anything more than about 30 articles per cluster. I suspect this is related to the "curse of dimensionality" for the vector model. (The reported dimension of the tfidf vectors is around 68,000.) I have played with the vectorization process (seq2sparse) and have implemented a Lucene Analyzer which invokes the Porter stemmer and the standard stop word filter. I have also cranked down the standard deviation coefficient (-xs) from 3.0 to 1.0 (since the canopies are intended only to seed some variety of KMeans this may be acceptable). My understanding is that this should decrease the dimension of the term vector by discarding terms which are either too rare or too frequent and thus relieve the pressure caused by "curse of dimensionality". Are there any other knobs or dials which are available to the canopy clustering procedure? (The SquaredEuclidean measure would seem to be less constrained but (i) it tends to overweight term frequency and (ii) appears to suffer from much the same problems as Tanimoto.) Your thoughts appreciated.
