Hi list

I am using the 'tm' package to review meeting notes at a school to identify terms frequently associated with 'learning', 'sports', and 'extra-mural' activities, and then to sort any terms according to these three headers in a way that could be supported statistically (as opposed to, say, my own bias, etc.).

To accomplish this, I have done the following:

(1) After the usual pre-processing of the text data, loading it as a corpus and then converting it into a document term matrix (called 'allTerms'), I have identified the 20 most frequently occurring terms in the meeting notes and extracted these into a named vector called 'freqTerms'. Many of the terms returned have nothing to do with any of the three themes of 'learning', 'sports', or 'extra-mural'.

(2) Therefore, I have also manually generated a list of terms and synonyms for 'learning' and 'sports', etc. (e.g. 'football', 'soccer', 'drama', 'chess', etc.) and then tested for the occurrence of each of these terms in the corpus, e.g.:

> allTerms['soccer']

and have come up with a list of some 30 terms together with their frequencies. I manually sorted these according to three headers 'learning', 'sports', and 'extra-mural' and dropped these into a table in a word processing document. Some of these terms are also in the freqTerms vector.

What I want to do now is to use cluster analysis (hclust, from the 'cluster' library) to plot a dendrogram of the terms I have manually checked and put into the table, in order to see how closely similar the terms are and whether they cluster in ways similar to the way as I manually sorted these under the table column headers of 'learning', 'sports', and 'extra-mural'.

To do this, I dropped these manually sorted terms into a data frame together with the associated values (which I called 'tes.df') and then tried plotting this as follows:

> dtes <- dist(tes.df, method = 'euclidean')
> dtesFreq <- hclust(dtes, method = 'ward.D')
> plot(dtesFreq, labels = names(tes.df))

However, I get an error message when trying to plot this: "Error in graphics:::plotHclust(n1, merge, height, order(x$order), hang, : invalid dendrogram input".

I'm clearly screwing something up, either in my source data.frame or in my setting hclust up, but don't know which, nor how.

More than just identifying the error however, I am interested in finding a smart (efficient/ elegant) way of checking the occurrence and frequency value of the terms that may be associated with 'sports', 'learning', and 'extra-mural' and extracting these into a matrix or data frame so that I can analyse and plot their clustering to see if how I associated these terms is actually supported statistically.

I'm sure that there must be a way of doing this in R, but I'm obviously not going about it correctly. Can anyone shine a light please?

Thanks for any help/ guidance.

Regards,
Sun

______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Reply via email to