Great, thanks Stephen. This is definitely an area that can use some attention, and I'm glad to see that you are looking into alternatives.
The metadata associated with a recording includes "subject", so it's certainly possible to load word lists specific to a given subject, and check whether a string obtained OCR or speech to text is a member of the generic language and the subject-specific union. The dictionary API would need to change, but I don't see a problem with that. Good point re: loading the word lists from filesystem instead of the classloader. The dictionary impl can be reloaded at runtime, though, so it shouldn't be too much of a hassle to reload new word lists. Any takers for working on this? Should I create a branch? Should we set up something on github? ;) Thanks, Josh On Mon, Mar 7, 2011 at 9:19 AM, Stephen Marquard <[email protected] > wrote: > Hi, > > I've recently been looking at dictionaries and language modelling as > part of preparatory work to integrating Sphinx for speech recognition. > The issues are similar to OCR recognition, and in fact related because > some of the approaches to language modelling for speech recognition > start by using keywords derived from accompanying slides. > > One challenge is that many university lectures can be expected to > contain specialized terms and vocabulary, which a restricted dictionary > will not include. OTOH a very large dictionary, especially from a corpus > such as Wikipedia, runs the risk of including many incorrect spellings > and non-words. However, we can expect that the specialized terms are > probably those most likely to be searched for, rather than common > English words. > > So far I've looked at a Wikipedia-derived dictionary, a dictionary from > the Google n-gram corpus at http://ngrams.googlelabs.com/datasets > (derived from scanned books in Google Books published up to 2000) and > one or two others. > > Using a manual transcription of a lecture on chemical pathology of the > liver as a reference text, it seems one needs a dictionary of around > 250K to 500K terms to cover almost all generic terms, and larger than > that to cover almost all technical terms. These are "naive" dictionaries > that include all variants rather than stem words, etc. > > I'm not really at the "recommend solution" stage of investigations yet, > but for now would say that it would probably be useful to allow the > dictionary to be loaded from the filesystem rather than classpath so > people can experiment with different dictionaries. > > Also we should support multiple languages - I'm not sure if recordings > have that as metadata yet? > > I'm happy to share my derived Wikipedia and Google N-gram English > dictionaries if anyone else would like to experiment. Wikipedia is GFDL > / CC-BY-SA and the Google n-gram data is CC-BY. > > Regards > Stephen > > > -- > Stephen Marquard, Learning Technologies Co-ordinator > Centre for Educational Technology, University of Cape Town > http://www.cet.uct.ac.za > Email / IM (Jabber/XMPP): [email protected] > Phone: +27-21-650-5037 Cell: +27-83-500-5290 > > > >>> Josh Holtzman <[email protected]> 3/7/2011 9:27 AM >>> > I've implemented a simple, in-memory dictionary service in order to > try > improving the quality of the text segments as displayed in the player. > The > idea is to create a smaller, "quality over quantity" dictionary than > what we > get from the Wikipedia extract. See MH-7077 for more details and the > attached diff. This implementation is currently English only, but can > certainly be extended to support multiple languages. If anyone is > interested in working on this, please let me know, and I'll create a > separate module in svn. > > Thanks, > Josh > > > > > > ### > UNIVERSITY OF CAPE TOWN > > This e-mail is subject to the UCT ICT policies and e-mail disclaimer > published on our website at > http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from > +27 21 650 9111. This e-mail is intended only for the person(s) to whom > it is addressed. If the e-mail has reached you in error, please notify > the author. If you are not the intended recipient of the e-mail you may > not use, disclose, copy, redirect or print the content. If this e-mail > is not related to the business of UCT it is sent by the sender in the > sender's individual capacity. > > ### > > _______________________________________________ > Matterhorn mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn > > > To unsubscribe please email > [email protected] > _______________________________________________ >
_______________________________________________ Matterhorn mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn To unsubscribe please email [email protected] _______________________________________________
