[
https://issues.apache.org/jira/browse/LUCENE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865894#action_12865894
]
Tom Burton-West commented on LUCENE-2393:
-----------------------------------------
Hi Mike,
Thanks for all your help.
If we replace the current HighFreqTerms with the HighFreqTermsWithTF should
there be a command line switch so that you could ask for the default behavior
of the current HighFreqTerms? Or perhaps the default should be the current
behavior and the switch should turn on the additional step of gathering and
reporting on the totalTF for the terms.
I haven't bench-marked it but I'm wondering if getting the totalTF could
take a significant additional amount of time for large indexes. When I ask
for the top 10,000 terms using HighFreqTermsWithTF for our 500,000 document
indexes it takes about 40 minutes to an hour. I'm guessing that most of that
time is taken in the first step of getting the top 10,000 terms by docFreq, but
still it seems that reading the data and calculating the totalTF for 10,000
terms might be a significant enough fraction of the total time that the option
to skip that step might be useful.
Tom
> Utility to output total term frequency and df from a lucene index
> -----------------------------------------------------------------
>
> Key: LUCENE-2393
> URL: https://issues.apache.org/jira/browse/LUCENE-2393
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/*
> Reporter: Tom Burton-West
> Priority: Trivial
> Attachments: LUCENE-2393, LUCENE-2393.patch, LUCENE-2393.patch,
> LUCENE-2393.patch, LUCENE-2393.patch, LUCENE-2393.patch, LUCENE-2393.patch
>
>
> This is a pair of command line utilities that provide information on the
> total number of occurrences of a term in a Lucene index. The first takes a
> field name, term, and index directory and outputs the document frequency for
> the term and the total number of occurrences of the term in the index (i.e.
> the sum of the tf of the term for each document). The second reads the
> index to determine the top N most frequent terms (by document frequency) and
> then outputs a list of those terms along with the document frequency and the
> total number of occurrences of the term. Both utilities are useful for
> estimating the size of the term's entry in the *prx files and consequent Disk
> I/O demands.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
