Hello,
I've been working on logs and metadata and I've been working on this
specific problem for quite a while.
if you have a constant, predefined number of fields associated to each
log, a relational database might perform just fine and offer you more
possibilities regarding log analysis (statistics, aggregation, min/max
... ). As a side note, you might not be aware that PostgreSQL ( > 8.3 )
comes with built-in full text search capabilities, so you don't have to
sacrifice the functionality. A column-based database would probably be
also worth trying.
On the other hand, if fields can vary in type and number for each log,
the relational database model will perform very badly : if you normalize
a lot you'll end up with a compact database schema that takes a lot of
time to query ( and queries can become hellishly hard to write); if you
don't you're losing most of what a database would bring you in the first
place. This is a very close problem to the one discussed here :
http://www.pui.ch/phred/archives/2005/04/tags-database-schemas.html this
link is worth a read if you want to go on with databases.
Do not forget also that unless you use a column-based database, you
might probably want to use indices on your tables in order to speed up
some lookups ... But it will come at a hefty price at insertion time
after you reach a certain amount of indexed logs.
Concerning my project (where we fall in the second case), current tests
with lucene are very encouraging. Sure, we lose some analytics
capabilities and the hard disk footprint is rather huge, but the query
response time, be it from full-text searches or complex queries on
specific fields, is outstanding.
Of course, in the end, it all depends on what you need to do. Just my
personal experience here ...
Regards,
Matthieu Huin
Le 14/01/2011 06:47, chensheng a écrit :
Hi, I have a similar case.
If I only index some fixed fields with single-word/number, instead of full-text
search,
is Lucene any better than a relational database?
------------------ Original ------------------
From: "Benzion G"<benzi...@yahoo.com>;
Date: Fri, Jan 14, 2011 01:15 AM
To: "java-user"<java-user@lucene.apache.org>;
Subject: Re: lucene-based log searcher?
Hi,
I almost finished these days to write such a tool.
It reads via FTPClient the log files and adds to Lucene index.
Via small Web application you can see/filter/sort the log messages.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
