Hi David,
The 'ingest and search logs' use case for a search engine is a
relatively new development - at least it wasn't talked about much before
Elasticsearch appeared. Elastic put a lot of effort into pushing
Elasticsearch as a Splunk alternative and with Logstash and Kibana for
ingestion and visualisation respectively, plus some clever aggregations
based on older work on facets in the middle, this became a big part of
why people chose Elasticsearch. Although there have been some efforts to
position Solr for similar use cases (Banana from Lucidworks an early
fork of Kibana, although that rather died on the vine (tree??), and it's
always been possible to use Logstash with Solr with some fiddling) it
never quite fit the same niche.
In the traditional 'text search' world however Solr and Elasticsearch
are pretty much equivalent in terms of what they can be made to do, and
I believe this is mainly where the Solr community has focused its
efforts. Some of us spend most of our time in this world of text
analysis, ranking and relevance (my employers OpenSource Connections
deliberately swerve from 'log ingest' projects and tend to pass these on
to partners or others). The skills required for each use case are
somewhat different. So I don't think you can say the popularity of Solr
has waned for everything, it just never really took off in the world of
logs. Certainly it's used by some huge companies for search applications
(Home Depot, Sony, Apple, Salesforce to name just a handful) and
continues to evolve and improve.
That said, if you can submit patches for RSolr or figure out better ways
for users to get started with log ingestion for Solr, I'm sure there
will be those who will be thankful - I just doubt there's a lot of
community will to push Solr as a serious alternative for the log
ingestion case. However I'm very happy to be corrected!
Best
Charlie
On 01/09/2022 01:43, MOSS, David wrote:
The popularity of Solr has waned in recent years with Elasticsearch
taking much of the “market share”.
I believe an important factor in this is the lack of options for
forwarding logs to Solr.
Most of the log forwarding options target Elasticsearch out of the
box. Splunk has its own dedicated universal forwarder. That makes it
very easy to choose Elasticsearch or Splunk. Those who try Solr beyond
the tutorial stage immediately run into problems forwarding log data
to Solr.
Logstash is a popular forwarding option that claims to target Solr.
When you try to use it however there is a bug in the ruby code for
RSolr that requires importing an older version and hand editing. It
does not work out of the box. Most of the other log forwarding options
suggest sending their logs to Logstash and having that forward them on
to Solr.
So, once a new user has finished testing with the tutorial data and is
all enthusiastic about Solr, the nightmare begins when they try to
send their own log data for indexing. I suspect most give up at this
point and go to Elasticsearch and Splunk. They make ingestion easy.
The solution?
The Solr community could make the effort to fix Logstash, or develop a
similar log forwarder that does work out of the box with Solr.
Difficulty with log forwarding is a dealbreaker for many people.
So, who is up for it?
Regards,
*David Moss*
*Technology Support Officer, Identity & Access Management,*
*Information Security Services*
Enterprise Technology Services | Information and Technologies Branch
Department of Education
*P: 0467441553 | M: 0467441553 | E: *david.m...@qed.qld.gov.au
<mailto:david.m...@qed.qld.gov.au>
Level 13| AM60 | 60 Albert Street | Brisbane QLD 4000
PO Box 15033 | City East QLD 4002
Pronouns He/Him/His <https://www.mypronouns.org/he-him>
***************************************************************************************************
IMPORTANT: This email and any attachments may contain legally
privileged, confidential or private information, and may be protected
by copyright. You may only use or disclose this information if you are
the intended recipient(s) and if you use it in an authorised way. No
other person is allowed to use, review, alter, transmit, disclose,
distribute, print or copy this email and any attachments without
appropriate authorisation.
If you are not the intended recipient(s) and the email was sent to you
by mistake, please notify the sender immediately by return email or
phone, destroy any hardcopies of this email and any attachments and
delete it from your system. Any legal privilege and confidentiality
attached to this email is not waived or destroyed by that mistake.
The Department of Education carries out monitoring, scanning and
blocking of emails and attachments sent from or to addresses within
the Department of Education for the purposes of operating, protecting,
maintaining and ensuring appropriate use of its computer network. It
is your responsibility to ensure that this email does not contain and
is not affected by computer viruses, defects or interference by third
parties or replication problems (including incompatibility with your
computer system).
The Department of Education does not accept any responsibility for any
loss or damage that may result from reliance on, or the use of, any
information contained in the email and any attachments.
***************************************************************************************************
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
Founding member of The Search Network <http://www.thesearchnetwork.com>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/wp-content/uploads/2020/08/ES_book_final_journal_version.pdf>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II
