Hi Luke,
We built Luwak, the basis of Solcolator, at my previous company Flax,
and donated it to Lucene. I've sent you a separate private email. I
don't know who built Solcolator sadly, it would indeed be awesome to get
this feature properly integrated into Solr.
Best
Charlie
On 13/12/2023 15:53, Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) wrote:
Hi All,
My team and I are trying to match documents against a set of queries. Both need
to be updated in real-time (at least on the order of seconds) and the system
needs to scale to potentially millions of queries. Although lucene monitor
greatly optimizes this kind of alerting logic, as far as we know, it won't
handle scaling as it isn't wrapped by a sophisticated runtime.
Reading through this thread https://lists.apache.org/list.html?dev@find_parent=true
pointed me at https://github.com/SOLR4189/solcolator. Overall, this project looks
promising but I'm concerned that it was contributed by an anonymous user and is lacking
basic maintenance, at the very least it needs dependency upgrades. My second concern with
it is that it ingests _documents_ via AddUpdateCommand and then writes the output to a
configurable sink. In my team's use case, it would be better to invert the flow and
process documents in the SearchHandler (so that the output can be written to
SolrQueryResponse) and AddUpdate the queries to take better advantage of solr's sharding.
The second concern is not as strong as the first as I am still thinking about how to
implement the "solcolator" within our constraints. Perhaps the
document-as-query approach has its own drawbacks that I am not seeing.
In either case I have two questions:
1. Has anyone here used solcolator and with what effect? I would be grateful to
learn specifics about the scale of your solution.
2. What do you think about forking or riffing off this project and adding a
document-as-query control flow?
As it stands now we are designing a system that coordinates lucene monitor by
directly interacting with zookeeper and it feels like we are reinventing some
of solr's capabilities. Any feedback anyone might have would be greatly
appreciated.
Many thanks,
Luke
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
Founding member of The Search Network and co-author of Searching the Enterprise
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II
