Thanks, Mengu (and Ere)!
It sounds like Zebra is really the most challenging aspect of scaling right now. Here's hoping that Elasticsearch is or will be stable enough to be used as the main search engine. Mengu: What do you do about cronjobs for a Koha installation running over multiple servers? Disable them on all but 1 VM for that Koha installation? Actually, now that I think about it, Koha Plugins could be a challenge, although once I improve the tooling for installing plugins, it should be manage plugins at scale. Thinking about RabbitMQ. that should be OK if each instance has a worker and the RabbitMQ servers are clustered (or at the very least centralized). Mengu: Is your Memcached clustered? Mengu: Do you do any log shipping, say with rsyslog? Sorry for all the questions. I like to think about as many details as I can heh. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-boun...@lists.koha-community.org> On Behalf Of Mengu Yazicioglu Sent: Friday, 12 June 2020 5:12 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Deploying Koha at scale Hi, I'd like to share some experiences of our installations and what we have done and we haven't. We've used many servers for apache part, and galera cluster for Mariadb part. We used nginx for load balancing and I shared a new entry in the wiki about our conf: https://wiki.koha-community.org/wiki/Nginx_conf_sample_for_load_balancing_wi th_many_servers The main problem always occured from Zebra site as you said David. Indexing for more millions records was horrible of course. After indexing it was easy to copy data and create new server while working in a huge VM environment. Not only indexing data but also with many requests from users also used a lot of resources of hardwares. Since Zebra part was the weak point, we preferred to power database servers and change a lot of code just to search directly from database instead of using Zebra. This gave us quite big flexibility in our installation. In some problems, if we need to do re-indexing Zebra, we used nigths of the day to do that and used some extra servers for indexing and when finished we copied the data. I personally spend a lot of time for Zebra things and I really hate its structure. I hope Elastic search will be good for cluster environments. We also thought Dockerized environment for a scale for large installations but we could not find necessary money and time to finish. Best regards Mengu Devinim Ltd. On 11.06.2020 11:03, Ere Maijala wrote: Hi, This is a welcome discussion, thanks for starting it! A couple of quick comments: 1. If you're running Elasticsearch and Z3950 Responder, I can't really see a need to run Zebra at all anymore. Is there something that it's really needed for? 3. Z3950 Responder is pretty lightweight, and it'll be the search index doing most of the work. But I understand the need from fault tolerance point of view. You could use YAZ Proxy or Metaproxy as the load balancer for multiple Z3950 Responders. However, that's still a single point managing the sessions. --Ere dc...@prosentient.com.au <mailto:dc...@prosentient.com.au> kirjoitti 11.6.2020 klo 10.52: Hi all, For a long time, I've been thinking about how Koha could scale better, and I'm wondering what other people are thinking about this topic. I think Mengu is probably the best expert on this topic that we have on Earth, so especially curious about his input. It's relatively easy to add more Apache web servers, more MySQL nodes, and more Elasticsearch nodes. but the same might not be said for other parts of Koha like the following: 1. Zebra a. In theory, you could load balance queries to a cluster of Zebra listeners b. But indexing updates to many Zebra databases could be tricky (I don't know if Zebra's database can support many readers). I suppose you could alter rebuild_zebra.pl to update multiple databases in a variety of different ways. I imagine Mengu could speak to this, as I recall him talking once about needing to do work to make Zebra scale? 2. I'm not familiar enough with the SIP server to say a. Off the top of my head, I think a TCP load balancer would be good enough here 3. Z3950 Responder? a. I'm not familiar with this daemon 4. What about cronjobs? Only 1 instance should execute the cronjobs. a. I suppose a person could disable cronjobs on all Koha nodes except for 1? It wouldn't be very robust in terms of availability but it would technically work most of the time. Any other thoughts? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
signature.asc
Description: PGP signature
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
