On 1/22/2019 10:36 AM, Matt Jadud wrote:
My initial wild guess is that the problem is in the queen. Going
by the
description it's way overly complicated, with plenty of opportunities
for something to get lost.
Perhaps. It's not a long driver, as code goes.
Length really isn't relevant. Your description hints that it is heavily
threaded - proportional to the number of places. Every additional
message queue is one more potential bottleneck and spot where something
can get lost.
Your whole architecture seems overly complicated for what it does.
Why
is the coordinator (queen) so intimately involved? Why not have the
workers process an ID end to end [read the source table, calculate,
update the result table]? All the coordinator should do is load
balance
and ensure that all the IDs actually get processed.
FWIW, to make sure that workers don't pull IDs on a duplicate basis, I
have one process farming them out. Hence, the worker requests an ID,
does the work, and submits the work back to the manager. The manager
doles out IDs, and then stores results.
I was only suggesting that the workers store their results to the DB
directly. The coordinator still can - and for load balancing, should -
dole out the work and check to make sure the workers finish and all the
IDs are processed.
The complexity of returning results to the coordinator and having it
write the DB is unnecessary.
And what is the DBMS?
They're two SQLite tables. The readers should never block readers, to
the best of my understanding, and there is only a single writer (to a
completely separate DB/table).
Ah. Now I understand the single writer approach. You might consider
changing to a server DBMS instead.
In any case I would work on reducing the complexity of the coordinator.
Workers should never need to "request" anything - the coordinator should
know whether they currently are busy or idle. And scads of router
threads really are not necessary - routers are a convenience, but at a
certain point they get in the way more than they help. It's really not
difficult [and with lots of channels, I think, often clearer] to
handle communications directly with put, get, and sync. I probably
would use only a couple of threads: one to distribute work, and one to
collect and write results to the database.
YMMV,
George
--
You received this message because you are subscribed to the Google Groups "Racket
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
