On 1/22/2019 10:36 AM, Matt Jadud wrote:

    My initial wild guess is that the problem is in the queen.  Going
    by the
    description it's way overly complicated, with plenty of opportunities
    for something to get lost.


Perhaps. It's not a long driver, as code goes.

Length really isn't relevant.  Your description hints that it is heavily threaded - proportional to the number of places.  Every additional message queue is one more potential bottleneck and spot where something can get lost.

    Your whole architecture seems overly complicated for what it does.
    Why
    is the coordinator (queen) so intimately involved?  Why not have the
    workers process an ID end to end [read the source table, calculate,
    update the result table]?  All the coordinator should do is load
    balance
    and ensure that all the IDs actually get processed.


FWIW, to make sure that workers don't pull IDs on a duplicate basis, I have one process farming them out. Hence, the worker requests an ID, does the work, and submits the work back to the manager. The manager doles out IDs, and then stores results.

I was only suggesting that the workers store their results to the DB directly.  The coordinator still can - and for load balancing, should - dole out the work and check to make sure the workers finish and all the IDs are processed.

The complexity of returning results to the coordinator and having it write the DB is unnecessary.


    And what is the DBMS?


They're two SQLite tables. The readers should never block readers, to the best of my understanding, and there is only a single writer (to a completely separate DB/table).

Ah.  Now I understand the single writer approach.  You might consider changing to a server DBMS instead.

In any case I would work on reducing the complexity of the coordinator.  Workers should never need to "request" anything - the coordinator should know whether they currently are busy or idle. And scads of router threads really are not necessary - routers are a convenience, but at a certain point they get in the way more than they help.   It's really not difficult  [and with lots of channels, I think, often clearer]  to handle communications directly with put, get, and sync.  I probably would use only a couple of threads: one to distribute work, and one to collect and write results to the database.

YMMV,
George

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to