[
https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13882627#comment-13882627
]
Anshum Gupta commented on SOLR-5477:
------------------------------------
bq. Why is it called a taskQueue in CoreAdminHandler? There is no queueing
happening here.
Changed it. Had that change on my machine before you mentioned :)
bq. Why is the taskQueue defined as a Map<String, Map<String, TaskObject>>? It
can simply be a Map<String, TaskObject>.
{quote} I don’t understand why we need three different maps for
running/completed/failure for overseer collection processor. My comment #2
applies here too. We can store the status in the value bytes instead of keeping
three different maps and moving the key around. What do you think? {quote}
It takes away the ability (or atleast makes it too complicated) to limit number
of tasks in a particular state e.g. limiting storage of 50 completed tasks only.
bq. The CoreAdminHandler.addTask with limit=true just removes a random (first?)
entry if the limit is reached.
It removes the first element. Its a synchronized LinkedHashMap so the iterator
preserves order and returns the first element.
bq. OverseerCollectionProcessor.requestStatus returns response with “success”
even if requestid is found in “running” or “failure” map
Success was supposed to mean that the task was found in a status map. It might
actually make sense to change it. Thanks for the suggestion.
bq. Although there is a provision to clear the overseer status maps by passing
requestid=1, it is never actually called.
The intention is for the user to explicitly call the API. There's no concept of
a map/queue in zk that maintains insertion state.
you'd have to check it, order it and then delete the apt one every time the
numChildren exceeds the limit. I thought it was best left to the user.
Will upload a patch with the following:
* Migrate API to also use the ASYNC CoreAdmin requests.
* Store the failed tasks information from CoreAdmin async calls in case of
Collection API requests.
* Tests for
** migratekey (and other calls) in ASYNC mode.
** Failing Collection API calls.
> Async execution of OverseerCollectionProcessor tasks
> ----------------------------------------------------
>
> Key: SOLR-5477
> URL: https://issues.apache.org/jira/browse/SOLR-5477
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Noble Paul
> Assignee: Anshum Gupta
> Attachments: SOLR-5477-CoreAdminStatus.patch, SOLR-5477.patch,
> SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch,
> SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch,
> SOLR-5477.patch, SOLR-5477.patch
>
>
> Typical collection admin commands are long running and it is very common to
> have the requests get timed out. It is more of a problem if the cluster is
> very large.Add an option to run these commands asynchronously
> add an extra param async=true for all collection commands
> the task is written to ZK and the caller is returned a task id.
> as separate collection admin command will be added to poll the status of the
> task
> command=status&id=7657668909
> if id is not passed all running async tasks should be listed
> A separate queue is created to store in-process tasks . After the tasks are
> completed the queue entry is removed. OverSeerColectionProcessor will perform
> these tasks in multiple threads
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
