[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990853#comment-13990853
]
Jun Rao commented on KAFKA-1430:
--------------------------------
Yes, it would be good to improve garbage collection in the purgatory too.
Currently, each watched item is kept in a watcher array list and a delayed
queue for expiration.
1. If a watched item is satisfied before expiration, the item is taken out of
the array list immediately, but has to wait until the expiration time before
it's taken out of the delayed queue. This is further guarded by a periodic
check. If the delayed queue becomes too big, we force a check to remove
satisfied items from the delayed queue. So, this part is not too bad, but a bit
ad hoc.
2. If an item expires, it's taken out of the delayed queue immediately.
However, it won't be taken out of the array list until the next update on the
watcher. If that update never comes, we actually will leak that item forever.
So, this seems to be a more serious issue.
One potential way of addressing #2 is to let ExpiredRequestReaper have access
to the watcher array list. Then everytime an item expires, we can also removes
it from the watcher list immediately.
Another way could be that we put the watcher list and the delayed queue in the
same entity and expose apis for accessing them in watch/update and
ExpiredRequestReaper. This way, we can probably take an item off both the list
and the queue at the same time. We can probably also get rid of the periodic
check in ExpiredRequestReaper.
> Purgatory redesign
> ------------------
>
> Key: KAFKA-1430
> URL: https://issues.apache.org/jira/browse/KAFKA-1430
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.8.2
> Reporter: Jun Rao
>
> We have seen 2 main issues with the Purgatory.
> 1. There is no atomic checkAndWatch functionality. So, a client typically
> first checks whether a request is satisfied or not and then register the
> watcher. However, by the time the watcher is registered, the registered item
> could already be satisfied. This item won't be satisfied until the next
> update happens or the delayed time expires, which means the watched item
> could be delayed.
> 2. FetchRequestPurgatory doesn't quite work. This is because the current
> design tries to incrementally maintain the accumulated bytes ready for fetch.
> However, this is difficult since the right time to check whether a fetch (for
> regular consumer) request is satisfied is when the high watermark moves. At
> that point, it's hard to figure out how many bytes we should incrementally
> add to each pending fetch request.
> The problem has been reported in KAFKA-1150 and KAFKA-703.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
