Ivan, I'm not super familiar with functions, couldn't a new field be added > to Function.Assignment, so that workerId X could be marked as > unschedulable. >
Currently, the assignment topic only holds data that provide a mapping between function instances and workers i.e. assignments. The topic is not responsible for keeping the state of any workers. There is currently no mechanism that manages any state of the worker. I don't think it is a good idea to pollute the existing assignment topic with state information about workers. They seem like two different things. If we really wanted to persist or log state changes for workers, we can use another topic but again i wonder if it is worth the hassle of implementing that. On Thu, Sep 2, 2021 at 12:26 PM Ivan Kelly <iv...@apache.org> wrote: > responses inline. > > > Marking a worker as unschedulable in a persisted manner is non trivial. > > We would have to introduce a new internal topic to track this or > leverage > > something like ZK. I didn't think it was worthwhile for us to add these > > new resources / potential dependencies for this feature. That is why I > > proposed that leaders track them just in memory and if a leader were to > > fail during a drain, a client can just simply restart the process. > I'm not super familiar with functions, couldn't a new field be added > to Function.Assignment, so that workerId X could be marked as > unschedulable. > > -Ivan >
