Ivan,
I'm not super familiar with functions, couldn't a new field be added
> to Function.Assignment, so that workerId X could be marked as
> unschedulable.
>
Currently, the assignment topic only holds data that provide a mapping
between function instances and workers i.e. assignments. The topic
responses inline.
> Marking a worker as unschedulable in a persisted manner is non trivial.
> We would have to introduce a new internal topic to track this or leverage
> something like ZK. I didn't think it was worthwhile for us to add these
> new resources / potential dependencies for this feat
Ivan, thanks for reviewing my proposal. I will answer your questions inline.
When a leader fails, does the new leader automatically create a new
> assignment, or does it continue with the assignment from the previous
> leader?
>
The new leader will resume scheduling duties with the current set of
> When the leader receives a request to drain a worker, it must first mark
> the worker as in the process to be drained i.e. blacklist the worker so
> that no new assignments can be assigned to it. We can perhaps just save the
> blacklist in memory. The worker should then create a new scheduling in
It is useful to be able to autoscale the number of Pulsar Function workers
when configuring the cluster to run instances as threads (ThreadRuntime) in
an environment like K8s. When scaling out the number of workers, the
"rebalance" endpoint can be involved after the scale out so that the new
worke
