Hi,
I'm working with Flink 1.20 (DataStream API, java 11) and I have a question
regarding the current capabilities for implementing a custom restart strategy .
I've gone through the documentation, but it’s not entirely clear whether it's
possible in this version to define a fully custom strategy beyond the built-in
ones like fixed-delay , failure-rate , etc.
Here's the use case I’m trying to address:
In my job, there are several points where I may encounter controlled
(recoverable or expected) exceptions—such as business validation failures.
Rather than handling these exceptions individually in each operator, I would
like to centralize the logic in one place.
The idea is to wrap all these controlled failure cases in a generic exception
(e.g., ControlledFailureException ) and intercept them from within a custom
restart strategy. When this specific exception is caught, I want to perform
some custom logic—specifically, send a message to a Kafka topic with error
details—and potentially avoid restarting the job altogether. For all other
unhandled exceptions, I would like Flink to apply a standard restart strategy
like fixed-delay .
I also considered using side outputs to emit these error messages, but as far
as I know, side outputs are limited to specific functions and wouldn’t help in
cases where the exception originates from, say, a SinkFunction .
So my questions are:
1.
Is it possible in Flink 1.20 to implement a fully custom restart strategy, with
conditional logic based on the type of exception thrown?
2.
If so, is there any guidance, example, or entry point in the API you would
recommend exploring?
3.
If not, is there any alternative approach to centralize the handling of
controlled exceptions across the pipeline without duplicating logic in every
operator?
Any insights or experience would be greatly appreciated.
Thanks in advance!
Best regards,
Aian
--
-----------------------------------------
Aian Cantabrana
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
telf.: +34 747421343
ofic.: +34 944272119
-----------------------------------------
Esta comunicación y su contenido, incluidos los datos personales, son
confidenciales, y para uso exclusivo del destinatario/s al/os que se dirige/n.
Si ha recibido este mensaje por error, le rogamos nos lo notifique
inmediatamente por esta misma vía y proceda a su eliminación. Consulte
información detallada sobre el tratamiento de datos en zylk.net
