Hi Ryan,
There are some limits, as the configs are stored inside an internal
topic of Connect. So the usual message size and producer request size
limitations apply.
You can reconfigure the internal topic to allow larger messages than the
default (I think it's 1MB), and the producer max request size for the
Connect worker.
Besides that, there is one extra bottleneck - Connect followers might
forward the task configs to the leader through the REST API. The default
request size limit of the REST client can also cause an issue, not sure
if there is a way to reconfigure that.
I think you should be able to find some kind of a related error message,
probably in the leader of the cluster, or the worker which was hosting
the connector itself.
Daniel
2022. 05. 19. 17:01 keltezéssel, Ryan Slominski írta:
Hi,
Are there limits to the size of configuration data passed via the
taskConfigs method of the Connector class? I'm observing a situation
where if I use a large configuration no tasks are created, and no log
messages appear in the connect log file. Using a smaller configuration
works. If there are limits, can I increase them? Also, it's probably a
good idea if Kafka were to log a warning message of some kind in this
scenario instead of silently failing. I'm using a custom Source Connector
and I have documented steps to reproduce the issue using Docker compose
here:
https://github.com/JeffersonLab/epics2kafka/issues/11
Thanks for any insights!
Ryan
--
Ezt az e-mailt az Avast víruskereső szoftver átvizsgálta.
https://www.avast.com/antivirus
