Re: Streamtasks not instantiating consumers on startup

2020-04-11 Thread Malcolm McFarland
Thanks Yi. We recovered this time around by processing all of the checkpoints manually, deleting and recreating the checkpoint topic, then re-populating the checkpoint topic with the saved offsets using the checkpoint tool,. If it happens again, I'll turn up logging to try and get more information.

Re: Streamtasks not instantiating consumers on startup

2020-04-10 Thread Yi Pan
Hi, Malcolm, Samza 0.14.1 is pretty old and if you have already upgraded Kafka to 2.2.1, I would highly recommend you to migrate to latest Samza version, which has Kafka 2.0 client. Depending on how you configure your broker, a 2.0 Kafka broker can be incompatible with older client like 0.8.6, sin

Streamtasks not instantiating consumers on startup

2020-04-09 Thread Malcolm McFarland
Hey folks, We're occasionally seeing an issue when starting Samza containers where none of the streamtasks for a job will instantiate consumers. We'll see the beginnings of an attempt to read the checkpoint stream and nothing further (including no errors). All of these streamtasks seem to be creat