So, I tried with another query which hangs and I get this, after its corresponding TERMINATE command:
[2020-04-17 15:25:15,796] INFO stream-client [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:263) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:1211) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:1211) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:1211) [2020-04-17 15:25:15,801] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220) [2020-04-17 15:25:15,806] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] Shutting down (org.apache.kafka.streams.processor.internals.StreamThread:1225) [2020-04-17 15:25:15,816] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] State transition from PENDING_SHUTDOWN to DEAD (org.apache.kafka.streams.processor.internals.StreamThread:220) [2020-04-17 15:25:15,816] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] Shutdown complete (org.apache.kafka.streams.processor.internals.StreamThread:1245) [2020-04-17 15:25:15,830] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] Shutting down (org.apache.kafka.streams.processor.internals.StreamThread:1225) [2020-04-17 15:25:15,839] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] State transition from PENDING_SHUTDOWN to DEAD (org.apache.kafka.streams.processor.internals.StreamThread:220) *[2020-04-17 15:25:15,839] INFO stream-thread [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] Shutdown complete (org.apache.kafka.streams.processor.internals.StreamThread:1245)* So, at least this query tries to shutdown but only a single thread remains alive, apparently. So: 1) CLI does not get any return (Query has been terminated) 2) If I open another terminal, if i DESCRIBE EXTENDED the corresponding stream, the query looks still attached to the stream and doesn't let me drop it 3) Consumer group does not get deleted If I restart the server: 1) then, again, the query looks still attached to the stream (DESCRIBE EXTENDED) 3) Still, I cannot drop the stream because the query is attached This is a really strange behaviour, does anyone have any insight or any pointers as to what I should look into? Thank you, Il giorno ven 17 apr 2020 alle ore 11:22 Federico D'Ambrosio < fedex...@gmail.com> ha scritto: > Looking back at the ksql-stream logs: > > I find for a query I'm trying to terminate this line: > > [2020-04-17 09:17:52,502] INFO stream-client > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_PREV_GARANZIE_INTERNAL_AVRO_180-2625ef40-c00a-4cdf-98ea-df239aa2f29d] > *Already in the pending shutdown state*, wait to complete shutdown > (org.apache.kafka.streams.KafkaStreams:849) > > and it just stays there, without any more information. > Restarting the server apparently doesn't help. > > Il giorno ven 17 apr 2020 alle ore 11:07 Federico D'Ambrosio < > fedex...@gmail.com> ha scritto: > >> Hi, >> >> We deployed 30-something queries which convert json events to avro, but >> lately I've noticed some of the queries are consuming very slowly, >> accumulating quite a bit of lag. >> In order to reset those streams, I tried to terminate those queries, to >> then recreate the streams, but the TERMINATE command from KSQL cli just >> hangs without any feedback. >> >> Ksql log is not much helpful, really just showing the POST request being >> sent. >> >> How should we proceed? Is there anyway to force deletion of a >> query/stream? >> What could be causing this issue? >> >> Thank you, >> Federico D'Ambrosio >> > > > -- > Federico D'Ambrosio > -- Federico D'Ambrosio
