Hi Federico, I'd think you could re-post this question here: https://github.com/confluentinc/ksql/issues.
Cheers, Guozhang On Fri, Apr 17, 2020 at 8:57 AM Federico D'Ambrosio <fedex...@gmail.com> wrote: > So, I tried with another query which hangs and I get this, after its > corresponding TERMINATE command: > > [2020-04-17 15:25:15,796] INFO stream-client > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43] > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.KafkaStreams:263) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319] > Informed to shut down > (org.apache.kafka.streams.processor.internals.StreamThread:1211) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319] > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.processor.internals.StreamThread:220) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] > Informed to shut down > (org.apache.kafka.streams.processor.internals.StreamThread:1211) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.processor.internals.StreamThread:220) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] > Informed to shut down > (org.apache.kafka.streams.processor.internals.StreamThread:1211) > [2020-04-17 15:25:15,801] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.processor.internals.StreamThread:220) > [2020-04-17 15:25:15,806] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] > Shutting down > (org.apache.kafka.streams.processor.internals.StreamThread:1225) > [2020-04-17 15:25:15,816] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] > State transition from PENDING_SHUTDOWN to DEAD > (org.apache.kafka.streams.processor.internals.StreamThread:220) > [2020-04-17 15:25:15,816] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320] > Shutdown complete > (org.apache.kafka.streams.processor.internals.StreamThread:1245) > [2020-04-17 15:25:15,830] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] > Shutting down > (org.apache.kafka.streams.processor.internals.StreamThread:1225) > [2020-04-17 15:25:15,839] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] > State transition from PENDING_SHUTDOWN to DEAD > (org.apache.kafka.streams.processor.internals.StreamThread:220) > *[2020-04-17 15:25:15,839] INFO stream-thread > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321] > Shutdown complete > (org.apache.kafka.streams.processor.internals.StreamThread:1245)* > > So, at least this query tries to shutdown but only a single thread remains > alive, apparently. So: > > 1) CLI does not get any return (Query has been terminated) > 2) If I open another terminal, if i DESCRIBE EXTENDED the corresponding > stream, the query looks still attached to the stream and doesn't let me > drop it > 3) Consumer group does not get deleted > > If I restart the server: > 1) then, again, the query looks still attached to the stream (DESCRIBE > EXTENDED) > 3) Still, I cannot drop the stream because the query is attached > > This is a really strange behaviour, does anyone have any insight or any > pointers as to what I should look into? > > Thank you, > > Il giorno ven 17 apr 2020 alle ore 11:22 Federico D'Ambrosio < > fedex...@gmail.com> ha scritto: > > > Looking back at the ksql-stream logs: > > > > I find for a query I'm trying to terminate this line: > > > > [2020-04-17 09:17:52,502] INFO stream-client > > > [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_PREV_GARANZIE_INTERNAL_AVRO_180-2625ef40-c00a-4cdf-98ea-df239aa2f29d] > > *Already in the pending shutdown state*, wait to complete shutdown > > (org.apache.kafka.streams.KafkaStreams:849) > > > > and it just stays there, without any more information. > > Restarting the server apparently doesn't help. > > > > Il giorno ven 17 apr 2020 alle ore 11:07 Federico D'Ambrosio < > > fedex...@gmail.com> ha scritto: > > > >> Hi, > >> > >> We deployed 30-something queries which convert json events to avro, but > >> lately I've noticed some of the queries are consuming very slowly, > >> accumulating quite a bit of lag. > >> In order to reset those streams, I tried to terminate those queries, to > >> then recreate the streams, but the TERMINATE command from KSQL cli just > >> hangs without any feedback. > >> > >> Ksql log is not much helpful, really just showing the POST request being > >> sent. > >> > >> How should we proceed? Is there anyway to force deletion of a > >> query/stream? > >> What could be causing this issue? > >> > >> Thank you, > >> Federico D'Ambrosio > >> > > > > > > -- > > Federico D'Ambrosio > > > > > -- > Federico D'Ambrosio > -- -- Guozhang
