veenaypatil commented on issue #6014:
URL: https://github.com/apache/hudi/issues/6014#issuecomment-1203915099
@minihippo the task logs only shows the consumer logs and it is stuck after
that.
```
2/08/03 11:02:40 INFO KafkaRDD: Computing topic
in.split.prod.hspay.subscriptions.partner_subscriptions, partition 1 offsets
3422973 -> 3497480
22/08/03 11:02:40 INFO KafkaRDD: Computing topic
in.split.prod.hspay.subscriptions.apple_partner_subscriptions, partition 0
offsets 6038717 -> 6222344
22/08/03 11:02:40 INFO KafkaRDD: Computing topic
in.split.prod.hspay.subscriptions.bsnl_partner_subscriptions, partition 1
offsets 31179 -> 32046
22/08/03 11:02:40 INFO CodeGenerator: Code generated in 15.559109 ms
22/08/03 11:02:40 INFO InternalKafkaConsumer: Initial fetch for
spark-executor-hudi_in_hspay_subs
in.split.prod.hspay.subscriptions.apple_partner_subscriptions-0 6038717
22/08/03 11:02:40 INFO InternalKafkaConsumer: Initial fetch for
spark-executor-hudi_in_hspay_subs
in.split.prod.hspay.subscriptions.partner_subscriptions-1 3422973
22/08/03 11:02:40 INFO KafkaConsumer: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-2,
groupId=spark-executor-hudi_in_hspay_subs] Seeking to offset 6038717 for
partition in.split.prod.hspay.subscriptions.apple_partner_subscriptions-0
22/08/03 11:02:40 INFO InternalKafkaConsumer: Initial fetch for
spark-executor-hudi_in_hspay_subs
in.split.prod.hspay.subscriptions.bsnl_partner_subscriptions-1 31179
22/08/03 11:02:40 INFO KafkaConsumer: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-3,
groupId=spark-executor-hudi_in_hspay_subs] Seeking to offset 3422973 for
partition in.split.prod.hspay.subscriptions.partner_subscriptions-1
22/08/03 11:02:40 INFO KafkaConsumer: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-1,
groupId=spark-executor-hudi_in_hspay_subs] Seeking to offset 31179 for
partition in.split.prod.hspay.subscriptions.bsnl_partner_subscriptions-1
22/08/03 11:02:40 INFO FetchSessionHandler: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-1,
groupId=spark-executor-hudi_in_hspay_subs] Error sending fetch request
(sessionId=307489077, epoch=2) to node 35: {}.
org.apache.kafka.common.errors.DisconnectException
22/08/03 11:02:40 INFO FetchSessionHandler: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-3,
groupId=spark-executor-hudi_in_hspay_subs] Error sending fetch request
(sessionId=1858238483, epoch=26) to node 7: {}.
org.apache.kafka.common.errors.DisconnectException
22/08/03 11:02:40 INFO Executor: Finished task 21.0 in stage 285.0 (TID
14173). 1611 bytes result sent to driver
22/08/03 11:02:40 INFO FetchSessionHandler: [Consumer
clientId=consumer-spark-executor-hudi_in_hspay_subs-2,
groupId=spark-executor-hudi_in_hspay_subs] Error sending fetch request
(sessionId=1695836972, epoch=3257) to node 11: {}.
org.apache.kafka.common.errors.DisconnectException
22/08/03 11:02:44 INFO Executor: Finished task 52.0 in stage 285.0 (TID
14197). 1611 bytes result sent to driver
22/08/03 11:04:25 INFO Executor: Finished task 29.0 in stage 285.0 (TID
14185). 1654 bytes result sent to driver
22/08/03 11:30:45 INFO BlockManager: Removing RDD 495
22/08/03 11:30:45 INFO BlockManager: Removing RDD 559
22/08/03 11:30:45 INFO BlockManager: Removing RDD 536
22/08/03 11:30:45 INFO BlockManager: Removing RDD 544
```
`[mapToPair at SparkWriteHelper.java:63]` This stage is running and I pasted
the above log from one of the running task.
This is causing a lot of issues when we try to kill the job and restart it.
It only happens for the first batch. Like today the first is running from 24hrs
<img width="992" alt="Screenshot 2022-08-03 at 6 26 19 PM"
src="https://user-images.githubusercontent.com/52563354/182613311-f3b9a93d-1f9e-4ea6-a7da-a693293571a6.png";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://user-images.githubusercontent.com/52563354/182613311-f3b9a93d-1f9e-4ea6-a7da-a693293571a6.png"](https://user-images.githubusercontent.com/52563354/182613311-f3b9a93d-1f9e-4ea6-a7da-a693293571a6.png"){kind=link}