abhiNB-star commented on issue #13410:
URL: https://github.com/apache/hudi/issues/13410#issuecomment-3135394749
@yihua @danny0405
after i migrated an existing table from hudi 0.13 to 0.14.1 it was running
good
but when i migrated it from 0.14.1 to 1.0.2
i am getting these kind of logs which are running from past 10-11 hours
`25/07/30 08:45:25 INFO SparkContext: Created broadcast 1897 from broadcast
at DAGScheduler.scala:1585
25/07/30 08:45:25 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 1901 (MapPartitionsRDD[3821] at mapToPair at
HoodieSparkEngineContext.java:175) (first 15 tasks are for partitions Vector(0))
25/07/30 08:45:25 INFO TaskSchedulerImpl: Adding task set 1901.0 with 1
tasks resource profile 0
25/07/30 08:45:25 INFO TaskSetManager: Starting task 0.0 in stage 1901.0
(TID 4113) (10.51.2.243, executor 3, partition 0, PROCESS_LOCAL, 10151 bytes)
25/07/30 08:45:25 INFO BlockManagerInfo: Added broadcast_1897_piece0 in
memory on 10.51.2.243:33361 (size: 90.8 KiB, free: 970.1 MiB)
25/07/30 08:45:25 INFO TaskSetManager: Finished task 0.0 in stage 1901.0
(TID 4113) in 68 ms on 10.51.2.243 (executor 3) (1/1)
25/07/30 08:45:25 INFO TaskSchedulerImpl: Removed TaskSet 1901.0, whose
tasks have all completed, from pool
25/07/30 08:45:25 INFO DAGScheduler: ResultStage 1901 (collectAsMap at
HoodieSparkEngineContext.java:178) finished in 0.093 s
25/07/30 08:45:25 INFO DAGScheduler: Job 1896 is finished. Cancelling
potential speculative or zombie tasks for this job
25/07/30 08:45:25 INFO TaskSchedulerImpl: Killing all running tasks in stage
1901: Stage finished
25/07/30 08:45:25 INFO DAGScheduler: Job 1896 finished: collectAsMap at
HoodieSparkEngineContext.java:178, took 0.097804 s
25/07/30 08:45:25 INFO HoodieLogFileReader: Closing Log file reader
.commits_.archive.1119_1-0-1
25/07/30 08:45:25 INFO CodecPool: Got brand-new compressor [.gz]
25/07/30 08:45:25 INFO LSMTimelineWriter: Writing schema
{"type":"record","name":"HoodieLSMTimelineInstant","namespace":"org.apache.hudi.avro.model","fields":[{"name":"instantTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"completionTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"action","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"metadata","type":["null","bytes"],"default":null},{"name":"plan","type":["null","bytes"],"default":null},{"name":"version","type":"int","default":1}]}
25/07/30 08:45:26 INFO BlockManagerInfo: Removed broadcast_1897_piece0 on
spark-fc6a30985a5a4d6d-driver-svc.spark-hood.svc:7079 in memory (size: 90.8
KiB, free: 886.7 MiB)
25/07/30 08:45:26 INFO BlockManagerInfo: Removed broadcast_1897_piece0 on
10.51.2.243:33361 in memory (size: 90.8 KiB, free: 970.2 MiB)
25/07/30 08:45:26 INFO BlockManagerInfo: Removed broadcast_1896_piece0 on
spark-fc6a30985a5a4d6d-driver-svc.spark-hood.svc:7079 in memory (size: 90.8
KiB, free: 886.8 MiB)
25/07/30 08:45:26 INFO BlockManagerInfo: Removed broadcast_1896_piece0 on
10.51.0.231:40967 in memory (size: 90.8 KiB, free: 970.2 MiB)
25/07/30 08:45:26 INFO SparkContext: Starting job: collectAsMap at
HoodieSparkEngineContext.java:178
25/07/30 08:45:26 INFO DAGScheduler: Got job 1897 (collectAsMap at
HoodieSparkEngineContext.java:178) with 1 output partitions
25/07/30 08:45:26 INFO DAGScheduler: Final stage: ResultStage 1902
(collectAsMap at HoodieSparkEngineContext.java:178)
25/07/30 08:45:26 INFO DAGScheduler: Parents of final stage: List()
25/07/30 08:45:26 INFO DAGScheduler: Missing parents: List()
25/07/30 08:45:26 INFO DAGScheduler: Submitting ResultStage 1902
(MapPartitionsRDD[3823] at mapToPair at HoodieSparkEngineContext.java:175),
which has no missing parents
25/07/30 08:45:26 INFO MemoryStore: Block broadcast_1898 stored as values in
memory (estimated size 241.3 KiB, free 886.6 MiB)
25/07/30 08:45:26 INFO MemoryStore: Block broadcast_1898_piece0 stored as
bytes in memory (estimated size 90.8 KiB, free 886.5 MiB)
25/07/30 08:45:26 INFO BlockManagerInfo: Added broadcast_1898_piece0 in
memory on spark-fc6a30985a5a4d6d-driver-svc.spark-hood.svc:7079 (size: 90.8
KiB, free: 886.7 MiB)
25/07/30 08:45:26 INFO SparkContext: Created broadcast 1898 from broadcast
at DAGScheduler.scala:1585
25/07/30 08:45:26 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 1902 (MapPartitionsRDD[3823] at mapToPair at
HoodieSparkEngineContext.java:175) (first 15 tasks are for partitions Vector(0))
25/07/30 08:45:26 INFO TaskSchedulerImpl: Adding task set 1902.0 with 1
tasks resource profile 0
25/07/30 08:45:26 INFO TaskSetManager: Starting task 0.0 in stage 1902.0
(TID 4114) (10.51.3.66, executor 4, partition 0, PROCESS_LOCAL, 10151 bytes)
25/07/30 08:45:26 INFO BlockManagerInfo: Added broadcast_1898_piece0 in
memory on 10.51.3.66:43689 (size: 90.8 KiB, free: 970.1 MiB)
25/07/30 08:45:26 INFO TaskSetManager: Finished task 0.0 in stage 1902.0
(TID 4114) in 89 ms on 10.51.3.66 (executor 4) (1/1)
25/07/30 08:45:26 INFO TaskSchedulerImpl: Removed TaskSet 1902.0, whose
tasks have all completed, from pool
25/07/30 08:45:26 INFO DAGScheduler: ResultStage 1902 (collectAsMap at
HoodieSparkEngineContext.java:178) finished in 0.117 s
25/07/30 08:45:26 INFO DAGScheduler: Job 1897 is finished. Cancelling
potential speculative or zombie tasks for this job
25/07/30 08:45:26 INFO TaskSchedulerImpl: Killing all running tasks in stage
1902: Stage finished
25/07/30 08:45:26 INFO DAGScheduler: Job 1897 finished: collectAsMap at
HoodieSparkEngineContext.java:178, took 0.118602 s
25/07/30 08:45:27 INFO CodecPool: Got brand-new compressor [.gz]
25/07/30 08:45:27 INFO LSMTimelineWriter: Writing schema
{"type":"record","name":"HoodieLSMTimelineInstant","namespace":"org.apache.hudi.avro.model","fields":[{"name":"instantTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"completionTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"action","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"metadata","type":["null","bytes"],"default":null},{"name":"plan","type":["null","bytes"],"default":null},{"name":"version","type":"int","default":1}]}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
