[
https://issues.apache.org/jira/browse/HUDI-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Goenka updated HUDI-8591:
--------------------------------
Description:
Even without setting any concurrency related configuration, deltastreamer code
is failing with `Caused by: java.lang.IllegalArgumentException: Property
hoodie.write.lock.zookeeper.lock_key not found`.
After setting zookeeper configs it works fine.
Full stack trace
```
Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable to
instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
at
org.apache.hudi.common.table.timeline.TimeGeneratorBase.getLockProvider(TimeGeneratorBase.java:93)
at
org.apache.hudi.common.table.timeline.TimeGeneratorBase.unlock(TimeGeneratorBase.java:117)
at
org.apache.hudi.common.table.timeline.SkewAdjustingTimeGenerator.generateTime(SkewAdjustingTimeGenerator.java:51)
at
org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.lambda$createNewInstantTime$0(HoodieInstantTimeGenerator.java:78)
at
java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:179)
at
org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.createNewInstantTime(HoodieInstantTimeGenerator.java:75)
at
org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:541)
at
org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:529)
at
org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:515)
at
org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:504)
at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:902)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:222)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:637)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
... 28 more
Caused by: java.lang.IllegalArgumentException: Property
hoodie.write.lock.zookeeper.lock_key not found
at
org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:358)
at
org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:332)
at
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.getLockKey(ZookeeperBasedLockProvider.java:55)
at
org.apache.hudi.client.transaction.lock.BaseZookeeperBasedLockProvider.<init>(BaseZookeeperBasedLockProvider.java:70)
at
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.<init>(ZookeeperBasedLockProvider.java:44)
... 33 more
```
was:
Even without setting any concurrency related configuration, deltastreamer code
is failing with `Caused by: java.lang.IllegalArgumentException: Property
hoodie.write.lock.zookeeper.lock_key not found`.
After setting zookeeper configs it works fine.
> Deltastreamer failing with zookeeper lock configs even with single_writer
> -------------------------------------------------------------------------
>
> Key: HUDI-8591
> URL: https://issues.apache.org/jira/browse/HUDI-8591
> Project: Apache Hudi
> Issue Type: Bug
> Components: deltastreamer
> Reporter: Aditya Goenka
> Priority: Blocker
> Fix For: 1.0.0
>
>
> Even without setting any concurrency related configuration, deltastreamer
> code is failing with `Caused by: java.lang.IllegalArgumentException: Property
> hoodie.write.lock.zookeeper.lock_key not found`.
> After setting zookeeper configs it works fine.
>
> Full stack trace
> ```
> Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable
> to instantiate class
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
> at
> org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
> at
> org.apache.hudi.common.table.timeline.TimeGeneratorBase.getLockProvider(TimeGeneratorBase.java:93)
> at
> org.apache.hudi.common.table.timeline.TimeGeneratorBase.unlock(TimeGeneratorBase.java:117)
> at
> org.apache.hudi.common.table.timeline.SkewAdjustingTimeGenerator.generateTime(SkewAdjustingTimeGenerator.java:51)
> at
> org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.lambda$createNewInstantTime$0(HoodieInstantTimeGenerator.java:78)
> at
> java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:179)
> at
> org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.createNewInstantTime(HoodieInstantTimeGenerator.java:75)
> at
> org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:541)
> at
> org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:529)
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:515)
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:504)
> at
> org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:902)
> at
> org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:222)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:637)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
> ... 28 more
> Caused by: java.lang.IllegalArgumentException: Property
> hoodie.write.lock.zookeeper.lock_key not found
> at
> org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:358)
> at
> org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:332)
> at
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.getLockKey(ZookeeperBasedLockProvider.java:55)
> at
> org.apache.hudi.client.transaction.lock.BaseZookeeperBasedLockProvider.<init>(BaseZookeeperBasedLockProvider.java:70)
> at
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.<init>(ZookeeperBasedLockProvider.java:44)
> ... 33 more
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
