nsivabalan commented on a change in pull request #2698:
URL: https://github.com/apache/hudi/pull/2698#discussion_r598345959
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
Review comment:
May be worth mentioning what all possible values for this
config(hoodie.writer.lock.provider) available out of the box. and the default
value if any.
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
+```
+
+There are 2 different server based lock providers that require different
configuration to be set.
+
+**`Zookeeper`** based lock provider
+
+```
+hoodie.writer.lock.zookeeper.url
+hoodie.writer.lock.zookeeper.port
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+hoodie.writer.lock.lock_key
+hoodie.writer.lock.zookeeper.zk_base_path
+```
+
+**`HiveMetastore`** based lock provider
+
+```
+hoodie.writer.lock.hivemetastore.database
+hoodie.writer.lock.hivemetastore.table
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+```
+
+`The HiveMetastore URI's are picked up from the hadoop configuration file
loaded during runtime.`
+
+## Datasource Writer
+
+The `hudi-spark` module offers the DataSource API to write (and read) a Spark
DataFrame into a Hudi table.
+
+Following is an example of how to use optimistic_concurrency_control via spark
datasource
+
+```java
+inputDF.write.format("hudi")
+ .options(getQuickstartWriteConfigs)
+ .option(PRECOMBINE_FIELD_OPT_KEY, "ts")
+ .option("hoodie.failed.writes.cleaner.policy", "LAZY")
+ .option("hoodie.write.concurrency.mode",
"optimistic_concurrency_control")
+ .option("hoodie.writer.lock.zookeeper.url", "zookeeper")
+ .option("hoodie.writer.lock.zookeeper.port", "2181")
+ .option("hoodie.writer.lock.wait_time_ms", "12000")
+ .option("hoodie.writer.lock.num_retries", "2")
+ .option("hoodie.writer.lock.lock_key", "test_table")
+ .option("hoodie.writer.lock.zookeeper.zk_base_path", "/test")
+ .option(RECORDKEY_FIELD_OPT_KEY, "uuid")
+ .option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath")
+ .option(TABLE_NAME, tableName)
+ .mode(Overwrite)
+ .save(basePath)
+```
+
+## DeltaStreamer
+
+The `HoodieDeltaStreamer` utility (part of hudi-utilities-bundle) provides
ways to ingest from different sources such as DFS or Kafka, with the following
capabilities.
+
+Using optimistic_concurrency_control via delta streamer requires adding the
above configs to the properties file that can be passed to the
+job. For example below, adding the configs to kafka-source.properties file and
passing them to deltastreamer will enable optimistic concurrency.
+A deltastreamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit --class
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls
packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props
file://${PWD}/hudi-utilities/src/test/resources/delta-streamer-config/kafka-source.properties
\
+ --schemaprovider-class
org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-deltastreamer-op \
+ --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+## Best Practices when using Optimistic Concurrency Control
+
+Concurrent Writing to Hudi tables requires acquiring a lock with either
Zookeeper or HiveMetastore. Due to several reasons you might want to configure
retries to allow your application to acquire the lock.
+1. Network connectivity or excessive load on servers increasing time for lock
acquisition resulting in timeouts
+2. Running a large number of concurrent jobs that are writing to the same hudi
table can result in contention during lock acquisition can cause timeouts
+3. In some scenarios of conflict resolution, Hudi commit operations might take
upto 10's of seconds while the lock is being held. This can result in timeouts
for other jobs waiting to acquire a lock.
+
+Set the correct native lock provider client retries. NOTE that sometimes these
settings are set on the server once and all clients inherit the same configs.
Please check your settings before enabling optimistic concurrency.
+
+```
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+```
+
+Set the correct hudi client retries for Zookeeper & HiveMetastore. This is
useful in cases when native client retry settings cannot be changed. Please
note that these retries will happen in addition to any native client retries
that you may have set.
+
+```
+hoodie.writer.lock.client.wait_time_ms
+hoodie.writer.lock.client.num_retries
+```
+
+*Setting the right values for these depends on a case by case basis; some
defaults have been provided for general cases.*
Review comment:
may be add a line about interface for LockProvider and users can add
their own implementation if interested.
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
+```
+
+There are 2 different server based lock providers that require different
configuration to be set.
+
+**`Zookeeper`** based lock provider
+
+```
+hoodie.writer.lock.zookeeper.url
Review comment:
can we also add hoodie.writer.lock.provider here.
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
+```
+
+There are 2 different server based lock providers that require different
configuration to be set.
+
+**`Zookeeper`** based lock provider
+
+```
+hoodie.writer.lock.zookeeper.url
+hoodie.writer.lock.zookeeper.port
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+hoodie.writer.lock.lock_key
+hoodie.writer.lock.zookeeper.zk_base_path
+```
+
+**`HiveMetastore`** based lock provider
+
+```
+hoodie.writer.lock.hivemetastore.database
Review comment:
same here as well hoodie.writer.lock.provider
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
+```
+
+There are 2 different server based lock providers that require different
configuration to be set.
+
+**`Zookeeper`** based lock provider
+
+```
+hoodie.writer.lock.zookeeper.url
+hoodie.writer.lock.zookeeper.port
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+hoodie.writer.lock.lock_key
+hoodie.writer.lock.zookeeper.zk_base_path
+```
+
+**`HiveMetastore`** based lock provider
+
+```
+hoodie.writer.lock.hivemetastore.database
+hoodie.writer.lock.hivemetastore.table
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+```
+
+`The HiveMetastore URI's are picked up from the hadoop configuration file
loaded during runtime.`
+
+## Datasource Writer
+
+The `hudi-spark` module offers the DataSource API to write (and read) a Spark
DataFrame into a Hudi table.
+
+Following is an example of how to use optimistic_concurrency_control via spark
datasource
+
+```java
+inputDF.write.format("hudi")
+ .options(getQuickstartWriteConfigs)
+ .option(PRECOMBINE_FIELD_OPT_KEY, "ts")
+ .option("hoodie.failed.writes.cleaner.policy", "LAZY")
+ .option("hoodie.write.concurrency.mode",
"optimistic_concurrency_control")
+ .option("hoodie.writer.lock.zookeeper.url", "zookeeper")
+ .option("hoodie.writer.lock.zookeeper.port", "2181")
+ .option("hoodie.writer.lock.wait_time_ms", "12000")
+ .option("hoodie.writer.lock.num_retries", "2")
+ .option("hoodie.writer.lock.lock_key", "test_table")
+ .option("hoodie.writer.lock.zookeeper.zk_base_path", "/test")
+ .option(RECORDKEY_FIELD_OPT_KEY, "uuid")
+ .option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath")
+ .option(TABLE_NAME, tableName)
+ .mode(Overwrite)
+ .save(basePath)
+```
+
+## DeltaStreamer
+
+The `HoodieDeltaStreamer` utility (part of hudi-utilities-bundle) provides
ways to ingest from different sources such as DFS or Kafka, with the following
capabilities.
+
+Using optimistic_concurrency_control via delta streamer requires adding the
above configs to the properties file that can be passed to the
+job. For example below, adding the configs to kafka-source.properties file and
passing them to deltastreamer will enable optimistic concurrency.
+A deltastreamer job can then be triggered as follows:
+
+```java
+[hoodie]$ spark-submit --class
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls
packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-*.jar` \
+ --props
file://${PWD}/hudi-utilities/src/test/resources/delta-streamer-config/kafka-source.properties
\
+ --schemaprovider-class
org.apache.hudi.utilities.schema.SchemaRegistryProvider \
+ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
+ --source-ordering-field impresssiontime \
+ --target-base-path file:\/\/\/tmp/hudi-deltastreamer-op \
+ --target-table uber.impressions \
+ --op BULK_INSERT
+```
+
+## Best Practices when using Optimistic Concurrency Control
+
+Concurrent Writing to Hudi tables requires acquiring a lock with either
Zookeeper or HiveMetastore. Due to several reasons you might want to configure
retries to allow your application to acquire the lock.
+1. Network connectivity or excessive load on servers increasing time for lock
acquisition resulting in timeouts
+2. Running a large number of concurrent jobs that are writing to the same hudi
table can result in contention during lock acquisition can cause timeouts
+3. In some scenarios of conflict resolution, Hudi commit operations might take
upto 10's of seconds while the lock is being held. This can result in timeouts
for other jobs waiting to acquire a lock.
+
+Set the correct native lock provider client retries. NOTE that sometimes these
settings are set on the server once and all clients inherit the same configs.
Please check your settings before enabling optimistic concurrency.
+
+```
+hoodie.writer.lock.wait_time_ms
+hoodie.writer.lock.num_retries
+```
+
+Set the correct hudi client retries for Zookeeper & HiveMetastore. This is
useful in cases when native client retry settings cannot be changed. Please
note that these retries will happen in addition to any native client retries
that you may have set.
+
+```
+hoodie.writer.lock.client.wait_time_ms
+hoodie.writer.lock.client.num_retries
+```
+
+*Setting the right values for these depends on a case by case basis; some
defaults have been provided for general cases.*
Review comment:
I am assuming you are going to cover about upgrade/downgrade steps else
where.
But even if not for downgrade, please call out if there are any steps
required to switch off multi-writing after trying it out. Lets say someone
tries multi-writing for sometime and then after running into some issues,
wishes to switch it off with 0.8.0. Is there anything to be considered, or just
disabling multi-writer config param is more than enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
