[
https://issues.apache.org/jira/browse/FLINK-23719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl updated FLINK-23719:
-------------------------
Description:
My application scenario is like this:
Use Flink SQL to write the batch data of hive table to a new HBase table every
day.
In this case, I think it is appropriate to disable WAL for the following
reasons:
1. If an error occurs, i can re-run it. No high security is required.
2. The write speed will be greatly improved. No additional writing to the wal
disk log
3. Save disk IO resources of the cluster
However, the HBase official website is described below,
h4. 70.6.10. Disabling the WAL
It is possible to disable the WAL, to improve performance in certain specific
situations. However, disabling the WAL puts your data at risk. The only
situation where this is recommended is during a bulk load. This is because, in
the event of a problem, the bulk load can be re-run with no risk of data loss.
The WAL is disabled by calling the HBase client field
{{Mutation.writeToWAL(false)}}. Use the
{{Mutation.setDurability(Durability.SKIP_WAL)}} and Mutation.getDurability()
methods to set and get the field’s value. There is no way to disable the WAL
for only a specific table.
[http://hbase.apache.org/2.3/book.html#hbase_default_configurations]
So i think it needs API level support, not properties support.
was:
My application scenario is like this:
Use flick SQL to write the batch data of hive table to a new HBase table every
day.
In this case, I think it is appropriate to disable WAL for the following
reasons:
1. If an error occurs, i can re-run it. No high security is required.
2. The write speed will be greatly improved. No additional writing to the wal
disk log
3. Save disk IO resources of the cluster
However, the HBase official website is described below,
h4. 70.6.10. Disabling the WAL
It is possible to disable the WAL, to improve performance in certain specific
situations. However, disabling the WAL puts your data at risk. The only
situation where this is recommended is during a bulk load. This is because, in
the event of a problem, the bulk load can be re-run with no risk of data loss.
The WAL is disabled by calling the HBase client field
{{Mutation.writeToWAL(false)}}. Use the
{{Mutation.setDurability(Durability.SKIP_WAL)}} and Mutation.getDurability()
methods to set and get the field’s value. There is no way to disable the WAL
for only a specific table.
[http://hbase.apache.org/2.3/book.html#hbase_default_configurations]
So i think it needs API level support, not properties support.
> Support switch WAL in Flink SQL DDL options
> --------------------------------------------
>
> Key: FLINK-23719
> URL: https://issues.apache.org/jira/browse/FLINK-23719
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / HBase
> Affects Versions: 1.13.2
> Reporter: Carl
> Priority: Major
>
> My application scenario is like this:
> Use Flink SQL to write the batch data of hive table to a new HBase table
> every day.
> In this case, I think it is appropriate to disable WAL for the following
> reasons:
> 1. If an error occurs, i can re-run it. No high security is required.
> 2. The write speed will be greatly improved. No additional writing to the wal
> disk log
> 3. Save disk IO resources of the cluster
>
> However, the HBase official website is described below,
> h4. 70.6.10. Disabling the WAL
> It is possible to disable the WAL, to improve performance in certain specific
> situations. However, disabling the WAL puts your data at risk. The only
> situation where this is recommended is during a bulk load. This is because,
> in the event of a problem, the bulk load can be re-run with no risk of data
> loss.
> The WAL is disabled by calling the HBase client field
> {{Mutation.writeToWAL(false)}}. Use the
> {{Mutation.setDurability(Durability.SKIP_WAL)}} and Mutation.getDurability()
> methods to set and get the field’s value. There is no way to disable the WAL
> for only a specific table.
> [http://hbase.apache.org/2.3/book.html#hbase_default_configurations]
>
> So i think it needs API level support, not properties support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
