AFAIK this is not possible; the client doesn't know anything about the
cluster configuration.
FLINK-15747 proposes to add an additional config option for controlling
the logging behavior.
The only workaround I can think of would be to create a custom Flink
distribution with a modified RocksDBStateBackend which always sets these
options by default.
On 23/04/2020 03:24, Bajaj, Abhinav wrote:
Bumping this one again to catch some attention.
*From: *"Bajaj, Abhinav" <abhinav.ba...@here.com>
*Date: *Monday, April 20, 2020 at 3:23 PM
*To: *"user@flink.apache.org" <user@flink.apache.org>
*Subject: *RocksDB default logging configuration
Hi,
Some of our teams ran into the disk space issues because of RocksDB
default logging configuration - FLINK-15068
<https://issues.apache.org/jira/browse/FLINK-15068>.
It seems the workaround suggested uses the OptionsFactory to set some
of the parameters from inside the job.
Since we provision the Flink cluster(version 1.7.1) for the teams, we
control the RocksDB statebackend configuration from flink-conf.yaml.
And it seems there isn’t any related RocksDB configuration
<https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#rocksdb-state-backend>
to set in flink-conf.yaml.
Is there a way for the job developer to retrieve the default
statebackend information from the cluster in the job and set the
DBOptions on top of it?
Appreciate the help!
~ Abhinav Bajaj
PS: Sharing below snippet as desired option if possible -
StreamExecutionEnvironment streamExecEnv =
StreamExecutionEnvironment./getExecutionEnvironment/();
StateBackend stateBackend = streamExecEnv.getDefaultStateBackend();
stateBackend.setOptions(new OptionsFactory() {
@Override
public DBOptions createDBOptions(DBOptions dbOptions) {
dbOptions.setInfoLogLevel(InfoLogLevel./WARN_LEVEL/);
dbOptions.setMaxLogFileSize(1024 * 1024)
return dbOptions;
}
@Override
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions
columnFamilyOptions) {
return columnFamilyOptions;
}
});
