Hi,
I am using direct kafka with checkpointing of offsets same as :
https://github.com/koeninger/kafka-exactly-once/blob/master/src/main/scala/example/IdempotentExample.scala
I need to change some parameters like db connection params :
username/password for db connection .
I stopped streaming gracefully ,changed parameters in config file and
restarted streaming.
*Issue : changed parameters username/password are not being considered.*
*Question* :
As per my understanding , Checkpointing should only save offsets of kafka
partitions and not the credentials of the db connection.
Why its picking old db connection params ?
I am declaring params in main method and not in setUpSsc(0 method.
My code is identical to that in the above program link as below:
val jdbcDriver = conf.getString("jdbc.driver")
val jdbcUrl = conf.getString("jdbc.url")
*val jdbcUser = conf.getString("jdbc.user")*
* val jdbcPassword = conf.getString("jdbc.password")*
// while the job doesn't strictly need checkpointing,
// we'll checkpoint to avoid replaying the whole kafka log in case of
failure
val checkpointDir = conf.getString("checkpointDir")
val ssc = StreamingContext.getOrCreate(
checkpointDir,
setupSsc(topics, kafkaParams, jdbcDriver, jdbcUrl, *jdbcUser*,
*jdbcPassword*, checkpointDir) _
)
--
Chandan Prakash
