I'm trying to add protected SSL credentials to the Kafka Storage Handler. This is my first jump into the pool.
I have it working where the creds for the keystore/truststore are in JCEKS files in HDFS and the KafkaStorageHandler class loads them into the job configuration based on some new TBLPROPERTIES. This works well, but, I've got the actual keystore and truststore for the Kafka consumer distributed to each node in my test cluster and residing on the local Linux FS. The Kafka consumer configs then reference them using /path/to/file. Now I want to also put the truststore/keystore into HDFS, then use the distributed cache to download them to the local filesystems and have the Kafka consumer config reference them relative to the container path. What I'm confused about is kind-of the what and where of execution environments. For example: A non-job query like 'SELECT * FROM <kafka topic>' does not spawn a Tez job and runs completely on the client (HS2). In this case, will my dist cache idea work for storage of the Kafka consumer SSL keystore/truststore files? What other mechanisms can I use to get a client on HS2 access to these if they are in HDFS? Thanks, Kris
