mimaison commented on code in PR #16650: URL: https://github.com/apache/kafka/pull/16650#discussion_r1686448199
########## docs/toc.html: ########## @@ -22,61 +22,62 @@ <ul class="toc"> <li><a href="#gettingStarted">1. Getting Started</a> <ul> - <li><a href="#introduction">1.1 Introduction</a> - <li><a href="#uses">1.2 Use Cases</a> - <li><a href="#quickstart">1.3 Quick Start</a> - <li><a href="#ecosystem">1.4 Ecosystem</a> - <li><a href="#upgrade">1.5 Upgrading</a> - <li><a href="#docker">1.6 Docker</a> + <li><a href="#introduction">1.1 Introduction</a></li> Review Comment: As per the HTML spec, the closing `li` tag is optional: https://www.w3.org/TR/2012/WD-html-markup-20121025/li.html#li-tags So let's undo these changes. ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. + </p> + + <p>You have the following options:</p> + <ul> + <li> + Use a custom provider by creating a Java class implementing the + <code>ConfigProvider</code> interface and packaging it into a JAR file. + </li> + <li>Use a built-in provider:</li> + <ul> + <li><code>DirectoryConfigProvider</code></li> + <li><code>EnvVarConfigProvider</code></li> + <li><code>FileConfigProvider</code></li> + </ul> + </ul> + + <p> + To use a configuration provider, specify it in your configuration file using the <code>config.providers</code> property. + </p> + <p> + Each provider can have its own set of parameters, which are passed in a specific format. + </p> + + <h4 class="anchor-heading"> + <a id="common_provider_params" class="anchor-link"></a> + <a href="#common_provider_params">Common Parameters</a> + </h4> + + <p>All configuration providers share a common way of passing parameters.</p> + <p> + You specify a comma-separated list of aliases and the fully-specified class that implements the configuration provider: + </p> + <pre><code>config.providers=provider1,provider2 +config.providers.provider1.class=com.example.Provider1 +config.providers.provider2.class=com.example.Provider2</code></pre> + <p> + Pass additional parameters to each provider as <code>config.providers.<provider_alias>.<parameter_key>=<parameter_value></code>: + </p> + <pre><code>config.providers.provider1.param1=value1 +config.providers.provider1.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="config_provider" class="anchor-link"></a> + <a href="#config_provider">ConfigProvider</a> + </h4> + + <p> + The <code>ConfigProvider</code> is a base interface for all configuration providers. Custom implementations of this interface can retrieve configuration data from various sources. You can then package and deploy the implementation as a JAR file and reference the provider's class in your configuration. + </p> + + <p><b>Example custom provider configuration</b></p> + <pre><code>config.providers=customProvider +config.providers.customProvider.class=com.example.customProvider +config.providers.customProvider.param1=value1 +config.providers.customProvider.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="directory_config_provider" class="anchor-link"></a> + <a href="#directory_config_provider">DirectoryConfigProvider</a> + </h4> + + <p> + The <code>DirectoryConfigProvider</code> retrieves configuration data from files stored in a specified directory. + </p> + <p> + Each file represents a key, and its content is the value. You specify the path to the directory where the configuration files are stored using the <code>dir</code> parameter. This provider is useful for loading multiple configuration files and for organizing configuration data into separate files. + </p> + + <p><b>Example <code>DirectoryConfigProvider</code> configuration</b></p> + <pre><code>config.providers=dirProvider +config.providers.dirProvider.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider +config.providers.dirProvider.param.dir=/path/to/config/directory</code></pre> + + <h4 class="anchor-heading"> + <a id="env_var_config_provider" class="anchor-link"></a> + <a href="#env_var_config_provider">EnvVarConfigProvider</a> + </h4> + + <p>The <code>EnvVarConfigProvider</code> retrieves configuration data from environment variables.</p> + <p>No specific parameters are required, as it reads directly from the specified environment variables.</p> + <p> + This provider is useful for configuring applications running in containers, for example, to load certificates or JAAS configuration from environment variables mapped from secrets. + </p> + + <p><b>Example <code>EnvVarConfigProvider</code> configuration</b></p> + <pre><code>config.providers=envVarProvider +config.providers.envVarProvider.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider</code></pre> + + <h4 class="anchor-heading"> + <a id="file_config_provider" class="anchor-link"></a> + <a href="#file_config_provider">FileConfigProvider</a> + </h4> + + <p>The <code>FileConfigProvider</code> retrieves configuration data from a single properties file.</p> + <p>You specify the path to the properties file using the <code>file</code> parameter.</p> + <p>This provider is useful for loading configuration data from mounted files.</p> + + <p><b>Example <code>FileConfigProvider</code> configuration</b></p> + <pre><code>config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.param.file=/path/to/config.properties</code></pre> + + <h4 class="anchor-heading"> + <a id="ref_config_provider" class="anchor-link"></a> + <a href="#ref_config_provider">Referencing configuration providers</a> + </h4> + <p> + To reference a property value supplied by a configuration provider, use the correct placeholder syntax: + </p> + <dl> + <dt><b><code>DirectoryConfigProvider</code></b></dt> + <dd> + <p><code>${dirProvider:<path_to_file>:<file_name>}</code></p> + </dd> + <dt><code><b>EnvVarConfigProvider</code></b></dt> + <dd> + <p><code>${envVarProvider:<enVar_name>}</code></p> + </dd> + + <dt><b><code>FileConfigProvider</code></b></dt> + <dd> + <p><code>${fileProvider:<path_and_filename>:<property>}</code></p> + </dd> + </dl> + <p> + Here’s an example that uses a file configuration provider with Kafka Connect to provide authentication credentials to a database for a connector. + </p> + <p> + First, create a <code>connector.properties</code> configuration file with the following credentials: Review Comment: Should we name that file `credentials.properties`, or `connector-credentials.properties` so it's not confused with the actual connector configurations that can also be provided as a file. ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. + </p> + + <p>You have the following options:</p> + <ul> + <li> + Use a custom provider by creating a Java class implementing the + <code>ConfigProvider</code> interface and packaging it into a JAR file. Review Comment: Same below for the built-in providers ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. + </p> + + <p>You have the following options:</p> + <ul> + <li> + Use a custom provider by creating a Java class implementing the + <code>ConfigProvider</code> interface and packaging it into a JAR file. Review Comment: Could we link to the javadoc for `ConfigProvider`? To point to the latest javadoc, you can use `{{version}}/javadoc/org/apache/kafka/common/config/provider/ConfigProvider.html` ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. Review Comment: `Configuration providers are primarily used with Kafka Connect.`: I don't think this is true. Configuration providers are used by all components. I wonder if we could drop this paragraph altogether. ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. + </p> + + <p>You have the following options:</p> + <ul> + <li> + Use a custom provider by creating a Java class implementing the + <code>ConfigProvider</code> interface and packaging it into a JAR file. + </li> + <li>Use a built-in provider:</li> + <ul> + <li><code>DirectoryConfigProvider</code></li> + <li><code>EnvVarConfigProvider</code></li> + <li><code>FileConfigProvider</code></li> + </ul> + </ul> + + <p> + To use a configuration provider, specify it in your configuration file using the <code>config.providers</code> property. + </p> + <p> + Each provider can have its own set of parameters, which are passed in a specific format. + </p> + + <h4 class="anchor-heading"> + <a id="common_provider_params" class="anchor-link"></a> + <a href="#common_provider_params">Common Parameters</a> + </h4> + + <p>All configuration providers share a common way of passing parameters.</p> + <p> + You specify a comma-separated list of aliases and the fully-specified class that implements the configuration provider: + </p> + <pre><code>config.providers=provider1,provider2 +config.providers.provider1.class=com.example.Provider1 +config.providers.provider2.class=com.example.Provider2</code></pre> + <p> + Pass additional parameters to each provider as <code>config.providers.<provider_alias>.<parameter_key>=<parameter_value></code>: + </p> + <pre><code>config.providers.provider1.param1=value1 +config.providers.provider1.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="config_provider" class="anchor-link"></a> + <a href="#config_provider">ConfigProvider</a> + </h4> + + <p> + The <code>ConfigProvider</code> is a base interface for all configuration providers. Custom implementations of this interface can retrieve configuration data from various sources. You can then package and deploy the implementation as a JAR file and reference the provider's class in your configuration. + </p> + + <p><b>Example custom provider configuration</b></p> + <pre><code>config.providers=customProvider +config.providers.customProvider.class=com.example.customProvider +config.providers.customProvider.param1=value1 +config.providers.customProvider.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="directory_config_provider" class="anchor-link"></a> + <a href="#directory_config_provider">DirectoryConfigProvider</a> + </h4> + + <p> + The <code>DirectoryConfigProvider</code> retrieves configuration data from files stored in a specified directory. + </p> + <p> + Each file represents a key, and its content is the value. You specify the path to the directory where the configuration files are stored using the <code>dir</code> parameter. This provider is useful for loading multiple configuration files and for organizing configuration data into separate files. + </p> + + <p><b>Example <code>DirectoryConfigProvider</code> configuration</b></p> + <pre><code>config.providers=dirProvider +config.providers.dirProvider.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider +config.providers.dirProvider.param.dir=/path/to/config/directory</code></pre> + + <h4 class="anchor-heading"> + <a id="env_var_config_provider" class="anchor-link"></a> + <a href="#env_var_config_provider">EnvVarConfigProvider</a> + </h4> + + <p>The <code>EnvVarConfigProvider</code> retrieves configuration data from environment variables.</p> + <p>No specific parameters are required, as it reads directly from the specified environment variables.</p> + <p> + This provider is useful for configuring applications running in containers, for example, to load certificates or JAAS configuration from environment variables mapped from secrets. + </p> + + <p><b>Example <code>EnvVarConfigProvider</code> configuration</b></p> + <pre><code>config.providers=envVarProvider +config.providers.envVarProvider.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider</code></pre> + + <h4 class="anchor-heading"> + <a id="file_config_provider" class="anchor-link"></a> + <a href="#file_config_provider">FileConfigProvider</a> + </h4> + + <p>The <code>FileConfigProvider</code> retrieves configuration data from a single properties file.</p> + <p>You specify the path to the properties file using the <code>file</code> parameter.</p> + <p>This provider is useful for loading configuration data from mounted files.</p> + + <p><b>Example <code>FileConfigProvider</code> configuration</b></p> + <pre><code>config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.param.file=/path/to/config.properties</code></pre> + + <h4 class="anchor-heading"> + <a id="ref_config_provider" class="anchor-link"></a> + <a href="#ref_config_provider">Referencing configuration providers</a> + </h4> + <p> + To reference a property value supplied by a configuration provider, use the correct placeholder syntax: + </p> + <dl> + <dt><b><code>DirectoryConfigProvider</code></b></dt> + <dd> + <p><code>${dirProvider:<path_to_file>:<file_name>}</code></p> + </dd> + <dt><code><b>EnvVarConfigProvider</code></b></dt> + <dd> + <p><code>${envVarProvider:<enVar_name>}</code></p> + </dd> + + <dt><b><code>FileConfigProvider</code></b></dt> + <dd> + <p><code>${fileProvider:<path_and_filename>:<property>}</code></p> + </dd> + </dl> + <p> + Here’s an example that uses a file configuration provider with Kafka Connect to provide authentication credentials to a database for a connector. + </p> + <p> + First, create a <code>connector.properties</code> configuration file with the following credentials: + </p> + <p><b>Example configuration file</b></p> +<pre><code>dbUsername=my-username +dbPassword=my-password</code></pre> + <p><em>NOTE: Ensure the appropriate permissions are set for securely accessing the properties file. + For additional security, consider encrypting sensitive information using a secret and implementing decryption in your application at runtime. + </em></p> + <p> + Specify a <code>FileConfigProvider</code> in the Kafka Connect configuration: + </p> + + <p><b>Example Kafka Connect configuration with an <code>FileConfigProvider</code></b></p> + <pre><code>config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.config.path=/path/to/connector-properties</code></pre> + + <p>Next, reference the properties from the file in the connector configuration.</p> + <p><b>Example connector configuration referencing file properties</b></p> + <pre><code>database.user=${file:/path/to/connector.properties:dbUsername} +database.password=${file:/path/to/connector.properties:dbPassword}</code></pre> + <p>The configuration provider reads and extracts the values from the properties file.</p> + + <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> Review Comment: We added a new section at 6.6, so we need to bump all section numbers below ########## docs/ops.html: ########## @@ -1247,7 +1247,168 @@ <h4 class="anchor-heading"><a id="prodconfig" class="anchor-link"></a><a href="# Our client configuration varies a fair amount between different use cases. - <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a href="#java">6.6 Java Version</a></h3> + <h3 class="anchor-heading"> + <a id="config_providers" class="anchor-link"></a> + <a href="#config_providers">6.6 Configuration Providers</a> + </h3> + + <p> + Use configuration providers to load configuration data from external sources. This might include sensitive information, such as passwords, API keys, or other credentials. + </p> + <p> + These configurations can be stored securely and dynamically retrieved as needed. Configuration providers are primarily used with Kafka Connect. For example, you can use providers to supply the credentials for Kafka Connect connectors. + </p> + + <p>You have the following options:</p> + <ul> + <li> + Use a custom provider by creating a Java class implementing the + <code>ConfigProvider</code> interface and packaging it into a JAR file. + </li> + <li>Use a built-in provider:</li> + <ul> + <li><code>DirectoryConfigProvider</code></li> + <li><code>EnvVarConfigProvider</code></li> + <li><code>FileConfigProvider</code></li> + </ul> + </ul> + + <p> + To use a configuration provider, specify it in your configuration file using the <code>config.providers</code> property. + </p> + <p> + Each provider can have its own set of parameters, which are passed in a specific format. + </p> + + <h4 class="anchor-heading"> + <a id="common_provider_params" class="anchor-link"></a> + <a href="#common_provider_params">Common Parameters</a> + </h4> + + <p>All configuration providers share a common way of passing parameters.</p> + <p> + You specify a comma-separated list of aliases and the fully-specified class that implements the configuration provider: + </p> + <pre><code>config.providers=provider1,provider2 +config.providers.provider1.class=com.example.Provider1 +config.providers.provider2.class=com.example.Provider2</code></pre> + <p> + Pass additional parameters to each provider as <code>config.providers.<provider_alias>.<parameter_key>=<parameter_value></code>: + </p> + <pre><code>config.providers.provider1.param1=value1 +config.providers.provider1.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="config_provider" class="anchor-link"></a> + <a href="#config_provider">ConfigProvider</a> + </h4> + + <p> + The <code>ConfigProvider</code> is a base interface for all configuration providers. Custom implementations of this interface can retrieve configuration data from various sources. You can then package and deploy the implementation as a JAR file and reference the provider's class in your configuration. + </p> + + <p><b>Example custom provider configuration</b></p> + <pre><code>config.providers=customProvider +config.providers.customProvider.class=com.example.customProvider +config.providers.customProvider.param1=value1 +config.providers.customProvider.param2=value2</code></pre> + + <h4 class="anchor-heading"> + <a id="directory_config_provider" class="anchor-link"></a> + <a href="#directory_config_provider">DirectoryConfigProvider</a> + </h4> + + <p> + The <code>DirectoryConfigProvider</code> retrieves configuration data from files stored in a specified directory. + </p> + <p> + Each file represents a key, and its content is the value. You specify the path to the directory where the configuration files are stored using the <code>dir</code> parameter. This provider is useful for loading multiple configuration files and for organizing configuration data into separate files. + </p> + + <p><b>Example <code>DirectoryConfigProvider</code> configuration</b></p> + <pre><code>config.providers=dirProvider +config.providers.dirProvider.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider +config.providers.dirProvider.param.dir=/path/to/config/directory</code></pre> + + <h4 class="anchor-heading"> + <a id="env_var_config_provider" class="anchor-link"></a> + <a href="#env_var_config_provider">EnvVarConfigProvider</a> + </h4> + + <p>The <code>EnvVarConfigProvider</code> retrieves configuration data from environment variables.</p> + <p>No specific parameters are required, as it reads directly from the specified environment variables.</p> + <p> + This provider is useful for configuring applications running in containers, for example, to load certificates or JAAS configuration from environment variables mapped from secrets. + </p> + + <p><b>Example <code>EnvVarConfigProvider</code> configuration</b></p> + <pre><code>config.providers=envVarProvider +config.providers.envVarProvider.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider</code></pre> + + <h4 class="anchor-heading"> + <a id="file_config_provider" class="anchor-link"></a> + <a href="#file_config_provider">FileConfigProvider</a> + </h4> + + <p>The <code>FileConfigProvider</code> retrieves configuration data from a single properties file.</p> + <p>You specify the path to the properties file using the <code>file</code> parameter.</p> + <p>This provider is useful for loading configuration data from mounted files.</p> + + <p><b>Example <code>FileConfigProvider</code> configuration</b></p> + <pre><code>config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.param.file=/path/to/config.properties</code></pre> + + <h4 class="anchor-heading"> + <a id="ref_config_provider" class="anchor-link"></a> + <a href="#ref_config_provider">Referencing configuration providers</a> + </h4> + <p> + To reference a property value supplied by a configuration provider, use the correct placeholder syntax: + </p> + <dl> + <dt><b><code>DirectoryConfigProvider</code></b></dt> + <dd> + <p><code>${dirProvider:<path_to_file>:<file_name>}</code></p> + </dd> + <dt><code><b>EnvVarConfigProvider</code></b></dt> + <dd> + <p><code>${envVarProvider:<enVar_name>}</code></p> + </dd> + + <dt><b><code>FileConfigProvider</code></b></dt> + <dd> + <p><code>${fileProvider:<path_and_filename>:<property>}</code></p> + </dd> + </dl> + <p> + Here’s an example that uses a file configuration provider with Kafka Connect to provide authentication credentials to a database for a connector. + </p> + <p> + First, create a <code>connector.properties</code> configuration file with the following credentials: + </p> + <p><b>Example configuration file</b></p> +<pre><code>dbUsername=my-username +dbPassword=my-password</code></pre> + <p><em>NOTE: Ensure the appropriate permissions are set for securely accessing the properties file. + For additional security, consider encrypting sensitive information using a secret and implementing decryption in your application at runtime. + </em></p> + <p> + Specify a <code>FileConfigProvider</code> in the Kafka Connect configuration: + </p> + + <p><b>Example Kafka Connect configuration with an <code>FileConfigProvider</code></b></p> + <pre><code>config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.config.path=/path/to/connector-properties</code></pre> + + <p>Next, reference the properties from the file in the connector configuration.</p> + <p><b>Example connector configuration referencing file properties</b></p> + <pre><code>database.user=${file:/path/to/connector.properties:dbUsername} Review Comment: In this example, `file` should be `fileProvider` as the first part of the configuration provider magic string is the alias given to the provider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org