> It would be nice to just have the config library to read the environment variables directly, instead of having to write them down in the file.
I guess it should be easy to have it with the existing homegrown configuration solution without completely moving to a new lib. For the other points such as security, versatile configuration format. Adopting a library like Gestalt Config looks good to me. Regards, Penghui On Thu, Feb 22, 2024 at 1:25 AM Matteo Merli <matteo.me...@gmail.com> wrote: > Good idea. > > It would be nice to just have the config library to read the environment > variables directly, instead of having to write them down in the file. > > > -- > Matteo Merli > <matteo.me...@gmail.com> > > > On Wed, Feb 21, 2024 at 1:28 AM Lari Hotari <lhot...@apache.org> wrote: > > > Hello everyone, > > > > I would like to bring up an issue with Pulsar's containers, specifically > > regarding the method of overriding configurations. For instance, the > > Apache Pulsar Helm chart employs "bin/apply-config-from-env.py > > conf/broker.conf" and "bin/gen-yml-from-env.py > > conf/functions_worker.yml" [1] to apply configurations passed in the > > environment to the configuration files in the container's root file > > system. > > This approach fails when the container's root file system is read-only > due > > to > > strict security policies (`readOnlyRootFilesystem` in > > `securityContext`). This issue has been reported as #22088 [2]. > > > > A temporary fix could involve using a temporary file to modify the > > configuration file when the filesystem is read-only. However, the Python > > script solution is not ideal, and we should consider eliminating it. In > > the long term, it would also be beneficial to remove the need for a > > shell script to start Pulsar, but that's a separate issue. > > > > For configuration handling, we need a solution that can apply overrides > > in memory, eliminating the need to modify on-disk files. Modern > > configuration frameworks can do this out-of-the-box. Currently, Pulsar > > uses a homegrown configuration framework. Instead of extending this > > framework, I propose we discuss replacing it with the Gestalt Config > > library [3]. This library, licensed under Apache-2.0, is a mature, > > well-established solution for configuration handling. > > > > Switching to Gestalt Config would allow us to move towards a more > > structured and modular configuration in Pulsar. Our current > > configuration is not modular, as it relies on a "god object" for > > configuration, which collects all possible configuration options. > > Gestalt Config offers modular usage patterns similar to those of > > Spring Boot's external configuration [4] and the MicroProfile Config [5] > > in Quarkus. However, Gestalt Config does not pull in other dependencies, > > giving it an advantage over Spring Boot and Quarkus configuration > > solutions. > > Other libraries in this category include the Typesafe config library [6] > > from Lightbend with HOCON [7], commonly used in Scala and Akka-based > > applications. > > > > Gestalt Config supports many configuration file formats, including flat > > properties files, yaml, json, toml, and even hocon. It also offers > > security features for reading secrets directly from Vault, AWS Secrets > > Manager, and GCP Secret Manager, without the need to use the file system > > or environment variables to inject secrets into the application > > configuration. This could significantly improve Pulsar's security > > posture. > > > > Pulsar's current "homegrown configuration framework" is quite simple, > > implemented in a few classes with the main logic in > > PulsarConfigurationLoader [8] and FieldParser [9] classes, called from > > the PulsarBrokerStarter class [10]. > > > > The main question is: should we continue extending Pulsar's homegrown > > configuration framework, or should we consider adopting a library like > > Gestalt Config for future configuration use case improvements for > > modularity, structured configuration, and security? > > > > Best regards, > > > > Lari > > > > References: > > 1 - > > > https://github.com/apache/pulsar-helm-chart/blob/29ea17b3fceef65160620b9018d0dd0449a168c5/charts/pulsar/templates/broker-statefulset.yaml#L210-L221 > > 2 - https://github.com/apache/pulsar/issues/22088 > > 3 - https://github.com/gestalt-config/gestalt > > 4 - > > > https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config > > 5 - https://microprofile.io/specifications/microprofile-config/ > > 6 - https://github.com/lightbend/config > > 7 - https://github.com/lightbend/config/blob/main/HOCON.md > > 8 - > > > https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PulsarConfigurationLoader.java > > 9 - > > > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/FieldParser.java > > 10 - > > > https://github.com/apache/pulsar/blob/db79096baaa3d7118aa026978a615ddc576f9183/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L69-L76 > > >
