Moderators note. Colin is not subscribed to the mailing list. I’ve added them to the cc:
> On Apr 10, 2024, at 9:56 AM, Colin Redmond <colin.redm...@outlook.com> wrote: > > Hello, > > I am the author of the Gestalt library and I am honored that you are > considering leveraging my library for your project. > > There are a lot of really great configuration libraries out there to chose > from, and I see you have already done your to do your due diligence in > reviewing many (i would recommend against Lightbend, as it has been put into > maintenance mode and does not support newer features liked java records). > > I think my zero dependency configuration library that is modular, extensible > and has a large number of features and functionality would be a great fit for > your project. It is great you already have a clear idea of how to leverage > the library to decouple your configs and increase your security posture. > > I am usually quick to reply to issues or bugs and I am open to pull requests. > If i can help answer any questions feel free to reach out. > > > Thanks, > > Colin Redmond > > > On 2024/02/21 09:26:19 Lari Hotari wrote: > > Hello everyone, > > > > I would like to bring up an issue with Pulsar's containers, specifically > > regarding the method of overriding configurations. For instance, the > > Apache Pulsar Helm chart employs "bin/apply-config-from-env.py > > conf/broker.conf" and "bin/gen-yml-from-env.py > > conf/functions_worker.yml" [1] to apply configurations passed in the > > environment to the configuration files in the container's root file system. > > This approach fails when the container's root file system is read-only due > > to > > strict security policies (`readOnlyRootFilesystem` in > > `securityContext`). This issue has been reported as #22088 [2]. > > > > A temporary fix could involve using a temporary file to modify the > > configuration file when the filesystem is read-only. However, the Python > > script solution is not ideal, and we should consider eliminating it. In > > the long term, it would also be beneficial to remove the need for a > > shell script to start Pulsar, but that's a separate issue. > > > > For configuration handling, we need a solution that can apply overrides > > in memory, eliminating the need to modify on-disk files. Modern > > configuration frameworks can do this out-of-the-box. Currently, Pulsar > > uses a homegrown configuration framework. Instead of extending this > > framework, I propose we discuss replacing it with the Gestalt Config > > library [3]. This library, licensed under Apache-2.0, is a mature, > > well-established solution for configuration handling. > > > > Switching to Gestalt Config would allow us to move towards a more > > structured and modular configuration in Pulsar. Our current > > configuration is not modular, as it relies on a "god object" for > > configuration, which collects all possible configuration options. > > Gestalt Config offers modular usage patterns similar to those of > > Spring Boot's external configuration [4] and the MicroProfile Config [5] > > in Quarkus. However, Gestalt Config does not pull in other dependencies, > > giving it an advantage over Spring Boot and Quarkus configuration solutions. > > Other libraries in this category include the Typesafe config library [6] > > from Lightbend with HOCON [7], commonly used in Scala and Akka-based > > applications. > > > > Gestalt Config supports many configuration file formats, including flat > > properties files, yaml, json, toml, and even hocon. It also offers > > security features for reading secrets directly from Vault, AWS Secrets > > Manager, and GCP Secret Manager, without the need to use the file system > > or environment variables to inject secrets into the application > > configuration. This could significantly improve Pulsar's security > > posture. > > > > Pulsar's current "homegrown configuration framework" is quite simple, > > implemented in a few classes with the main logic in > > PulsarConfigurationLoader [8] and FieldParser [9] classes, called from > > the PulsarBrokerStarter class [10]. > > > > The main question is: should we continue extending Pulsar's homegrown > > configuration framework, or should we consider adopting a library like > > Gestalt Config for future configuration use case improvements for > > modularity, structured configuration, and security? > > > > Best regards, > > > > Lari > > > > References: > > 1 - > > https://github.com/apache/pulsar-helm-chart/blob/29ea17b3fceef65160620b9018d0dd0449a168c5/charts/pulsar/templates/broker-statefulset.yaml#L210-L221 > > 2 - https://github.com/apache/pulsar/issues/22088 > > 3 - https://github.com/gestalt-config/gestalt > > 4 - > > https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config > > 5 - https://microprofile.io/specifications/microprofile-config/ > > 6 - https://github.com/lightbend/config > > 7 - https://github.com/lightbend/config/blob/main/HOCON.md > > 8 - > > https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PulsarConfigurationLoader.java > > 9 - > > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/FieldParser.java > > 10 - > > https://github.com/apache/pulsar/blob/db79096baaa3d7118aa026978a615ddc576f9183/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L69-L76 > >
