All,
Following log4jshell advisory [1], I want to start a discussion thread around
what next steps should we follow.
Overview: The log4j 1.x has limited feature set and didn't have the same attack
surface as log4j 2.x, this saved the ACS community from log4jshell [1]. In
addition, our uber/fat jar build process since (last few years) only bundles
classes from dependency jars that CloudStack management server codebase
requires; this may benefit that certain classes which could be exploit-able in
1.x weren't actually shadowed/bundled in the cloudstack-management uber/fat
jar[2]. This strategy reduces the CloudStack management uber/far jar's attack
surface greatly.
What is used in CloudStack: (details investigated with my colleague Daan)
* Basic logging feature (console/file/rsyslog appenders, log levels, log4j
config file)
* Context-aware logging (this includes full/abbreviated class/package name,
and sometimes specific context,
LogContext and CallContext have some example use of MDC, NDC)
* Stacktrace upon exception, a custom CglibThrowableRenderer class
* Jar dependencies used: log4j v1.2.17, and apache-log4j-extras v1.2.17
As the next step, I think it would be better to refactor the logging usage to a
new loggging utility in cloud-utils which can be used throughout the codebase,
and this utility can in-turn wrap whatever logging library we use and reduce
our dependency on any logging library/version and swiftly change logging
library. The utility may also allow us to centrally add custom cloudstack
feature (such as context-aware logging, manage stacktrace output etc).
I did some research to look at possible options for the community:
1. Status-Quo, stick with log4j 1.x: this is not ideal, as v1.2.17 was the
last release in 1.x which has reach EOL [3].
2. Migrate to 2.x: this is probably what is advised by the Apache Logging
PMC [4] since 1.x has EOL, however:
* 1.x pretty much sufficed our requirements of a logging library and 2.x
has a large attack surface.
*
* Other Apache projects (such as hadoop, hive etc) are facing while
migrating their large codebases to log4j 2.x; the migration will require time,
effort, and testing. The 2.x migration guide would require a lot of manual
effort, there is no tooling to do this at scale.
* On Twitter, some of the Logging PMC have advised that their v3
architecture will make log4j modular and lower attach surfaces.
* My opinion: Logging library shouldn't try to do anything beyond
logging, I'm less confident in migrating to 2.x after seeing how the CVE was
handled by the Logging PMC and I didn't have a good interaction with them in a
ML discussion thread. For an ASF TLP that is asked to adhere to the 'community
over code' spirit, the user community isn't listend to and in a ML thread, it
was deemed that any 1.x fork/maintenance would be considered "hostile" - this
has led to interested sponsors and PMCs of other Apache projects joining forces
with Ceki to continue log4j 1.x fork outside of ASF (see below).
3. Migrate to 1.x fork, drop-in replacement: Ceki (one of the developers of
log4j 1.x, also behind sl4j and logback) has started a fork of 1.x under the
name reload4j with which they continue 1.x support and address bugs/CVEs. They
have already published v1.2.18 https://github.com/qos-ch/reload4j this is
already finding a lot of support with a lot of users.
4. Migrate to logback: This is another project by Ceki/qos-ch, which aims to
continue 1.x work while keeping logging library robust/small, uses sl4j.
https://logback.qos.ch/
Thoughts, what/where should we go, any other alternatives we should explore?
[1] https://blogs.apache.org/cloudstack/entry/log4jshell
[2] https://maven.apache.org/plugins/maven-shade-plugin/
[3]
https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
[4] https://logging.apache.org/log4j/2.x/
Regards.
