avshenuk opened a new issue, #16425:
URL: https://github.com/apache/pinot/issues/16425
According to the new plugin system (and verified from the code), each plugin
folder and all its .jar files are isolated and loaded via the whole
ClassWorld/Realm loading system, which is great and it works but...
there is one critical flaw, and it's not in the Java code. It's in the Bash
scripts loading Pinot.
In `appAssemblerScriptTemplate` we can see that absolutely every single JAR
from the plugin folders (regardless whether it uses the old or new loading) is
added to the classpath:
```
# Set $PLUGINS_CLASSPATH for plugin jars to be put into classpath.
# $PLUGINS_DIR and $PLUGINS_INCLUDE are used if $PLUGINS_CLASSPATH is not
set.
# $PLUGINS_DIR is semi-colon separated list of plugin directories, default
to '"$BASEDIR"/plugins' if not set.
# $PLUGINS_INCLUDE is semi-colon separated plugins name, e.g.
pinot-avro;pinot-batch-ingestion-standalone. Default is not set, which means
load all the plugin jars.
# If the same plugin is found in multiple directories, it will only be
picked up from the first directory it was found. The directories in
$PLUGINS_DIR are traversed by their order.
if [ -z "$PLUGINS_CLASSPATH" ] ; then
....
PLUGIN_JARS=$(find "$DIR" -name \*.jar)
for PLUGIN_JAR in $PLUGIN_JARS ; do
if [ -n "$PLUGINS_CLASSPATH" ] ; then
PLUGINS_CLASSPATH=$PLUGINS_CLASSPATH:$PLUGIN_JAR
else
PLUGINS_CLASSPATH=$PLUGIN_JAR
fi
done
fi
...
if [ -n "$PLUGINS_CLASSPATH" ] ; then
CLASSPATH=$CLASSPATH:$PLUGINS_CLASSPATH
fi
```
So as a result, any dependency you are using: Pinot or not, is going to
override whatever Pinot is using internally.
Already tested this and got a mismatch of classes.
P.S. There is also a small issue in that the Bash script overrides the
plugins dir System property:
```
if [ -z "$PLUGINS_CLASSPATH" ] ; then
if [ -z "$PLUGINS_DIR" ] ; then
PLUGINS_DIR="$BASEDIR"/plugins
...
if [ -z "$PLUGINS_DIR" ] ; then
PLUGINS_DIR=$BASEDIR/plugins
fi
ALL_JAVA_OPTS="$ALL_JAVA_OPTS -Dplugins.dir=$PLUGINS_DIR"
```
Which means the System property set in e.g. K8S Helm charts is completely
ignored. This is what is currently present in the chart
```
env:
- name: JAVA_OPTS
value: "{{ .Values.server.jvmOpts }}
-Dlog4j2.configurationFile={{ .Values.server.log4j2ConfFile }} -Dplugins.dir={{
.Values.server.pluginsDir }}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
