layoaster opened a new issue, #19385:
URL: https://github.com/apache/druid/issues/19385
### Affected Version
34.0.0
### Description
I'm not able to pass the JVM flags to the peons when running tasks natively
on k8s (`druid-kubernetes-overlord-extensions`). Tasks fail upon startup
because they complain about not having enough DirectMemory space.
#### Peon logs:
```
2026-04-28T11:48:28+00:00 startup service overlord
2026-04-28T11:48:29,608 INFO [main]
org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator
6.2.5.Final
2026-04-28T11:48:29,972 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-basic-security], jars:
druid-basic-security-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,974 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar,
druid-datasketches-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,976 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-distinctcount], jars:
druid-distinctcount-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,979 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kafka-indexing-service], jars:
druid-kafka-indexing-service-34.0.0.jar, kafka-clients-3.9.1.jar,
lz4-java-1.8.0.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid
extension dependencies [[]]
2026-04-28T11:48:30,003 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kubernetes-overlord-extensions], jars:
druid-kubernetes-overlord-extensions-34.0.0.jar,
jackson-dataformat-yaml-2.18.4.jar, jackson-datatype-jsr310-2.18.4.jar,
kubernetes-client-7.2.0.jar, kubernetes-client-api-7.2.0.jar,
kubernetes-httpclient-vertx-7.2.0.jar,
kubernetes-model-admissionregistration-7.2.0.jar,
kubernetes-model-apiextensions-7.2.0.jar, kubernetes-model-apps-7.2.0.jar,
kubernetes-model-autoscaling-7.2.0.jar, kubernetes-model-batch-7.2.0.jar,
kubernetes-model-certificates-7.2.0.jar, kubernetes-model-common-7.2.0.jar,
kubernetes-model-coordination-7.2.0.jar, kubernetes-model-core-7.2.0.jar,
kubernetes-model-discovery-7.2.0.jar, kubernetes-model-events-7.2.0.jar,
kubernetes-model-extensions-7.2.0.jar, kubernetes-model-flowcontrol-7.2.0.jar,
kubernetes-model-gatewayapi-7.2.0.jar, kubernetes-model-metrics-7.2.0.jar,
kubernetes-model-networking-7.2.0.jar, kubernetes-model
-node-7.2.0.jar, kubernetes-model-policy-7.2.0.jar,
kubernetes-model-rbac-7.2.0.jar, kubernetes-model-resource-7.2.0.jar,
kubernetes-model-scheduling-7.2.0.jar, kubernetes-model-storageclass-7.2.0.jar,
netty-buffer-4.1.122.Final.jar, netty-codec-4.1.122.Final.jar,
netty-codec-dns-4.1.122.Final.jar, netty-codec-http-4.1.122.Final.jar,
netty-codec-http2-4.1.122.Final.jar, netty-codec-socks-4.1.122.Final.jar,
netty-common-4.1.122.Final.jar, netty-handler-4.1.122.Final.jar,
netty-handler-proxy-4.1.122.Final.jar, netty-resolver-4.1.122.Final.jar,
netty-resolver-dns-4.1.122.Final.jar, netty-transport-4.1.122.Final.jar,
netty-transport-native-unix-common-4.1.122.Final.jar, slf4j-api-2.0.16.jar,
snakeyaml-2.3.jar, snakeyaml-engine-2.9.jar, vertx-auth-common-4.5.14.jar,
vertx-core-4.5.14.jar, vertx-web-client-4.5.14.jar,
vertx-web-common-4.5.14.jar, zjsonpatch-7.2.0.jar. Druid extension dependencies
[[]]
2026-04-28T11:48:30,055 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-lookups-cached-global], jars:
druid-lookups-cached-global-34.0.0.jar, mapdb-1.0.8.jar. Druid extension
dependencies [[]]
2026-04-28T11:48:30,056 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-multi-stage-query], jars:
druid-multi-stage-query-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,077 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-parquet-extensions], jars: aircompressor-2.0.2.jar,
avro-1.11.4.jar, commons-logging-1.3.5.jar,
druid-parquet-extensions-34.0.0.jar, hadoop-client-api-3.3.6.jar,
hadoop-client-runtime-3.3.6.jar, javax.annotation-api-1.3.2.jar,
jsr305-3.0.2.jar, parquet-avro-1.15.2.jar, parquet-column-1.15.2.jar,
parquet-common-1.15.2.jar, parquet-encoding-1.15.2.jar,
parquet-format-structures-1.15.2.jar, parquet-hadoop-1.15.2.jar,
parquet-jackson-1.15.2.jar, slf4j-api-2.0.16.jar, snappy-java-1.1.10.7.jar,
zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,088 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-protobuf-extensions], jars:
druid-protobuf-extensions-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,092 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-s3-extensions], jars:
aws-java-sdk-core-1.12.784.jar, aws-java-sdk-sts-1.12.784.jar,
commons-codec-1.16.1.jar, commons-logging-1.3.5.jar,
druid-s3-extensions-34.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.16.jar,
jackson-dataformat-cbor-2.18.4.jar, jmespath-java-1.12.784.jar,
joda-time-2.14.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,095 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [postgresql-metadata-storage], jars:
checker-qual-3.42.0.jar, postgresql-42.7.2.jar,
postgresql-metadata-storage-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,098 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [statsd-emitter], jars: asm-9.8.jar, asm-analysis-7.1.jar,
asm-commons-9.8.jar, asm-tree-7.1.jar, asm-util-7.1.jar,
java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar,
jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar,
jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar,
jnr-x86asm-1.0.2.jar, statsd-emitter-34.0.0.jar. Druid extension dependencies
[[]]
2026-04-28T11:48:30,233 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-basic-security], jars:
druid-basic-security-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,235 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar,
druid-datasketches-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,238 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-distinctcount], jars:
druid-distinctcount-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,239 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kafka-indexing-service], jars:
druid-kafka-indexing-service-34.0.0.jar, kafka-clients-3.9.1.jar,
lz4-java-1.8.0.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid
extension dependencies [[]]
2026-04-28T11:48:30,240 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kubernetes-overlord-extensions], jars:
druid-kubernetes-overlord-extensions-34.0.0.jar,
jackson-dataformat-yaml-2.18.4.jar, jackson-datatype-jsr310-2.18.4.jar,
kubernetes-client-7.2.0.jar, kubernetes-client-api-7.2.0.jar,
kubernetes-httpclient-vertx-7.2.0.jar,
kubernetes-model-admissionregistration-7.2.0.jar,
kubernetes-model-apiextensions-7.2.0.jar, kubernetes-model-apps-7.2.0.jar,
kubernetes-model-autoscaling-7.2.0.jar, kubernetes-model-batch-7.2.0.jar,
kubernetes-model-certificates-7.2.0.jar, kubernetes-model-common-7.2.0.jar,
kubernetes-model-coordination-7.2.0.jar, kubernetes-model-core-7.2.0.jar,
kubernetes-model-discovery-7.2.0.jar, kubernetes-model-events-7.2.0.jar,
kubernetes-model-extensions-7.2.0.jar, kubernetes-model-flowcontrol-7.2.0.jar,
kubernetes-model-gatewayapi-7.2.0.jar, kubernetes-model-metrics-7.2.0.jar,
kubernetes-model-networking-7.2.0.jar, kubernetes-model
-node-7.2.0.jar, kubernetes-model-policy-7.2.0.jar,
kubernetes-model-rbac-7.2.0.jar, kubernetes-model-resource-7.2.0.jar,
kubernetes-model-scheduling-7.2.0.jar, kubernetes-model-storageclass-7.2.0.jar,
netty-buffer-4.1.122.Final.jar, netty-codec-4.1.122.Final.jar,
netty-codec-dns-4.1.122.Final.jar, netty-codec-http-4.1.122.Final.jar,
netty-codec-http2-4.1.122.Final.jar, netty-codec-socks-4.1.122.Final.jar,
netty-common-4.1.122.Final.jar, netty-handler-4.1.122.Final.jar,
netty-handler-proxy-4.1.122.Final.jar, netty-resolver-4.1.122.Final.jar,
netty-resolver-dns-4.1.122.Final.jar, netty-transport-4.1.122.Final.jar,
netty-transport-native-unix-common-4.1.122.Final.jar, slf4j-api-2.0.16.jar,
snakeyaml-2.3.jar, snakeyaml-engine-2.9.jar, vertx-auth-common-4.5.14.jar,
vertx-core-4.5.14.jar, vertx-web-client-4.5.14.jar,
vertx-web-common-4.5.14.jar, zjsonpatch-7.2.0.jar. Druid extension dependencies
[[]]
2026-04-28T11:48:30,242 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-lookups-cached-global], jars:
druid-lookups-cached-global-34.0.0.jar, mapdb-1.0.8.jar. Druid extension
dependencies [[]]
2026-04-28T11:48:30,243 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-multi-stage-query], jars:
druid-multi-stage-query-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,250 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-parquet-extensions], jars: aircompressor-2.0.2.jar,
avro-1.11.4.jar, commons-logging-1.3.5.jar,
druid-parquet-extensions-34.0.0.jar, hadoop-client-api-3.3.6.jar,
hadoop-client-runtime-3.3.6.jar, javax.annotation-api-1.3.2.jar,
jsr305-3.0.2.jar, parquet-avro-1.15.2.jar, parquet-column-1.15.2.jar,
parquet-common-1.15.2.jar, parquet-encoding-1.15.2.jar,
parquet-format-structures-1.15.2.jar, parquet-hadoop-1.15.2.jar,
parquet-jackson-1.15.2.jar, slf4j-api-2.0.16.jar, snappy-java-1.1.10.7.jar,
zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,251 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-protobuf-extensions], jars:
druid-protobuf-extensions-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,252 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-s3-extensions], jars:
aws-java-sdk-core-1.12.784.jar, aws-java-sdk-sts-1.12.784.jar,
commons-codec-1.16.1.jar, commons-logging-1.3.5.jar,
druid-s3-extensions-34.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.16.jar,
jackson-dataformat-cbor-2.18.4.jar, jmespath-java-1.12.784.jar,
joda-time-2.14.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,253 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [postgresql-metadata-storage], jars:
checker-qual-3.42.0.jar, postgresql-42.7.2.jar,
postgresql-metadata-storage-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,254 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [statsd-emitter], jars: asm-9.8.jar, asm-analysis-7.1.jar,
asm-commons-9.8.jar, asm-tree-7.1.jar, asm-util-7.1.jar,
java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar,
jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar,
jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar,
jnr-x86asm-1.0.2.jar, statsd-emitter-34.0.0.jar. Druid extension dependencies
[[]]
2026-04-28T11:48:30,519 INFO [main] org.apache.druid.cli.CliPeon - Running
peon in k8s mode
2026-04-28T11:48:30,926 INFO [main]
org.apache.druid.emitter.statsd.DimensionConverter - Using metric dimensions at
types at [/opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json]
2026-04-28T11:48:30,947 INFO [main] org.apache.druid.guice.JsonConfigurator
- Skipping property [druid.emitter.prometheus.addServiceAsLabel]: one of it's
prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,947 INFO [main] org.apache.druid.guice.JsonConfigurator
- Skipping property [druid.emitter.prometheus.namespace]: one of it's prefixes
[druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator
- Skipping property [druid.emitter.prometheus.strategy]: one of it's prefixes
[druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator
- Skipping property [druid.emitter.statsd.dimensionMapPath]: one of it's
prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator
- Skipping property [druid.emitter.prometheus.port]: one of it's prefixes
[druid] is also used as a property key.
2026-04-28T11:48:30,968 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Using emitter
[org.apache.druid.emitter.statsd.StatsDEmitter@2fac80a8] for metrics and
alerts, with dimensions [{version=34.0.0}].
2026-04-28T11:48:31,474 INFO [main]
org.apache.druid.java.util.metrics.SigarUtil - Loading sigar native lib at
tmpPath[/tmp/sigar10852341354949080755/libsigar-amd64-linux.so]
2026-04-28T11:48:31,502 INFO [main]
org.apache.druid.server.metrics.MetricsModule - Loaded 7 monitors:
org.apache.druid.java.util.metrics.JvmMonitor,
org.apache.druid.java.util.metrics.JvmCpuMonitor,
org.apache.druid.java.util.metrics.JvmThreadsMonitor,
org.apache.druid.server.metrics.ServiceStatusMonitor,
org.apache.druid.query.ExecutorServiceMonitor,
org.apache.druid.curator.DruidConnectionStateListener,
org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2026-04-28T11:48:31,507 INFO [main] org.apache.druid.cli.CliPeon - Starting
up with processors [2], memory [116,391,936], maxMemory [2,147,483,648],
directMemory [2,147,483,648]. Properties follow.
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.authorizerName:
MyBasicMetadataAuthorizer
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialIterations:
10000
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialsValidator.type:
metadata
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.initialAdminPassword:
<masked>
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.initialInternalClientPassword:
<masked>
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.skipOnFailure: false
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticator.MyBasicMetadataAuthenticator.type: basic
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authenticatorChain: ["MyBasicMetadataAuthenticator"]
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminRole: admin
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminUser: admin
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authorizer.MyBasicMetadataAuthorizer.type: basic
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.auth.authorizers: ["MyBasicMetadataAuthorizer"]
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.cache.sizeInBytes: 400Mi
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.cache.type: caffeine
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter: statsd
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.prometheus.addServiceAsLabel: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.prometheus.dimensionMapPath:
/opt/druid/conf/druid/cluster/_common/metricsMapping.json
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.prometheus.namespace: druid
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.prometheus.port: 9001
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.prometheus.strategy: exporter
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.dimensionMapPath:
/opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.dogstatsd: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.dogstatsdServiceAsTag: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.hostname: otel-collector.otel-collector.svc.cluster.local
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.includeHost: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.emitter.statsd.port: 8125
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.escalator.authorizerName: MyBasicMetadataAuthorizer
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.escalator.internalClientPassword: <masked>
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.escalator.internalClientUsername: druid_system
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.escalator.type: basic
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.extensions.loadList: ["druid-basic-security", "druid-datasketches",
"druid-distinctcount", "druid-kafka-indexing-service",
"druid-kubernetes-overlord-extensions", "druid-lookups-cached-global",
"druid-multi-stage-query", "druid-parquet-extensions",
"druid-protobuf-extensions", "druid-s3-extensions",
"postgresql-metadata-storage", "statsd-emitter"]
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.logs.disableAcl: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.logs.s3Bucket: REDACTED
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.logs.s3Prefix: tasks
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.logs.type: s3
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.runner.javaOptsArray: ["-server", "-Xms2900M", "-Xmx2900M",
"-XX:MaxDirectMemorySize=4G", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8",
"-Djava.io.tmpdir=/druid/data/temp/", "-XX:+ExitOnOutOfMemoryError",
"-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager",
"-XX:+HeapDumpOnOutOfMemoryError", "-XX:+UseG1GC",
"--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED",
"--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED",
"--add-opens=java.base/java.lang=ALL-UNNAMED",
"--add-opens=java.base/java.io=ALL-UNNAMED",
"--add-opens=java.base/java.nio=ALL-UNNAMED",
"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"]
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.runner.type: k8s
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.task.baseTaskDir: /druid/data/temp
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.indexer.task.encapsulatedTask: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.lookup.enableLookupSyncOnStartup: false
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.metadata.storage.connector.connectURI:
jdbc:postgresql://druid-postgresql-cluster.druid.svc:5432/druid
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.metadata.storage.connector.createTables: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.metadata.storage.connector.password: <masked>
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.metadata.storage.connector.user: druid
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.metadata.storage.type: postgresql
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.monitoring.emissionPeriod: PT10S
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.monitoring.monitors: ["org.apache.druid.java.util.metrics.JvmMonitor",
"org.apache.druid.java.util.metrics.JvmCpuMonitor",
"org.apache.druid.java.util.metrics.JvmThreadsMonitor",
"org.apache.druid.server.metrics.ServiceStatusMonitor"]
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.peon.mode: remote
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.plaintextPort: 8100
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.port: 8100
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.processing.buffer.sizeBytes: 400MiB
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.processing.intermediaryData.storage.type: deepstore
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.processing.numMergeBuffers: 4
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.processing.numThreads: 4
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.processing.tmpDir: /druid/data/tmp/queries
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.query.groupBy.maxOnDiskStorage: 2000000000
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.realtime.cache.populateCache: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.realtime.cache.useCache: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.request.logging.delegate.type: slf4j
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.request.logging.queryTimeThresholdMs: 60000
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - *
druid.request.logging.sqlQueryTimeThresholdMs: 60000
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.request.logging.type: filtered
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.s3.accessKey: { "type": "environment", "variable": "AWS_ACCESS_KEY_ID" }
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.s3.secretKey: <masked>
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.server.http.defaultQueryTimeout: 120000
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.server.http.numThreads: 130
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.service: druid/peon
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.startup.logging.logProperties: true
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.storage.baseKey: segments
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.storage.bucket: s73-dremel-druid-deep-storage-staging
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.storage.disableAcl: true
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.storage.type: s3
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.zk.paths.base: /druid
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.zk.service.compress: false
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
druid.zk.service.host: druid-zookeeper.druid.svc
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
file.encoding: UTF-8
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
file.separator: /
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
java.class.path:
/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/RoaringBitmap-0.9.49.jar:lib/accessors-smart-2.5.2.jar:lib/airline-2.8.4.jar:lib/airline-io-2.8.4.jar:lib/annotations-26.0.2.jar:lib/antlr4-runtime-4.5.3.jar:lib/aopalliance-1.0.jar:lib/apiguardian-api-1.1.2.jar:lib/asm-9.8.jar:lib/asm-commons-9.8.jar:lib/asm-tree-9.8.jar:lib/async-http-client-3.0.2.jar:lib/audience-annotations-0.12.0.jar:lib/avatica-core-1.25.0.jar:lib/avatica-metrics-1.25.0.jar:lib/avatica-server-1.25.0.jar:lib/aws-java-sdk-core-1.12.784.jar:lib/aws-java-sdk-ec2-1.12.784.jar:lib/aws-java-sdk-kms-1.12.784.jar:lib/aws-java-sdk-s3-1.12.784.jar:lib/aws-java-sdk-sts-1.12.784.jar:lib/caffeine-2.8.0.jar:lib/calcite-core-1.37.0.jar:lib/calcite-linq4j-1.37.0.jar:lib/checker-qual-3.48.1.jar:lib/classmate-1.5.1.jar:lib/commons-beanutils-1.11.0.jar:lib/commons-codec-1.16.1.jar:lib/commons-collections-3.
2.2.jar:lib/commons-collections4-4.2.jar:lib/commons-compiler-3.1.11.jar:lib/commons-compress-1.26.0.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-io-2.17.0.jar:lib/commons-lang3-3.17.0.jar:lib/commons-logging-1.3.5.jar:lib/commons-math3-3.6.1.jar:lib/commons-pool2-2.2.jar:lib/commons-text-1.13.1.jar:lib/compress-lzf-1.0.4.jar:lib/cron-scheduler-0.1.jar:lib/cron-utils-9.2.0.jar:lib/curator-client-5.8.0.jar:lib/curator-framework-5.8.0.jar:lib/curator-recipes-5.8.0.jar:lib/curator-x-discovery-5.8.0.jar:lib/datasketches-java-4.2.0.jar:lib/datasketches-memory-2.2.0.jar:lib/derby-10.14.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/derbynet-10.14.2.0.jar:lib/disruptor-3.3.6.jar:lib/druid-aws-common-34.0.0.jar:lib/druid-gcp-common-34.0.0.jar:lib/druid-indexing-hadoop-34.0.0.jar:lib/druid-indexing-service-34.0.0.jar:lib/druid-processing-34.0.0.jar:lib/druid-server-34.0.0.jar:lib/druid-services-34.0.0.jar:lib/druid-sql-34.0.0.jar:lib/elasticache-java-cluster-client-1.2.0.jar:lib/error_prone_annota
tions-2.35.1.jar:lib/failureaccess-1.0.1.jar:lib/fastutil-8.5.4.jar:lib/fastutil-core-8.5.4.jar:lib/fastutil-extra-8.5.4.jar:lib/google-api-client-2.2.0.jar:lib/google-http-client-1.42.3.jar:lib/google-http-client-apache-v2-1.42.3.jar:lib/google-http-client-gson-1.42.3.jar:lib/google-http-client-jackson2-1.42.3.jar:lib/google-oauth-client-1.34.1.jar:lib/graal-sdk-22.3.5.jar:lib/grpc-context-1.27.2.jar:lib/gson-2.10.1.jar:lib/guava-32.1.3-jre.jar:lib/guice-5.1.0.jar:lib/guice-assistedinject-5.1.0.jar:lib/guice-servlet-5.1.0.jar:lib/hibernate-validator-6.2.5.Final.jar:lib/httpclient-4.5.13.jar:lib/httpclient5-5.2.1.jar:lib/httpcore-4.4.16.jar:lib/httpcore5-5.2.3.jar:lib/httpcore5-h2-5.2.jar:lib/icu4j-73.2.jar:lib/ipaddress-5.3.4.jar:lib/istack-commons-runtime-3.0.11.jar:lib/j2objc-annotations-2.8.jar:lib/jackson-annotations-2.18.4.jar:lib/jackson-core-2.18.4.1.jar:lib/jackson-databind-2.18.4.jar:lib/jackson-dataformat-cbor-2.18.4.jar:lib/jackson-dataformat-smile-2.18.4.jar:lib/jackson
-datatype-guava-2.18.4.jar:lib/jackson-datatype-joda-2.18.4.jar:lib/jackson-jaxrs-base-2.18.4.jar:lib/jackson-jaxrs-json-provider-2.18.4.jar:lib/jackson-jaxrs-smile-provider-2.18.4.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-guice-2.18.4.jar:lib/jackson-module-jaxb-annotations-2.18.4.jar:lib/jakarta.activation-1.2.2.jar:lib/jakarta.activation-api-1.2.2.jar:lib/jakarta.el-3.0.4.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jakarta.validation-api-2.0.2.jar:lib/jakarta.xml.bind-api-2.3.3.jar:lib/janino-3.1.11.jar:lib/javax.el-api-3.0.0.jar:lib/javax.inject-1.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-runtime-2.3.3.jar:lib/jboss-logging-3.4.1.Final.jar:lib/jcl-over-slf4j-2.0.16.jar:lib/jcodings-1.0.50.jar:lib/jdbi-2.63.1.jar:lib/jersey-core-1.19.4.jar:lib/jersey-guice-1.19.4.jar:lib/jersey-server-1.19.4.jar:lib/jersey-servlet-1.19.4.jar:lib/jetty-client-9.4.57.v20241219.jar:lib/jetty-continuation-9.4.57.v20241219.jar:lib/jetty-http-9.4.57.v20241219.jar:lib/jetty-io-9.4.57.v20241219.ja
r:lib/jetty-proxy-9.4.57.v20241219.jar:lib/jetty-rewrite-9.4.57.v20241219.jar:lib/jetty-security-9.4.57.v20241219.jar:lib/jetty-server-9.4.57.v20241219.jar:lib/jetty-servlet-9.4.57.v20241219.jar:lib/jetty-servlets-9.4.57.v20241219.jar:lib/jetty-util-9.4.57.v20241219.jar:lib/jetty-util-ajax-9.4.57.v20241219.jar:lib/jmespath-java-1.12.784.jar:lib/jna-5.13.0.jar:lib/jna-platform-5.13.0.jar:lib/joda-time-2.14.0.jar:lib/joni-2.1.34.jar:lib/js-22.3.5.jar:lib/js-scriptengine-22.3.5.jar:lib/json-path-2.9.0.jar:lib/json-simple-1.1.1.jar:lib/json-smart-2.5.2.jar:lib/jsr305-3.0.2.jar:lib/jsr311-api-1.1.1.jar:lib/jts-core-1.19.0.jar:lib/jts-io-common-1.19.0.jar:lib/log4j-1.2-api-2.22.1.jar:lib/log4j-api-2.22.1.jar:lib/log4j-core-2.22.1.jar:lib/log4j-jul-2.22.1.jar:lib/log4j-slf4j2-impl-2.22.1.jar:lib/lz4-java-1.8.0.jar:lib/maven-artifact-3.6.0.jar:lib/maven-builder-support-3.6.0.jar:lib/maven-model-3.6.0.jar:lib/maven-model-builder-3.6.0.jar:lib/maven-repository-metadata-3.6.0.jar:lib/maven-res
olver-api-1.3.1.jar:lib/maven-resolver-connector-basic-1.3.1.jar:lib/maven-resolver-impl-1.3.1.jar:lib/maven-resolver-provider-3.6.0.jar:lib/maven-resolver-spi-1.3.1.jar:lib/maven-resolver-transport-http-1.3.1.jar:lib/maven-resolver-util-1.3.1.jar:lib/metrics-core-4.2.22.jar:lib/netty-3.10.6.Final.jar:lib/netty-buffer-4.1.122.Final.jar:lib/netty-codec-4.1.122.Final.jar:lib/netty-codec-dns-4.1.122.Final.jar:lib/netty-codec-http-4.1.122.Final.jar:lib/netty-codec-socks-4.1.122.Final.jar:lib/netty-common-4.1.122.Final.jar:lib/netty-handler-4.1.122.Final.jar:lib/netty-handler-proxy-4.1.122.Final.jar:lib/netty-resolver-4.1.122.Final.jar:lib/netty-resolver-dns-4.1.122.Final.jar:lib/netty-transport-4.1.122.Final.jar:lib/netty-transport-native-unix-common-4.1.122.Final.jar:lib/opencensus-api-0.31.1.jar:lib/opencensus-contrib-http-util-0.31.1.jar:lib/opencsv-4.6.jar:lib/oshi-core-6.4.4.jar:lib/plexus-component-annotations-1.7.1.jar:lib/plexus-interpolation-1.25.jar:lib/plexus-utils-3.1.0.jar:
lib/proj4j-1.2.2.jar:lib/protobuf-java-3.25.5.jar:lib/regex-22.3.5.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/resilience4j-core-1.3.1.jar:lib/rhino-1.7.14.jar:lib/shims-0.9.49.jar:lib/sigar-1.6.5.132.jar:lib/slf4j-api-2.0.16.jar:lib/snappy-java-1.1.10.7.jar:lib/truffle-api-22.3.5.jar:lib/txw2-2.3.3.jar:lib/uzaygezen-core-0.2.jar:lib/vavr-0.10.2.jar:lib/vavr-match-0.10.2.jar:lib/web-console-34.0.0.jar:lib/xz-1.9.jar:lib/zookeeper-3.8.4.jar:lib/zookeeper-jute-3.8.4.jar:lib/zstd-jni-1.5.2-3.jar:
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
java.class.version: 61.0
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - *
java.home: /usr/lib/jvm/java-17-openjdk-amd64
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.io.tmpdir: /tmp
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.library.path:
/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.runtime.name: OpenJDK Runtime Environment
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.runtime.version: 17.0.15+6-Debian-1deb12u1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.specification.maintenance.version: 1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.specification.name: Java Platform API Specification
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.specification.vendor: Oracle Corporation
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.specification.version: 17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vendor: Debian
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vendor.url: https://tracker.debian.org/openjdk-17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vendor.url.bug: https://bugs.debian.org/openjdk-17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.version: 17.0.15
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.version.date: 2025-04-15
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.compressedOopsMode: 32-bit
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.info: mixed mode, sharing
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.name: OpenJDK 64-Bit Server VM
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.specification.name: Java Virtual Machine Specification
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.specification.vendor: Oracle Corporation
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.specification.version: 17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.vendor: Debian
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
java.vm.version: 17.0.15+6-Debian-1deb12u1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - *
jdk.debug: release
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
line.separator:
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
log4j.shutdownHookEnabled: true
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
log4j2.is.webapp: false
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
native.encoding: UTF-8
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
os.arch: amd64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
os.name: Linux
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
os.version: 6.12.79-101.147.amzn2023.x86_64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
path.separator: :
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.arch.data.model: 64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.boot.library.path: /usr/lib/jvm/java-17-openjdk-amd64/lib
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.cpu.endian: little
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.io.unicode.encoding: UnicodeLittle
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.java.command: org.apache.druid.cli.Main internal peon --taskId
index_kafka_cs-decisions-realtime_2ccb995f736b135_bohodcjo /druid/data/temp 1
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.java.launcher: SUN_STANDARD
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.jnu.encoding: UTF-8
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
user.dir: /opt/druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
user.home: /opt/druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
user.language: en
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
user.name: druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - *
user.timezone: GMT
2026-04-28T11:48:31,658 INFO [main]
org.apache.druid.query.DruidProcessingConfig - Detected max direct memory size
of [2,147,483,648] bytes
2026-04-28T11:48:31,669 INFO [main]
org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage
location strategy[LeastBytesUsedStorageLocationSelectorStrategy].
2026-04-28T11:48:31,669 INFO [main]
org.apache.druid.segment.loading.SegmentLocalCacheManager - Number of threads
to load segments into page cache - on bootstrap: [0], on download: [0].
2026-04-28T11:48:31,700 INFO [main] org.apache.druid.guice.StorageNodeModule
- Segment cache not configured on ServerType [indexer-executor]. It will not be
assignable for segment placement
2026-04-28T11:48:31,724 INFO [main] org.eclipse.jetty.util.log - Logging
initialized @2792ms to org.eclipse.jetty.util.log.Slf4jLog
2026-04-28T11:48:31,738 INFO [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http
connector with port [8100]
2026-04-28T11:48:31,814 WARN [main]
org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is
inefficient for short content, break even is size 23
2026-04-28T11:48:31,833 ERROR [main] org.apache.druid.cli.CliPeon - Error
when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following
errors:
1) Not enough direct memory. Please adjust -XX:MaxDirectMemorySize,
druid.processing.buffer.sizeBytes, druid.processing.numThreads, or
druid.processing.numMergeBuffers: maxDirectMemory[2,147,483,648],
memoryNeeded[3,774,873,600] = druid.processing.buffer.sizeBytes[419,430,400] *
(druid.processing.numMergeBuffers[4] + druid.processing.numThreads[4] + 1)
at PeonProcessingModule.getMergeBufferPool(PeonProcessingModule.java:123)
\_ installed by: Modules$OverrideModule -> Modules$OverrideModule ->
PeonProcessingModule
at
PeonProcessingModule.getGroupByResourcesReservationPool(PeonProcessingModule.java:146)
\_ for 1st parameter
at
PeonProcessingModule.getGroupByResourcesReservationPool(PeonProcessingModule.java:146)
\_ installed by: Modules$OverrideModule -> Modules$OverrideModule ->
PeonProcessingModule
at GroupingEngine.<init>(GroupingEngine.java:135)
\_ for 3rd parameter
at GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:59)
\_ for 1st parameter
at QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:90)
\_ installed by: Modules$OverrideModule -> Modules$OverrideModule ->
QueryRunnerFactoryModule
while locating GroupByQueryRunnerFactory
while locating QueryRunnerFactory annotated with
@Element(setName=,uniqueId=107, type=MAPBINDER, keyType=Class<? extends Query>)
at
DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:50)
\_ for 1st parameter
while locating DefaultQueryRunnerFactoryConglomerate
at QueryableModule.configure(QueryableModule.java:56)
\_ installed by: Modules$OverrideModule -> Modules$OverrideModule ->
QueryableModule
at QueryLifecycleFactory.<init>(QueryLifecycleFactory.java:60)
\_ for 1st parameter
at QueryLifecycleFactory.class(QueryLifecycleFactory.java:60)
at QueryResource.<init>(QueryResource.java:111)
\_ for 1st parameter
at QueryResource.class(QueryResource.java:72)
while locating QueryResource
1 error
======================
Full classname legend:
======================
DefaultQueryRunnerFactoryConglomerate:
"org.apache.druid.query.DefaultQueryRunnerFactoryConglomerate"
Element: "com.google.inject.internal.Element"
GroupByQueryRunnerFactory:
"org.apache.druid.query.groupby.GroupByQueryRunnerFactory"
GroupingEngine:
"org.apache.druid.query.groupby.GroupingEngine"
Modules$OverrideModule:
"com.google.inject.util.Modules$OverrideModule"
PeonProcessingModule:
"org.apache.druid.guice.PeonProcessingModule"
Query: "org.apache.druid.query.Query"
QueryLifecycleFactory:
"org.apache.druid.server.QueryLifecycleFactory"
QueryResource:
"org.apache.druid.server.QueryResource"
QueryRunnerFactory:
"org.apache.druid.query.QueryRunnerFactory"
QueryRunnerFactoryModule:
"org.apache.druid.guice.QueryRunnerFactoryModule"
QueryableModule:
"org.apache.druid.guice.QueryableModule"
========================
End of classname legend:
========================
at
com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
~[guice-5.1.0.jar:?]
at
com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1104)
~[guice-5.1.0.jar:?]
at
com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1134)
~[guice-5.1.0.jar:?]
at
org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:150)
~[druid-processing-34.0.0.jar:34.0.0]
at
org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137)
[druid-services-34.0.0.jar:34.0.0]
at
org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94)
[druid-services-34.0.0.jar:34.0.0]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:390)
[druid-services-34.0.0.jar:34.0.0]
at org.apache.druid.cli.Main.main(Main.java:112)
[druid-services-34.0.0.jar:34.0.0]
```
I've configured the Overlord to pick a custom Pod template with the
`customTemplateAdapter`.
#### Overlord configuration
```
# https://druid.apache.org/docs/latest/configuration/index.html#overlord
druid.service=druid/overlord
druid.plaintextPort=8090
# Extensions
druid.extensions.loadList=["druid-basic-security",
"druid-datasketches", "druid-distinctcount", "druid-kafka-indexing-service",
"druid-kubernetes-overlord-extensions", "druid-lookups-cached-global",
"druid-multi-stage-query", "druid-parquet-extensions",
"druid-protobuf-extensions", "druid-s3-extensions",
"postgresql-metadata-storage", "prometheus-emitter"]
#
https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations
druid.indexer.storage.type=metadata
druid.indexer.storage.recentlyFinishedThreshold=PT24H
druid.indexer.tasklock.batchSegmentAllocation=true
druid.indexer.tasklock.batchAllocationWaitTime=0
druid.indexer.queue.startDelay=PT30S
# Monitoring
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor",
"org.apache.druid.java.util.metrics.JvmCpuMonitor",
"org.apache.druid.java.util.metrics.JvmThreadsMonitor",
"org.apache.druid.server.metrics.ServiceStatusMonitor",
"org.apache.druid.server.metrics.TaskCountStatsMonitor",
"org.apache.druid.server.metrics.TaskSlotCountStatsMonitor"]
## Tasks Metadata/Logs Management
##
https://druid.apache.org/docs/latest/operations/clean-metadata-store/#indexer-task-logs
# Cleanup of task logs and its associated metadata
druid.indexer.logs.kill.enabled=true
# 24 hours in milliseconds
druid.indexer.logs.kill.durationToRetain=86400000
# 5 min in milliseconds
druid.indexer.logs.kill.initialDelay=300000
# 6 hours in milliseconds
druid.indexer.logs.kill.delay=21600000
## Segments metadata cache
##
https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental
druid.manager.segments.useCache=ifSynced
## MiddleManager-less
#
https://druid.apache.org/docs/latest/development/extensions-core/k8s-jobs/
druid.indexer.runner.type=k8s
druid.indexer.task.encapsulatedTask=true
druid.indexer.runner.namespace=druid
druid.indexer.runner.capacity=600
druid.indexer.runner.k8sjobLaunchTimeout=PT10M
druid.indexer.runner.taskCleanupDelay=P2D
druid.indexer.runner.taskCleanupInterval=PT20M
druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.debugJobs=true
# Pod Templates per task type
druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.k8s.podTemplate.base=/druid/task_templates/default/pod_template.yaml
druid.indexer.runner.k8s.podTemplate.index_kafka=/druid/task_templates/realtime/pod_template.yaml
## Peons
druid.indexer.runner.peonMonitors=["org.apache.druid.java.util.metrics.JvmMonitor",
"org.apache.druid.java.util.metrics.JvmCpuMonitor",
"org.apache.druid.java.util.metrics.JvmThreadsMonitor",
"org.apache.druid.server.metrics.ServiceStatusMonitor",
"org.apache.druid.client.cache.CacheMonitor"]
```
#### Pod Template (for `index_kafka` tasks) as ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: druid-analytics-task-template-realtime
namespace: druid
data:
pod_template.yaml: |-
apiVersion: v1
kind: PodTemplate
metadata:
name: druid-task-default
labels:
app: druid
component: task
druid_cr: analytics
task-tmpl: realtime
template:
metadata:
labels:
app: druid
component: task
druid_cr: analytics
task-tmpl: realtime
spec:
serviceAccountName: druid-analytics-overlord
restartPolicy: Never
securityContext:
fsGroup: 1000
runAsGroup: 1000
runAsUser: 1000
imagePullSecrets:
- name: docker-registry-credentials
containers:
- name: main
image: <REDACTED>/druid:34.0.0-1.0.0
imagePullPolicy: IfNotPresent
command:
- sh
- -c
args:
- |
/peon.sh /druid/data/temp 1
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT
value: "true"
envFrom:
- secretRef:
name: druid-credentials
- secretRef:
name: druid-kafka-credentials
- secretRef:
name: druid-s3-credentials
ports:
- containerPort: 8100
name: druid-port
protocol: TCP
resources:
limits:
cpu: 2000m
memory: 8Gi
requests:
cpu: 2000m
memory: 7Gi
volumeMounts:
- name: common-config-volume
mountPath: /opt/druid/conf/druid/cluster/_common
readOnly: true
- name: task-peon-config-volume
mountPath:
/opt/druid/conf/druid/cluster/master/coordinator-overlord
readOnly: true
- name: temp-data-volume
mountPath: /druid/data/temp
volumes:
# configmaps
- name: common-config-volume
configMap:
name: analytics-druid-common-config
defaultMode: 420
- name: task-peon-config-volume
configMap:
defaultMode: 420
name: druid-analytics-task-template-realtime-config
# storage
- name: temp-data-volume
emptyDir: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- druid-data-mm
tolerations:
- key: druid
value: data-mm
operator: Equal
effect: NoSchedule
topologySpreadConstraints:
- topologyKey: "topology.kubernetes.io/zone"
maxSkew: 1
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: druid
component: task
druid_cr: analytics
```
#### Config files for the task/pod as ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: druid-analytics-task-template-realtime-config
namespace: druid
data:
log4j2.xml: |-
<?xml version="1.0" encoding="UTF-8" ?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{ISO8601} %p [%t] %c -%notEmpty{
[%markerSimpleName]} %m%n"/>
</Console>
</Appenders>
<Loggers>
<!-- AsyncLogger instead of Logger -->
<AsyncLogger
name="org.apache.druid.curator.inventory.CuratorInventoryManager" level="debug"
additivity="false">
<AppenderRef ref="Console"/>
</AsyncLogger>
<AsyncLogger name="org.apache.druid.client.BatchServerInventoryView"
level="debug" additivity="false">
<AppenderRef ref="Console"/>
</AsyncLogger>
<!-- Make extra sure nobody adds logs in a bad way that can hurt
performance -->
<AsyncLogger name="org.apache.druid.client.ServerInventoryView"
level="debug" additivity="false">
<AppenderRef ref="Console"/>
</AsyncLogger>
<AsyncLogger name
="org.apache.druid.java.util.http.client.pool.ChannelResourceFactory"
level="info" additivity="false">
<AppenderRef ref="Console"/>
</AsyncLogger>
<AsyncLogger name="org.apache.druid.jetty.RequestLog"
additivity="false" level="info">
<AppenderRef ref="Console"/>
</AsyncLogger>
<Root level="info">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
jvm.config: |-
-server
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=/druid/data/temp/
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Dorg.jboss.logging.provider=slf4j
-Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger
-Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown
-Dlog4j.shutdownHookEnabled=true
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-Xms2900M
-Xmx2900M
-XX:MaxDirectMemorySize=4G
runtime.properties: |
druid.service=druid/peon
druid.port=8100
druid.plaintextPort=8100
# Extensions
druid.extensions.loadList=["druid-basic-security", "druid-datasketches",
"druid-distinctcount", "druid-kafka-indexing-service",
"druid-kubernetes-overlord-extensions", "druid-lookups-cached-global",
"druid-multi-stage-query", "druid-parquet-extensions",
"druid-protobuf-extensions", "druid-s3-extensions",
"postgresql-metadata-storage", "statsd-emitter"]
druid.peon.mode=remote
druid.indexer.runner.type=k8s
druid.indexer.task.encapsulatedTask=true
druid.processing.intermediaryData.storage.type=deepstore
#
https://druid.apache.org/docs/latest/configuration/index.html#middlemanager-configuration
druid.indexer.task.baseTaskDir=/druid/data/temp
# HTTP server
#
https://druid.apache.org/docs/latest/configuration/index.html#indexer-concurrent-requests
# Sum of `druid.broker.http.numConnections` across all the brokers in
the cluster
druid.server.http.numThreads=130
# Processing threads and buffers
druid.processing.numThreads=4
druid.processing.numMergeBuffers=4
druid.processing.buffer.sizeBytes=400MiB
# Monitoring
druid.emitter=statsd
druid.emitter.statsd.hostname=otel-collector.otel-collector.svc.cluster.local
druid.emitter.statsd.port=8125
druid.emitter.statsd.dimensionMapPath=/opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json
druid.emitter.statsd.includeHost=true
druid.emitter.statsd.dogstatsd=true
druid.emitter.statsd.dogstatsdServiceAsTag=true
# Query processing
druid.processing.tmpDir=/druid/data/tmp/queries
#https://druid.apache.org/docs/latest/configuration/#peon-caching
# Query cache
druid.realtime.cache.useCache=true
druid.realtime.cache.populateCache=true
druid.cache.sizeInBytes=400Mi
# Query Performance
# Default timeout (120 seconds) can be overridden at query context
druid.server.http.defaultQueryTimeout=120000
# GroupBy merging buffer per-query spilling to disk (2GB)
druid.query.groupBy.maxOnDiskStorage=2000000000
```
As you can see on Peon logs, the _runtime.properties_ file is properly
picked but not the JVM flags on _jvm.config_.
I also tried setting the `druid.indexer.runner.javaOptsArray` on the
`runtime.properties` but it didn't work either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
