Fryuni opened a new issue, #18330:
URL: https://github.com/apache/druid/issues/18330
### Affected Version
`33.0.0`
### Description
I'm seeing the error `Attempting to publish with empty segment set, but
total row count was not 0`.
It is the same error as #9845, which was supposed to be fixed by #14995 that
landed on v29. But this is happening on `v33`.
This only happens when task replicas are created to avoid problems when
workers are terminated (by the K8s cluster auto-scaler, for example). Any task
with `replicas` greater than 1 reaches the same error.
It doesn't happen to only one of the replicas. It seems to be a race, one
task wins for a few segments and the other wins for other segments. Then both
fail and nothing gets commited. Setting replicas back to one makes this issue
go away, but then ingestion have downtimes whenever a worker is stopped.
<details>
<summary>Log of a failed task</summary>
```
2025-07-26T14:31:11+00:00 startup service overlord
2025-07-26 14:31:18.336 INFO net.spy.memcached.MemcachedConnection: Setting
retryQueueSize to -1
2025-07-26 14:31:18.351 INFO net.spy.memcached.MemcachedConnection: Added
{QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null,
toWrite=0, interested=0} to connect queue
2025-07-26 14:31:18.372 INFO net.spy.memcached.MemcachedConnection: Setting
retryQueueSize to -1
2025-07-26 14:31:18.373 INFO net.spy.memcached.MemcachedConnection: Added
{QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null,
toWrite=0, interested=0} to connect queue
2025-07-26 14:31:18.377 INFO net.spy.memcached.MemcachedConnection: Setting
retryQueueSize to -1
2025-07-26 14:31:18.378 INFO net.spy.memcached.MemcachedConnection: Added
{QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null,
toWrite=0, interested=0} to connect queue
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a
provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as
a provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering
org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a
provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering
org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a
provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering
org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper as a
provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering
org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper
as a provider class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.StatusResource as a root resource
class
Jul 26, 2025 2:31:20 PM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017
03:20 PM'
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.server.initialization.jetty.CustomExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper
to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to
GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to
GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.security.basic.BasicSecurityResourceFilter to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.security.basic.authentication.endpoint.BasicAuthenticatorResource
to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding
org.apache.druid.security.basic.authorization.endpoint.BasicAuthorizerResource
to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.security.pac4j.Pac4jCallbackResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.http.security.StateResourceFilter to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.http.SegmentListerResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.http.HistoricalResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.QueryResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.segment.realtime.ChatHandlerResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.http.security.ConfigResourceFilter to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.query.lookup.LookupListeningResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.query.lookup.LookupIntrospectionResource to
GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.druid.server.StatusResource to
GuiceManagedComponentProvider with the scope "Singleton"
14:36:50.981
[[index_kafka_goalByExperience_d9e37bb95ea1354_jocncobe]-publish] ERROR
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error
while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=681,
sequenceName='index_kafka_goalByExperience_d9e37bb95ea1354_681',
assignments=[], startOffsets={KafkaTopicPartition{partition=23, topic='null',
multiTopicPartition=false}=9469798, KafkaTopicPartition{partition=22,
topic='null', multiTopicPartition=false}=9498999,
KafkaTopicPartition{partition=25, topic='null',
multiTopicPartition=false}=10047313, KafkaTopicPartition{partition=24,
topic='null', multiTopicPartition=false}=9658869,
KafkaTopicPartition{partition=27, topic='null',
multiTopicPartition=false}=9504586, KafkaTopicPartition{partition=26,
topic='null', multiTopicPartition=false}=9595552,
KafkaTopicPartition{partition=29, topic='null',
multiTopicPartition=false}=9606074, KafkaTopicPartition{partition=28,
topic='null', multiTopicPartition=f
alse}=9340750, KafkaTopicPartition{partition=1, topic='null',
multiTopicPartition=false}=9665107, KafkaTopicPartition{partition=0,
topic='null', multiTopicPartition=false}=9467462,
KafkaTopicPartition{partition=3, topic='null',
multiTopicPartition=false}=9377332, KafkaTopicPartition{partition=2,
topic='null', multiTopicPartition=false}=9570694,
KafkaTopicPartition{partition=5, topic='null',
multiTopicPartition=false}=9462158, KafkaTopicPartition{partition=4,
topic='null', multiTopicPartition=false}=9457911,
KafkaTopicPartition{partition=7, topic='null',
multiTopicPartition=false}=9511729, KafkaTopicPartition{partition=6,
topic='null', multiTopicPartition=false}=9447538,
KafkaTopicPartition{partition=9, topic='null',
multiTopicPartition=false}=9470660, KafkaTopicPartition{partition=8,
topic='null', multiTopicPartition=false}=9381318,
KafkaTopicPartition{partition=11, topic='null',
multiTopicPartition=false}=9522858, KafkaTopicPartition{partition=10,
topic='null', multiTopicPartition=
false}=9720465, KafkaTopicPartition{partition=13, topic='null',
multiTopicPartition=false}=9639213, KafkaTopicPartition{partition=12,
topic='null', multiTopicPartition=false}=9905181,
KafkaTopicPartition{partition=15, topic='null',
multiTopicPartition=false}=9435236, KafkaTopicPartition{partition=14,
topic='null', multiTopicPartition=false}=10042299,
KafkaTopicPartition{partition=17, topic='null',
multiTopicPartition=false}=9405092, KafkaTopicPartition{partition=16,
topic='null', multiTopicPartition=false}=9485933,
KafkaTopicPartition{partition=19, topic='null',
multiTopicPartition=false}=9355762, KafkaTopicPartition{partition=18,
topic='null', multiTopicPartition=false}=9665043,
KafkaTopicPartition{partition=21, topic='null',
multiTopicPartition=false}=9593813, KafkaTopicPartition{partition=20,
topic='null', multiTopicPartition=false}=9612303}, exclusiveStartPartitions=[],
endOffsets={KafkaTopicPartition{partition=23, topic='null',
multiTopicPartition=false}=9469798, KafkaTopicPart
ition{partition=22, topic='null', multiTopicPartition=false}=9497963,
KafkaTopicPartition{partition=25, topic='null',
multiTopicPartition=false}=10047313, KafkaTopicPartition{partition=24,
topic='null', multiTopicPartition=false}=9657756,
KafkaTopicPartition{partition=27, topic='null',
multiTopicPartition=false}=9504586, KafkaTopicPartition{partition=26,
topic='null', multiTopicPartition=false}=9595552,
KafkaTopicPartition{partition=29, topic='null',
multiTopicPartition=false}=9606074, KafkaTopicPartition{partition=28,
topic='null', multiTopicPartition=false}=9340750,
KafkaTopicPartition{partition=1, topic='null',
multiTopicPartition=false}=9665107, KafkaTopicPartition{partition=0,
topic='null', multiTopicPartition=false}=9467462,
KafkaTopicPartition{partition=3, topic='null',
multiTopicPartition=false}=9377332, KafkaTopicPartition{partition=2,
topic='null', multiTopicPartition=false}=9570694,
KafkaTopicPartition{partition=5, topic='null',
multiTopicPartition=false}=9462158, KafkaTo
picPartition{partition=4, topic='null', multiTopicPartition=false}=9457911,
KafkaTopicPartition{partition=7, topic='null',
multiTopicPartition=false}=9511729, KafkaTopicPartition{partition=6,
topic='null', multiTopicPartition=false}=9447538,
KafkaTopicPartition{partition=9, topic='null',
multiTopicPartition=false}=9470660, KafkaTopicPartition{partition=8,
topic='null', multiTopicPartition=false}=9381318,
KafkaTopicPartition{partition=11, topic='null',
multiTopicPartition=false}=9521747, KafkaTopicPartition{partition=10,
topic='null', multiTopicPartition=false}=9720465,
KafkaTopicPartition{partition=13, topic='null',
multiTopicPartition=false}=9638252, KafkaTopicPartition{partition=12,
topic='null', multiTopicPartition=false}=9904681,
KafkaTopicPartition{partition=15, topic='null',
multiTopicPartition=false}=9435236, KafkaTopicPartition{partition=14,
topic='null', multiTopicPartition=false}=10042299,
KafkaTopicPartition{partition=17, topic='null',
multiTopicPartition=false}=9405092,
KafkaTopicPartition{partition=16, topic='null',
multiTopicPartition=false}=9485933, KafkaTopicPartition{partition=19,
topic='null', multiTopicPartition=false}=9355762,
KafkaTopicPartition{partition=18, topic='null',
multiTopicPartition=false}=9665043, KafkaTopicPartition{partition=21,
topic='null', multiTopicPartition=false}=9593813,
KafkaTopicPartition{partition=20, topic='null',
multiTopicPartition=false}=9612303}, sentinel=false, checkpointed=true}]
org.apache.druid.java.util.common.ISE: Attempting to publish with empty
segment set, but total row count was not 0: [1565].
at
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610)
~[druid-server-33.0.0.jar:33.0.0]
at
org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289)
~[druid-server-33.0.0.jar:33.0.0]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169)
[guava-32.0.1-jre.jar:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
14:37:15.007 [task-runner-0-priority-0] ERROR
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Encountered exception in run() before persisting.
java.util.concurrent.ExecutionException:
org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment
set, but total row count was not 0: [1565].
at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
~[guava-32.0.1-jre.jar:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.checkPublishAndHandoffFailure(SeekableStreamIndexTaskRunner.java:985)
~[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:634)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:295)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:152)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
[guava-32.0.1-jre.jar:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.java.util.common.ISE: Attempting to publish with
empty segment set, but total row count was not 0: [1565].
at
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610)
~[druid-server-33.0.0.jar:33.0.0]
at
org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289)
~[druid-server-33.0.0.jar:33.0.0]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169)
~[guava-32.0.1-jre.jar:?]
... 3 more
14:37:15.463 [task-runner-0-priority-0] ERROR
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Encountered exception while running task.
java.util.concurrent.ExecutionException:
org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment
set, but total row count was not 0: [1565].
at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
~[guava-32.0.1-jre.jar:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.checkPublishAndHandoffFailure(SeekableStreamIndexTaskRunner.java:985)
~[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:634)
~[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:295)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:152)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449)
[druid-indexing-service-33.0.0.jar:33.0.0]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
[guava-32.0.1-jre.jar:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.java.util.common.ISE: Attempting to publish with
empty segment set, but total row count was not 0: [1565].
at
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610)
~[druid-server-33.0.0.jar:33.0.0]
at
org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289)
~[druid-server-33.0.0.jar:33.0.0]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257)
~[guava-32.0.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169)
~[guava-32.0.1-jre.jar:?]
... 3 more
Finished peon task
```
</details>
- Cluster size:
- Coordinator: 3
- Overlord: 2
- Brokers: 6
- Routers: 6
- Historicals: 15
- Peons: one per task (using Overlord Kubernetes Extensions)
- Configuration in use:
- <details>
<summary>Common config</summary>
```
# Disable property logging at startup
druid.startup.logging.logProperties=false
# Extensions
druid.extensions.loadList=["druid-basic-security","postgresql-metadata-storage","druid-google-extensions","druid-pac4j","druid-kafka-indexing-service","druid-lookups-cached-global","druid-multi-stage-query","druid-virtual-columns","druid-avro-extensions","druid-datasketches","druid-moving-average-query","druid-time-min-max","druid-stats","druid-kubernetes-overlord-extensions"]
# Service discovery
druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator
druid.indexer.runner.type=httpRemote
druid.indexer.runner.timeout=PT5H
druid.indexer.queue.startDelay=PT5S
druid.indexer.storage.type=metadata
# Zookeeper discovery
druid.zk.service.enabled=true
druid.zk.service.host=<redacted>
# Metadata Store
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<redacted>
druid.metadata.storage.connector.createTables=true
druid.metadata.storage.connector.user=druid
# Deep Storage
druid.storage.type=google
druid.google.bucket=<redacted>
# Indexer logs
druid.indexer.logs.type=google
druid.indexer.logs.bucket=<redacted>
druid.indexer.logs.prefix=taskLogs
druid.indexer.logs.kill.enabled=false
druid.indexer.logs.kill.durationToRetain=604800000
# Storage type of double columns
druid.indexing.doubleStorage=double
# Security
druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password",
"password", "key", "token", "pwd"]
druid.server.http.enableRequestLimit=true
druid.server.http.defaultQueryTimeout=60000
druid.server.http.maxQueryTimeout=300000
# Authentication
druid.auth.authenticator.allowAll.type=allowAll
druid.auth.authorizer.allowAll.type=allowAll
druid.auth.authenticatorChain=["allowAll", "basic", "pac4j"]
druid.lookup.enableLookupSyncOnStartup=true
druid.lookup.lookupTier=broker
druid.javascript.enabled=false
druid.emmiter.logging.logLevel=warn
# Metrics
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor"]
druid.query.groupBy.maxOnDiskStorage=1073741824
druid.query.groupBy.applyLimitPushDownToSegment=true
# SQL
druid.sql.enable=true
# Planning SQL query when there is aggregate distinct in the statement
druid.sql.planner.useGroupingSetForExactDistinct=true
# Expression processing config
druid.expressions.useStrictBooleans=true
# Http client
druid.global.http.eagerInitialization=false
# Cache
druid.cache.type=memcached
druid.cache.hosts<redacted>
druid.cache.numConnections=3
```
</details>
- <details>
<summary>Overlord config</summary>
```
druid.service=druid/overlord
druid.indexer.storage.type=metadata
druid.indexer.queue.startDelay=PT30S
druid.indexer.tasklock.batchSegmentAllocation=true
druid.indexer.task.encapsulatedTask=true
druid.indexer.fork.property.druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.type=k8s
druid.indexer.queue.maxSize=60
druid.indexer.runner.namespace=analytics
druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.k8s.podTemplate.base=/druid/tasktemplate/default/default-task-template.yaml
druid.indexer.runner.primaryContainerName=main
druid.indexer.runner.debugJobs=false
druid.indexer.runner.maxTaskDuration=PT4H
druid.indexer.runner.taskCleanupDelay=PT2H
druid.indexer.runner.taskCleanupInterval=PT10M
druid.indexer.runner.taskJoinTimeout=PT10M
druid.indexer.runner.K8sjobLaunchTimeout=PT1H
druid.indexer.runner.pendingTasksRunnerNumThreads=2
druid.indexer.runner.labels={"app.kubernetes.io/name":"Druid","app.kubernetes.io/component":"Peon"}
druid.indexer.runner.graceTerminationPeriodSeconds=PT1M
druid.manager.segments.useCache=ifSynced
druid.manager.segments.pollDuration=PT5M
```
</details>
- <details>
<summary>Peon config</summary>
```
druid.startup.logging.logProperties=true
druid.port=8100
druid.service=druid/peon
druid.server.http.numThreads=3
druid.processing.numMergeBuffers=2
druid.indexer.task.baseTaskDir=/druid/data
druid.peon.mode=remote
druid.indexer.task.encapsulatedTask=true
druid.indexer.fork.property.druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.queue.startDelay=PT30S
druid.indexer.storage.type=metadata
druid.indexer.task.encapsulatedTask=true
druid.indexer.runner.type=k8s
druid.indexer.queue.maxSize=60
druid.indexer.runner.namespace=analytics
druid.processing.intermediaryData.storage.type=deepstore
druid.realtime.cache.useCache=true
druid.realtime.cache.populateCache=true
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
