We are testing an upgrade to 0.10.1 from 0.9.1 and noticed a regression. When
starting a stream job that consumes a topic that does not yet exist, the job
dies with the following exception:
Exception in thread "main" java.lang.IllegalArgumentException: No tasks found.
Likely due to no input partitions. Can't run a job with no tasks.
at
org.apache.samza.container.grouper.task.GroupByContainerCount.validateTasks(GroupByContainerCount.java:193)
at
org.apache.samza.container.grouper.task.GroupByContainerCount.balance(GroupByContainerCount.java:86)
at
org.apache.samza.coordinator.JobModelManager$.refreshJobModel(JobCoordinator.scala:278)
at
org.apache.samza.coordinator.JobModelManager$.jobModelGenerator$1(JobCoordinator.scala:211)
at
org.apache.samza.coordinator.JobModelManager$.initializeJobModel(JobCoordinator.scala:217)
at
org.apache.samza.coordinator.JobModelManager$.getJobCoordinator(JobCoordinator.scala:122)
at
org.apache.samza.coordinator.JobModelManager$.apply(JobCoordinator.scala:106)
at
org.apache.samza.coordinator.JobModelManager$.apply(JobCoordinator.scala:112)
at
org.apache.samza.job.local.ThreadJobFactory.getJob(ThreadJobFactory.scala:40)
at org.apache.samza.job.JobRunner.run(JobRunner.scala:129)
at org.apache.samza.job.JobRunner$.main(JobRunner.scala:66)
at org.apache.samza.job.JobRunner.main(JobRunner.scala)
The root cause seems to be commit 920f803a2e3dab809f4d7bb518259b0f4164407f from
SAMZA-971. From what I can tell passing partitionsMetadataOnly = true to the
StreamMetadataCache in JobModelManager#getInputStreamPartitions is what's
causing this this behavior. The input topic is still created, but the proper
partition metadata is not returned, resulting in an empty set being returned.
The behavior of Kafka here is screwy, but this still seems like a regression.
The old behavior is nice because it doesn't require that producer systems come
up before the stream processors.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com>
[email protected]<mailto:[email protected]>
________________________________
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
