t; >> > 2017-09-20 03:21:02.060 INFO o.a.s.c.kafka.KafkaCheckpointManager
> > >> [main]
> > >> > -
> > >> > Got offset 0 for topic __samza_checkpoint_ver_1_for_test-job_1 and
> > >> > partition 0. Attempting to fetch messages for checkpoint log.
> > >> > 2017-09-20 03:21:02.072 INFO o.a.s.c.kafka.KafkaCheckpointManager
> > >> [main]
> > >> > -
> > >> > Get latest offset 42890599 for topic
> > >> > __samza_checkpoint_ver_1_for_test-job_1 and partition 0.
> > >> >
> > >> > Looking at this line in KafkaCheckpointManager
> > >> > <https://github.com/apache/samza/blob/0.9.1/samza-kafka/
> > >> > src/main/scala/org/apache/samza/checkpoint/kafka/
> > >> > KafkaCheckpointManager.scala#L275>,
> > >> > it seems to indicate that the loop iterates from 0 to 42890599 and
> > make
> > >> > requests for each.
> > >> >
> > >> > Questions:
> > >> > 1. What does that loop do exactly?
> > >> > 2. Is this an expected behaviour? Is "Got offset 0 for topic ..."
> > >> normal?
> > >> > 3. Any ideas on how to fix this?
> > >> >
> > >> > Thanks,
> > >> > Xiaochuan Yu
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Jagadish V,
> > >> Graduate Student,
> > >> Department of Computer Science,
> > >> Stanford University
> > >>
> > >
> >
>
--
All the best
Liu Bo
> at org.apache.samza.coordinator.JobModelManager$.apply(
> JobCoordinator.scala:115)
> at org.apache.samza.job.local.ThreadJobFactory.getJob(
> ThreadJobFactory.scala:40)
> at org.apache.samza.job.JobRunner.run(JobRunner.scala:125)
> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:65)
> at org.apache.samza.job.JobRunner.main(JobRunner.scala)
>
>
--
All the best
Liu Bo
ee the log “Begin to close files” are not printed out and of
> course the logic is not executed.
>
> 2. The hadoop cluster I use is “HDP-2.5.0”,the log aggregation is also
> enabled, but logs of containers can not be collected, only the log of am
> can be seen.
>
>
>
>
t; BaseMessageChooser and SystemProducer will not be called and the
> container
> > log will be removed automatically, how can resolve this?
> >
> > Thanks.
> >
> >
> > ShuQi
>
--
All the best
Liu Bo
own.
2016-09-07 23:03:50.289 [main] SamzaContainer [INFO] Shutting down consumer
multiplexer.
--
All the best
Liu Bo
is topic in the email list, I guess I have to wait for a
while.
On 10 May 2016 at 01:24, Yi Pan wrote:
> Hi, Bo,
>
> I embedded my answers in-between:
>
> On Sun, May 8, 2016 at 9:00 PM, Liu Bo wrote:
>
> > The other thing is log retention is set to 24 hour or 30GB.
02:25, Jagadish Venkatraman wrote:
> Hi,
>
> Is your checkpoint topic log compacted? That may help in reducing the size
> of the log.
>
> On Sat, May 7, 2016 at 2:35 AM, Liu Bo wrote:
>
> > Hi group
> >
> > I've got this problem while doing maintenance of o
help will be much of my appreciation.
BTW: I think the checkpoint manager could start from the end of the
checkpoint, and try to "look backward" for the offset for current partition
instead of reading all of them.
--
All the best
Liu Bo
h 15, 2016, David Yu wrote:
> >
> > > Our samza job is consuming from a Kafka topic. AFAIU, samza will auto
> > > assign the job a consumer group id and client id. However, I'm not able
> > to
> > > see that showing up under zookeeper. Am I missing something?
> > >
> >
>
--
All the best
Liu Bo
;t have a "recommended" GC for the users. It would probably also
> depend on the application's usage pattern as well. Our perf partner Tao
> Feng @LinkedIn may have some more insights.
>
> @Tao, do you have any comments on this?
>
> -Yi
>
> On Sun, Jan 31, 2016
nce on samza GC tuning for
discussion? Thanks in advance.
--
All the best
Liu Bo
properties:
systems.kafka.streams.metrics.samza.msg.serde=metrics
ref: http://samza.apache.org/learn/documentation/0.9/container/metrics.html
Metrics are written to kafka correctly after adding this to the metrics
config.
Hope it's correct for 0.10.
--
All the best
Liu Bo
12 matches
Mail list logo
