(source.log.readInto(readBuffer, position))
for (entry <- messages.shallowIterator) {
ByteBufferMessageSet.writeMessage(writeBuffer, entry.message,
entry.offset)
On Thu, Jan 21, 2016 at 3:13 PM, Chen Song wrote:
> I am testing compact policy on a topic and got the following exc
(LogCleaner.scala:354)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:321)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:320)
at scala.collection.immutable.List.foreach(List.scala:318)
--
Chen Song
Any thoughts on this topic. Not sure if others have seen the same spike as
us.
On Tue, Nov 17, 2015 at 3:51 PM, Chen Song wrote:
> BTW, we are running Kafka 0.8.2.2.
>
> On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote:
>
>> We have a cluster of 32 nodes cluster, with a few t
BTW, we are running Kafka 0.8.2.2.
On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote:
> We have a cluster of 32 nodes cluster, with a few topics. When we set 100
> partitions for each topic, the overall CPU usage is 30%, and when we
> increase to 400 partitions, the CPU goes up over
broker CPU usage. However,
this seems to be a limiting factor to parallelism (mostly for consumer
perspective) because all I test is just 400 partitions.
Not sure if this is being addressed in an way. Any thoughts on this?
--
Chen Song
> log.dirs=/mnt/kafka-logs
> >>>
> >>> # The default number of log partitions per topic. More partitions allow
> >>> greater
> >>> # parallelism for consumption, but this will also result in more files
> >>> across
> >>> # the brokers.
> >>> num.partitions=16
> >>>
> >>> # The minimum age of a log file to be eligible for deletion
> >>> log.retention.hours=1
> >>>
> >>> # The maximum size of a log segment file. When this size is reached a
> new
> >>> log segment will be created.
> >>> log.segment.bytes=536870912
> >>>
> >>> # The interval at which log segments are checked to see if they can be
> >>> deleted according
> >>> # to the retention policies
> >>> log.retention.check.interval.ms=6
> >>>
> >>> # By default the log cleaner is disabled and the log retention policy
> >> will
> >>> default to just delete segments after their retention expires.
> >>> # If log.cleaner.enable=true is set the cleaner will be enabled and
> >>> individual logs can then be marked for log compaction.
> >>> log.cleaner.enable=false
> >>>
> >>> # Timeout in ms for connecting to zookeeper
> >>> zookeeper.connection.timeout.ms=100
> >>>
> >>> auto.leader.rebalance.enable=true
> >>> controlled.shutdown.enable=true
> >>>
> >>>
> >>> Thanks in advance.
> >>>
> >>>
> >>>
> >>> Carles Sistare
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> http://khangaonkar.blogspot.com/
> >>
> >
>
>
--
Chen Song
me, any topic that had broker 1 as a leader were not working. ZK
> thought that everything was ok and in sync.
>
> Restarting broker 1 fixed the broken topics for a bit, until broker 1 was
> reassigned as leader of some topics, at which point it broke again.
> Restarting broker 2 fixed it ().
>
> We're using kafka-2.10.0_0.8.2.0. Could anyone explain what happened, and
> (most importantly) how we stop it happening again in the future?
>
> Many thanks,
> SimonC
>
>
--
Chen Song
Sorry this is meant to go to spark users. Ignore this thread.
On Fri, Jan 23, 2015 at 2:25 PM, Chen Song wrote:
> I am running into some problems with Spark Streaming when reading from
> Kafka.I used Spark 1.2.0 built on CDH5.
> The example is based on:
>
> https://github.com/ap
quot;Streaming" tab, it show messages below:
Batch Processing Statistics
No statistics have been generated yet.
Am I doing anything wrong on the parallel receiving part?
--
Chen Song
the new producer which allows sending multiple
> in-flight requests whereas the first script use the current (old) producer
> which sends only one request at a time to a certain broker.
>
> The new producer will be officially released in 0.8.2.
>
> Guozhang
>
>
> On Tue, Jul
/s and 1,000,000 records/s.
I am wondering what makes these 2 implementations perform so differently?
--
Chen Song
11 matches
Mail list logo
