quot;Streaming" tab, it show messages below:
Batch Processing Statistics
No statistics have been generated yet.
Am I doing anything wrong on the parallel receiving part?
--
Chen Song
Sorry this is meant to go to spark users. Ignore this thread.
On Fri, Jan 23, 2015 at 2:25 PM, Chen Song wrote:
> I am running into some problems with Spark Streaming when reading from
> Kafka.I used Spark 1.2.0 built on CDH5.
> The example is based on:
>
> https://github.com/ap
me, any topic that had broker 1 as a leader were not working. ZK
> thought that everything was ok and in sync.
>
> Restarting broker 1 fixed the broken topics for a bit, until broker 1 was
> reassigned as leader of some topics, at which point it broke again.
> Restarting broker 2 fixed it ().
>
> We're using kafka-2.10.0_0.8.2.0. Could anyone explain what happened, and
> (most importantly) how we stop it happening again in the future?
>
> Many thanks,
> SimonC
>
>
--
Chen Song
> log.dirs=/mnt/kafka-logs
> >>>
> >>> # The default number of log partitions per topic. More partitions allow
> >>> greater
> >>> # parallelism for consumption, but this will also result in more files
> >>> across
> >>> # the brokers.
> >>> num.partitions=16
> >>>
> >>> # The minimum age of a log file to be eligible for deletion
> >>> log.retention.hours=1
> >>>
> >>> # The maximum size of a log segment file. When this size is reached a
> new
> >>> log segment will be created.
> >>> log.segment.bytes=536870912
> >>>
> >>> # The interval at which log segments are checked to see if they can be
> >>> deleted according
> >>> # to the retention policies
> >>> log.retention.check.interval.ms=6
> >>>
> >>> # By default the log cleaner is disabled and the log retention policy
> >> will
> >>> default to just delete segments after their retention expires.
> >>> # If log.cleaner.enable=true is set the cleaner will be enabled and
> >>> individual logs can then be marked for log compaction.
> >>> log.cleaner.enable=false
> >>>
> >>> # Timeout in ms for connecting to zookeeper
> >>> zookeeper.connection.timeout.ms=100
> >>>
> >>> auto.leader.rebalance.enable=true
> >>> controlled.shutdown.enable=true
> >>>
> >>>
> >>> Thanks in advance.
> >>>
> >>>
> >>>
> >>> Carles Sistare
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> http://khangaonkar.blogspot.com/
> >>
> >
>
>
--
Chen Song
broker CPU usage. However,
this seems to be a limiting factor to parallelism (mostly for consumer
perspective) because all I test is just 400 partitions.
Not sure if this is being addressed in an way. Any thoughts on this?
--
Chen Song
BTW, we are running Kafka 0.8.2.2.
On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote:
> We have a cluster of 32 nodes cluster, with a few topics. When we set 100
> partitions for each topic, the overall CPU usage is 30%, and when we
> increase to 400 partitions, the CPU goes up over
Any thoughts on this topic. Not sure if others have seen the same spike as
us.
On Tue, Nov 17, 2015 at 3:51 PM, Chen Song wrote:
> BTW, we are running Kafka 0.8.2.2.
>
> On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote:
>
>> We have a cluster of 32 nodes cluster, with a few t
(LogCleaner.scala:354)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:321)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:320)
at scala.collection.immutable.List.foreach(List.scala:318)
--
Chen Song
(source.log.readInto(readBuffer, position))
for (entry <- messages.shallowIterator) {
ByteBufferMessageSet.writeMessage(writeBuffer, entry.message,
entry.offset)
On Thu, Jan 21, 2016 at 3:13 PM, Chen Song wrote:
> I am testing compact policy on a topic and got the following exc
/s and 1,000,000 records/s.
I am wondering what makes these 2 implementations perform so differently?
--
Chen Song
the new producer which allows sending multiple
> in-flight requests whereas the first script use the current (old) producer
> which sends only one request at a time to a certain broker.
>
> The new producer will be officially released in 0.8.2.
>
> Guozhang
>
>
> On Tue, Jul
11 matches
Mail list logo
