See KAFKA-7124
Cheers
On Sun, Jul 1, 2018 at 3:26 AM, Debraj Manna <subharaj.ma...@gmail.com>
wrote:
> Is there any bug filed for this ?
>
> On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Code snippet from ReassignPartitionsCommand.scala :
> >
> > "log_dirs" -> replicas.map(r =>
> > replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
> > tp.partition, r), AnyLogDir)).asJava
> >
> > We know that the appearance of "any" was due to the OrElse clause.
> > Arguably there is a bug in the above code that the number of AnyLogDir
> > should match the length of the replicas list, or "log_dirs" should be
> > omitted in such case.
> >
> > On Sat, Jun 30, 2018 at 12:06 AM, Manikumar <manikumar.re...@gmail.com>
> > wrote:
> >
> > > It will be taken as "any" directory for each replica, which means
> replica
> > > will placed on any one of the
> > > configured directory on that broker.
> > >
> > > Since it is "log_dirs" optional, you can remove from the json.
> > >
> > > On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <
> subharaj.ma...@gmail.com>
> > > wrote:
> > >
> > > > It is problem on my side. The code was changing the replicas count
> but
> > > not
> > > > the log_dirs. Since I am migrating from 0.10 this part of the code
> was
> > > not
> > > > changed.
> > > >
> > > > I have a follow up question what is the default value of log_dirs if
> I
> > > > don't specify it in reassignment.json ?
> > > >
> > > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <
> > subharaj.ma...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I am generating the reassignent.json like below
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > > > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s'
> |tail
> > > -1 >
> > > > /home/ubuntu/deploy/kafka/reassignment.json"
> > > > >
> > > > > Then I am doing the reassignment using the generated file
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --execute --reassignment-json-file
> > > > /home/ubuntu/deploy/kafka/reassignment.json
> > > > >
> > > > > kafka-reassign-partitions.sh helps states
> > > > >
> > > > > The JSON file with the partition reassignment configurationThe
> format
> > > to
> > > > >> use is -
> > > > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas":
> [1,2,3],
> > > > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> > > "log_dirs"
> > > > is
> > > > >> optional. When it is specified, its length must equal the length
> of
> > > the
> > > > >> replicas list. The value in this list can be either "any" or the
> > > > absolution
> > > > >> path of the log directory on the broker. If absolute log directory
> > > path
> > > > is
> > > > >> specified, it is currently required that the replica has not
> already
> > > > been
> > > > >> created on that broker. The replica will then be created in the
> > > > specified
> > > > >> log directory on the broker later.
> > > > >
> > > > >
> > > > > So it appears reassignment json that is generated by
> > > > > kafka-reassign-partions.sh is creating an issue with logdirs. Is
> this
> > > > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> > > from
> > > > my
> > > > > side. ?
> > > > >
> > > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> > > subharaj.ma...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Please find the server.properties from one of the broker.
> > > > >>
> > > > >> broker.id=0
> > > > >> port=9092
> > > > >> num.network.threads=3
> > > > >> num.io.threads=8
> > > > >> socket.send.buffer.bytes=102400
> > > > >> socket.receive.buffer.bytes=102400
> > > > >> socket.request.max.bytes=104857600
> > > > >> log.dirs=/var/lib/kafka/kafka-logs
> > > > >> num.recovery.threads.per.data.dir=1
> > > > >> log.retention.hours=36
> > > > >> log.retention.bytes=1073741824
> > > > >> log.segment.bytes=536870912
> > > > >> log.retention.check.interval.ms=300000
> > > > >> log.cleaner.enable=false
> > > > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > > > >> message.max.bytes=15000000
> > > > >> replica.fetch.max.bytes=15000000
> > > > >> auto.create.topics.enable=true
> > > > >> zookeeper.connection.timeout.ms=6000
> > > > >> unclean.leader.election.enable=false
> > > > >> delete.topic.enable=false
> > > > >> offsets.topic.replication.factor=1
> > > > >> transaction.state.log.replication.factor=1
> > > > >> transaction.state.log.min.isr=1
> > > > >>
> > > > >> I have placed server.log from a broker at
> > https://gist.github.com/deb
> > > > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > > > >>
> > > > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > > > >>
> > > > >>> Seems to be related to KIP-113.
> > > > >>>
> > > > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> > > content
> > > > ?
> > > > >>>
> > > > >>> If you can pastebin logs from broker, that should help.
> > > > >>>
> > > > >>> Thanks
> > > > >>>
> > > > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > > > subharaj.ma...@gmail.com
> > > > >>> >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi
> > > > >>> >
> > > > >>> > I altered a topic like below in kafka 1.1.0
> > > > >>> >
> > > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > > > >>> 127.0.0.1:2181
> > > > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > > > >>> >
> > > > >>> > But whenever I am trying to verify the reassignment it is
> showing
> > > the
> > > > >>> > below exception
> > > > >>> >
> > > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > > > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > > > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > > > >>> >
> > > > >>> > Partitions reassignment failed due to Size of replicas list
> > > Vector(3,
> > > > >>> 0, 2) is different from size of log dirs list Vector(any) for
> > > partition
> > > > >>> Topic3-7
> > > > >>> > kafka.common.AdminCommandFailedException: Size of replicas
> list
> > > > >>> Vector(3, 0, 2) is different from size of log dirs list
> Vector(any)
> > > for
> > > > >>> partition Topic3-7
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > > >>> ReassignPartitionsCommand.scala:262)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > > >>> ReassignPartitionsCommand.scala:251)
> > > > >>> > at scala.collection.Iterator$
> class.foreach(Iterator.scala:
> > > 891)
> > > > >>> > at scala.collection.AbstractIterator.foreach(
> > > Iterator.scala:133
> > > > >>> 4)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > > >>> sCommand.scala:251)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > > >>> sCommand.scala:250)
> > > > >>> > at scala.collection.immutable.
> List.foreach(List.scala:392)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> > > > >>> > at scala.collection.immutable.
> List.foreach(List.scala:392)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > parsePartitionReassig
> > > > >>> nmentData(ReassignPartitionsCommand.scala:249)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > verifyAssignment(Reas
> > > > >>> signPartitionsCommand.scala:90)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > verifyAssignment(Reas
> > > > >>> signPartitionsCommand.scala:84)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > main(ReassignPartitio
> > > > >>> nsCommand.scala:58)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand.
> > > main(ReassignPartition
> > > > >>> sCommand.scala)
> > > > >>> >
> > > > >>> >
> > > > >>> > My reassignment.json & server.properties is attached. Same
> thing
> > > used
> > > > >>> to
> > > > >>> > work fine in kafka 0.10. Can someone let me what is going
> wrong?
> > Is
> > > > >>> > anything changed related to this in kafka 1.1.0 ?
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
