I was testing a scenario of a server crash while the Kafka is still running
and passing messages. (just did a sudo reboot). By default, I am storing
the logs at /tmp/kafka-log.
So in case an EC2 server restart, am I going to loose the data if I don't
store kafka logs in an EBS drive?
Also, I have another doubt, Does zookeeper stores Kafka info (viz. topics,
brokers list, consumer groups) at /tmp/zookeeper/version-2/log.x ?
I am using python zookeeper client: *kazoo* and here we can find various
Kafka info like '/brokers/topics', '/brokers/topics/topic/partitions' ,
'/consumers' , '/consumers/consumer/offsets'. Where can I get this info
directly?
~Gourab
On Fri, Oct 21, 2016 at 8:13 PM, Ben Davison <ben.davi...@7digital.com>
wrote:
> Are you restarting or terminating the instance? If your terminating, and
> don't have an EBS drive attached you will lose all data on the drive.
>
> On Fri, Oct 21, 2016 at 2:09 PM, Gourab Chowdhury <gourab....@gmail.com>
> wrote:
>
> > I am running kafka in a EC2 server. While running if there is a sudden
> > restart of server, and then we restart the kafka broker, we can't find
> the
> > details of all the topics.
> >
> > bin/kafka-topics.sh --describe --zookeeper <hostname:port>
> > gives no result
> >
> > bin/kafka-console-producer.sh --broker-list <hostname:port> --topic test
> > yields this particular error as follows:
> >
> > WARN Error while fetching metadata with correlation id n :
> > {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.
> NetworkClient)
> >
> > Is the details of topics from the zookeeper getting deleted while
> restart?
> > or this is something else?
> >
> > Also how can I cope with this in in real life production?
> >
> > Regards,
> > Gourab Chowdhury,
> >
>
> --
>
>
> This email, including attachments, is private and confidential. If you have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> Registered in England and Wales. Registered No. 04843573.
>
