You are going to lose everything you store in /tmp. In a production system
you never configure Kafka or zookeeper to store critical data in /tmp.
This has nothing to do with AWS or EBS it is just standard Linux than
everything under /tmp is deleted when Linux reboots.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Fri, Oct 21, 2016 at 10:38 AM, Gourab Chowdhury <gourab....@gmail.com>
wrote:
> I was testing a scenario of a server crash while the Kafka is still running
> and passing messages. (just did a sudo reboot). By default, I am storing
> the logs at /tmp/kafka-log.
>
> So in case an EC2 server restart, am I going to loose the data if I don't
> store kafka logs in an EBS drive?
>
> Also, I have another doubt, Does zookeeper stores Kafka info (viz. topics,
> brokers list, consumer groups) at /tmp/zookeeper/version-2/log.x ?
>
> I am using python zookeeper client: *kazoo* and here we can find various
> Kafka info like '/brokers/topics', '/brokers/topics/topic/partitions' ,
> '/consumers' , '/consumers/consumer/offsets'. Where can I get this info
> directly?
>
> ~Gourab
>
> On Fri, Oct 21, 2016 at 8:13 PM, Ben Davison <ben.davi...@7digital.com>
> wrote:
>
> > Are you restarting or terminating the instance? If your terminating, and
> > don't have an EBS drive attached you will lose all data on the drive.
> >
> > On Fri, Oct 21, 2016 at 2:09 PM, Gourab Chowdhury <gourab....@gmail.com>
> > wrote:
> >
> > > I am running kafka in a EC2 server. While running if there is a sudden
> > > restart of server, and then we restart the kafka broker, we can't find
> > the
> > > details of all the topics.
> > >
> > > bin/kafka-topics.sh --describe --zookeeper <hostname:port>
> > > gives no result
> > >
> > > bin/kafka-console-producer.sh --broker-list <hostname:port> --topic
> test
> > > yields this particular error as follows:
> > >
> > > WARN Error while fetching metadata with correlation id n :
> > > {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.
> > NetworkClient)
> > >
> > > Is the details of topics from the zookeeper getting deleted while
> > restart?
> > > or this is something else?
> > >
> > > Also how can I cope with this in in real life production?
> > >
> > > Regards,
> > > Gourab Chowdhury,
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>
