Re: no space left error

2015-01-14 Thread Jun Rao
Yes, that would be useful. Thanks, Jun On Tue, Jan 6, 2015 at 2:07 PM, Sa Li wrote: > BTW, I found the the /kafka/logs also getting biger and bigger, like > controller.log and state-change.logs. should I launch a cron the clean them > up regularly or there is way to delete them regularly? > >

Re: no space left error

2015-01-06 Thread Sa Li
BTW, I found the the /kafka/logs also getting biger and bigger, like controller.log and state-change.logs. should I launch a cron the clean them up regularly or there is way to delete them regularly? thanks AL On Tue, Jan 6, 2015 at 2:01 PM, Sa Li wrote: > Hi, All > > We fix the problem, I lik

Re: no space left error

2015-01-06 Thread Sa Li
Hi, All We fix the problem, I like to share the what the problem is in case someone come across the similar issues. We add the data drive for each node /dev/sdb1 , but specify the wrong path in server.properties, which means the data was written into the wrong drive /dev/sda2, quickly eat up all t

Re: no space left error

2015-01-06 Thread Sa Li
Thanks the reply, the disk is not full: root@exemplary-birds:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 133G 3.4G 123G 3% / none4.0K 0 4.0K 0% /sys/fs/cgroup udev 32G 4.0K 32G 1% /dev tmpfs 6.3G 764K 6.3G 1% /run

Re: no space left error

2015-01-06 Thread Joe Stein
There are two parts to this 1) How to prevent Kafka from filling up disks which https://issues.apache.org/jira/browse/KAFKA-1489 is trying to deal with (I set the ticket to unassigned just now since i don't think anyone is working on it and was assigned by default, could be wrong though so assign

Re: no space left error

2015-01-06 Thread David Birdsong
I'm keen to hear about how to work one's way out of a filled partition since I've run into this many times after having tuned retention bytes or retention (time?) incorrectly. The proper path to resolving this isn't obvious based on my many harried searches through documentation. I often end up st

Re: no space left error

2015-01-06 Thread Otis Gospodnetic
Hi, Your disk is full. You should probably have something that checks/monitors disk space and alerts you when it's full. Maybe you can point Kafka to a different, larger disk or partition. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Suppor

Re: no space left error

2015-01-06 Thread Sa Li
Continue this issue, when I restart the server, like bin/kafka-server-start.sh config/server.properties it will fails to start the server, like [2015-01-06 20:00:55,441] FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.InternalE

Re: no space left error

2015-01-06 Thread Sa Li
the complete error message: -su: cannot create temp file for here-document: No space left on device OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_root/19721 Try using the -Djava.io.tmpdir= option to select an alternate temp location. [2015-01-06 19

no space left error

2015-01-06 Thread Sa Li
Hi, All I am doing performance test on our new kafka production server, but after sending some messages (even faked message by using bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance), it comes out the error of connection, and shut down the brokers, after that, I see such e