This may not be helpful, but the first thing I've learned to check in similar situations is whether there is significant time-drift between VMs and actual hardware. Some combination of time-drift and a time-sensitive security check could be causing this. IIRC, CentOS has a funky issue with getting time, even when properly set up... At least I recall that issue on my last gig with CentOS VMs.
HTH... --JohnB On Thu, Feb 18, 2016 at 7:34 AM, Paul Jensen <paul.jen...@spottradingllc.com > wrote: > Running on Centos with Kafka 0.8.2.1 > > I’m experiencing periodic disconnects. I’ve enabled keepalives, though I > don’t think that’s related to the issue. > > The fact that this happens at granularity of 5 minutes and always at 45 > seconds after is interesting. This may be some internal network issue, but > wanted to see if anyone else had encountered this. > > > [paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep > Disconnected logs/20160217.log | grep -v bootstrap > > 17:05:45.679 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 18:05:45.674 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 18:40:46.281 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 18:55:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 19:35:45.673 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 19:50:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 20:10:46.286 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 20:50:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 21:00:45.980 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 22:15:45.678 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 22:35:45.673 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > 23:35:45.676 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: > Disconnected > > [paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep > Disconnected logs/20160217.log | grep bootstrap > > 17:15:45.984 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 18:20:45.674 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 18:30:45.983 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 19:15:45.673 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 20:00:45.987 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 20:25:45.676 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 20:35:45.982 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 21:15:45.678 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 21:35:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 21:50:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 22:00:45.978 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 22:50:45.672 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected > > 23:50:45.674 INFO > [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR > fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnectedd > > > Thanks, > > Paul > > > > > ________________________________ > > Paul Jensen > Software Developer IV > > Spot Trading L.L.C > 440 South LaSalle St., Suite 2800 > Chicago, IL 60605 > Office: 312.362.4550 > Direct: 312-362-4711 > Fax: 312.362.4551 > paul.jen...@spottradingllc.com > www.spottradingllc.com<http://www.spottradingllc.com/> > > ________________________________ > > The information contained in this message may be privileged and > confidential and protected from disclosure. If the reader of this message > is not the intended recipient, or an employee or agent responsible for > delivering this message to the intended recipient, you are hereby notified > that any dissemination, distribution or copying of this communication is > strictly prohibited. If you have received this communication in error, > please notify us immediately by replying to the message and deleting it > from your computer. Thank you. Spot Trading, LLC > > >
