Thank you Sean for your response. We are also suspecting the same and analyzing/troubleshooting it around queries associated timestamp.
Thanks, Mahesh Daksha On Tue, Apr 9, 2019 at 7:08 PM Durity, Sean R <[email protected]> wrote: > My first suspicion would be to look at the server times in the cluster. It > looks like other cases where a write occurs (with no errors) but the data > is not retrieved as expected. If the write occurs with an earlier timestamp > than the existing data, this is the behavior you would see. The write would > occur, but it would not be the latest data to be retrieved. The write looks > like it fails silently, but it actually does exactly what it is designed to > do. > > > > Sean Durity > > > > *From:* Mahesh Daksha <[email protected]> > *Sent:* Tuesday, April 09, 2019 9:10 AM > *To:* [email protected] > *Subject:* [EXTERNAL] Issue while updating a record in 3 node cassandra > cluster deployed using kubernetes > > > > Hello All, > > I have a 3 node cassandra cluster with Replication factor as 2 and > read-write consistency set to QUORUM. We are using Spring data cassandra. > All infrastructure is deployed using kubernetes. > > Now in normal use case many records gets inserted to cassandra table. Then > we try to modify/update one of the record using save method of repo, like > below: > > ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta); > > After execution of above statement we never see any exception or error. > But still this update state goes silent/fail intermittently. That is at > times the record in the db gets updated successfully where as other time it > fails. Also in the above query when we print *tmpRec* it contains the > updated and correct value every time. Still in the db these updated values > doesn't get reflected. > > We check the the cassandra transport TRACE logs on all nodes and found the > our queries are getting logged there and are being executed also with out > any error or exception. > > Now another weird observation is this all thing works erfectly fine if I > am using single cassandra node (in kubernetes) or if we deploy above infra > using ansible (even works for 3 nodes for Ansible). > > It looks some issue is specifically with the kubernetes 3 node deployment > of cassandra. Primarily looks like replication among nodes causing this. > > Please suggest. > > > > > I have a 3 node cassandra cluster with Replication factor as 2 and > read-write consistency set to QUORUM. We are using Spring data cassandra. > All infrastructure is deployed using kubernetes. > > Now in normal use case many records gets inserted to cassandra table. Then > we try to modify/update one of the record using save method of repo, like > below: > > ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta); > > After execution of above statement we never see any exception or error. > But still this update fail intermittently. That is when we check the record > in the db sometime it gets updated successfully where as other time it > fails. Also in the above query when we print *tmpRec* it contains the > updated and correct value. Still in the db these updated values doesnt get > reflected. > > We check the the cassandra transport TRACE logs on all nodes and found the > our queries are getting logged there and are being executed also. > > Now another weird observation is this all thing works if I am using single > cassandra node (in kubernetes) or if we deploy above infra using ansible > (even works for 3 nodes for Ansible). > > It looks some issue is specifically with the kubernetes 3 node deployment > of cassandra. Primarily looks like replication among nodes causing this. > > Please suggest. > > Below are the contents of my cassandra Docker file: > > FROM ubuntu:16.04 > > > > RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils > net-tools && apt-get clean && \ > > addgroup testuser && useradd -g testuser testuser && usermod --password > testuser testuser; > > > > RUN mkdir -p /opt/test && \ > > mkdir -p /opt/test/data; > > > > ADD jre8.tar.gz /opt/test/ > > ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/ > > > > RUN chmod 755 -R /opt/test/jre && \ > > ln -s /opt/test/jre/bin/java /usr/bin/java && \ > > mv /opt/test/apache-cassandra* /opt/test/cassandra; > > > > RUN mkdir -p /opt/test/cassandra/logs; > > > > ENV JAVA_HOME /opt/test/jre > > RUN export JAVA_HOME > > > > COPY version.txt /opt/test/cassandra/version.txt > > > > WORKDIR /opt/test/cassandra/bin/ > > > > RUN mkdir -p /opt/test/data/saved_caches && \ > > mkdir -p /opt/test/data/commitlog && \ > > mkdir -p /opt/test/data/hints && \ > > chown -R testuser:testuser /opt/test/data && \ > > chown -R testuser:testuser /opt/test; > > > > USER testuser > > > > CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e > 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' > ../conf/conf.yml > ../conf/cassandra.yaml && rm ../conf/conf.yml && > ./cassandra -f > > Please note conf.yml is basically cassandra.yml file having properties > related to cassandra. > > > > Thanks, > > Mahesh Daksha > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
