Also I am not sure it matters, but I just realized the keyspace created has replication factor of 2 when my Cassandra is really just a single node. Is Cassandra smart enough to ignore the RF of 2 and work with only 1 single node?
On Mon, Apr 20, 2015 at 8:23 PM, Jimmy Lin <y2klyf+w...@gmail.com> wrote: > hi, > there were only a few (4 of them across 4 minutes with around 200ms), so > shouldn't be the reason > > The system log has tons of > INFO [MigrationStage:1] 2015-04-20 11:03:21,880 ColumnFamilyStore.java > (line 633) Enqueuing flush of Memtable-schema_keyspaces@2079381036(138/1215 > serialized/live bytes, 3 ops) > INFO [MigrationStage:1] 2015-04-20 11:03:21,900 ColumnFamilyStore.java > (line 633) Enqueuing flush of > Memtable-schema_columnfamilies@1283263314(1036/3946 > serialized/live bytes, 24 ops) > INFO [MigrationStage:1] 2015-04-20 11:03:21,921 ColumnFamilyStore.java > (line 633) Enqueuing flush of Memtable-schema_columns > > But that could be just normal given that our unit tests are doing lot of > droping keyspace and creating keyspace/tables. > > I read the MigrationStage thread pool is default to one, so wondering if > that could be a reason it may be doing something that block others? > > > > On Mon, Apr 20, 2015 at 2:40 PM, Sebastian Estevez < > sebastian.este...@datastax.com> wrote: > >> Can you grep for GCInspector in your system.log? Maybe you have long GC >> pauses. >> >> All the best, >> >> >> [image: datastax_logo.png] <http://www.datastax.com/> >> >> Sebastián Estévez >> >> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com >> >> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: >> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] >> <https://twitter.com/datastax> [image: g+.png] >> <https://plus.google.com/+Datastax/about> >> <http://feeds.feedburner.com/datastax> >> >> <http://cassandrasummit-datastax.com/> >> >> DataStax is the fastest, most scalable distributed database technology, >> delivering Apache Cassandra to the world’s most innovative enterprises. >> Datastax is built to be agile, always-on, and predictably scalable to any >> size. With more than 500 customers in 45 countries, DataStax is the >> database technology and transactional backbone of choice for the worlds >> most innovative companies such as Netflix, Adobe, Intuit, and eBay. >> >> On Mon, Apr 20, 2015 at 12:19 PM, Jimmy Lin <y2klyf+w...@gmail.com> >> wrote: >> >>> Yes, sometimes it is create table and sometime it is create index. >>> It doesn't happen all the time, but feel like if multiple tests trying >>> to do schema change(create or drop), Cassandra has a long delay on the >>> schema change statements. >>> >>> I also just read about "auto_snapshot", and I turn it off but still no >>> luck. >>> >>> >>> >>> On Mon, Apr 20, 2015 at 6:42 AM, Jim Witschey <jim.witsc...@datastax.com >>> > wrote: >>> >>>> Jimmy, >>>> >>>> What's the exact command that produced this trace? Are you saying that >>>> the 16-second wait in your trace what times out in your CREATE TABLE >>>> statements? >>>> >>>> Jim Witschey >>>> >>>> Software Engineer in Test | jim.witsc...@datastax.com >>>> >>>> On Sun, Apr 19, 2015 at 7:13 PM, Jimmy Lin <y2klyf+w...@gmail.com> >>>> wrote: >>>> > hi, >>>> > we have some unit tests that run parallel that will create tmp >>>> keyspace, and >>>> > tables and then drop them after tests are done. >>>> > >>>> > From time to time, our create table statement run into "All hosts(s) >>>> for >>>> > query failed... Timeout during read" (from datastax driver) error. >>>> > >>>> > We later turn on tracing, and record something in the following. >>>> > See below between "===" , Native_Transport-Request thread and >>>> MigrationStage >>>> > thread, there was like 16 seconds doing something. >>>> > >>>> > Any idea what that 16 seconds Cassandra was doing? We can work around >>>> that >>>> > but increasing our datastax driver timeout value, but wondering if >>>> there is >>>> > actually better way to solve this? >>>> > >>>> > thanks >>>> > >>>> > >>>> > >>>> > ---------------- tracing ---------- >>>> > >>>> > >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 58730d97-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> > Key cache hit for sstable 95588 | 127.0.0.1 | 1592 | >>>> > Native-Transport-Requests:102 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 58730d98-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> Seeking >>>> > to partition beginning in data file | 127.0.0.1 | 1593 | >>>> > Native-Transport-Requests:102 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 58730d99-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> Merging >>>> > data from memtables and 3 sstables | 127.0.0.1 | 1595 | >>>> > Native-Transport-Requests:102 >>>> > >>>> > ===================== >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 58730d9a-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> > Read 3 live and 0 tombstoned cells | 127.0.0.1 | 1610 | >>>> > Native-Transport-Requests:102 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a40-e6e2-11e4-823d-93572f3db015 >>>> > | Executing seq scan across 1 sstables for >>>> > (min(-9223372036854775808), min(-9223372036854775808)] | 127.0.0.1 | >>>> > 16381594 | MigrationStage:1 >>>> > ===================== >>>> > >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a41-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> Seeking >>>> > to partition beginning in data file | 127.0.0.1 | 16381782 | >>>> > MigrationStage:1 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a42-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381787 | >>>> > MigrationStage:1 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a43-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> Seeking >>>> > to partition beginning in data file | 127.0.0.1 | 16381789 | >>>> > MigrationStage:1 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a44-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381791 | >>>> > MigrationStage:1 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a45-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> Seeking >>>> > to partition beginning in data file | 127.0.0.1 | 16381792 | >>>> > MigrationStage:1 >>>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>>> 62364a46-e6e2-11e4-823d-93572f3db015 >>>> > | >>>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381794 | >>>> > MigrationStage:1 >>>> > . >>>> > . >>>> > . >>>> > >>>> >>> >>> >> >
