The disk full bug is fixed in the -final artifacts and the 1.0.0 svn branch.
On Wed, Oct 12, 2011 at 10:16 AM, Günter Ladwig <guenter.lad...@kit.edu> wrote:
> Hi,
>
> I tried running cfstats on other nodes. It works on all except two nodes.
> Then I tried scrubbing the OSP CF on one of the nodes where it fails
> (actually the node where the first exception I reported happened), but got
> this exception in the log:
>
> [...]
> INFO 14:58:00,604 Scrub of
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9528-Data.db')
> complete: 44378258 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO 14:58:00,604 Scrubbing
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9709-Data.db')
> INFO 14:58:46,958 GC for ParNew: 347 ms for 1 collections, 1172065472 used;
> max is 2057306112
> INFO 14:59:51,360 Scrub of
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9709-Data.db')
> complete: 3833789 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO 14:59:51,361 Scrubbing
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9708-Data.db')
> INFO 14:59:56,355 Scrub of
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9708-Data.db')
> complete: 179681 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO 14:59:56,356 Scrubbing
> SSTableReader(path='/data/cassandra/data/KeyspaceCumulus/OSP-h-9530-Data.db')
> ERROR 14:59:56,393 Fatal exception in thread
> Thread[CompactionExecutor:7,5,RMI Runtime]
> java.io.IOException: disk full
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:477)
> at
> org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:465)
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63)
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:217)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> The disk is nowhere near full:
>
> root@ubuntu:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/vda1 1.9G 654M 1.2G 36% /
> none 2.0G 164K 2.0G 1% /dev
> none 2.0G 0 2.0G 0% /dev/shm
> none 2.0G 40K 2.0G 1% /var/run
> none 2.0G 0 2.0G 0% /var/lock
> none 2.0G 0 2.0G 0% /lib/init/rw
> none 1.9G 654M 1.2G 36% /var/lib/ureadahead/debugfs
> /dev/vdb 79G 44G 32G 59% /data
> /dev/vdc 4.0G 137M 3.7G 4% /data2
>
> /data contains the data directory, /data2 the commitlog.
>
> This is the sstable on disk:
>
> -rw-r--r-- 2 cassandra cassandra 3946478 2011-10-10 07:51
> /data/cassandra/data/KeyspaceCumulus/OSP-h-9530-CompressionInfo.db
> -rw-r--r-- 2 cassandra cassandra 3207243773 2011-10-10 07:51
> /data/cassandra/data/KeyspaceCumulus/OSP-h-9530-Data.db
> -rw-r--r-- 2 cassandra cassandra 1758976 2011-10-10 07:51
> /data/cassandra/data/KeyspaceCumulus/OSP-h-9530-Filter.db
> -rw-r--r-- 2 cassandra cassandra 71879406 2011-10-10 07:51
> /data/cassandra/data/KeyspaceCumulus/OSP-h-9530-Index.db
> -rw-r--r-- 2 cassandra cassandra 4284 2011-10-10 07:51
> /data/cassandra/data/KeyspaceCumulus/OSP-h-9530-Statistics.db
>
> I'm currently scrubbing the CF on the other node where cfstats fails, but
> it's not done yet.
>
> Cheers,
> Günter
>
> On 12.10.2011, at 16:28, Jonathan Ellis wrote:
>
>> Try scrubbing the CF ("nodetool scrub") and see if that fixes it.
>>
>> If not, then at least we have a reproducible problem. :)
>>
>> On Tue, Oct 11, 2011 at 4:43 PM, Günter Ladwig <guenter.lad...@kit.edu>
>> wrote:
>>> Hi all,
>>>
>>> I'm seeing the same problem on my 1.0.0-rc2 cluster. However, I do not have
>>> 5000, but just three (compressed) CFs.
>>>
>>> The exception does not happen for the Migrations CF, but for one of mine:
>>>
>>> Keyspace: KeyspaceCumulus
>>> Read Count: 816
>>> Read Latency: 8.926029411764706 ms.
>>> Write Count: 16808336
>>> Write Latency: 0.03914435902518846 ms.
>>> Pending Tasks: 0
>>> Column Family: OSP
>>> SSTable count: 22
>>> Space used (live): 22319610951
>>> Space used (total): 22227585112
>>> Number of Keys (estimate): 87322624
>>> Memtable Columns Count: 56028
>>> Memtable Data Size: 54362270
>>> Memtable Switch Count: 154
>>> Read Count: 277
>>> Read Latency: NaN ms.
>>> Write Count: 10913659
>>> Write Latency: NaN ms.
>>> Pending Tasks: 0
>>> Key cache: disabled
>>> Row cache: disabled
>>> Compacted row minimum size: 125
>>> Compacted row maximum size: 9223372036854775807
>>> Exception in thread "main" java.lang.IllegalStateException: Unable to
>>> compute ceiling for max when histogram overflowed
>>> at
>>> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:170)
>>> at
>>> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.java:275)
>>> [...snip…]
>>>
>>> I also had a look at the stats using JMX. The other CFs work fine, the only
>>> problem seems to be this one. In JMX it shows 'Unavailable' for the row
>>> mean size and also that ridiculous value for the max size.
>>>
>>> The cluster consists of 15 nodes. The keyspace has three CFs (SPO, OSP and
>>> POS) of which only two contain any data (POS is empty), and uses
>>> replication factor 2. In total, there are about 2 billion columns in each
>>> CF. The data distribution is different between the two CFs. The row sizes
>>> for SPO should be fairly evenly distributed whereas OSP will have a few
>>> very wide rows and a large number of small rows.
>>>
>>> Here is the output from describe:
>>>
>>> Keyspace: KeyspaceCumulus:
>>>
>>> Replication Strategy:
>>> org.apache.cassandra.locator.SimpleStrategy
>>> Durable Writes: true
>>> Options: [replication_factor:2]
>>> Column Families:
>>> ColumnFamily: OSP
>>> Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
>>> Default column value validator:
>>> org.apache.cassandra.db.marshal.UTF8Type
>>> Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
>>> Row cache size / save period in seconds / keys to save : 0.0/0/all
>>> Key cache size / save period in seconds: 0.0/0
>>> GC grace seconds: 0
>>> Compaction min/max thresholds: 4/32
>>> Read repair chance: 0.0
>>> Replicate on write: false
>>> Built indexes: []
>>> Compaction Strategy:
>>> org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
>>> Compression Options:
>>> sstable_compression:
>>> org.apache.cassandra.io.compress.SnappyCompressor
>>> ColumnFamily: POS
>>> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
>>> Default column value validator:
>>> org.apache.cassandra.db.marshal.UTF8Type
>>> Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
>>> Row cache size / save period in seconds / keys to save : 0.0/0/all
>>> Key cache size / save period in seconds: 0.0/0
>>> GC grace seconds: 0
>>> Compaction min/max thresholds: 4/32
>>> Read repair chance: 0.0
>>> Replicate on write: false
>>> Built indexes: [POS.index_p]
>>> Column Metadata:
>>> Column Name: !o
>>> Validation Class: org.apache.cassandra.db.marshal.UTF8Type
>>> Column Name: !p
>>> Validation Class: org.apache.cassandra.db.marshal.UTF8Type
>>> Index Name: index_p
>>> Index Type: KEYS
>>> Compaction Strategy:
>>> org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
>>> Compression Options:
>>> sstable_compression:
>>> org.apache.cassandra.io.compress.SnappyCompressor
>>> ColumnFamily: SPO
>>> Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
>>> Default column value validator:
>>> org.apache.cassandra.db.marshal.UTF8Type
>>> Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
>>> Row cache size / save period in seconds / keys to save : 0.0/0/all
>>> Key cache size / save period in seconds: 0.0/0
>>> GC grace seconds: 0
>>> Compaction min/max thresholds: 4/32
>>> Read repair chance: 0.0
>>> Replicate on write: false
>>> Built indexes: []
>>> Compaction Strategy:
>>> org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
>>> Compression Options:
>>> sstable_compression:
>>> org.apache.cassandra.io.compress.SnappyCompressor
>>>
>>> If you need additional information, let me know.
>>>
>>> Cheers,
>>> Günter
>>>
>>> On 04.10.2011, at 10:20, aaron morton wrote:
>>>
>>>> That row has a size of 819 peta bytes, so something is odd there. The
>>>> error is a result of that value been so huge. When you rant he same script
>>>> on 0.8.6 what was the max size of the Migrations CF ?
>>>>
>>>> As Jonathan says, it's unlikely anyone would have tested creating 5000
>>>> CF's. Most people only create a few 10's of CF's at most.
>>>>
>>>> either use fewer CF's or…
>>>>
>>>> * dump the Migrations CF using sstable2json to take a look around
>>>> * work out steps to reproduce and report it on Jira
>>>>
>>>> Hope that helps.
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Cassandra Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 4/10/2011, at 11:30 AM, Ramesh Natarajan wrote:
>>>>
>>>>> We recreated the schema using the same input file on both clusters and
>>>>> they are running identical load.
>>>>>
>>>>> Isn't the exception thrown in the system CF?
>>>>>
>>>>> this line looks strange:
>>>>>
>>>>> Compacted row maximum size: 9223372036854775807
>>>>>
>>>>> thanks
>>>>> Ramesh
>>>>>
>>>>> On Mon, Oct 3, 2011 at 5:26 PM, Jonathan Ellis <jbel...@gmail.com> wrote:
>>>>> Looks like you have unexpectedly large rows in your 1.0 cluster but
>>>>> not 0.8. I guess you could use sstable2json to manually check your
>>>>> row sizes.
>>>>>
>>>>> On Mon, Oct 3, 2011 at 5:20 PM, Ramesh Natarajan <rames...@gmail.com>
>>>>> wrote:
>>>>>> It happens all the time on 1.0. It doesn't happen on 0.8.6. Is there any
>>>>>> thing I can do to check?
>>>>>> thanks
>>>>>> Ramesh
>>>>>>
>>>>>> On Mon, Oct 3, 2011 at 5:15 PM, Jonathan Ellis <jbel...@gmail.com> wrote:
>>>>>>>
>>>>>>> My suspicion would be that it has more to do with "rare case when
>>>>>>> running with 5000 CFs" than "1.0 regression."
>>>>>>>
>>>>>>> On Mon, Oct 3, 2011 at 5:00 PM, Ramesh Natarajan <rames...@gmail.com>
>>>>>>> wrote:
>>>>>>>> We have about 5000 column family and when we run the nodetool cfstats
>>>>>>>> it
>>>>>>>> throws out this exception... this is running 1.0.0-rc1
>>>>>>>> This seems to work on 0.8.6. Is this a bug in 1.0.0?
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> Ramesh
>>>>>>>> Keyspace: system
>>>>>>>> Read Count: 28
>>>>>>>> Read Latency: 5.8675 ms.
>>>>>>>> Write Count: 3
>>>>>>>> Write Latency: 0.166 ms.
>>>>>>>> Pending Tasks: 0
>>>>>>>> Column Family: Schema
>>>>>>>> SSTable count: 4
>>>>>>>> Space used (live): 4293758276
>>>>>>>> Space used (total): 4293758276
>>>>>>>> Number of Keys (estimate): 5376
>>>>>>>> Memtable Columns Count: 0
>>>>>>>> Memtable Data Size: 0
>>>>>>>> Memtable Switch Count: 0
>>>>>>>> Read Count: 3
>>>>>>>> Read Latency: NaN ms.
>>>>>>>> Write Count: 0
>>>>>>>> Write Latency: NaN ms.
>>>>>>>> Pending Tasks: 0
>>>>>>>> Key cache capacity: 53
>>>>>>>> Key cache size: 2
>>>>>>>> Key cache hit rate: NaN
>>>>>>>> Row cache: disabled
>>>>>>>> Compacted row minimum size: 104
>>>>>>>> Compacted row maximum size: 1955666
>>>>>>>> Compacted row mean size: 1508515
>>>>>>>> Column Family: HintsColumnFamily
>>>>>>>> SSTable count: 0
>>>>>>>> Space used (live): 0
>>>>>>>> Space used (total): 0
>>>>>>>> Number of Keys (estimate): 0
>>>>>>>> Memtable Columns Count: 0
>>>>>>>> Memtable Data Size: 0
>>>>>>>> Memtable Switch Count: 0
>>>>>>>> Read Count: 5
>>>>>>>> Read Latency: NaN ms.
>>>>>>>> Write Count: 0
>>>>>>>> Write Latency: NaN ms.
>>>>>>>> Pending Tasks: 0
>>>>>>>> Key cache capacity: 1
>>>>>>>> Key cache size: 0
>>>>>>>> Key cache hit rate: NaN
>>>>>>>> Row cache: disabled
>>>>>>>> Compacted row minimum size: 0
>>>>>>>> Compacted row maximum size: 0
>>>>>>>> Compacted row mean size: 0
>>>>>>>> Column Family: LocationInfo
>>>>>>>> SSTable count: 1
>>>>>>>> Space used (live): 6947
>>>>>>>> Space used (total): 6947
>>>>>>>> Number of Keys (estimate): 128
>>>>>>>> Memtable Columns Count: 0
>>>>>>>> Memtable Data Size: 0
>>>>>>>> Memtable Switch Count: 2
>>>>>>>> Read Count: 20
>>>>>>>> Read Latency: NaN ms.
>>>>>>>> Write Count: 3
>>>>>>>> Write Latency: NaN ms.
>>>>>>>> Pending Tasks: 0
>>>>>>>> Key cache capacity: 1
>>>>>>>> Key cache size: 1
>>>>>>>> Key cache hit rate: NaN
>>>>>>>> Row cache: disabled
>>>>>>>> Compacted row minimum size: 73
>>>>>>>> Compacted row maximum size: 258
>>>>>>>> Compacted row mean size: 185
>>>>>>>> Column Family: Migrations
>>>>>>>> SSTable count: 4
>>>>>>>> Space used (live): 4315909643
>>>>>>>> Space used (total): 4315909643
>>>>>>>> Number of Keys (estimate): 512
>>>>>>>> Memtable Columns Count: 0
>>>>>>>> Memtable Data Size: 0
>>>>>>>> Memtable Switch Count: 0
>>>>>>>> Read Count: 0
>>>>>>>> Read Latency: NaN ms.
>>>>>>>> Write Count: 0
>>>>>>>> Write Latency: NaN ms.
>>>>>>>> Pending Tasks: 0
>>>>>>>> Key cache capacity: 5
>>>>>>>> Key cache size: 0
>>>>>>>> Key cache hit rate: NaN
>>>>>>>> Row cache: disabled
>>>>>>>> Compacted row minimum size: 5839589
>>>>>>>> Compacted row maximum size: 9223372036854775807
>>>>>>>> Exception in thread "main" java.lang.IllegalStateException: Unable to
>>>>>>>> compute ceiling for max when histogram overflowed
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:170)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.java:275)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>>>>>>> at
>>>>>>>> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
>>>>>>>> at
>>>>>>>> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
>>>>>>>> at
>>>>>>>>
>>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
>>>>>>>> at
>>>>>>>>
>>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>>>>>>> at
>>>>>>>>
>>>>>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>>>>>>> at
>>>>>>>>
>>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>>>>>>> at
>>>>>>>>
>>>>>>>> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>> at
>>>>>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>>>>>>> at
>>>>>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jonathan Ellis
>>>>>>> Project Chair, Apache Cassandra
>>>>>>> co-founder of DataStax, the source for professional Cassandra support
>>>>>>> http://www.datastax.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jonathan Ellis
>>>>> Project Chair, Apache Cassandra
>>>>> co-founder of DataStax, the source for professional Cassandra support
>>>>> http://www.datastax.com
>>>>>
>>>>
>>>
>>> --
>>> Dipl.-Inform. Günter Ladwig
>>>
>>> Karlsruhe Institute of Technology (KIT)
>>> Institute AIFB
>>>
>>> Englerstraße 11 (Building 11.40, Room 250)
>>> 76131 Karlsruhe, Germany
>>> Phone: +49 721 608-47946
>>> Email: guenter.lad...@kit.edu
>>> Web: www.aifb.kit.edu
>>>
>>> KIT – University of the State of Baden-Württemberg and National Large-scale
>>> Research Center of the Helmholtz Association
>>>
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
> --
> Dipl.-Inform. Günter Ladwig
>
> Karlsruhe Institute of Technology (KIT)
> Institute AIFB
>
> Englerstraße 11 (Building 11.40, Room 250)
> 76131 Karlsruhe, Germany
> Phone: +49 721 608-47946
> Email: guenter.lad...@kit.edu
> Web: www.aifb.kit.edu
>
> KIT – University of the State of Baden-Württemberg and National Large-scale
> Research Center of the Helmholtz Association
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
