> /var/log/cassandra$ cat system.log | grep "Compacting large" | grep -E > "[0-9]+ bytes" -o | cut -d " " -f 1 | awk '{ foo = $1 / 1024 / 1024 ; > print foo "MB" }' | sort -nr | head -n 50
> Is it bad signal? Sorry, I do not know what this is outputting. >> As I can see in cfstats, compacted row maximum size: 386857368 ! Yes. Having rows in the 100's of MB is will cause problems. Doubly so if they are large super columns. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/09/2012, at 5:07 AM, Denis Gabaydulin <gaba...@gmail.com> wrote: > And some stuff from log: > > > /var/log/cassandra$ cat system.log | grep "Compacting large" | grep -E > "[0-9]+ bytes" -o | cut -d " " -f 1 | awk '{ foo = $1 / 1024 / 1024 ; > print foo "MB" }' | sort -nr | head -n 50 > 3821.55MB > 3337.85MB > 1221.64MB > 1128.67MB > 930.666MB > 916.4MB > 861.114MB > 843.325MB > 711.813MB > 706.992MB > 674.282MB > 673.861MB > 658.305MB > 557.756MB > 531.577MB > 493.112MB > 492.513MB > 492.291MB > 484.484MB > 479.908MB > 465.742MB > 464.015MB > 459.95MB > 454.472MB > 441.248MB > 428.763MB > 424.028MB > 416.663MB > 416.191MB > 409.341MB > 406.895MB > 397.314MB > 388.27MB > 376.714MB > 371.298MB > 368.819MB > 366.92MB > 361.371MB > 360.509MB > 356.168MB > 355.012MB > 354.897MB > 354.759MB > 347.986MB > 344.109MB > 335.546MB > 329.529MB > 326.857MB > 326.252MB > 326.237MB > > Is it bad signal? > > On Fri, Sep 21, 2012 at 8:22 PM, Denis Gabaydulin <gaba...@gmail.com> wrote: >> Found one more intersting fact. >> As I can see in cfstats, compacted row maximum size: 386857368 ! >> >> On Fri, Sep 21, 2012 at 12:50 PM, Denis Gabaydulin <gaba...@gmail.com> wrote: >>> Reports - is a SuperColumnFamily >>> >>> Each report has unique identifier (report_id). This is a key of >>> SuperColumnFamily. >>> And a report saved in separate row. >>> >>> A report is consisted of report rows (may vary between 1 and 500000, >>> but most are small). >>> >>> Each report row is saved in separate super column. Hector based code: >>> >>> superCfMutator.addInsertion( >>> report_id, >>> "Reports", >>> HFactory.createSuperColumn( >>> report_row_id, >>> mapper.convertObject(object), >>> columnDefinition.getTopSerializer(), >>> columnDefinition.getSubSerializer(), >>> inferringSerializer >>> ) >>> ); >>> >>> We have two frequent operation: >>> >>> 1. count report rows by report_id (calculate number of super columns >>> in the row). >>> 2. get report rows by report_id and range predicate (get super columns >>> from the row with range predicate). >>> >>> I can't see here a big super columns :-( >>> >>> On Fri, Sep 21, 2012 at 3:10 AM, Tyler Hobbs <ty...@datastax.com> wrote: >>>> I'm not 100% that I understand your data model and read patterns correctly, >>>> but it sounds like you have large supercolumns and are requesting some of >>>> the subcolumns from individual super columns. If that's the case, the >>>> issue >>>> is that Cassandra must deserialize the entire supercolumn in memory >>>> whenever >>>> you read *any* of the subcolumns. This is one of the reasons why composite >>>> columns are recommended over supercolumns. >>>> >>>> >>>> On Thu, Sep 20, 2012 at 6:45 AM, Denis Gabaydulin <gaba...@gmail.com> >>>> wrote: >>>>> >>>>> p.s. Cassandra 1.1.4 >>>>> >>>>> On Thu, Sep 20, 2012 at 3:27 PM, Denis Gabaydulin <gaba...@gmail.com> >>>>> wrote: >>>>>> Hi, all! >>>>>> >>>>>> We have a cluster with virtual 7 nodes (disk storage is connected to >>>>>> nodes with iSCSI). The storage schema is: >>>>>> >>>>>> Reports:{ >>>>>> 1:{ >>>>>> 1:{"value1":"some val", "value2":"some val"}, >>>>>> 2:{"value1":"some val", "value2":"some val"} >>>>>> ... >>>>>> }, >>>>>> 2:{ >>>>>> 1:{"value1":"some val", "value2":"some val"}, >>>>>> 2:{"value1":"some val", "value2":"some val"} >>>>>> ... >>>>>> } >>>>>> ... >>>>>> } >>>>>> >>>>>> create keyspace osmp_reports >>>>>> with placement_strategy = 'SimpleStrategy' >>>>>> and strategy_options = {replication_factor : 4} >>>>>> and durable_writes = true; >>>>>> >>>>>> use osmp_reports; >>>>>> >>>>>> create column family QueryReportResult >>>>>> with column_type = 'Super' >>>>>> and comparator = 'BytesType' >>>>>> and subcomparator = 'BytesType' >>>>>> and default_validation_class = 'BytesType' >>>>>> and key_validation_class = 'BytesType' >>>>>> and read_repair_chance = 1.0 >>>>>> and dclocal_read_repair_chance = 0.0 >>>>>> and gc_grace = 432000 >>>>>> and min_compaction_threshold = 4 >>>>>> and max_compaction_threshold = 32 >>>>>> and replicate_on_write = true >>>>>> and compaction_strategy = >>>>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' >>>>>> and caching = 'KEYS_ONLY'; >>>>>> >>>>>> ============================================= >>>>>> >>>>>> Read/Write CL: 2 >>>>>> >>>>>> Most of the reports are small, but some of them could have a half >>>>>> mullion of rows (xml). Typical operations on this dataset is: >>>>>> >>>>>> count report rows by report_id (top level id of super column); >>>>>> get columns (report_rows) by range predicate and limit for given >>>>>> report_id. >>>>>> >>>>>> A data is written once and hasn't never been updated. >>>>>> >>>>>> So, time to time a couple of nodes crashes with OOM exception. Heap >>>>>> dump says, that we have a lot of super columns in memory. >>>>>> For example, I see one of the reports is in memory entirely. How it >>>>>> could be possible? If we don't load the whole report, cassandra could >>>>>> whether do this for some internal reasons? >>>>>> >>>>>> What should we do to avoid OOMs? >>>> >>>> >>>> >>>> >>>> -- >>>> Tyler Hobbs >>>> DataStax >>>>