That's the cause of the imbalance -- an excessively large sstable which suggests to me that at some point you performed a manual major compaction with nodetool compact.
If the table is using STCS, there won't be other compaction partners in the near future so you split the sstable manually with sstablesplit (offline tool so requires C* on the node to be shutdown temporarily). On Mon, Jun 12, 2017 at 1:25 PM, Xihui He <xihu...@gmail.com> wrote: > Hi Vladimir, > > The disks size are all the same, 1.8T as show in df, and only used by > cassandra. > It seems to me that maybe the compacted lagetst file is on data01 which > uses 1.1T. > > Thanks, > Xihui > > On 11 June 2017 at 17:26, Vladimir Yudovin <vla...@winguzone.com> wrote: > >> Hi, >> >> Do your disks have the same size? AFAK Cassandra distributes data with >> proportion to disk size, i.e. keeps the same percent of busy space. >> >> Best regards, Vladimir Yudovin, >> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting* >> >> >> ---- On Wed, 07 Jun 2017 06:15:48 -0400 *Xihui He <xihu...@gmail.com >> <xihu...@gmail.com>>* wrote ---- >> >> Dear All, >> >> We are using multiple disks per node and find the data is not evenly >> distributed (data01 uses 1.1T, but data02 uses 353G). Is this expected? If >> data01 becomes full, would the node be still writable? We are using 2.2.6. >> >> Thanks, >> Xihui >> >> data_file_directories: >> - /data00/cassandra >> - /data01/cassandra >> - /data02/cassandra >> - /data03/cassandra >> - /data04/cassandra >> >> df >> /dev/sde1 1.8T 544G 1.2T 32% /data03 >> /dev/sdc1 1.8T 1.1T 683G 61% /data01 >> /dev/sdf1 1.8T 491G 1.3T 29% /data04 >> /dev/sdd1 1.8T 353G 1.4T 21% /data02 >> /dev/sdb1 1.8T 285G 1.5T 17% /data00 >> >> root@n9-016-015:~# du -sh /data01/cassandra/album_media_feature/* >> 143M /data01/cassandra/album_media_feature/media_feature_blur-066 >> e5700c41511e5beacf197ae340934 >> 4.4G /data01/cassandra/album_media_feature/media_feature_c1-dbadf >> 930c41411e5974743d3a691d887 >> 56K /data01/cassandra/album_media_feature/media_feature_duplicat >> e-09d4b380c41511e58501e9aa37be91a5 >> 16K /data01/cassandra/album_media_feature/media_feature_emotion- >> b8570470054d11e69fb88f073bab8267 >> 240M /data01/cassandra/album_media_feature/media_feature_exposure >> -f55449c0c41411e58f5c9b66773b60c3 >> 649M /data01/cassandra/album_media_feature/media_feature_group-f8 >> de0cc0c41411e5827b995f709095c8 >> 22G /data01/cassandra/album_media_feature/media_feature_multi_cl >> ass-cf3bb72006c511e69fb88f073bab8267 >> 44K /data01/cassandra/album_media_feature/media_feature_pool5-11 >> 85b200c41511e5b7d8757e25e34d67 >> 15G /data01/cassandra/album_media_feature/media_feature_poster-f >> cf45850c41411e597bb1507d1856305 >> 8.0K /data01/cassandra/album_media_feature/media_feature_quality- >> 155d9500c41511e5974743d3a691d887 >> 17G /data01/cassandra/album_media_feature/media_feature_quality_ >> rc-51babf50dba811e59fb88f073bab8267 >> 8.7G /data01/cassandra/album_media_feature/media_feature_scene-00 >> 8a5050c41511e59ebcc3582d286c8d >> 8.0K /data01/cassandra/album_media_feature/media_region_features_ >> v4-29a0cd10150611e6bd3e3f41faa2612a >> 971G /data01/cassandra/album_media_feature/media_region_features_ >> v5-1b805470a3d711e68121757e9ac51b7b >> >> root@n9-016-015:~# du -sh /data02/cassandra/album_media_feature/* >> 1.6G /data02/cassandra/album_media_feature/media_feature_blur-066 >> e5700c41511e5beacf197ae340934 >> 44G /data02/cassandra/album_media_feature/media_feature_c1-dbadf >> 930c41411e5974743d3a691d887 >> 64K /data02/cassandra/album_media_feature/media_feature_duplicat >> e-09d4b380c41511e58501e9aa37be91a5 >> 75G /data02/cassandra/album_media_feature/media_feature_emotion- >> b8570470054d11e69fb88f073bab8267 >> 2.0G /data02/cassandra/album_media_feature/media_feature_exposure >> -f55449c0c41411e58f5c9b66773b60c3 >> 21G /data02/cassandra/album_media_feature/media_feature_group-f8 >> de0cc0c41411e5827b995f709095c8 >> 336M /data02/cassandra/album_media_feature/media_feature_multi_cl >> ass-cf3bb72006c511e69fb88f073bab8267 >> 44K /data02/cassandra/album_media_feature/media_feature_pool5-11 >> 85b200c41511e5b7d8757e25e34d67 >> 2.0G /data02/cassandra/album_media_feature/media_feature_poster-f >> cf45850c41411e597bb1507d1856305 >> 8.0K /data02/cassandra/album_media_feature/media_feature_quality- >> 155d9500c41511e5974743d3a691d887 >> 17G /data02/cassandra/album_media_feature/media_feature_quality_ >> rc-51babf50dba811e59fb88f073bab8267 >> 141M /data02/cassandra/album_media_feature/media_feature_scene-00 >> 8a5050c41511e59ebcc3582d286c8d >> 8.0K /data02/cassandra/album_media_feature/media_region_features_ >> v4-29a0cd10150611e6bd3e3f41faa2612a >> 93G /data02/cassandra/album_media_feature/media_region_features_ >> v5-1b805470a3d711e68121757e9ac51b7b >> >> root@n9-016-015:~# du -sh /data03/cassandra/album_media_feature/* >> 4.3G /data03/cassandra/album_media_feature/media_feature_blur-066 >> e5700c41511e5beacf197ae340934 >> 19G /data03/cassandra/album_media_feature/media_feature_c1-dbadf >> 930c41411e5974743d3a691d887 >> 72K /data03/cassandra/album_media_feature/media_feature_duplicat >> e-09d4b380c41511e58501e9aa37be91a5 >> 2.8G /data03/cassandra/album_media_feature/media_feature_emotion- >> b8570470054d11e69fb88f073bab8267 >> 105M /data03/cassandra/album_media_feature/media_feature_exposure >> -f55449c0c41411e58f5c9b66773b60c3 >> 15G /data03/cassandra/album_media_feature/media_feature_group-f8 >> de0cc0c41411e5827b995f709095c8 >> 23G /data03/cassandra/album_media_feature/media_feature_multi_cl >> ass-cf3bb72006c511e69fb88f073bab8267 >> 44K /data03/cassandra/album_media_feature/media_feature_pool5-11 >> 85b200c41511e5b7d8757e25e34d67 >> 17G /data03/cassandra/album_media_feature/media_feature_poster-f >> cf45850c41411e597bb1507d1856305 >> 8.0K /data03/cassandra/album_media_feature/media_feature_quality- >> 155d9500c41511e5974743d3a691d887 >> 294M /data03/cassandra/album_media_feature/media_feature_quality_ >> rc-51babf50dba811e59fb88f073bab8267 >> 5.1G /data03/cassandra/album_media_feature/media_feature_scene-00 >> 8a5050c41511e59ebcc3582d286c8d >> 8.0K /data03/cassandra/album_media_feature/media_region_features_ >> v4-29a0cd10150611e6bd3e3f41faa2612a >> 228G /data03/cassandra/album_media_feature/media_region_features_ >> v5-1b805470a3d711e68121757e9ac51b7b >> >> >> >
