Hi Sam, Your finding is interesting. Effectively, if the number of bytes to skip is larger than the remaining bytes in the buffer + the buffer size it could be faster to use seek. Feel free to open a JIRA ticket and attach your patch. It will be great if you could add to the ticket your table schema as well as some information on your environment (e.g. disk type).
On Tue, Apr 17, 2018 at 8:53 PM, Sam Klock <skl...@akamai.com> wrote: > Thanks (and apologies for the delayed response); that was the kind of > feedback we were looking for. > > We backported the fix for CASSANDRA-10657 to 3.0.16, and it partially > addresses our problem in the sense that it does limit the data sent on > the wire. The performance is still extremely poor, however, due to the > fact that Cassandra continues to read large volumes of data from disk. > (We've also confirmed this behavior in 3.11.2.) > > With a bit more investigation, we now believe the problem (after > CASSNDRA-10657 is applied) is in RebufferingInputStream.skipBytes(), > which appears to read bytes in order to skip them. The subclass used in > our case, RandomAccessReader, exposes a seek(), so we overrode > skipBytes() in it to make use of seek(), and that seems to resolve the > problem. > > This change is intuitively much safer than the one we'd originally > identified, but we'd still like to confirm with you folks whether it's > likely safe and, if so whether it's also potentially worth contributing. > > Thanks, > Sk > > > On 2018-03-22 18:16, Benjamin Lerer wrote: > >> You should check the 3.x release. CASSANDRA-10657 could have fixed your >> problem. >> >> >> On Thu, Mar 22, 2018 at 9:15 PM, Benjamin Lerer < >> benjamin.le...@datastax.com >> >>> wrote: >>> >> >> Syvlain explained the problem in CASSANDRA-4536: >>> " Let me note that in CQL3 a row that have no live column don't exist, so >>> we can't really implement this with a range slice having an empty columns >>> list. Instead we should do a range slice with a full-row slice predicate >>> with a count of 1, to make sure we do have a live column before including >>> the partition key. " >>> >>> By using ColumnFilter.selectionBuilder(); you do not select all the >>> columns. By consequence, some partitions might be returned while they >>> should not. >>> >>> On Thu, Mar 22, 2018 at 6:24 PM, Sam Klock <skl...@akamai.com> wrote: >>> >>> Cassandra devs, >>>> >>>> We use workflows in some of our clusters (running 3.0.15) that involve >>>> "SELECT DISTINCT key FROM..."-style queries. For some tables, we >>>> observed extremely poor performance under light load (i.e., a small >>>> number of rows per second and frequent timeouts), which we eventually >>>> traced to replicas shipping entire rows (which in some cases could store >>>> on the order of MBs of data) to service the query. That surprised us >>>> (partly because 2.1 doesn't seem to behave this way), so we did some >>>> digging, and we eventually came up with a patch that modifies >>>> SelectStatement.java in the following way: if the selection in the query >>>> only includes the partition key, then when building a ColumnFilter for >>>> the query, use: >>>> >>>> builder = ColumnFilter.selectionBuilder(); >>>> >>>> instead of: >>>> >>>> builder = ColumnFilter.allColumnsBuilder(); >>>> >>>> to initialize the ColumnFilter.Builder in gatherQueriedColumns(). That >>>> seems to repair the performance regression, and it doesn't appear to >>>> break any functionality (based on the unit tests and some smoke tests we >>>> ran involving insertions and deletions). >>>> >>>> We'd like to contribute this patch back to the project, but we're not >>>> convinced that there aren't subtle correctness issues we're missing, >>>> judging both from comments in the code and the existence of >>>> CASSANDRA-5912, which suggests optimizing this kind of query is >>>> nontrivial. >>>> >>>> So: does this change sound safe to make, or are there corner cases we >>>> need to account for? If there are corner cases, are there plausibly >>>> ways of addressing them at the SelectStatement level, or will we need to >>>> look deeper? >>>> >>>> Thanks, >>>> SK >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>>> >>>> >>>> >>> >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
