At a theoretical level assuming it could be implemented with a magic wand, would there be value to having a dual set of queues/threadpools at each of the SEDA stages inside cassandra for a two-tier of priority? Such that you could mark queries that return pages and pages of data as lower-priority while smaller single-partition queries could be marked/defaulted as normal priority, such that the lower-priority queues are only served if the normal priority queues are empty?
I suppose rough equivalency to this would be dual-datacenter with an analysis cluster to serve the "slow" queries and a frontline one for the higher priority stuff. However, it has come up several times that I'd like to run a one-off maintenance job/query against production that could not be easily changed (can't just throw up a DC), and while I can do app-level throttling with some pain and sweat, it would seem something like this could do lower-priority work in a somewhat-loaded cluster without impacting the main workload.
