Is the node selection based on key deterministic across multiple clients? If it is, that sounds plausible. For this particular workload it's definitely possible to have a hot key / spot, but it was surprising it wasn't three nodes that got hot, it was just one.
On Mon, Nov 14, 2016 at 6:26 PM, Alex Popescu <al...@datastax.com> wrote: > I'm wondering if what you are seeing is https://datastax-oss. > atlassian.net/browse/PYTHON-643 (that could still be a sign of a > potential data hotspot) > > On Sun, Nov 13, 2016 at 10:57 PM, Andrew Bialecki < > andrew.biale...@klaviyo.com> wrote: > >> We're using the "default" TokenAwarePolicy. Our nodes are spread across >> different racks within one datacenter. I've turned on debug logging for the >> Python driver, but it doesn't look like it logs which Casandra node each >> request goes to, but maybe I haven't got the right logging set to debug. >> >> On Mon, Nov 14, 2016 at 12:39 AM, Ben Slater <ben.sla...@instaclustr.com> >> wrote: >> >>> What load balancing policies are you using in your client code ( >>> https://datastax.github.io/python-driver/api/cassandra/policies.html)? >>> >>> Cheers >>> Ben >>> >>> On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki < >>> andrew.biale...@klaviyo.com> wrote: >>> >>>> We have an odd situation where all of a sudden of our cluster started >>>> seeing a disproportionate number of writes go to one node. We're using the >>>> Python driver version 3.7.1. I'm not sure if this is a driver issue or >>>> possibly a network issue causing requests to get routed in an odd way. It's >>>> not absolute, there are requests going to all nodes. >>>> >>>> Tried restarting the problematic node, no luck (those are the quiet >>>> periods). Tried restarting the clients, also no luck. Checked nodetool >>>> status and ownership is even across the cluster. >>>> >>>> Curious if anyone's seen this behavior before. Seems like the next step >>>> will be to debug the client and see why it's choosing that node. >>>> >>>> [image: Inline image 1] >>>> >>>> >>>> -- >>>> AB >>>> >>> >> >> >> -- >> AB >> > > > > -- > Bests, > > Alex Popescu | @al3xandru > Sen. Product Manager @ DataStax > > > > -- AB
