You're correct in that the timeout is only driver side. The server will have its own timeouts configured in the cassandra.yaml file.
I suspect either that you have a node down in your cluster (or 4), or your queries are gradually getting slower. This kind of aligns with the slow query statements in your logs. Are you making changes/updates to the partitions that you are querying? It could be that the partitions are now spread across multiple SSTables and thus slowing things down. You should perform a trace to get a better idea of the issue. A hacky workaround would be to increase your read timeouts server side (read_timeout_in_ms), however this will mask underlying data model issues.
