Hey Jon, This sounds exciting and pretty useful, thanks.
Looking forward to using tlp-stress for validating 15066 performance. We should touch base some time next week to pick a comprehensive set of workloads and versions, perhaps? > On 12 Apr 2019, at 16:34, Jon Haddad <j...@jonhaddad.com> wrote: > > I don't want to derail the discussion about Stabilizing Internode > Messaging, so I'm starting this as a separate thread. There was a > comment that Josh made [1] about doing performance testing with real > clusters as well as a lot of microbenchmarks, and I'm 100% in support > of this. We've been working on some tooling at TLP for the last > several months to make this a lot easier. One of the goals has been > to help improve the 4.0 testing process. > > The first tool we have is tlp-stress [2]. It's designed with a "get > started in 5 minutes" mindset. My goal was to ship a stress tool that > ships with real workloads out of the box that can be easily tweaked, > similar to how fio allows you to design a disk workload and tweak it > with paramaters. Included are stress workloads that stress LWTs (two > different types), materialized views, counters, time series, and > key-value workloads. Each workload can be modified easily to change > compaction strategies, concurrent operations, number of partitions. > We can run workloads for a set number of iterations or a custom > duration. We've used this *extensively* at TLP to help our customers > and most of our blog posts that discuss performance use it as well. > It exports data to both a CSV format and auto sets up prometheus for > metrics collection / aggregation. As an example, we were able to > determine that the compression length set on the paxos tables imposes > a significant overhead when using the Locking LWT workload, which > simulates locking and unlocking of rows. See CASSANDRA-15080 for > details. > > We have documentation [3] on the TLP website. > > The second tool we've been working on is tlp-cluster [4]. This tool > is designed to help provision AWS instances for the purposes of > testing. To be clear, I don't expect, or want, this tool to be used > for production environments. It's designed to assist with the > Cassandra build process by generating deb packages or re-using the > ones that have already been uploaded. Here's a short list of the > things you'll care about: > > 1. Create instances in AWS for Cassandra using any instance size and > number of nodes. Also create tlp-stress instances and a box for > monitoring > 2. Use any available build of Cassandra, with a quick option to change > YAML config. For example: tlp-stress use 3.11.4 -c > concurrent_writes:256 > 3. Do custom builds just by pointing to a local Cassandra git repo. > They can be used the same way as #2. > 4. tlp-stress is automatically installed on the stress box. > 5. Everything's installed with pure bash. I considered something more > complex, but since this is for development only, it turns out the > simplest tool possible works well and it means it's easily > configurable. Just drop in your own bash script starting with a > number in a XX_script_name.sh format and it gets run. > 6. The monitoring box is running Prometheus. It auto scrapes > Cassandra using the Instaclustr metrics library. > 7. Grafana is also installed automatically. There's a couple sample > graphs there now. We plan on having better default graphs soon. > > For the moment it installs java 8 only but that should be easily > fixable to use java 11 to test ZGC (it's on my radar). > > Documentation for tlp-cluster is here [5]. > > There's still some things to work out in the tool, and we've been > working hard to smooth out the rough edges. I still haven't announced > anything WRT tlp-cluster on the TLP blog, because I don't think it's > quite ready for public consumption, but I think the folks on this list > are smart enough to see the value in it even if it has a few warts > still. > > I don't consider myself familiar enough with the networking patch to > give it a full review, but I am qualified to build tools to help test > it and go through the testing process myself. From what I can tell > the patch is moving the codebase in a positive direction and I'd like > to help build confidence in it so we can get it merged in. > > We'll continue to build out and improve the tooling with the goal of > making it easier for people to jump into the QA side of things. > > Jon > > [1] > https://lists.apache.org/thread.html/742009c8a77999f4b62062509f087b670275f827d0c1895bf839eece@%3Cdev.cassandra.apache.org%3E > [2] https://github.com/thelastpickle/tlp-stress > [3] http://thelastpickle.com/tlp-stress/ > [4] https://github.com/thelastpickle/tlp-cluster > [5] http://thelastpickle.com/tlp-cluster/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
