Yes, limits are good, in case of a solution for virtual tables. I can imagine that it will be use just for the sake of diagnostics what a node is doing and so on and even we lift the limit to something like 1000, that is already a plenty to show a respective operator (as a person) what a node does in the particular moment or some time ago.
For a more "forensics" approach it might be ideal to put it to chronicle queues to have it persistent and an offline debugging but that is something the other (pluggable) implementation would cover. It should be also possible to tune this in runtime, e.g. the limit, not sure how hard this is to implement in the run time when diagnostics "framework" is started, however we should be at least able to turn it off completely on demand. I am not sure yet how the implementation in case of virtual tables fits into the overall picture of "pluggability". I would say that is a completely standalone approach. Virtual tables are quite unique in this regard. However, I might somehow "force it" to implement some extension point / interface just for the sake of having it all on the same approach. On Sat, 24 Jul 2021 at 10:34, Mick Semb Wever <m...@apache.org> wrote: > > Some input… > > We don't need a CEP for this: Diagnostic Events already exists, as do > Virtual Tables. But if you want to take it to a CEP, that's to be > encouraged :) > > Virtual Tables should remain virtual, i.e. they shouldn't be backed by > custom system_* tables solely for the sake of persisting the virtual table > data. Furthermore, I'd avoid a) the write amplification this would cause, > and b) putting queue data into a system table. (Let's not repeat the hints > table). > > A big +1 to adding a virtual table that reads `DiagnosticEventPersistence. > instance().getEvents(..)` > That is, adding a virtual table for diagnostic events that honours the > already existing backend store (whatever it is memory or binlog). This will > be an effective and elegant solution to CASSANDRA-13472 (thank you for > bringing this to light!) > > And, a big +1 to adding local persistence of diagnostic events using binlog > (ChronicleQueue). > > > > My concern is that there would be "too much data" in them and it might grow > and grow as a node is live, … > > > It is in the same category as auditing and fql. The limit is there as it is > expected that a client is collecting these events, and that the store > (memory or binlog) is not the permanent archive. Check out how Reaper > collects and archives diagnostic events: > https://github.com/thelastpickle/cassandra-reaper/blob/master/src/server/src/main/java/io/cassandrareaper/service/DiagEventPoller.java --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
