We can also do an education campaign to get people to migrate. There will be good reasons to do it.
On Wed, Mar 5, 2025 at 12:33 PM Jeff Jirsa <jji...@gmail.com> wrote: > > I think widely accepted that otel in general has won this stage of > observability, as most metrics systems allow it and most saas providers > support it. So Jon’s point there is important. > > The promise of unifying logs/traces/metrics usually (aka wide events) is far > more important in the tracing side of our observability than in the areas we > use Codahale/DropWizard. > > Scott: if we swap, we can (probably should) deprecate like everything else, > and run both side by side for a release so people don’t lose metrics entirely > on bounce? FF both, to control double cost during the transition. > > > > > On Mar 5, 2025, at 8:21 PM, C. Scott Andreas <sc...@paradoxica.net> wrote: > > No strong opinion on particular choice of metrics library. > > My primary feedback is that if we swap metrics implementations and the new > values are *different*, we can anticipate broad user confusion/interest. > > In particular if latency stats are reported higher post-upgrade, we should > expect users to interpret this as a performance regression, dedicating > significant resources to investigating the change, and expending credibility > with stakeholders in their systems. > > - Scott > > On Mar 5, 2025, at 11:57 AM, Benedict <bened...@apache.org> wrote: > > > I really like the idea of integrating tracing, metrics and logging frameworks. > > I would like to have the time to look closely at the API before we decide to > adopt it though. I agree that a widely deployed API has inherent benefits, > but any API we adopt also shapes future evolution of our capabilities. > Hopefully this is also a good API that allows us plenty of evolutionary > headroom. > > > On 5 Mar 2025, at 19:45, Josh McKenzie <jmcken...@apache.org> wrote: > > > > if the plan is to rip out something old and unmaintained and replace with > something new, I think there's a huge win to be had by implementing the > standard that everyone's using now. > > Strong +1 on anything that's an ecosystem integration inflection point. The > added benefit here is that if we architect ourselves to gracefully integrate > with whatever system's are ubiquitous today, we'll inherit the migration work > that any new industry-wide replacement system would need to do to become the > new de facto standard. > > On Wed, Mar 5, 2025, at 2:23 PM, Jon Haddad wrote: > > Thank you for the replies. > > Dmitry: Based on some other patches you've worked on and your explanation > here, it looks like you're optimizing the front door portion of write path - > very cool. Testing it in isolation with those settings makes sense if your > goal is to push write throughput as far as you can, something I'm very much > on board with, and is a key component to pushing density and reducing cost. > I'm spinning up a 5.0 cluster now to run a test, so I'll run a load test > similar to what you've done and try to reproduce your results. I'll also > review the JIRA to get more familiar with what you're working on. > > Benedict: I agree with your line of thinking around optimizing the cost of > metrics. As we push both density and multi-tenancy, there's going to be more > and more demand for clusters with hundreds or thousands of tables. Maybe > tens of thousands. Reducing overhead for something that's O(N * M) (multiple > counters per table) will definitely be a welcome improvement. There's always > more stuff that's going to get in the way, but it's an elephant and I > appreciate every bite. > > My main concern with metrics isn't really compatibility, and I don't have any > real investment in DropWizard. I don't know if there's any real value in > putting in effort to maintain compatibility, but I'm just one sample, so I > won't make a strong statement here. > > It would be *very nice* we moved to metrics which implement the Open > Telemetry Metrics API [1], which I think solves multiple issues at once: > > * We can use either one of the existing implementations (OTel SDK) or our own > * We get a "free" upgrade that lets people tap into the OTel ecosystem > * It paves the way for OTel traces with ZipKin [2] / Jaeger [3] > * We can use the ubiquitous Otel instrumentation agent to send metrics to the > OTel collector, meaning people can collect at a much higher frequency than > today > * OTel logging is a significant improvement over logback, you can coorelate > metrics + traces + logs together. > > Anyways, if the plan is to rip out something old and unmaintained and replace > with something new, I think there's a huge win to be had by implementing the > standard that everyone's using now. > > All this is very exciting and I appreciate the discussion! > > Jon > > [1] https://opentelemetry.io/docs/languages/java/api/ > [2] https://zipkin.io/ > [3] https://www.jaegertracing.io/ > > > > > On Wed, Mar 5, 2025 at 2:58 AM Dmitry Konstantinov <netud...@gmail.com> wrote: > > Hi Jon > > >> Is there a specific workload you're running where you're seeing it take > >> up a significant % of CPU time? Could you share some metrics, profile > >> data, or a workload so I can try to reproduce your findings? > Yes, I have shared the workload generation command (sorry, it is in > cassandra-stress, I have not yet adopted your tool but want to do it soon :-) > ), setup details and async profiler CPU profile in CASSANDRA-20250 > A summary: > > it is a plain insert-only workload to assert a max throughput capacity for a > single node: ./tools/bin/cassandra-stress "write n=10m" -rate threads=100 > -node myhost > small amount of data per row is inserted, local SSD disks are used, so CPU is > a primary bottleneck in this scenario (while it is quite synthetic in my real > business cases CPU is a primary bottleneck as well) > I used 5.1 trunk version (similar results I have for 5.0 version while I was > checking CASSANDRA-20165) > I enabled trie memetables + offheap objects mode > I disabled compaction > a recent nightly build is used for async-profiler > my hardware is quite old: on-premise VM, Linux 4.18.0-240.el8.x86_64, > OpenJdk-11.0.26+4, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 16 cores > link to CPU profile ("codahale" code: 8.65%) > -XX:+DebugNonSafepoints option is enabled to improve the profile precision > > > On Wed, 5 Mar 2025 at 12:38, Benedict Elliott Smith <bened...@apache.org> > wrote: > > Some quick thoughts of my own… > > === Performance === > - I have seen heap dumps with > 1GiB dedicated to metric counters. This patch > should improve this, while opening up room to cut it further, steeply. > - The performance improvement in relative terms for the metrics being > replaced is rather dramatic - about 80%.. We can also improve this further. > - Cheaper metrics (in terms of both cpu and memory) means we can readily have > more of them, exposing finer-grained details. This is hard to understate the > value of. > > === Reporting === > - We’re already non-standard for our most important metrics, because we had > to replace the Codahale histogram years ago > - We can continue implementing the Codahale interfaces, so that exporting > libraries have minimal work to support us > - We can probably push patches upstream to a couple of selected libraries we > consider important > - I would anyway also support picking a new reporting framework to support, > but I would like us to do this with great care to avoid repeating our > mistakes. I won’t have cycles to actually implement this, so it would be down > to others to decide if they are willing to undertake this work > > I think the fallback option for now, however, is to abuse unsafe to allow us > to override the implementation details of Codahale metrics. So we can > decouple the performance discussion for now from the deprecation discussion, > but I think we should have a target of deprecating Codahale/DropWizard for > the reasons Dmitry outlines, however we decide to do it. > > On 4 Mar 2025, at 21:17, Jon Haddad <j...@rustyrazorblade.com> wrote: > > I've got a few thoughts... > > On the performance side, I took a look at a few CPU profiles from past > benchmarks and I'm seeing DropWizard taking ~ 3% of CPU time. Is there a > specific workload you're running where you're seeing it take up a significant > % of CPU time? Could you share some metrics, profile data, or a workload so > I can try to reproduce your findings? In my testing I've found the majority > of the overhead from metrics to come from JMX, not DropWizard. > > On the operator side, inventing our own metrics lib means risks making it > harder to instrument Cassandra. There are libraries out there that allow you > to tap into DropWizard metrics directly. For example, Sarma Pydipally did a > presentation on this last year [1] based on some code I threw together. > > If you're planning on making it easier to instrument C* by supporting sending > metrics to the OTel collector [2], then I could see the change being a net > win as long as the perf is no worse than the status quo. > > It's hard to know the full extent of what you're planning and the impact, so > I'll save any opinions till I know more about the plan. > > Thanks for bringing this up! > Jon > > [1] > https://planetcassandra.org/leaf/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra-business-platform-team/ > [2] https://opentelemetry.io/docs/collector/ > > On Tue, Mar 4, 2025 at 12:40 PM Dmitry Konstantinov <netud...@gmail.com> > wrote: > > Hi all, > > After a long conversation with Benedict and Maxim in CASSANDRA-20250 I would > like to raise and discuss a proposal to deprecate Dropwizard/Codahale metrics > usage in the next major release of Cassandra server and drop it in the > following major release. > Instead of it our own Java API and implementation should be introduced. For > the next major release Dropwizard/Codahale API is still planned to support by > extending Codahale implementations, to give potential users of this API > enough time for transition. > The proposal does not affect JMX API for metrics, it is only about local Java > API changes within Cassandra server classpath, so it is about the cases when > somebody outside of Cassandra server code relies on Codahale API in some kind > of extensions or agents. > > Reasons: > 1) Codahale metrics implementation is not very efficient from CPU and memory > usage point of view. In the past we already replaced default Codahale > implementations for Reservoir with our custom one and now in CASSANDRA-20250 > we (Benedict and I) want to add a more efficient implementation for Counter > and Meter logic. So, in total we do not have so much logic left from the > original library (mostly a MetricRegistry as container for metrics) and the > majority of logic is implemented by ourselves. > We use metrics a lot along the read and write paths and they contribute a > visible overhead (for example for plain write load it is about 9-11% > according to async profiler CPU profile), so we want them to be highly > optimized. > From memory perspective Counter and Meter are built based on LongAdder and > they are quite heavy for the amounts which we create and use. > > 2) Codahale metrics does not provide any way to replace Counter and Meter > implementations. There are no full functional interfaces for these entities + > MetricRegistry has casts/checks to implementations and cannot work with > anything else. > I looked through the already reported issues and found the following similar > and unsuccessful attempt to introduce interfaces for metrics: > https://github.com/dropwizard/metrics/issues/2186 > as well as other older attempts: > https://github.com/dropwizard/metrics/issues/252 > https://github.com/dropwizard/metrics/issues/264 > https://github.com/dropwizard/metrics/issues/703 > https://github.com/dropwizard/metrics/pull/487 > https://github.com/dropwizard/metrics/issues/479 > https://github.com/dropwizard/metrics/issues/253 > > So, the option to request an extensibility from Codahale metrics does not > look real.. > > 3) It looks like the library is in maintenance mode now, 5.x version is on > hold and many integrations are also not so alive. > The main benefit to use Codahale metrics should be a huge amount of > reporters/integrations but if we check carefully the list of reporters > mentioned here: > https://metrics.dropwizard.io/4.2.0/manual/third-party.html#reporters > we can see that almost all of them are dead/archived. > > 4) In general, exposing other 3rd party libraries as our own public API > frequently creates too many limitations and issues (Guava is another typical > example which I saw previously, it is easy to start but later you struggle > more and more). > > Does anyone have any questions or concerns regarding this suggestion? > -- > Dmitry Konstantinov > > > > > -- > Dmitry Konstantinov > >
