On 11/21/2013 11:14 AM, Romain Manni-Bucau wrote: > Hi > > first maybe the thread should move over sirona list (I wouldn't bother > commons ;)): http://sirona.incubator.apache.org/mail-lists.html > > about hot topics I think we have: > 1) Gui enhancement (a jira is about using another js library to draw > graphes and get an analysis/correlation view, another topic is the GUI > configuration: how to make dashboard etc...) > 2) Storage backends (we support mainly cassandra and in memory ATM but > a rotating JDBC impl could be great too) > 3) Alerting is not yet done and needs to be in final versions > 4) Agent/Plugins enhancement (we have a few today) > 5) Surely a better configuration > 6) Counter implementation with less locks (even if thanks to the stat > impl we use we reduced the lock duration we still rely too much on > lock IMO and I'm sure we can do better). > 7) etc.. ;) > > > It is a new project so all help is welcomed!
Hi Romain, thanks for the pointers, I will take a closer look, some of them would really be interesting to work on. Thanks, Thomas > > PS: we have an IRC channel on freenode: #apache-sirona > > > > > Romain Manni-Bucau > Twitter: @rmannibucau > Blog: http://rmannibucau.wordpress.com/ > LinkedIn: http://fr.linkedin.com/in/rmannibucau > Github: https://github.com/rmannibucau > > > > 2013/11/21 Thomas Neidhart <thomas.neidh...@gmail.com>: >> I did take a look at sirona, and it looks really nice. >> Whats the best way to contribute to it? Any open issues you would like to >> work somebody on? >> >> Thomas >> >> >> On Wed, Nov 20, 2013 at 5:38 PM, Romain Manni-Bucau >> <rmannibu...@gmail.com>wrote: >> >>> yep but this is not enough, almost each "component" is doing n++. This >>> is fast but since done under a lock if it can be removed it should. >>> Romain Manni-Bucau >>> Twitter: @rmannibucau >>> Blog: http://rmannibucau.wordpress.com/ >>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>> Github: https://github.com/rmannibucau >>> >>> >>> >>> 2013/11/20 Phil Steitz <phil.ste...@gmail.com>: >>>> On 11/20/13, 7:26 AM, Romain Manni-Bucau wrote: >>>>> next version (rewrite/fork): >>>>> >>> https://svn.apache.org/repos/asf/incubator/sirona/trunk/core/src/main/java/org/apache/sirona/counters/OptimizedStatistics.java >>>>> >>>>> was easier to centralize everything in a single class >>>> >>>> That is exactly what I meant about the moment statistics. They can >>>> share state. The problem is this approach eliminates the >>>> pluggability and also cuts out the other stats. We can probably >>>> have it both ways by allowing you to turn off stuff you don't want >>>> and just having the moment stats share state. >>>> >>>> Phil >>>>> Romain Manni-Bucau >>>>> Twitter: @rmannibucau >>>>> Blog: http://rmannibucau.wordpress.com/ >>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>>>> Github: https://github.com/rmannibucau >>>>> >>>>> >>>>> >>>>> 2013/11/20 Phil Steitz <phil.ste...@gmail.com>: >>>>>> On 11/20/13, 12:43 AM, Romain Manni-Bucau wrote: >>>>>>> Hi >>>>>>> >>>>>>> A quick mail to give some feedbacks of my tests. >>>>>>> >>>>>>> I started to hack a bit to get rid of not used stats by sirona, >>>>>>> typically I do ATM: >>>>>>> >>>>>>> setSumsqImpl(NoopStat.INSTANCE); >>>>>>> setSumLogImpl(NoopStat.INSTANCE); >>>>>>> setGeoMeanImpl(NoopStat.INSTANCE); >>>>>>> >>>>>>> (NoopStat is a mock of StorelessUnivariateStatistic doijg nothing) >>>>>>> >>>>>>> Another point which could be improoved is the duplication of info >>>>>>> accross sub StorelessUnivariateStatistic (typically n computed several >>>>>>> times for instance). >>>>>> Good point. Its kind of funny that simplest way to solve the >>>>>> problem you mention at the end is to remove the flexibility that you >>>>>> use in the beginning - i.e., to no longer use separate stats >>>>>> instances to compute the bundled statistics. The setup is the way >>>>>> it is precisely so that you can plug in alternative impls. I had >>>>>> never thought of no-op-ing instances to suppress things, but it does >>>>>> work. Having stats share state data is a little tricky but in >>>>>> theory possible. The moment stats at least are set up to support >>>>>> this. Patches are welcome. If you don't mind opening a JIRA to >>>>>> suggest eliminating repeated computations that would be great. >>>>>> >>>>>> Phil >>>>>>> Romain Manni-Bucau >>>>>>> Twitter: @rmannibucau >>>>>>> Blog: http://rmannibucau.wordpress.com/ >>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>>>>>> Github: https://github.com/rmannibucau >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2013/11/6 Phil Steitz <phil.ste...@gmail.com>: >>>>>>>> On 11/6/13 9:05 AM, Romain Manni-Bucau wrote: >>>>>>>>> Great! >>>>>>>>> >>>>>>>>> Btw not sure for sirona we oculd use it. One constraint on >>> sirona-core >>>>>>>>> is to stay self contained. We already shade math3 so shading pool2 >>> too >>>>>>>>> would start to create a big jar for this need. I'll try to bench >>>>>>>>> deeper next week too. >>>>>>>> OK - and any ideas you have about how to implement something >>>>>>>> lightweight inside [math] much appreciated. >>>>>>>> >>>>>>>> Phil >>>>>>>>> Romain Manni-Bucau >>>>>>>>> Twitter: @rmannibucau >>>>>>>>> Blog: http://rmannibucau.wordpress.com/ >>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>>>>>>>> Github: https://github.com/rmannibucau >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2013/11/6 Phil Steitz <phil.ste...@gmail.com>: >>>>>>>>>> On 11/6/13 8:47 AM, Romain Manni-Bucau wrote: >>>>>>>>>>> well pool are based on locks so I'm not sure (it would need deep >>>>>>>>>>> benchs on a real app) it does worth it >>>>>>>>>> Commons pool2 uses pretty lightweight locking and using a pool of >>>>>>>>>> instances achieves the basic objective of reducing contention for >>>>>>>>>> the single sync lock on one SummaryStatistics object. I bet it >>>>>>>>>> would improve throughput over the single-instance approach if >>>>>>>>>> maxActive, maxIdle were tuned. If I get some time to play with >>>>>>>>>> this, I will report back with some benchmarks. >>>>>>>>>> >>>>>>>>>> Phil >>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>> Twitter: @rmannibucau >>>>>>>>>>> Blog: http://rmannibucau.wordpress.com/ >>>>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>>>>>>>>>> Github: https://github.com/rmannibucau >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2013/11/6 Phil Steitz <phil.ste...@gmail.com>: >>>>>>>>>>>> On 11/5/13 11:26 PM, Romain Manni-Bucau wrote: >>>>>>>>>>>>> Hehe, right. >>>>>>>>>>>>> >>>>>>>>>>>>> I looked a bit more today and LongAdder is only a part of the >>>>>>>>>>>>> solution. The stat computation still needs to lock to get acces >>> to >>>>>>>>>>>>> previous values (N -> N+1). Basically the gain wouldn't be as >>>>>>>>>>>>> important as I thought :(. >>>>>>>>>>>> Right, but I think your original idea of maintaining a pool of >>>>>>>>>>>> instances (fewer that one per thread) to be periodically >>> aggregated >>>>>>>>>>>> is a good one. See below. >>>>>>>>>>>>> As I said before we'll wait a bit to gather feedbacks, if it >>> blocks >>>>>>>>>>>>> I'll come back trying to find + propose a solution. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in all cases for your answers! >>>>>>>>>>>> A workaround that I have started playing with (partly for other >>>>>>>>>>>> benchmarking reasons) might be to actually use a pool for the >>> stats >>>>>>>>>>>> objects that the monitoring threads use. Using a pool would >>> allow >>>>>>>>>>>> you to monitor and tune the parameters. We now have (well, once >>> the >>>>>>>>>>>> VOTE in progress completes :) a decently performing pool >>>>>>>>>>>> implementation. The tricky bit is locking the instances during >>>>>>>>>>>> aggregation. One way to handle this would be to have the >>> factory's >>>>>>>>>>>> passivate method and the aggregation thread contend for locks on >>> the >>>>>>>>>>>> pooled stats instances. The only contention would be when >>>>>>>>>>>> aggregation is copying individual instances and contention would >>> be >>>>>>>>>>>> with at most one client thread (waiting to proceed in passivate). >>>>>>>>>>>> >>>>>>>>>>>> Phil >>>>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>>>> Twitter: @rmannibucau >>>>>>>>>>>>> Blog: http://rmannibucau.wordpress.com/ >>>>>>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau >>>>>>>>>>>>> Github: https://github.com/rmannibucau >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2013/11/5 Phil Steitz <phil.ste...@gmail.com>: >>>>>>>>>>>>>> On 11/5/13 9:57 AM, Romain Manni-Bucau wrote: >>>>>>>>>>>>>>> @Phil: hmm can be but the framework would create its own >>> overhead which >>>>>>>>>>>>>>> would be avoided with a dedicated solution, no? Well thought >>> gain was great >>>>>>>>>>>>>>> for small investment but ok to postpone it >>>>>>>>>>>>>> As I said, patches welcome. Go for it. My point about the >>>>>>>>>>>>>> framework was that when you actually get this implemented >>> inside, >>>>>>>>>>>>>> e.g. SummaryStatistics, you will have built a mini-framework. >>>>>>>>>>>>>> Whatever overhead it has, it will have ;) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Phil >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Le 5 nov. 2013 18:54, "Romain Manni-Bucau" < >>> rmannibu...@gmail.com> a écrit : >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Well I didnt test sirona in prod but when using jamon (same >>> kind of >>>>>>>>>>>>>>>> framework) locks were creating a serious overhead on some >>> benches. Not the >>>>>>>>>>>>>>>> most important but enough to try to solve it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That said we are not yet in 1.0 so Im ok to wait for more >>> serious >>>>>>>>>>>>>>>> feedbacks if you think it is better >>>>>>>>>>>>>>>> Le 5 nov. 2013 18:48, "Ted Dunning" <ted.dunn...@gmail.com> >>> a écrit : >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Nov 4, 2013 at 10:09 PM, Romain Manni-Bucau >>>>>>>>>>>>>>>>> <rmannibu...@gmail.com>wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Oh sorry, that's what I said early, in a real app no or >>> not enough to >>>>>>>>>>>>>>>>> be an >>>>>>>>>>>>>>>>>> issue buy on simple apps or very high thrououtput apps yes. >>>>>>>>>>>>>>>>>> Le 5 nov. 2013 07:00, "Ted Dunning" < >>> ted.dunn...@gmail.com> a écrit : >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> That isn't what I meant. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Do you really think that more than one metric has to >>> update >>>>>>>>>>>>>>>>> (increment, >>>>>>>>>>>>>>>>>>> say) at precisely the same time? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I realize that is what you said. Do you have any serious >>> examples where >>>>>>>>>>>>>>>>> metrics have to be updated all or nothing? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>>>> >>>>>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>> >>>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>> >>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>> >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>> For additional commands, e-mail: dev-h...@commons.apache.org >>> >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org