Hi Team, Thanks again for the recommendations and the articles.
One quick question on the PERCENTILE function. Does this support weighted percentiles? I see that the t-digest library supports this. Weighting is critical for accurate reporting of medians with survey data. WILL On Wed, Oct 12, 2022 at 2:03 AM Xiaoxiang Yu <x...@apache.org> wrote: > Hi, Will > Glad to see you have complete the 'basic path' of kylin4_on_cloud, > which provided > some tools which make deployment of Kylin much easier than before. But I > think to > make Kylin provide satisfying performance(response time, concurrency), > user must > have enough knowledge of Apache Spark and Apache Kylin. I think this > article maybe > helpful: > https://kylin.apache.org/blog/2021/06/17/Why-did-Youzan-choose-Kylin4 . > > > -- > *Best wishes to you ! * > *From ：**Xiaoxiang Yu* > > > > At 2022-10-12 02:21:13, "Will Glass-Husain" <wgl...@forio.com> wrote: > >Thank you -- very helpful. > > > >Regarding limits on the number of dimensions. What are the > >compute/storage constraints on this? For a given query: > >* Where is the data stored > >* Which nodes is the computation occurring on? > > > >I am trying to figure out -- if we have a large number of dimensions, what > >part of the cloud based kylin needs to be increased (I'm doing the setup > >from the kylin4_on_cloud branch) > > > >Thanks, WILL > > > >On Tue, Oct 11, 2022 at 1:20 AM Xiaoxiang Yu <x...@apache.org> wrote: > > > >> 1) The criteria for filtering (e.g. selecting sex='male') and grouping > >> (e.g. > >> group by state) should be dimensions - is this correct? > >> Yes, besides Kylin has limit of 63 dimensions at maximum. But you should > >> be aware of 'The Curse of Dimensionality'. > >> > >> 2.1) Items that I would like to sum should be measures, is that right? > >> Yes. > >> > >> 2.2) Is there a limit to the number of measures? > >> No, there isn't such limit. > >> > >> 3) Did Kylin support sum(expression)? > >> From mysql doc > >> https://dev.mysql.com/doc/refman/5.7/en/aggregate-functions.html#function_sum > >> , > >> we know MySQL supports it. > >> For Kylin, Kylin should support it for Kylin 3.X and the future version > >> 5.x. But unluckily, Kylin 4.x didn't support sum exprssion, and Kylin 4.x > >> is the version you are using. > >> > >> 4) Does Kylin support MEDIAN? > >> > >> Yes, Kylin should support but I didn't test it. In fact, Kylin has a > >> measure PERCENTILE, and I think 50th percentile is equal to MEDIAN, am I > >> right? > >> > >> -- > >> *Best wishes to you ! * > >> *From ：**Xiaoxiang Yu* > >> > >> > >> > >> At 2022-10-11 14:03:14, "Will Glass-Husain" <wgl...@forio.com> wrote: > >> >Hi, > >> > > >> >Thanks for the recent help as I set up my first Kylin system. I have a > >> >question regarding proper design of a cube to run some > >> >demographic queries. I want to make this accessible in a webapp, with > >> >reasonable response time. > >> > > >> >I have a CSV file with about 80 columns on sex, race, state, age, internet > >> >access, job, etc. > >> > > >> >Can you advise regarding proper cube design? > >> > > >> >1) The criteria for filtering (e.g. selecting sex='male') and grouping > >> >(e.g. group by state) should be dimensions - is this correct? > >> > > >> >2) Items that I would like to sum should be measures, is that right? Is > >> >there a limit to the number of measures? I want to report out up to 300 > >> >different measures aggregated by the dimensions. > >> > > >> >3) > >> >In MySQL, I am querying for different values like this > >> > > >> >select SUM((married=1) * weight) as MARRIED_1, SUM((married=2) * weight) > >> >as > >> >MARRIED_2 from data group by state; > >> > > >> >This returns the total number of weighted records for records where > >> >married > >> >is 1 and where married is 2. > >> > > >> >Question - is there a way to do this in the Kylin query? Or do I need > >> >to > >> >pre-compute my weights and create columns MARRIED_1 and MARRIED_2 in the > >> >source data, then sum it in Kylin. > >> > > >> >4) This is a tricky one. Does Kylin support MEDIAN? In MySQL, there's > >> >no > >> >MEDIAN function but we can calculate it by counting all the records, then > >> >selecting the record at an offset of half the records. I want to > >> >calculate "median" (not mean) for age and some other variables. > >> > > >> >Thanks for any tips. > >> > > >> >Best, WILL > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >-- > >> >William Glass-Husain /forio | +1 (415) 440 7500 x802 | forio.com > >> ><http://www.forio.com/> > >> > >> > > > >-- > >William Glass-Husain /forio | +1 (415) 440 7500 x802 | forio.com > ><http://www.forio.com/> > > -- William Glass-Husain /forio | +1 (415) 440 7500 x802 | forio.com <http://www.forio.com/>
