Hi Asim, I heard a lot of Kylin users may have such wish(remove Hadoop component), currently kylin comunity have already implement RDBMS as Metadata Store(default is HBase), next plan will be add parquet/druid as storage layer(to replace HBase) and use Spark instead of MapReduce. Maybe newer version of Kylin which deployed in cloud will only depend on MYSQL(metadata), Spark/Parquet(as storage layer and compute engine), S3 as distributed storage. Currently there are branchs https://github.com/apache/kylin/tree/kylin-on-parquet & https://github.com/apache/kylin/tree/kylin-on-druid, but these feature have some limitation and maybe enough mature. We may wait for further development work. ----------------- ----------------- Best wishes to you ! From ：Xiaoxiang Yu
At 2019-07-25 11:29:22, "Asim Ali" <[email protected]> wrote: >Hi All, >I tried using Kylin on Hadoop environment, but overhead of hadoop is too >much >for our medium scale need. >Is there any way we can use kylin Olap engine with minimal requirements of >underlaying storage layer. >What are the best practices and architecture to support this, where we >possibly can use it without hadoop components. >Thanks > >Asim Ali >*Software Developer* > >Email: [email protected] <[email protected]> > >Phone: 1300 855 642 <1300855642> >Website: www.easyemployer.com > >[image: easyemployer] <http://www.easyemployer.com/> > > >On Fri, 21 Jun 2019 at 15:22, 敏丞 <[email protected]> wrote: > >> Hi, >> After check cube.json provided by you, I can reproduce this error in >> my development env. Looks like this kind exception occurs when you have a >> cube which *have both Raw measure and CountDistinct(Bitmap) on the same >> column*. I find the reason should be the Raw Measure choose the wrong >> dictionary (AppendTrieDictionary cannot used to decode). Maybe you should >> try use two cubes in this situation. >> And if you don't mind, I have a question, have you ever use this type >> of query "select * from FACT_TABLE" in old version of kylin in such kind of >> cube(raw measure and count_distinct both on the same column) and get >> correct result successfully? >> >> >> If you have find anything wrong or other information, please let me >> know. Thank you. >> >> >> >> >> *-----------------* >> *-----------------* >> *Best wishes to you ! * >> *From ：**Xiaoxiang Yu* >> >> >> At 2019-06-20 16:00:10, "[email protected]" >> <[email protected]> wrote: >> >hi,all: >> > >> > I got an error like "AppendTrieDictionary can't retrieve value from id" >> > while query the cube data, the following is the cube info and exception >> > info. >> > >> >I found the same error in this >> >thread(https://lists.apache.org/thread.html/63981bc08ef7b97c41921ed692de79ef9a744f6329192e5199074ba3@%3Cdev.kylin.apache.org%3E), >> > but I just use the bitmaps (count distinct) as measure, and never use it >> >as dimension. >> > >> >So please help me to resolve it. >> > >> >cube data: >> >{ >> > "uuid": "eb0b4a32-fbc0-b197-b3f0-4c9cd5fb3916", >> > "last_modified": 1561014544528, >> > "version": "2.6.2.0", >> > "name": "dev_cube_user_currency", >> > "is_draft": false, >> > "model_name": "user_currency", >> > "description": "", >> > "null_string": null, >> > "dimensions": [ >> > { >> > "name": "TYPE", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "TYPE", >> > "derived": null >> > }, >> > { >> > "name": "SUB_TYPE", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "SUB_TYPE", >> > "derived": null >> > }, >> > { >> > "name": "SOURCE_TYPE", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "SOURCE_TYPE", >> > "derived": null >> > }, >> > { >> > "name": "SOURCE", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "SOURCE", >> > "derived": null >> > }, >> > { >> > "name": "STATUS", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "STATUS", >> > "derived": null >> > }, >> > { >> > "name": "CREATE_TIME", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "CREATE_TIME", >> > "derived": null >> > }, >> > { >> > "name": "UPDATE_TIME", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "UPDATE_TIME", >> > "derived": null >> > }, >> > { >> > "name": "END_TIME", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "END_TIME", >> > "derived": null >> > }, >> > { >> > "name": "ACTIVATION_TIME", >> > "table": "DEV_DWD_USER_CURRENCY", >> > "column": "ACTIVATION_TIME", >> > "derived": null >> > }, >> > { >> > "name": "CODE", >> > "table": "DEV_DIM_CURRENCY_TYPE", >> > "column": null, >> > "derived": [ >> > "CODE" >> > ] >> > }, >> > { >> > "name": "SHORT_NAME", >> > "table": "DEV_DIM_CURRENCY_TYPE", >> > "column": null, >> > "derived": [ >> > "SHORT_NAME" >> > ] >> > } >> > ], >> > "measures": [ >> > { >> > "name": "_COUNT_", >> > "function": { >> > "expression": "COUNT", >> > "parameter": { >> > "type": "constant", >> > "value": "1" >> > }, >> > "returntype": "bigint" >> > } >> > }, >> > { >> > "name": "SUM_TOTAL", >> > "function": { >> > "expression": "SUM", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.TOTAL" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "SUM_BALANCE", >> > "function": { >> > "expression": "SUM", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.BALANCE" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "SUM_VALID_PERIOD", >> > "function": { >> > "expression": "SUM", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" >> > }, >> > "returntype": "bigint" >> > } >> > }, >> > { >> > "name": "MAX_TOTAL", >> > "function": { >> > "expression": "MAX", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.TOTAL" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "MAX_BALANCE", >> > "function": { >> > "expression": "MAX", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.BALANCE" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "MAX_VALID_PERIOD", >> > "function": { >> > "expression": "MAX", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" >> > }, >> > "returntype": "integer" >> > } >> > }, >> > { >> > "name": "MIN_TOTAL", >> > "function": { >> > "expression": "MIN", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.TOTAL" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "MIN_BALANCE", >> > "function": { >> > "expression": "MIN", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.BALANCE" >> > }, >> > "returntype": "double" >> > } >> > }, >> > { >> > "name": "MIN_VALID_PERIOD", >> > "function": { >> > "expression": "MIN", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" >> > }, >> > "returntype": "integer" >> > } >> > }, >> > { >> > "name": "RAW_PARENT_ID", >> > "function": { >> > "expression": "RAW", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.PARENT_ID" >> > }, >> > "returntype": "raw" >> > } >> > }, >> > { >> > "name": "RAW_ORDER_NO", >> > "function": { >> > "expression": "RAW", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.ORDER_NO" >> > }, >> > "returntype": "raw" >> > } >> > }, >> > { >> > "name": "COUNT_PARENTID", >> > "function": { >> > "expression": "COUNT_DISTINCT", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.PARENT_ID" >> > }, >> > "returntype": "bitmap" >> > } >> > }, >> > { >> > "name": "COUNT_ORDERNO", >> > "function": { >> > "expression": "COUNT_DISTINCT", >> > "parameter": { >> > "type": "column", >> > "value": "DEV_DWD_USER_CURRENCY.ORDER_NO" >> > }, >> > "returntype": "bitmap" >> > } >> > } >> > ], >> > "dictionaries": [ >> > { >> > "column": "DEV_DWD_USER_CURRENCY.PARENT_ID", >> > "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder" >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.ORDER_NO", >> > "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder" >> > } >> > ], >> > "rowkey": { >> > "rowkey_columns": [ >> > { >> > "column": "DEV_DWD_USER_CURRENCY.TYPE", >> > "encoding": "integer:2", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.SUB_TYPE", >> > "encoding": "integer:2", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.SOURCE_TYPE", >> > "encoding": "integer:2", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.STATUS", >> > "encoding": "integer:2", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.CREATE_TIME", >> > "encoding": "time", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.UPDATE_TIME", >> > "encoding": "time", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.END_TIME", >> > "encoding": "time", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.ACTIVATION_TIME", >> > "encoding": "time", >> > "encoding_version": 1, >> > "isShardBy": false >> > }, >> > { >> > "column": "DEV_DWD_USER_CURRENCY.SOURCE", >> > "encoding": "fixed_length:10", >> > "encoding_version": 1, >> > "isShardBy": false >> > } >> > ] >> > }, >> > "hbase_mapping": { >> > "column_family": [ >> > { >> > "name": "F1", >> > "columns": [ >> > { >> > "qualifier": "M", >> > "measure_refs": [ >> > "_COUNT_", >> > "SUM_TOTAL", >> > "SUM_BALANCE", >> > "SUM_VALID_PERIOD", >> > "MAX_TOTAL", >> > "MAX_BALANCE", >> > "MAX_VALID_PERIOD", >> > "MIN_TOTAL", >> > "MIN_BALANCE", >> > "MIN_VALID_PERIOD", >> > "RAW_PARENT_ID", >> > "RAW_ORDER_NO" >> > ] >> > } >> > ] >> > }, >> > { >> > "name": "F2", >> > "columns": [ >> > { >> > "qualifier": "M", >> > "measure_refs": [ >> > "COUNT_PARENTID", >> > "COUNT_ORDERNO" >> > ] >> > } >> > ] >> > } >> > ] >> > }, >> > "aggregation_groups": [ >> > { >> > "includes": [ >> > "DEV_DWD_USER_CURRENCY.TYPE", >> > "DEV_DWD_USER_CURRENCY.SUB_TYPE", >> > "DEV_DWD_USER_CURRENCY.SOURCE_TYPE", >> > "DEV_DWD_USER_CURRENCY.SOURCE", >> > "DEV_DWD_USER_CURRENCY.STATUS", >> > "DEV_DWD_USER_CURRENCY.CREATE_TIME", >> > "DEV_DWD_USER_CURRENCY.UPDATE_TIME", >> > "DEV_DWD_USER_CURRENCY.END_TIME", >> > "DEV_DWD_USER_CURRENCY.ACTIVATION_TIME" >> > ], >> > "select_rule": { >> > "hierarchy_dims": [], >> > "mandatory_dims": [], >> > "joint_dims": [] >> > } >> > } >> > ], >> > "signature": "eiGIwgaD57yCz/jfbRCscg==", >> > "notify_list": [], >> > "status_need_notify": [ >> > "ERROR", >> > "DISCARDED", >> > "SUCCEED" >> > ], >> > "partition_date_start": 0, >> > "partition_date_end": 3153600000000, >> > "auto_merge_time_ranges": [], >> > "volatile_range": 0, >> > "retention_range": 0, >> > "engine_type": 2, >> > "storage_type": 2, >> > "override_kylin_properties": {}, >> > "cuboid_black_list": [], >> > "parent_forward": 3, >> > "mandatory_dimension_set_list": [], >> > "snapshot_table_desc_list": [] >> >} >> > >> >=====exception info==================== >> >org.apache.kylin.rest.exception.InternalErrorException: >> >AppendTrieDictionary can't retrieve value from id >> >while executing SQL: "select * from mmears_dev.DEV_DWD_USER_CURRENCY limit >> >10" >> > at >> > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:441) >> > at >> > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:367) >> > at >> > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93) >> > at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source) >> > at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >> > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) >> > at >> > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) >> > at >> > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) >> > at >> > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) >> > at >> > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) >> > at >> > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) >> > at >> > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) >> > at >> > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) >> > at >> > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) >> > at >> > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) >> > at >> > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >> > at >> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) >> > at >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) >> > at >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158) >> > at >> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) >> > at >> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) >> > at >> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) >> > at >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) >> > at >> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) >> > at >> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) >> > at >> > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) >> > at >> > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >> > at >> > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209) >> > at >> > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) >> > at >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >> > at >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) >> > at >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) >> > at >> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) >> > at >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) >> > at >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) >> > at >> > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) >> > at >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) >> > at >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) >> > at >> > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) >> > at >> > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) >> > at >> > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) >> > at >> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) >> > at >> > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) >> > at >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at >> > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >> > at java.lang.Thread.run(Thread.java:748) >> > >> >>
