Hi, Being able to aggregate on the query side is a great idea and would allow us to transfer less data as well as having a full query representation of the visualization.
However creating a SQL query dynamically is a pretty difficult task, and might be too much for that scope. Also I see some possible problems with this method: - Changing the pivot or simple filtering would mean running the query again - Being able to make pivot style SQL query would be really hard, we would need multiple sub-queries or even some times multiple queries (I tried a few times and could have the result wanted only with visualization side pivot). It would end up with really bad SQL queries, especially with the Hive SQL or Spark SQL limitations and would take way more time to process. On Sun, Mar 22, 2015 at 10:08 PM, IT CTO <[email protected]> wrote: > Hi, > > The Chart library features sounds promising. > As for the data engine - one thing that I think is missing is the ability > to use the visualization to drive the aggregation in the SQL. today, you > first write the SQL, you execute it, *limited by the number of results sent > to the client*, and then you use viz to understand the results. > Alternatively, if through the visualization I can generate a better SQL > which returns returns an aggregated data-set then I can analyze a bigger > amount of data. > > I hope I was clear enough in my explanation :-) > > Eran > > > On Fri, Mar 20, 2015 at 8:21 AM, madhuka udantha <[email protected] > > > wrote: > > > Hi, > > > > Here is my proposing ideas. > > According to COMDEV-119 jira. Charts are hard coded until now and data > > transformation issue was highlighted since different charts have > different > > pivot fields eg: Area charts, Scatter, Surface charts, Bubble charts, > Radar > > charts. etc.. > > > > To solve this I am introducing a two major component one is called 'Chart > > library' and 'Data transformation engine'. Chart library is located where > > it shows the chats that are currently plugged. There we can plug chart > > types and those can be reused. > > > > *Chart library features * > > > > - Users can select the chart from library > > - Those charts are pluggable to library > > - Charts can be plugged by config(json)/UI with wizard > > - Configuration/Meta file of the chart contains interface, libs, > themes > > and a data transformation types/mappings > > > > > > > > *Data Transformation Engine* > > 'Data transformation engine' contains data transformation modules. Those > > modules are also pluggable to engine. Those have connections to charts. > > Data transformation engine sit between the data (sql) and chart. So this > > module converts data and map them to each chart pivot field > > > > - This module will look at pivot fields of the chart > > - Selected attributes of the SQL query > > - Attribute value operations improvement (string split, value > > aggregation, round number round) > > > > > > Another improvement that I notice is that > > > > - Query Edit auto-completion support (with Ctrl+space) > > > > > > Your ideas are welcome here > > Thanks > > > > On Fri, Mar 20, 2015 at 10:57 AM, madhuka udantha < > > [email protected]> > > wrote: > > > > > Hi All, > > > > > > I'm Udantha, MSc. Student at University of Moratuwa. This GSoC 2015 > > > project, 0COMDEV-1190 captures my interest. > > > > > > I have abundant experiences of visualization techniques creating > numerous > > > dashboards[1,2] with javascript, html5, angularJS, d3 charting etc. > > > > > > My current research area comprises of big data where I have worked with > > > various types of data sets. Also I'm working with cluster > representation > > > and classification techniques where visualization amounts to a > > considerable > > > part. I was following COMDEV-119 (jira) with Alexander Bezzubov and > > CORNEAU > > > Damien for more than week. > > > > > > Thanks > > > > > > [1] http://wso2.com/products/user-engagement-server/ > > > [2] https://github.com/wso2/jaggery > > > -- > > > Cheers, > > > Madhuka Udantha > > > http://madhukaudantha.blogspot.com > > > > > > > > > > > -- > > Cheers, > > Madhuka Udantha > > http://madhukaudantha.blogspot.com > > > > > > -- > Eran | CTO >
