Ok, I am making progress. I guess all those JavaScript tools are not great when it comes to plotting millions of points but I am happy to downsample on the server side and send less points
- I am using flot instead of highcharts - Currently, the user is uploading a csv file. I don't do any parsing at this stage. However, I rather keep the file (under uploads) and parse it on request. Having said that this will become a lot more slick soon. This is my first application. One thing that puzzles me for now... A user has to login to upload a file (that's good), but he can then also modify or delete entries in the SQL database created by others. How can I make sure he/she only deletes rows he/she has created in the first place. All users should be able to see all files though. Here's a link: https://tschmelz.pythonanywhere.com/csv I will soon post it to my Github (username tschm) thomas On Sunday, 20 October 2013 15:38:07 UTC+2, Niphlod wrote: > > first things first: are you sure that highcharts can handle 10*100k points > to draw a graph ? > As for the storage, you can do anything you like: if the data doesn't > change that much, storing into the database will be a long process only on > the first time. > On the other end, if you need to fetch 100k records and transform them to > json, it's going to take some time. > Not sure on how much you'll gain from parsing i.e. a csv file instead of a > querying a db.... > if the returning json object is , let's say, 10 mb, it's always gonna feel > heavy. > > On Sunday, October 20, 2013 9:11:07 AM UTC+2, Thomas S wrote: >> >> >> Hi, >> >> I have created a standard application relying on Pandas and PyQt4 to >> browse through a Pandas Dataframe. >> A dataframe is essentially a dictionary of time series data. >> >> I am new to web2py but I have experience with Pandas and matplotlib. >> I am also tempted to embed www.highcharts.com into my application. >> >> Before I dig into web2py I would like to know which route is probably >> most promising. >> >> Should I parse the dataframe on the webserver and write it into a SQL >> database? I guess that could be slow? >> Such a dataframe may consist of a dictionary with 100 elements each >> several 100k points. >> >> Should I parse a time series onto request into a json format and export >> to javaScript? >> In this case how could I provide a way to generate a menu from the keys >> in the dictionary. >> E.g. user clicks on a key, python does all the computations for some >> stats. >> >> The plan is to upload the data using csv files. >> >> So, I am a bit lost by the wide range of possibilities in web2py. I would >> be delighted if you would like to get involved in this open source project. >> The main goal for now is to learn web2py :-) >> >> Please find the Github of the original application here >> https://github.com/tschm/PandasMonitor >> >> Sorry for being so unprecise in my questions but it just reflects that I >> don't have a very precise plan at this stage. >> >> Kind regards >> Thomas >> > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
