I heard a lot of good about HDF5 file format to hande important volume of data hierachical (mean you can query what ever data you need without load the full data set into a json for instance) : http://en.wikipedia.org/wiki/Hierarchical_Data_Format
It very much faster then postgres (sure postgres is not the faster backend but it scale gracefully)... The intend of this file format is to be used in conjonction with a DB. If I remember Pandas can write HDF5, not sure which lib it uses, there is two major lib in python which have different set of feature, one is more fancy but not support all the HDF5 feature and the other is supporting "all" the feature but is less sexy... Richard On Tue, Oct 22, 2013 at 6:54 AM, Cliff Kachinske <cjk...@gmail.com> wrote: > use the rows field in auth_permission as described here. > > http://web2py.com/books/default/chapter/29/09/access-control#Authorization > > > On Tuesday, October 22, 2013 5:40:49 AM UTC-4, Thomas S wrote: >> >> Ok, I am making progress. I guess all those JavaScript tools are not >> great when it comes to plotting millions of points but I am happy to >> downsample on the server side and send less points >> >> - I am using flot instead of highcharts >> >> - Currently, the user is uploading a csv file. I don't do any parsing at >> this stage. However, I rather keep the file (under uploads) and parse it on >> request. Having said that this will become a lot more slick soon. This is >> my first application. >> >> One thing that puzzles me for now... >> >> A user has to login to upload a file (that's good), but he can then also >> modify or delete entries in the SQL database created by others. How can I >> make sure he/she only deletes rows he/she has created in the first place. >> All users should be able to see all files though. >> >> Here's a link: >> https://tschmelz.**pythonanywhere.com/csv<https://tschmelz.pythonanywhere.com/csv> >> >> I will soon post it to my Github (username tschm) >> thomas >> >> >> On Sunday, 20 October 2013 15:38:07 UTC+2, Niphlod wrote: >>> >>> first things first: are you sure that highcharts can handle 10*100k >>> points to draw a graph ? >>> As for the storage, you can do anything you like: if the data doesn't >>> change that much, storing into the database will be a long process only on >>> the first time. >>> On the other end, if you need to fetch 100k records and transform them >>> to json, it's going to take some time. >>> Not sure on how much you'll gain from parsing i.e. a csv file instead of >>> a querying a db.... >>> if the returning json object is , let's say, 10 mb, it's always gonna >>> feel heavy. >>> >>> On Sunday, October 20, 2013 9:11:07 AM UTC+2, Thomas S wrote: >>>> >>>> >>>> Hi, >>>> >>>> I have created a standard application relying on Pandas and PyQt4 to >>>> browse through a Pandas Dataframe. >>>> A dataframe is essentially a dictionary of time series data. >>>> >>>> I am new to web2py but I have experience with Pandas and matplotlib. >>>> I am also tempted to embed www.highcharts.com into my application. >>>> >>>> Before I dig into web2py I would like to know which route is probably >>>> most promising. >>>> >>>> Should I parse the dataframe on the webserver and write it into a SQL >>>> database? I guess that could be slow? >>>> Such a dataframe may consist of a dictionary with 100 elements each >>>> several 100k points. >>>> >>>> Should I parse a time series onto request into a json format and export >>>> to javaScript? >>>> In this case how could I provide a way to generate a menu from the keys >>>> in the dictionary. >>>> E.g. user clicks on a key, python does all the computations for some >>>> stats. >>>> >>>> The plan is to upload the data using csv files. >>>> >>>> So, I am a bit lost by the wide range of possibilities in web2py. I >>>> would be delighted if you would like to get involved in this open source >>>> project. >>>> The main goal for now is to learn web2py :-) >>>> >>>> Please find the Github of the original application here >>>> https://github.com/tschm/**PandasMonitor<https://github.com/tschm/PandasMonitor> >>>> >>>> Sorry for being so unprecise in my questions but it just reflects that >>>> I don't have a very precise plan at this stage. >>>> >>>> Kind regards >>>> Thomas >>>> >>> -- > Resources: > - http://web2py.com > - http://web2py.com/book (Documentation) > - http://github.com/web2py/web2py (Source code) > - https://code.google.com/p/web2py/issues/list (Report Issues) > --- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to web2py+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
