Hi Jim, > How can I do this?I can get the page data of current moment as a > string using urllib.urlopen and store it in a variable, say > current_page_data.But ,how do I store yesterday's data so that I can > compare the two?Do I need to store it as a field in database?(I am not > very familiar with database programming..I am using postgres for web > app development with django.So I don't know if storing such a large > string like that is possible).
Yes, it's definitely possible. DBs support very large text fields nowadays, I wouldn't worry about it. Postgres's text field type is only limited by the 1GB value-size limit general to all Postgres DB fields. > I considered writing the data to a file..but suppose a lot of users > want to do the comparison of 2 versions of many pages..?I would be > creating so many files =usercount * number_of_url_by_each_user .If > the time duration between 2 comparisons was some seconds/minutes then > I can just use 2 local variables andmay not need persistence.But ,to > compare data of yesterday and today I may need some such mechanism.. You will definitely want persistence, but if you do choose to go the files route I wouldn't worry about the storage space. If you have enough users for that to be a problem you should be able to afford the space, cloud-based storage like Amazon's S3 is cheap and easy. If you're really resource constrained you can always just compress\decompress the files, text compresses very well. It'll add some processing overhead for sure, but there's no magic bullet that's not going to cost you some form of resource or another. Just limit the number of versions that can be kept and remove the old-ones on a rolling basis... Good luck, Jonathan > I guess this may be a common problem and there may some solution for > this. > Can someone comment/suggest how this can be done? > thanks > jim > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-us...@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-us...@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
