From: "Fred Cisin" <ci...@xenosoft.com> > They literally refused to understand that a > dataset could exist that would be too large to fit into memory.
In the UK, the Home Secretary wants to force all ISPs to store and keep (reasonably) easily searchable logs of all URLs accessed by all their customers. This, I think, is an example of "big data" … possibly so big that backup is impracticable. When I did daily analysis of the urls accessed by the 2000 or so students at a University*, that daily database was pushing what I could get into memory on my desktop system (well specified) … now imagine 20 million users for 365 days - I'd be doubtful that even todays mainframes and supercomputers could do that. * When I questioned the legality (and morality) of doing this the answer was "we believe it is legal" (and "stop complaining and keep digging"). Andy
