Hey all,
I was wondering if anyone could give me some pointers on a good approach for temporally clustering a data set I have. The data set consists of web page crawl data - for the sake of this we'll just focus on two important pieces - a tuple containing a uri, the uri's domain, and unix timestamp. The nature of the crawler dictates that I visit a set of pages on each domain at some regular interval, so I want to group all those visits together temporally into a single "domain crawled" output line. The naive approach has me converting the unix timestamp to an ISO date, truncating to the nearest day, and then grouping. But the problem with this is that a single domain might be crawled across a day boundary (of course), so ideally there would be some sort of "rolling window" where these results should be included as well instead of resulting in two crawls when it really was just a single crawl that crossed the day boundary. Any ideas how I might approach this? Thanks for any ideas! Cheers, -Zach
