Technically, we don't need to nor are we required to skip *exactly* one minute. The real question is not 'has 60 seconds passed', but 'have we checked cron in this minute'. I would thus check not for a 60 sec offset, but rather if the minute of cron start matches.
Also in line 51: s.enter(60 - now % 60, 1, self.launch, ()) it would probably be better to use int(now) to avoid the sleep/time functions rounding/timing errors potentially result in a 59.99... On Apr 13, 3:18 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > I do not like very much the idea of setting ctime/mtime but I do not > like the current mechanism either. It is not just the pickle overhead. > There seem to be precision issues that make it difficult to skip > exactly one minute. For example right now newcron contains: > > if startup or self.now - start > 59.99 > > 59.99 should be 60 but that causes cron to skip some calls. 59.99 is > OK most of the cases but it may still cause some false positives. > > I do not know have a simple solution to this problem. It seems to me > we need to store more info, not less, even start/stop times may not be > sufficient information. We may also need to id each schedules call to > cron and detect not only whether a cron task is running but which of > past scheduled tasks are running. > > Bottom line: the part of cron/newcron that runs every 1 minute need to > be redesigned completely in my view. > > Massimo > > On Apr 13, 6:23 am, AchipA <attila.cs...@gmail.com> wrote: > > > Hey, I don't have a problem with that, just saying the lock-and-read > > mechanism can increase the overhead/latency significantly in certain > > setups. If a two-file approach is a problem, would you consider > > avoiding the cPickle read if I can find a way to do it via setting > > ctime/mtime (we set a mtime older then ctime, and when we finish, we > > set now() as mtime) ? > > > Alternatively, if it turns out that can't be done on all platforms and > > we only need to check when it started and whether it is running, we > > could simplify running checks by file size. When the cron starts, we > > create the file with zero size, and when it finishes we just write a > > space to it. That way we can check whether it's running or not just by > > looking at the file size (a lot cheaper that reading/pickling). > > > The bottom line is that I would like to avoid having to do locking for > > the read-checks (we write only once a minute, but read quite a bit > > more as we scale upwards - like on a shared volume or on a busy site > > with softcron). > > > On Apr 12, 11:27 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > On Apr 12, 1:19 pm, AchipA <attila.cs...@gmail.com> wrote: > > > > > Why do we need the time range ? If the tasks are overlapping it's > > > > their responsibility to handle that (I know this is arguable, but > > > > that's how 'standard' cron works). > > > > This is also as it works in newcron. The problem is that if for any > > > reason, the main process that loops and spans the tasks gets stuck, it > > > may give rise to a proliferation of processes that may crash the os. > > > The current mechanism is similar to the one you originally implemented > > > but you used a n additional file to determine if the cron was > > > completed. I use the completion date. > > > > > Also, we can easily store two > > > > timestamps (slightly hackish, but mtime and ctime can be set > > > > separately), would have to check whether that is supported on all > > > > platforms. Of course there are many other ways of reading data without > > > > opening files, I'm just pondering about alternatives as the current > > > > locking mechanism causes some problems on my shared-volume based multi- > > > > server setup (that's why I used 'move' originally as it's atomic and > > > > works well with netwok shares). > > > > > On Apr 12, 5:12 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > Because they os timestamp only can only tell you when a task has > > > > > started (or stopped, depending on when it was created) it does not > > > > > contain enough information to give you a time range (time and stop). > > > > > Cron needs to know when the previous crondance started and whether is > > > > > was completed or not. The original implementation was doing the check > > > > > using locks and that resulted in a large number of try... except... > > > > > The current implementation removes most of the try.. except... (people > > > > > complained about that) and just stores start_time, stop_time > > > > > explicitly in a picke. > > > > > > On Apr 12, 8:00 am, AchipA <attila.cs...@gmail.com> wrote: > > > > > > > To correct myself, it seems the cron in web2py no longer uses the > > > > > > filesystem timestamps, but cPickles timestamps from/to the lock > > > > > > file. > > > > > > I'm not sure why Massimo changed it, but this *is* a bigger overhead > > > > > > than it was previously (as it needs to do file locking and > > > > > > cPickle.load() on every single request - as opposed to a simple > > > > > > cached > > > > > > non-locking filesystem call). > > > > > > > On Apr 1, 8:20 pm, AchipA <attila.cs...@gmail.com> wrote: > > > > > > > > Exactly, hardcron checks once a minute, softcron checks on each > > > > > > > page > > > > > > > load. The 'check' is calling a function or two and comparing a > > > > > > > file's > > > > > > > timestamp, so not *that* much more expensive. > > > > > > > > On Apr 1, 7:51 pm, Jonathan Lundell <jlund...@pobox.com> wrote: > > > > > > > > > On Apr 1, 2010, at 10:37 AM, AchipA wrote: > > > > > > > > > > There is some overhead, but efficiency is a disputable term - > > > > > > > > > there is > > > > > > > > > certainly more overhead than hardcron, but IMO not in a way > > > > > > > > > that would > > > > > > > > > affect overall performance unless you're running it on a site > > > > > > > > > that has > > > > > > > > > hundreds of thousands of hits per day... > > > > > > > > > Perhaps we could change (or eliminate) the wording. How about > > > > > > > > simply 'Using softcron'? > > > > > > > > > I'm curious: what is the extra overhead of soft vs hardcron? > > > > > > > > Just that it does a test on each page access? I'm guessing > > > > > > > > that's pretty cheap. > > > > > > > > > > On Apr 1, 5:40 pm, Jonathan Lundell <jlund...@pobox.com> > > > > > > > > > wrote: > > > > > > > > >> Section 4.17 (cron) mentions hard vs softcrondefaults, but > > > > > > > > >> doesn't say how to override them. > > > > > > > > > >> Section 4.1 (cli) doesn't list --softcron > > > > > > > > > >> The startup message for softcronsays: 'Using softcron (but > > > > > > > > >> this is not very efficient)' > > > > > > > > > >> In what sense "not efficient"? I understand that the timing > > > > > > > > >> is less consistent, but is there really more overhead? > > > > > > > > >> softcron seems like a pretty reasonable choice if all you're > > > > > > > > >> doing it deleting expired sessions. > > -- To unsubscribe, reply using "remove me" as the subject.
