I agree with this approach. However, the OP specifically asked about "how to update status file".Renaming files is the wrong way to synchronize a crawler. Use a database that has ACID properties, such as SQLite. Far fewer I/O operations are required for small updates. It's not the 1980s any more.
-- http://mail.python.org/mailman/listinfo/python-list