On Thursday, April 2, 2015 at 9:06:49 AM UTC-4, Chris Angelico wrote:
> On Thu, Apr 2, 2015 at 11:46 PM, Saran A <ahlusar.ahluwa...@gmail.com> wrote:
> > @ChrisA - this is a smaller function that will take the most updated file.
> > My intention is the following:
> >
> > * Monitor a folder for files that are dropped throughout the day
> >
> > * When a file is dropped in the folder the program should scan the file
> >
> > o IF all the contents in the file have the same length (let's assume line
> > length)
> >
> > o THEN the file should be moved to a "success" folder and a text file
> > written indicating the total number of records/lines/words processed
> >
> > o IF the file is empty OR the contents are not all of the same length
> >
> > o THEN the file should be moved to a "failure" folder and a text file
> > written indicating the cause for failure (for example: Empty file or line
> > 100 was not the same length as the rest).
> >
>
> Sounds like a perfect job for inotify, then. Your function will be
> called whenever there's a new file.
>
> > Here is the code I have written:
> >
> > def initialize_logger(output_dir):
> > logger = logging.getLogger()
> > ...
> > def file_len(filename
> > with open(filename) as f:
> > for i, l in enumerate(f):
> > pass
> > return i + 1
>
> These functions are all getting defined inside your
> initialize_logger() function. I suspect you want them to be flush left
> instead.
>
> > def copyFile(src, dest):
> > try:
> > shutil.copy(src, dest)
> > # eg. src and dest are the same file
> > except shutil.Error as e:
> > print('Error: %s' % e)
> > # eg. source or destination doesn't exist
> > except IOError as e:
> > print('Error: %s' % e.strerror)
>
> Recommendation: Skip the try/except, and just let exceptions bubble
> up. Don't just print out messages and keep going.
>
> > def move_to_failure_folder_and_return_error_file():
> > os.mkdir('Failure')
> > copyFile(filename, 'Failure')
> > initialize_logger('rootdir/Failure')
> > logging.error("Either this file is empty or the lines")
>
> This doesn't move the file, it copies it. Is that your intention?
>
> Moving a file is pretty easy. Just use os.rename().
>
> > if __name__ == '__main__':
> > import sys
> > validate_files(sys.argv[1:])
>
> I've no idea what validate_files() does, as you haven't included that.
>
> I think you could code this fairly efficiently as a simple callback
> off pyinotify, or if you're not on Linux, with one of the equivalent
> services. What you're doing here (watching for files, looking inside
> them, and moving them when done) is pretty common around the world.
>
> ChrisA
@ChrisA
validate_files will:
#double check for record time and record length - logic to be written to either
pass to Failure or Success folder respectively. I welcome your thoughts on
this.
def validate_files():
creation = time.ctime(os.path.getctime(added))
lastmod = time.ctime(os.path.getmtime(added))
Does this address the issue. I particularly like writing my own exceptions as
they provide me with more information on what could be the root cause. I know
that in other circumstances, try and except are not the best practice. I
appreciate the reminder though.
Does this modification to copyFile do the job of moving the file? I haven't
written a test yet.
Thanks for catching the indentation for the helper functions.
def copyFile(src, dest):
> > try:
> > shutil.rename(src, dest)
> > # eg. src and dest are the same file
> > except shutil.Error as e:
> > print('Error: %s' % e)
> > # eg. source or destination doesn't exist
> > except IOError as e:
> > print('Error: %s' % e.strerror)
--
https://mail.python.org/mailman/listinfo/python-list
