Is it possible that the OS is increasing the swap file size (What is you OS?) due to increasing memory demands? Try setting a minimum swap file size in the OS configuration. You should pick a value for the minimum size slightly above the size after a few runs of the program.
You might also want to carefully inspect your code to eliminate unnecessary extra data copies.
