On Thu, Aug 11, 2011 at 7:40 AM, <przemol...@poczta.fm> wrote: > I am not a database developer so I don't want to change the whole process > of data flow between applications in my company. Another process is > reading this XML from particular Oracle table so I have to put the final XML > there.
I think you may be looking at a submission to http://www.thedailywtf.com/ soon. You seem to be working in a rather weird dataflow. :( Under the circumstances, you're probably going to want to go with the original ''.join() option. > This server has 256 GB of RAM so memory is not a problem. > Also the select which fetches the data is sorted. That is why I have to > carefully divide into subtasks and then merge it in correct order. There's no guarantee that all of that 256GB is available to you, of course. What may be the easiest way is to do the select in a single process, then partition it and use the Python multiprocessing module to split the job into several parts. Then you need only concatenate the handful of strings. You'll need to do some serious profiling, though, to ascertain where the bottleneck really is. Is it actually slow doing the concatenation, or is it taking more time reading/writing the disk? Is it actually all just taking time due to RAM usage? Proper string concatenation doesn't need a huge amount of CPU. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
