< http://marcio.io/2015/07/cheap-mapreduce-in-go/>
tl;dr Sometimes you don’t need overly complex infrastructures or systems to do a job well. In this case, we were running these exact same aggregations over close to 20 EMR instances that would take a few minutes to execute the entire MapReduce job over hundreds of Gigabytes of data each day. When we decided to take a look at this problem again, we rewrote this task using Go, and we now simply run this on a single 8-core machine and the whole daily execution takes about 10 minutes. We cut a lot of the costs associated with maintaining and running these EMR systems and we just schedule this Go app to run once a day over our daily dataset. You can find the entire code here: https://gist.github.com/mcastilho/e051898d129b44e2f502 © -- |:**THE BEER-WARE LICENSE** (Revision 42): | <ca...@golang.it> wrote this mail. As long as you retain | this notice you can do whatever you want with this stuff. | If we meet some day, and you think this stuff is worth it, | you can buy me a beer in return. | --Carlo Miron :
_______________________________________________ Python mailing list Python@lists.python.it http://lists.python.it/mailman/listinfo/python
