Hi, I recently confronted the problem of building data stream pipeline in golang to build some etl programs.
I found it was not so simple to use the golang idioms. Using them i noticed i needed deep care and understanding to produce correct code. I also felt they was uselessly verbose and repetitive. I have searched for existing libraries produced by the community but I failed to find a suitable one for my need. Something like ratchet (https://github.com/dailyburn/ratchet) was close to it, but its unlikely i use such complex api. I came to write my own for this purpose, you can find it here https://github.com/clementauger/st To achieve it i have been extensively using the reflection API, which is really helpful. Unfortunately, I have been using interface{} almost everywhere, and that would be difficult to change. In exchange this api provides lots of flexibility. For comparison and introduction purposes i have rewrote the pipeline walk file example available in the blog at https://blog.golang.org/pipelines The version i provide is available at https://github.com/clementauger/st/blob/master/examples/walkfiles/main.go This version implements a slightly more complex flow mecanism as it bufferizes path, however, it is much easier, shorter (<150 LOC), and i believe clearer code than the original version. Where the last property is subject to personal opinion, yet, I encourage you to consider it. Overall, while this implementation is slower because there are tons of additionnal indirections, I think it is still interesting to consider to compensate that with easier and appropriate bufferization and paralellisation whenever possible to achieve better performance. I intended that this library helps to make that happens as easily and smoothly as possible. For your information you can also find a bulk version of the original bounded version provided in the blog at https://github.com/clementauger/x/blob/master/01_pipeline/bounded_bulk.go. This last program is more suitable for full comparison with the version i provide. I hope to engage in interesting conversations around that, thanks for reading. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
