unsubscribe

2021-05-06 Thread Simon Gauld

Re: How avoid blocking when decompressing large GZIP files.

2021-04-23 Thread Simon Gauld
it to read a portion of the > file. I don’t know, however, how this might (or might not) work with > compression. > > [1] > https://beam.apache.org/blog/splittable-do-fn-is-available/ > [2] > https://beam.apache.org/documentation/programming-guide/#splittable-dofns > > Than

How avoid blocking when decompressing large GZIP files.

2021-04-23 Thread Simon Gauld
Hello, I am trying to apply a transformation to each row in a reasonably large (1b row) gzip compressed CSV. The first operation is to assign a sequence number, in this case 1,2,3.. The second operation is the actual transformation. I would like to apply the sequence number *as* each row is rea