Hi, Thanx for the clear answer.
Yes indeed, the byte position could be sufficient. I currently just need to: · have some kind of line order, so I can understand which line is the first/last · also it is preferable to have output in same order of the input From: [External] Ryan Skraba <[email protected]> Sent: Thursday, November 07, 2019 3:41 PM To: [email protected] Subject: Re: Reading file with line number Hello! Generally, for big data, line number (or similar "sequence id") is an antipattern. Knowing any line number implies knowing the previous line... and even this simple relationship breaks the ability to process elements in parallel. It *should* be possible to efficiently know the starting byte position (not row number) of an element in a file, but none of the file format IOs (TextIO, AvroIO) expose this for now. Would that be an alternative to your use case? If your files are small enough to be processed in one task, you can use https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/sdk/io/FileIO.html#readMatches--<https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_releases_javadoc_2.16.0_org_apache_beam_sdk_io_FileIO.html-23readMatches-2D-2D&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=zQK9HvoMl_nlcIbODGHLneVRKPFZYyMxJ2r0MyzkJgU&s=l7lEG2-rfaNGFgfS2Csbfc2W2HpNpkC5h8RmxQ5U-Gk&e=> and stream through the file yourself while counting rows. Best regards, Ryan On Wed, Nov 6, 2019 at 7:57 AM Gershi, Noam <[email protected]<mailto:[email protected]>> wrote: Hi, How can I read a file and having in transformation the line number information? [citi_logo_mail][citi_logo_mail]Noam Gershi Software Developer T: +972 (3) 7405718 [Mail_signature_blue]
