Hi Madan, If your source has a parallelism > 1, then when the CSV file is split, only one of the operators will get the split with the header row.
So in that case, how would you communicate the column name->index information to the other operators? If you force a parallelism of 1 for the source, then I’m pretty sure you’re guaranteed that the file will be processed in order. — Ken > On Oct 31, 2018, at 12:50 AM, madan <madan.yella...@gmail.com> wrote: > > Hi, > > When we are splitting a csv file into multiple parts we are not sure which > part is read first. Is there any way to make sure first part with header is > read first ? I need to read header line first to store column name vs index > and use this index for processing next records. > > I could read header line from the file before submitting job to the flink, > but that way we are opening the file 2 times. Is there any better way to do > this? Please suggest. > > -- > Thank you. -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra
