subject:"\\r\\n in csv output"

Re: \r\n in csv output

2020-03-23 Thread Vipul Rajan

You can use newAPIHadoopFile import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.conf.Configuration import org.apache.hadoop.mapreduce.lib.input.TextInputFormat val conf = new Configuration conf.set("textinputformat.record.delimiter", "\r\n") val df

Re: \r\n in csv output

2020-03-23 Thread Steven Parkes

Hrm ... looks like we were setting this in the past although it looks like it was being ignored ... On Mon, Mar 23, 2020 at 12:53 PM Steven Parkes wrote: > SPARK-26108 / PR#23080 > added a require on

\r\n in csv output

2020-03-23 Thread Steven Parkes

SPARK-26108 / PR#23080 added a require on CSVOptions#lineSeparator to be a single character. AFAICT, this keeps us from writing CSV files with \r\n line terminators. Wondering if this was intended or