Hi, All.
Could you give me some opinion?
There is an old SPARK issue, SPARK-11374, about removing header lines from text
file.
Currently, Spark supports removing CSV header lines by the following way.
```
scala> spark.read.option("header","true").csv("/data").show
+---+---+
| c1| c2|
+---+---+
| 1| a|
| 2| b|
+---+---+
```
In SQL world, we can support that like the Hive way, `skip.header.line.count`.
```
scala> sql("CREATE TABLE t1 (id INT, value VARCHAR(10)) ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/data'
TBLPROPERTIES('skip.header.line.count'='1')")
scala> sql("SELECT * FROM t1").show
+---+-----+
| id|value|
+---+-----+
| 1| a|
| 2| b|
+---+-----+
```
Although I made a PR for this based on the JIRA issue, I want to know this is
really needed feature.
Is it need for your use cases? Or, it's enough for you to remove them in a
preprocessing stage.
If this is too old and not proper in these days, I'll close the PR and JIRA
issue as WON'T FIX.
Thank you for all in advance!
Bests,
Dongjoon.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
