Re: Re: Does Table API connector, csv, has some option to ignore some columns

[podunk]

This is really surprising.

When you import data from a file, you really rarely need to import everything from that file. Most often it is several columns.

So the program that reads the file should be able to do this - this is the ABC of working with data.

 

Often the suggestion is "you can write your script". Sure. I can. I can write the entire program here - from scratch.

But I use a ready-made program to avoid writing my scripts.

 

 

Sent: Tuesday, July 12, 2022 at 12:24 AM
From: "Alexander Fedulov" <alexan...@ververica.com>
To: pod...@gmx.com
Cc: "user" <user@flink.apache.org>
Subject: Re: Re: Does Table API connector, csv, has some option to ignore some columns

Hi podunk,

 

no, this is currently not possible: 
> Currently, the CSV schema is derived from table schema. [1]

 

So the Table schema is used to define how Jackson CSV parses the lines and hence needs to be complete.

 

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/csv/

 

Best,

Alexander Fedulov

 

On Mon, Jul 11, 2022 at 5:43 PM <pod...@gmx.com> wrote:

No, I did not mean.

I said 'Does Table API connector, CSV, has some option to ignore some columns in source file?'

 

 

Sent: Monday, July 11, 2022 at 5:28 PM
From: "Xuyang" <xyzhong...@163.com>
To: pod...@gmx.com
Cc: user@flink.apache.org
Subject: Re:Re: Does Table API connector, csv, has some option to ignore some columns

Hi, did you mean `insert into table1 select col1, col2, col3 ... from table2`?

 

If this doesn't meet your requirement, what about using UDF to custom what you want in runtime.

 

--

    Best！

    Xuyang

 

 

 

在 2022-07-11 16:10:00，pod...@gmx.com 写道：

I want to control what I insert in table not what I get from table.

 

 

Sent: Monday, July 11, 2022 at 3:37 AM
From: "Shengkai Fang" <fskm...@gmail.com>
To: pod...@gmx.com
Cc: "user" <user@flink.apache.org>
Subject: Re: Does Table API connector, csv, has some option to ignore some columns

Hi. 

 

In Flink SQL, you can select the column that you wants in the query. For example, you can use 

 

```

SELECT col_a, col_b FROM some_table;

```

 

Best,

Shengkai

 

 

<pod...@gmx.com> 于2022年7月9日周六 01:48写道：

Does Table API connector, CSV, has some option to ignore some columns in source file?

For instance read only first, second, nine... but not the others?

 

Or any other trick?

CREATE TABLE some_table (
  some_id BIGINT,
  ...
) WITH (
 'format' = 'csv',
 ...
)