ruanwenjun commented on code in PR #5873:
URL: https://github.com/apache/seatunnel/pull/5873#discussion_r1408703608
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/TextReadStrategy.java:
##########
@@ -94,32 +96,21 @@ public void read(String path, String tableId,
Collector<SeaTunnelRow> output)
.forEach(
line -> {
try {
- SeaTunnelRow seaTunnelRow =
-
deserializationSchema.deserialize(line.getBytes());
- if (!readColumns.isEmpty()) {
- // need column projection
- Object[] fields;
- if (isMergePartition) {
- fields =
- new Object
- [readColumns.size()
- +
partitionsMap.size()];
- } else {
- fields = new
Object[readColumns.size()];
- }
- for (int i = 0; i < indexes.length;
i++) {
- fields[i] =
seaTunnelRow.getField(indexes[i]);
- }
- seaTunnelRow = new
SeaTunnelRow(fields);
- }
- if (isMergePartition) {
- int index =
seaTunnelRowType.getTotalFields();
- for (String value :
partitionsMap.values()) {
- seaTunnelRow.setField(index++,
value);
+ if (StringUtils.isBlank(rowDelimiter)) {
+ SeaTunnelRow seaTunnelRow =
+ deserializeRow(line,
partitionsMap, tableId);
+ output.collect(seaTunnelRow);
+ } else {
+ String[] rows =
line.split(rowDelimiter, -1);
Review Comment:
> I don’t understand why it doesn’t work in most cases. The processing logic
is to use custom delimiters to split the read lines. In the example data you
gave, the delimiters need to be escaped as "\\|",There is no problem. Could you
please tell me your processing logic?
For this case, the content will be split into `a,b,c,d`, `e,f`, `g,h`,
`j,k,l,m`, am I right?
But he correct content should be `a,b,c,d`, `e,f,g,h`, `j,k,l,m`, you still
use `\n` to split the row.
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/TextReadStrategy.java:
##########
@@ -94,32 +96,21 @@ public void read(String path, String tableId,
Collector<SeaTunnelRow> output)
.forEach(
line -> {
try {
- SeaTunnelRow seaTunnelRow =
-
deserializationSchema.deserialize(line.getBytes());
- if (!readColumns.isEmpty()) {
- // need column projection
- Object[] fields;
- if (isMergePartition) {
- fields =
- new Object
- [readColumns.size()
- +
partitionsMap.size()];
- } else {
- fields = new
Object[readColumns.size()];
- }
- for (int i = 0; i < indexes.length;
i++) {
- fields[i] =
seaTunnelRow.getField(indexes[i]);
- }
- seaTunnelRow = new
SeaTunnelRow(fields);
- }
- if (isMergePartition) {
- int index =
seaTunnelRowType.getTotalFields();
- for (String value :
partitionsMap.values()) {
- seaTunnelRow.setField(index++,
value);
+ if (StringUtils.isBlank(rowDelimiter)) {
+ SeaTunnelRow seaTunnelRow =
+ deserializeRow(line,
partitionsMap, tableId);
+ output.collect(seaTunnelRow);
+ } else {
+ String[] rows =
line.split(rowDelimiter, -1);
Review Comment:
> I don’t understand why it doesn’t work in most cases. The processing logic
is to use custom delimiters to split the read lines. In the example data you
gave, the delimiters need to be escaped as "\\|",There is no problem. Could you
please tell me your processing logic?
For this case, the content will be split into `a,b,c,d`, `e,f`, `g,h`,
`j,k,l,m`, am I right?
But he correct content should be `a,b,c,d`, `e,f,g,h`, `j,k,l,m`, you still
use `\n` to split the row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
