xy720 opened a new pull request #4532:
URL: https://github.com/apache/incubator-doris/pull/4532
## Proposed changes
There is a 4 columns source data:
```
1|1|jim|2|
2|1|grace|2|
3|2|tom|2|
4|3|bush|3|
5|3|helen|3|
5|3|helen|6|
6|3|helen|3|
6|3|helen|3|
...
```
Given the same column terminator '|', broker load determines that it is 5
columns, and spark load determines that it is 4 columns.
And there is another 4 columns source
```
1|1|jim|2
2|1|grace|2
3|2|tom|2
4|3|bush|3
5|3|helen|3
5|3|helen|6
6|3|helen|3
6|3|helen|3
...
```
Given the same column terminator '|', both the broker load and spark load
determines that it is 4 columns.
**To Reproduce**
Steps to reproduce the behavior:
1. Submit a broker load.
```
load label ssb_db.broker_load_label
(
data infile ("hdfs://ymy-host:port/user/palo/table1")
into table test_tbl
COLUMNS TERMINATED BY "|"
(k1,k2,name,clicks )
)
with broker "doris" ("username" = "test", "password" = "test");
```
2. Submit a spark load.
```
load label ssb_db.spark_load_label
( data infile ("hdfs://ymy-host:port/user/palo/table1")
into table test_tbl
COLUMNS TERMINATED BY "|"
(k1,k2,name,clicks )
) with resource "spark0"
("spark.executor.memory" = "24g", "spark.executor.cores" = "2",
"spark.executor.instances" = "8");
```
3. Broker Load will report an error "quality not good enough to cancel"
**The reson of this bug**
This is because the first character and the last character of a line are not
considered to be delimeter in spark load.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [x] I have create an issue on (Fix #ISSUE), and have described the
bug/feature there in detail
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
