rwzdk opened a new pull request, #13020:
URL: https://github.com/apache/seatunnel/pull/13020
## Symptom
The HTTP source fails the whole job with an `IndexOutOfBoundsException` on
perfectly valid JSON, whenever the configured `json_field` paths match a
different number of elements and `json_filed_missed_return_null` is enabled.
Minimal reproduction — two paths over an array of different lengths:
```json
{"a":[1],"b":[10,20]}
```
```hocon
source {
Http {
url = "http://...";
json_field_missed_return_null = true
json_field = {
a = "$.a[*]"
b = "$.b[*]"
}
}
}
```
→ `java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1`
## Root cause
`seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/util/JsonPathProcessorImpl.java:108`
```java
protected List<List<String>> dataFlip(List<List<String>> results) {
List<List<String>> datas = new ArrayList<>();
for (int i = 0; i < results.size(); i++) {
List<String> result = results.get(i);
if (i == 0) {
// rows are created from the FIRST path's result size
for (Object o : result) { ... datas.add(row); }
} else {
for (int j = 0; j < result.size(); j++) {
...
List<String> row = datas.get(j); // <-- no bound check
row.add(val);
}
}
}
```
`dataFlip` assumes every per-path list has the same length as the first one.
It builds exactly `results.get(0).size()` rows, then indexes `datas.get(j)`
with `j` running up to a *later* path's size.
That assumption holds when consistency is validated — but `processJsonData`
deliberately **skips** that validation when `json_filed_missed_return_null` is
enabled:
```java
// Only validate consistency if jsonFiledMissedReturnNull is false
boolean shouldValidate = !isJsonFiledMissedReturnNull();
if (shouldValidate) {
validateResultsConsistency(results, paths);
}
return dataFlip(results);
```
So the exact configuration that is *supposed* to tolerate missing/short
fields is the one that crashes.
## Fix
Build one row per element of the **longest** path, and pad shorter paths
with `null`, preserving index alignment:
```java
int rowCount = 0;
for (List<String> result : results) {
if (result != null && result.size() > rowCount) {
rowCount = result.size();
}
}
for (int i = 0; i < rowCount; i++) {
datas.add(new ArrayList<>(results.size()));
}
for (int pathIndex = 0; pathIndex < results.size(); pathIndex++) {
List<String> result = results.get(pathIndex);
int size = result == null ? 0 : result.size();
for (int rowIndex = 0; rowIndex < rowCount; rowIndex++) {
Object o = rowIndex < size ? result.get(rowIndex) : null;
String val = o == null ? null : o.toString();
datas.get(rowIndex).add(val);
}
}
```
Note this keeps the existing behaviour for the equal-length case (the only
case reachable when validation is on), so no regression there.
## Test
New `JsonPathProcessorDataFlipTest` (4 cases):
1. `$.a[*]`=1, `$.b[*]`=2 — shorter **first** path (crashed before);
2. `$.a[*]`=2, `$.b[*]`=1 — shorter **later** path (crashed before);
3. equal cardinality — rows still aligned (`[[1,10],[2,20]]`);
4. a path matching **nothing** — padded with `null`, first column order
preserved.
**Before (real output)**
```
[ERROR] Tests run: 4, Failures: 1, Errors: 1, Skipped: 0 <<< FAILURE!
[ERROR]
JsonPathProcessorDataFlipTest.shorterFirstPathMustNotThrowIndexOutOfBounds:65
Unequal path cardinality must not crash the reader
==> Unexpected exception thrown: java.lang.IndexOutOfBoundsException:
Index 1 out of bounds for length 1
[ERROR]
JsonPathProcessorDataFlipTest.missingFieldPadsWithNullInsteadOfShifting:120 »
IndexOutOfBounds
```
**After (real output)**
```
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -- in
...util.JsonPathProcessorDataFlipTest
```
**Full `connector-http-base` module suite (no regressions)**
```
[INFO] Tests run: 6 -- HttpClientProviderTest
[INFO] Tests run: 5 -- HttpParameterCredentialSchemeTest
[INFO] Tests run: 13 -- HttpFactoryTest
[INFO] Tests run: 3 -- HttpSourceReaderInternalPollNextTest
[INFO] Tests run: 17 -- HttpSourceReaderUpdateRequestParamTest
[INFO] Tests run: 1 -- JsonFieldMissedReturnNullComplexTest
[INFO] Tests run: 1 -- JsonFieldMissedReturnNullTest
[INFO] Tests run: 1 -- JsonFieldMissedReturnNullTreeFeatureTest
[INFO] Tests run: 5 -- HttpSinkBatchWriterTest
[INFO] Tests run: 5 -- BinaryResponseCollectorTest
[INFO] Tests run: 11 -- FilenameExtractorTest
[INFO] Tests run: 4 -- JsonPathProcessorDataFlipTest
[INFO] Tests run: 72, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
```
Commands run:
```shell
export JAVA_HOME=/path/to/jdk17
# compile the module and its upstream deps (offline, using the local ~/.m2
cache)
mvn -o -q -pl seatunnel-connectors-v2/connector-http/connector-http-base -am
\
-DskipTests -Dspotless.check.skip=true install
# the new regression test
mvn -o -pl seatunnel-connectors-v2/connector-http/connector-http-base \
-Dspotless.check.skip=true -Dcheckstyle.skip=true \
-Dtest=JsonPathProcessorDataFlipTest -DfailIfNoTests=false test
# full module suite
mvn -o -pl seatunnel-connectors-v2/connector-http/connector-http-base \
-Dspotless.check.skip=true -Dcheckstyle.skip=true test
# formatting (Apache spotless / AOSP style)
mvn -o -pl seatunnel-connectors-v2/connector-http/connector-http-base
spotless:apply
```
**What was not run (stated plainly):** only `connector-http-base` was built
and tested, not the full reactor — the repo is huge and most other modules need
network/DB/Docker services. There is no end-to-end HTTP-source job run against
a live endpoint; the proof is at the unit level (the exact `dataFlip` contract
plus the full module suite). `spotless:apply` was run so the new files match
the project's formatter.
## Notes
- The `ArrayJsonPathProcessor` subclass always pre-sizes its rows, so it
cannot hit this path; the fix is in the shared base class `dataFlip` that it
also inherits.
- Configuration that previously crashed now yields rows padded with `null`,
matching the documented intent of `json_field_missed_return_null`.
Prepared with an AI coding agent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
