PDGGK opened a new pull request, #13181:
URL: https://github.com/apache/seatunnel/pull/13181
### Purpose of this pull request
Fixes #9537, which the stale bot closed without a fix.
With `array_mode = true`, `HttpSinkWriter.flush()` serializes each row to
JSON and adds the text to the request array with `ArrayNode.add(String)`. That
adds a string node, so the request body is an array of JSON strings rather than
an array of objects:
```
["{\"id\":1,\"name\":\"xiaoming1\",\"age\":18}","{\"id\":2,\"name\":\"xiaoming2\",\"age\":19}","{\"id\":3,\"name\":\"xiaoming3\",\"age\":20}"]
```
An endpoint that expects an array of objects rejects this body, as in the
issue (`http response status code:[400]`). Object mode (`array_mode = false`)
is not affected.
This PR parses each serialized row with `mapper.readTree(...)` and adds it
as a JSON value. Both serializers that `HttpSinkWriter` is used with in this
repo produce JSON: the default `JsonSerializationSchema` and the WeChat sink's
`WeChatBotMessageSerializationSchema`.
### Does this PR introduce _any_ user-facing change?
Yes. With `array_mode = true`, the request body is now
`[{"id":1,...},{"id":2,...}]`. The line has not changed since batch writes were
added in #9292, so 2.3.11 through 3.0.0 and current `dev` send the array of
strings.
I sent the issue's three rows through `HttpSinkWriter` (with the real
`HttpClientProvider` and `array_mode = true`) to a small HTTP server that
accepts only a JSON array of objects. Both ran in Docker.
| Jars | Body received | Result |
|---|---|---|
| 3.0.0 release | `["{\"id\":1,\"name\":\"xiaoming1\",\"age\":18}", ...]` |
400. The writer logs the error and `close()` returns normally, so the batch is
lost. |
| `dev` (f54bb58) | same | 400. The write fails with
`HttpConnectorException` (`HTTP-02`). |
| `dev` + this PR | `[{"id":1,"name":"xiaoming1","age":18}, ...]` | 200 |
### How was this patch tested?
- Added `HttpSinkBatchWriterTest#testArrayModeSendsEachRowAsJsonObject`. It
writes three rows in array mode, parses the captured request body, and checks
that it is an array of three objects with the expected `id`, `name` and `age`.
The existing `testArrayModeWithBatch` only checks that the body starts with `[`
and ends with `]`.
- Without the change to `HttpSinkWriter`, it fails with
`["{\"id\":1,\"name\":\"user1\",\"age\":20}", ...] ==> expected: <true> but
was: <false>`.
- With the change, the `connector-http-base` tests pass on JDK 17: 69
tests, 0 failures.
- `mvn spotless:check -pl
seatunnel-connectors-v2/connector-http/connector-http-base` passes.
#12376 also edits `HttpSinkBatchWriterTest`. The new test here is placed
before `testArrayModeWithBatch`, so it does not touch the methods that PR
changes.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]