This is an automated email from the ASF dual-hosted git repository.
liugddx pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 447d25e879 [Docs][Connector-V2][Http]Reconstruct the Http connector
document (#4962)
447d25e879 is described below
commit 447d25e879c7a1bb8db05e681c0f08c9693e176e
Author: monster <[email protected]>
AuthorDate: Sun Jun 25 22:02:33 2023 +0800
[Docs][Connector-V2][Http]Reconstruct the Http connector document (#4962)
Co-authored-by: chenzy15 <[email protected]>
---
docs/en/connector-v2/sink/Http.md | 72 ++++++--------
docs/en/connector-v2/source/Http.md | 188 ++++++++++++++++++++----------------
2 files changed, 134 insertions(+), 126 deletions(-)
diff --git a/docs/en/connector-v2/sink/Http.md
b/docs/en/connector-v2/sink/Http.md
index 0ccc1b785c..099943ce86 100644
--- a/docs/en/connector-v2/sink/Http.md
+++ b/docs/en/connector-v2/sink/Http.md
@@ -2,57 +2,45 @@
> Http sink connector
-## Description
+## Support Those Engines
-Used to launch web hooks using data.
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
-> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`],
the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
-
-**Tips: Http sink only support `post json` webhook and the data from source
will be treated as body content in web hook.**
-
-## Key features
+## Key Features
- [ ] [exactly-once](../../concept/connector-v2-features.md)
+- [ ] [cdc](../../concept/connector-v2-features.md)
-## Options
-
-| name | type | required | default value |
-|-----------------------------|--------|----------|---------------|
-| url | String | Yes | - |
-| headers | Map | No | - |
-| params | Map | No | - |
-| retry | int | No | - |
-| retry_backoff_multiplier_ms | int | No | 100 |
-| retry_backoff_max_ms | int | No | 10000 |
-| common-options | | no | - |
-
-### url [String]
-
-http request url
-
-### headers [Map]
-
-http headers
-
-### params [Map]
-
-http params
+## Description
-### retry [int]
+Used to launch web hooks using data.
-The max retry times if request http return to `IOException`
+> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`],
the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
-### retry_backoff_multiplier_ms [int]
+**Tips: Http sink only support `post json` webhook and the data from source
will be treated as body content in web hook.**
-The retry-backoff times(millis) multiplier if request http failed
+## Supported DataSource Info
-### retry_backoff_max_ms [int]
+In order to use the Http connector, the following dependencies are required.
+They can be downloaded via install-plugin.sh or from the Maven central
repository.
-The maximum retry-backoff times(millis) if request http failed
+| Datasource | Supported Versions |
Dependency |
+|------------|--------------------|------------------------------------------------------------------------------------------------------------|
+| Http | universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-http)
|
-### common options
+## Sink Options
-Sink plugin common parameters, please refer to [Sink Common
Options](common-options.md) for details
+| Name | Type | Required | Default |
Description
|
+|-----------------------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------|
+| url | String | Yes | - | Http request url
|
+| headers | Map | No | - | Http headers
|
+| params | Map | No | - | Http params
|
+| retry | Int | No | - | The max retry
times if request http return to `IOException`
|
+| retry_backoff_multiplier_ms | Int | No | 100 | The
retry-backoff times(millis) multiplier if request http failed
|
+| retry_backoff_max_ms | Int | No | 10000 | The maximum
retry-backoff times(millis) if request http failed
|
+| common-options | | No | - | Sink plugin
common parameters, please refer to [Sink Common Options](common-options.md) for
details |
## Example
@@ -60,11 +48,11 @@ simple:
```hocon
Http {
- url = "http://localhost/test/webhook";
- headers {
- token = "9e32e859ef044462a257e1fc76730066"
- }
+ url = "http://localhost/test/webhook";
+ headers {
+ token = "9e32e859ef044462a257e1fc76730066"
}
+}
```
## Changelog
diff --git a/docs/en/connector-v2/source/Http.md
b/docs/en/connector-v2/source/Http.md
index b63e9daaae..291835b93e 100644
--- a/docs/en/connector-v2/source/Http.md
+++ b/docs/en/connector-v2/source/Http.md
@@ -2,11 +2,13 @@
> Http source connector
-## Description
+## Support Those Engines
-Used to read data from Http.
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
-## Key features
+## Key Features
- [x] [batch](../../concept/connector-v2-features.md)
- [x] [stream](../../concept/connector-v2-features.md)
@@ -15,66 +17,112 @@ Used to read data from Http.
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)
-## Options
-
-| name | type | required | default value |
-|-----------------------------|---------|----------|---------------|
-| url | String | Yes | - |
-| schema | Config | No | - |
-| schema.fields | Config | No | - |
-| json_field | Config | No | - |
-| content_json | String | No | - |
-| format | String | No | json |
-| method | String | No | get |
-| headers | Map | No | - |
-| params | Map | No | - |
-| body | String | No | - |
-| poll_interval_ms | int | No | - |
-| retry | int | No | - |
-| retry_backoff_multiplier_ms | int | No | 100 |
-| retry_backoff_max_ms | int | No | 10000 |
-| enable_multi_lines | boolean | No | false |
-| common-options | config | No | - |
-
-### url [String]
-
-http request url
-
-### method [String]
-
-http request method, only supports GET, POST method.
-
-### headers [Map]
-
-http headers
-
-### params [Map]
-
-http params
-
-### body [String]
-
-http body
-
-### poll_interval_ms [int]
+## Description
-request http api interval(millis) in stream mode
+Used to read data from Http.
-### retry [int]
+## Key features
-The max retry times if request http return to `IOException`
+- [x] [batch](../../concept/connector-v2-features.md)
+- [x] [stream](../../concept/connector-v2-features.md)
+- [ ] [exactly-once](../../concept/connector-v2-features.md)
+- [ ] [column projection](../../concept/connector-v2-features.md)
+- [ ] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)
-### retry_backoff_multiplier_ms [int]
+Supported DataSource Info
+-------------------------
+
+In order to use the Http connector, the following dependencies are required.
+They can be downloaded via install-plugin.sh or from the Maven central
repository.
+
+| Datasource | Supported Versions |
Dependency |
+|------------|--------------------|------------------------------------------------------------------------------------------------------------|
+| Http | universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-http)
|
+
+## Source Options
+
+| Name | Type | Required | Default |
Description
|
+|-----------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------|
+| url | String | Yes | - | Http request
url.
|
+| schema | Config | No | - | Http and
seatunnel data structure mapping
|
+| schema.fields | Config | No | - | The schema
fields of upstream data
|
+| json_field | Config | No | - | This parameter
helps you configure the schema,so this parameter must be used with schema.
|
+| content_json | String | No | - | This parameter
can get some json data.If you only need the data in the 'book' section,
configure `content_field = "$.store.book.*"`. |
+| format | String | No | json | The format of
upstream data, now only support `json` `text`, default `json`.
|
+| method | String | No | get | Http request
method, only supports GET, POST method.
|
+| headers | Map | No | - | Http headers.
|
+| params | Map | No | - | Http params.
|
+| body | String | No | - | Http body.
|
+| poll_interval_ms | Int | No | - | Request http
api interval(millis) in stream mode.
|
+| retry | Int | No | - | The max retry
times if request http return to `IOException`.
|
+| retry_backoff_multiplier_ms | Int | No | 100 | The
retry-backoff times(millis) multiplier if request http failed.
|
+| retry_backoff_max_ms | Int | No | 10000 | The maximum
retry-backoff times(millis) if request http failed
|
+| enable_multi_lines | Boolean | No | false |
|
+| common-options | | No | - | Source plugin
common parameters, please refer to [Source Common Options](common-options.md)
for details |
+
+## How to Create a Http Data Synchronization Jobs
-The retry-backoff times(millis) multiplier if request http failed
+```hocon
+env {
+ execution.parallelism = 1
+ job.mode = "BATCH"
+}
-### retry_backoff_max_ms [int]
+source {
+ Http {
+ result_table_name = "http"
+ url = "http://mockserver:1080/example/http";
+ method = "GET"
+ format = "json"
+ schema = {
+ fields {
+ c_map = "map<string, string>"
+ c_array = "array<int>"
+ c_string = string
+ c_boolean = boolean
+ c_tinyint = tinyint
+ c_smallint = smallint
+ c_int = int
+ c_bigint = bigint
+ c_float = float
+ c_double = double
+ c_bytes = bytes
+ c_date = date
+ c_decimal = "decimal(38, 18)"
+ c_timestamp = timestamp
+ c_row = {
+ C_MAP = "map<string, string>"
+ C_ARRAY = "array<int>"
+ C_STRING = string
+ C_BOOLEAN = boolean
+ C_TINYINT = tinyint
+ C_SMALLINT = smallint
+ C_INT = int
+ C_BIGINT = bigint
+ C_FLOAT = float
+ C_DOUBLE = double
+ C_BYTES = bytes
+ C_DATE = date
+ C_DECIMAL = "decimal(38, 18)"
+ C_TIMESTAMP = timestamp
+ }
+ }
+ }
+ }
+}
-The maximum retry-backoff times(millis) if request http failed
+# Console printing of the read Http data
+sink {
+ Console {
+ parallelism = 1
+ }
+}
+```
-### format [String]
+## Parameter Interpretation
-the format of upstream data, now only support `json` `text`, default `json`.
+### format
when you assign format is `json`, you should also assign schema option, for
example:
@@ -126,13 +174,7 @@ connector will generate data as the following:
|----------------------------------------------------------|
| {"code": 200, "data": "get success", "success": true} |
-### schema [Config]
-
-#### fields [Config]
-
-the schema fields of upstream data
-
-### content_json [String]
+### content_json
This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
@@ -207,7 +249,7 @@ Here is an example:
- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
-### json_field [Config]
+### json_field
This parameter helps you configure the schema,so this parameter must be used
with schema.
@@ -268,28 +310,6 @@ source {
- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
-### common options
-
-Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
-
-## Example
-
-simple:
-
-```hocon
-Http {
- url = "https://tyrantlucifer.com/api/getDemoData";
- schema {
- fields {
- code = int
- message = string
- data = string
- ok = boolean
- }
- }
-}
-```
-
## Changelog
### 2.2.0-beta 2022-09-26
