xiaofan2022 commented on code in PR #5561:
URL: https://github.com/apache/seatunnel/pull/5561#discussion_r1339464449
##########
docs/en/connector-v2/source/Http.md:
##########
@@ -42,24 +42,31 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
## Source Options
-| Name | Type | Required | Default |
Description
|
-|-----------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------|
-| url | String | Yes | - | Http request
url.
|
-| schema | Config | No | - | Http and
seatunnel data structure mapping
|
-| schema.fields | Config | No | - | The schema
fields of upstream data
|
-| json_field | Config | No | - | This parameter
helps you configure the schema,so this parameter must be used with schema.
|
-| content_json | String | No | - | This parameter
can get some json data.If you only need the data in the 'book' section,
configure `content_field = "$.store.book.*"`. |
-| format | String | No | json | The format of
upstream data, now only support `json` `text`, default `json`.
|
-| method | String | No | get | Http request
method, only supports GET, POST method.
|
-| headers | Map | No | - | Http headers.
|
-| params | Map | No | - | Http params,the
program will automatically add http header application/x-www-form-urlencoded.
|
-| body | String | No | - | Http body,the
program will automatically add http header application/json,body is jsonbody.
|
-| poll_interval_millis | Int | No | - | Request http
api interval(millis) in stream mode.
|
-| retry | Int | No | - | The max retry
times if request http return to `IOException`.
|
-| retry_backoff_multiplier_ms | Int | No | 100 | The
retry-backoff times(millis) multiplier if request http failed.
|
-| retry_backoff_max_ms | Int | No | 10000 | The maximum
retry-backoff times(millis) if request http failed
|
-| enable_multi_lines | Boolean | No | false |
|
-| common-options | | No | - | Source plugin
common parameters, please refer to [Source Common Options](common-options.md)
for details |
+| Name | Type | Required | Default |
Description
|
+|--------------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| url | String | Yes | - | Http request
url.
|
+| schema | Config | No | - | Http and
seatunnel data structure mapping
|
+| schema.fields | Config | No | - | The schema
fields of upstream data
|
+| json_field | Config | No | - | This
parameter helps you configure the schema,so this parameter must be used with
schema.
|
+| pageing | Config | No | - | This
parameter is used for paging queries
|
+| pageing.page_field | String | No | - | This
parameter is used to specify the page field name in the request parameter
|
+| *pageing.max_page_size* | Int | No | 10000 | Change the
parameter to control the maximum number of pages (if using paging please ensure
that this value is greater than the target number of pages otherwise it may
cause early *data accuracy* problems) |
+| pageing.total_page_size | Int | No | - | This
parameter is used to control the total number of pages
|
+| pageing.total_page_field_path | String | No | - | This
parameter to get the total number of pages in response
|
+| pageing.json_verify_expression | String | No | - | This
parameter is used verify that the condition is met through json path
|
+| pageing.json_verify_value | String | No | - | This
parameter must be configured after json_verify_expression is configured (make
sure verify value is in this parameter).
|
Review Comment:
These two parameters are primarily used in cases where there is no total
page count field in the API response to determine the number of iterations.
They allow you to pinpoint a field using a JSON path and then decide whether to
continue the task based on whether a specific condition is met (typically,
whether the target value is found in json_verify_value).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
