Rishavbhattarai opened a new pull request, #12844:
URL: https://github.com/apache/seatunnel/pull/12844
Yes, I'm contributing to Apache SeaTunnel.
Closes #12822
### Purpose of this pull request
This adds a Typeform source connector, `connector-http-typeform`, built on
`connector-http-base` the same way the Asana and Intercom connectors are. It
reads the responses of one form from `GET /forms/{form_id}/responses` in batch
mode. The scope is the one settled in #12822: responses only, personal access
token only.
### What changed
- New module `connector-http-typeform` with `TypeformSource`,
`TypeformSourceFactory`, `TypeformSourceReader` and the option and parameter
classes. Plugin name and factory identifier are both `Typeform`.
- Auth is `Authorization: Bearer <token>` from the required `api_key`
option. Required `form_id` is validated to letters, digits, `_` and `-`, so it
cannot change the URL path.
- `base_url` defaults to `https://api.typeform.com`. EU accounts can set
`https://api.eu.typeform.com` or `https://api.typeform.eu`.
- Pagination uses the existing `CURSOR` type with no shared-client change.
The cursor goes back as `before` and is read from `$.items[-1:].token`, the
token of the last response on the page. An empty page ends the read.
`page_size` defaults to 100 and is validated to 1..1000.
- `since`, `until` and `response_type` are passed to Typeform only when set.
`sort` is not exposed, because `before` paging depends on the default
newest-first order.
- Output columns: `response_id`, `landing_id`, `token`, `landed_at`,
`submitted_at`, and `answers` and `hidden` as raw JSON strings.
- The reader retries HTTP 429 and 5xx with bounded exponential backoff,
reusing the existing `retry` options, as in Intercom. Other 4xx responses are
not retried.
- The reader also spaces requests at least 500 ms apart. This was not in the
design on #12822. Typeform allows 2 requests per second per account, so without
pacing a large form would keep hitting 429 and waiting out the backoff.
Typeform's own JS client waits 500 ms between requests too.
- A second `validateCredentialScheme()` call after the URL and
`Authorization` header are set, so an `http://` base_url logs the cleartext
warning (the same fix as in #12831).
- Streaming mode is rejected.
- Registered in `plugin-mapping.properties`, `config/plugin_config`,
`connector-http/pom.xml`, `seatunnel-dist/pom.xml` and the e2e `pom.xml`.
- English and Chinese docs and changelogs.
### API facts and evidence
#12822 asked for evidence for these. All of it comes from Typeform's docs
and official client, with no credentials used.
- Auth: `Authorization: Bearer {your_access_token}` in the curl examples of
the [hands-on guide](https://www.typeform.com/developers/get-started/hands-on/)
and the [Responses
walkthrough](https://www.typeform.com/developers/responses/walkthroughs/);
`bearer ${token}` in
[Typeform/js-api-client](https://github.com/Typeform/js-api-client)
`src/create-client.ts`.
- Cursor: the walkthrough says "The value for the `before` parameter should
be the `token` value for the 25th response." The
[reference](https://www.typeform.com/developers/responses/reference/retrieve-responses/)
describes `before` as exclusive.
- Max page size: 1000 (walkthrough: "Max. value is 1000"; JS client
`MAX_RESULTS_PAGE_SIZE = 1000`).
- Rate limit: 2 requests per second per account ([get
started](https://www.typeform.com/developers/get-started/)).
### Behavior to know about
- `answers` is NULL when a response has no answers, either an empty array or
a missing key. This comes from `JsonPathProcessorImpl.extractValue`, which
returns null for an empty list. It is documented and pinned in a unit test
rather than changed here. An empty `hidden` object stays `"{}"`.
- Each run is a bounded snapshot, optionally narrowed by `since` and
`until`. Nothing is persisted. After a failure the job restarts from the first
page and can emit rows it already sent. Every row carries `response_id`, so a
sink can deduplicate or upsert. No exactly-once claim.
- Responses submitted during a read arrive at the newest end and do not
shift the pages being read.
- Typeform says responses from roughly the last 30 minutes may not be
returned yet. The connector docs say so too.
- The token goes only into the `Authorization` header, never into query
params, and the connector logs status codes and attempt counts only. I did not
check whether the engine masks `api_key` when it logs the job config.
### Not in this PR
OAuth, webhooks, the forms, workspaces and Create API endpoints, flattening
`answers`, `sort`, persisted incremental state, streaming mode, `Retry-After`
and shared HTTP-client changes.
### Dependencies
None added. `okhttp3:mockwebserver` is test scope only, as in the Asana and
Intercom modules.
### How it was tested
Ran locally with JDK 11:
- 20 unit tests in `connector-http-typeform` pass, and `spotless:check`
passes. They use `MockWebServer` and cover:
- paging with `before` set to the last response's `token`, with `token`
and `response_id` given different values so the test shows which one is sent,
and an empty last page ending the read
- `answers` with 10 answer types (text, boolean, email, an integer above
2^53, a decimal, date, choices, choice with `other`, file_url, payment) coming
out byte-identical to the source JSON, including key order, escaping and
unicode; `hidden` key order preserved
- the edge cases: `answers: []` to NULL, `hidden: {}` to `"{}"`, explicit
nulls and missing keys to NULL, an unknown answer type passing through
unchanged, and a JSON-looking hidden value staying an escaped string
- `since`, `until` and `response_type` sent on every page, and not sent
when unset or blank
- validation of `api_key`, `form_id` (including path characters),
`page_size` and the backoff values, and that error messages do not contain the
token
- retry on 429 and 5xx, no retry on 401, giving up after the cap, the
interrupt during backoff, and the 500 ms pacing
- I broke the code on purpose to check the tests catch it. Switching the
cursor path to `response_id` makes the paging test fail, and removing the
pacing makes the pacing test fail.
Not run locally:
- The E2E test `connector-typeform-e2e` is written but not run, because the
starter modules it depends on are not built on my machine. The mock server
returns 401 without the Bearer header, a 429 on the first request, and chooses
each page by the `before` cursor, answering 400 to any other cursor, so the job
fails if the cursor is not sent back. I'm relying on CI for it.
- I have not tested against a live Typeform account. All requests in the
tests go to a mock server.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01LP6NzY6hpzUKArjJCC4Mv1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
