Pierre Villard created NIFI-14837:
-------------------------------------
Summary: Performance improvement GitHub Registry Client
Key: NIFI-14837
URL: https://issues.apache.org/jira/browse/NIFI-14837
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Pierre Villard
Assignee: Pierre Villard
I'm using the GitHub Registry Client in my NiFi instance. I have about 50
process groups that are versioned. Every process group matches a versioned flow
that may have tens of commits.
When I want to change version, the current implementation will list all
commits, and {*}for each commit{*}, will make an API call to GiHub in order to
retrieve some specific informations (commit message, commit date, etc).
This is extremely ineffective and changing the version a flow ends up taking a
very long time. For some cases with many commits, I cannot change version
because the call in the NiFi UI would time out before the backend has sent back
the full list of commits with all of the information.
This becomes very not friendly and barely usable. This will also impact the API
rate limits a lot.
This Jira is to introduce multiple improvements that are making all of this
MUCH better.
* The GitHub client being used is initialized with an optional OkHttp client
cache (see [https://hub4j.github.io/github-api/)]
{quote}This library comes with a pluggable connector to use different HTTP
client implementations through {{{}HttpConnector{}}}. In particular, this means
you can use [OkHttp|https://square.github.io/okhttp/], so we can make use of
its HTTP response cache. Making a conditional request against the GitHub API
and receiving a 304 response [does not count against the rate
limit|https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#conditional-requests].
{quote} * Adding a LRU cache to the client with a fixed size of 1000 commits
maximum in order to keep an internal cache of commit SHA to commit details.
* Expose a property to limit the number of commits retrieved. The client does
not ensure a chronological order but guarantees a topological ordering. So it
should be chronological except in some specific edge cases like rebase, merge
commits, cherry-pick, commits with manual dates, etc. However, this is very
unlikely to happen with a normal usage of the client. Regardless the default is
to retrieve all commits like it is right now.
* Adding a Rate Abuse Limit Handler to log an error when abusing the API
limits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
