[
https://issues.apache.org/jira/browse/NIFI-16359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jannik Rebmann updated NIFI-16359:
----------------------------------
Description:
We are using GitLabFlowRegistryClient against a self-hosted GitLab instance.
Our infrastructure team flagged a high load on the GitLab server that is being
generated by the NiFi flow registry client. With a larger number of versioned
process groups, NiFi's periodic version checks and version listings translate
into a high volume of GitLab REST API calls (repository/commits,
repository/tree, repository/files).
Looking into it, NIFI-14837 (fixed in 2.6.0, PR #10186) already reduced GitHub
API load in GitHubFlowRegistryClient by
(a) adding an in-memory LRU cache of commit details keyed by SHA (max 1000), and
(b) limiting commit listing to the first page instead of paging the full
history.
These optimizations were applied only to GitHubRepositoryClient. The sibling
GitLabRepositoryClient (nifi-gitlab-bundle) still has no caching and, in
getCommits(path, branch), calls CommitsApi.getCommits(projectPath, branch,
resolvedPath), which pages the entire commit history for the path (default
100/page). This is the most likely driver of the load our GitLab team is seeing.
Proposal: apply the same two improvements to GitLabRepositoryClient - an
internal commit cache keyed by commit SHA, and a bounded commit listing (e.g.
first page / configurable or hard limit) - mirroring the approach in PR #10186.
Reference: NIFI-14837, PR [https://github.com/apache/nifi/pull/10186]
was:
We are using GitLabFlowRegistryClient against a self-hosted GitLab instance.
Our infrastructure team flagged a high load on the GitLab server that is being
generated by the NiFi flow registry client. With a larger number of versioned
process groups, NiFi's periodic version checks and version listings translate
into a high volume of GitLab REST API calls (repository/commits,
repository/tree, repository/files).
Looking into it, NIFI-14837 (fixed in 2.6.0, PR #10186) already reduced GitHub
API load in GitHubFlowRegistryClient by (a) adding an in-memory LRU cache of
commit details keyed by SHA (max 1000), and (b) limiting commit listing to the
first page instead of paging the full history.
These optimizations were applied only to GitHubRepositoryClient. The sibling
GitLabRepositoryClient (nifi-gitlab-bundle) still has no caching and, in
getCommits(path, branch), calls CommitsApi.getCommits(projectPath, branch,
resolvedPath), which pages the
entire commit history for the path (default 100/page). This is the most likely
driver of the load our GitLab team is seeing.
Proposal: apply the same two improvements to GitLabRepositoryClient - an
internal commit cache keyed by commit SHA, and a bounded commit listing (e.g.
first page / configurable or hard limit) - mirroring the approach in PR #10186.
Reference: NIFI-14837, PR https://github.com/apache/nifi/pull/10186
> Port GitHubFlowRegistryClient API-call optimizations (NIFI-14837) to
> GitLabFlowRegistryClient
> ---------------------------------------------------------------------------------------------
>
> Key: NIFI-16359
> URL: https://issues.apache.org/jira/browse/NIFI-16359
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Flow Versioning
> Affects Versions: 2.12.0
> Reporter: Jannik Rebmann
> Priority: Major
>
> We are using GitLabFlowRegistryClient against a self-hosted GitLab instance.
> Our infrastructure team flagged a high load on the GitLab server that is
> being generated by the NiFi flow registry client. With a larger number of
> versioned process groups, NiFi's periodic version checks and version listings
> translate into a high volume of GitLab REST API calls (repository/commits,
> repository/tree, repository/files).
> Looking into it, NIFI-14837 (fixed in 2.6.0, PR #10186) already reduced
> GitHub API load in GitHubFlowRegistryClient by
> (a) adding an in-memory LRU cache of commit details keyed by SHA (max 1000),
> and
> (b) limiting commit listing to the first page instead of paging the full
> history.
> These optimizations were applied only to GitHubRepositoryClient. The sibling
> GitLabRepositoryClient (nifi-gitlab-bundle) still has no caching and, in
> getCommits(path, branch), calls CommitsApi.getCommits(projectPath, branch,
> resolvedPath), which pages the entire commit history for the path (default
> 100/page). This is the most likely driver of the load our GitLab team is
> seeing.
> Proposal: apply the same two improvements to GitLabRepositoryClient - an
> internal commit cache keyed by commit SHA, and a bounded commit listing (e.g.
> first page / configurable or hard limit) - mirroring the approach in PR
> #10186.
> Reference: NIFI-14837, PR [https://github.com/apache/nifi/pull/10186]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
