[
https://issues.apache.org/jira/browse/CALCITE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17940778#comment-17940778
]
Julian Hyde commented on CALCITE-6728:
--------------------------------------
[~mbudiu], 1. Please add a link (so that this case can be found from the other
end). 2. It's better if you just write the Jira id (CALCITE-6933) rather than
paste the URL (https://issues.apache.org/jira/browse/CALCITE-6933).
> Lazy loading of database tables and schemas
> -------------------------------------------
>
> Key: CALCITE-6728
> URL: https://issues.apache.org/jira/browse/CALCITE-6728
> Project: Calcite
> Issue Type: Wish
> Components: core
> Affects Versions: 1.38.0
> Reporter: Ulrich Kramer
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.39.0
>
>
> We are facing the issue that our Database contains 500000 schemas with up to
> 500000 tables each.
> Caching all these schemas and tables is not an option
> 1. It will require a lot of memory
> 2. The eviction of the cache must happen quite often since it's likely that
> every second one of these table is changed.
> We worked around this issue by introducing some kind of lazy loading. We did
> this in the following way:
> Instead of having pairs of methods to lookup tables and schemas (e.g.
> {{getTable}} and {{{}getTableNames{}}}), we introduced a new {{Lookup}}
> interface, which bundles these two methods
> {code:java}
> public interface Lookup<T> {
> /**
> * Returns an entity with a given name, or null if not found.
> * The name is matched case sensitive.
> *
> * @param name Name
> * @return Entity, or null
> */
> @Nullable T get(String name) ;
> /**
> * Returns a named entity with a given name ignoring the case, or null if
> not found.
> *
> * @param name Name
> * @return Entity, or null
> */
> @Nullable Named<T> getIgnoreCase(String name) ;
> /**
> * Returns the names of the entities in matching pattern.
> *
> * @return Names of the entities
> */
> Set<String> getNames(LikePattern pattern);
> }
> {code}
> and modified the {{Schema}} interface accordingly
> {code:java}
> public interface Schema {
> /**
> * Returns a lookup object to find tables
> *
> * @return Lookup
> */
> Lookup<Table> tables();
> /**
> * Returns a lookup object to find schemas
> *
> * @return Lookup
> */
> Lookup<? extends Schema> subSchemas();
> ...
> }
> {code}
> Most of the changes can be found in [this
> commit|https://github.com/sap-contributions/calcite/commit/edfbf29209b02b08d895da5fa1f61044bd207f3b#diff-a6ee53ffb06bb5a6d6a732372eb341a0bbaa374f6757b50f94bdc05ec71567d4].
> With theses changes we have been able to manage the huge amount of schemas
> and tables.
> Here are my questions related to my wish/proposal:
> - Does it make sense to provide a PR wich includes these modifications
> although they might break compatibility with older versions?
> - Does someone else have a great idea to introduce lazy loading with a
> smaller change?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
