Hi Hang and Jark
Thank you very much for your reply.
@Jark
> 1. Could you move the CatalogStore registration API to the "Public
Interface" section?
Indeed, it is more reasonable.
> 2. We should prefix "table." for the CatalogStore configuration.
Sure, the names that you provided are indeed more logical and easier to
understand.
> 3. About the open/close method in CatalogStoreFactory.
I also agree with the proposed requirement scenario and design. I have
already made modifications to the interface.
@Hang
> 1. The `CatalogStore` need the `open`/`close` methods to init or close
the resource.
I have already added the missing method.
> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
Sorry for the misspelling, I updated this part.
> 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
to init or close its resource
This is indeed a scenario that needs to be considered, and I have added
relevant methods accordingly.
Additionally, I have included some relevant description information in the
documentation to help others understand
why the open/close methods need to be added.
Best,
Feng
On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <imj...@gmail.com> wrote:
> Hi Jing,
>
> Thank you for the update.
>
> 1. Could you move the CatalogStore registration API to the "Public
> Interface" section?
> "Proposed Changes" is more like a place to describe the implementation
> details.
>
> 2. We should prefix "table." for the CatalogStore configuration. Besides,
> the config key
> name should be hierarchical[1]. Therefore, it may be better to use:
> "table.catalog-store.kind"
> "table.catalog-store.file.path"
>
> 3. I think Hang's suggestions make sense.
>
> Others look good to me.
>
> Best,
> Jark
>
> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ruanhang1...@gmail.com> wrote:
>
> > Hi, Feng.
> >
> > Thanks for the update.
> > The current design looks good to me. I have some minor comments.
> >
> > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> the
> > resource. For example, when storing the information in MySQL, the store
> > needs to open and close the connections.
> >
> > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >
> > 3. About the usage in the sql gateway.
> > Considering the usage in sql gateway, the sql gateway may create a
> > CatalogStore for each session.
> > If we are using the MySqlCatalogStore, there would be so many
> connections.
> > How can we reuse the connection among these sessions?
> > I think sql gateway need to maintain a connection pool in
> > the CatalogStoreFactory and each session get its own connection from the
> > pool when it is created.
> > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> init
> > or close its resource.
> > Or is there a better way?
> >
> > Best,
> > Hang
> >
> > Feng Jin <jinfeng1...@gmail.com> 于2023年6月2日周五 14:45写道：
> >
> > > Thanks Jingsong.
> > >
> > > > Just naming, maybe `createCatalog` in TableEnv
> > >
> > > +1 For this, I have already updated FLIP.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <jingsongl...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Feng,
> > > >
> > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > functions are converted to createxxx from registerxxx.
> > > >
> > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi jark, thanks for your suggestion.
> > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > CatalogStore should be immutable once TableEnv is created.
> Otherwise,
> > > > there
> > > > > might be some data inconsistencies when CatalogStore is changed.
> > > > >
> > > > > Yes, We should initialize the CatalogStore when creating the
> > TableEnv.
> > > > > Therefore, my current proposal is to add a method to configure the
> > > > > CatalogStore in EnvironmentSettings.
> > > > >
> > > > > // Initialize a catalog Store
> > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > >
> > > > > // set up the Table API
> > > > > final EnvironmentSettings settings =
> > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > .withCatalogStore(catalogStore)
> > > > > .build();
> > > > >
> > > > > final TableEnvironment tableEnv =
> > > > TableEnvironment.create(settings);
> > > > >
> > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > not an interface method?
> > > > >
> > > > > Sorry, While I did refer to the Catalog interface, I agree that as
> a
> > > new
> > > > > interface, the CatalogStoreFactory should not have a default method
> > but
> > > > an
> > > > > interface method. I have already modified the interface.
> > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > deprecated`registerCatalog`.
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > > persisted catalog store) is used.
> > > > >
> > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > >
> > > > >
> > > > > [1].
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <imj...@gmail.com> wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > The current design looks pretty good to me. I just have some
> minor
> > > > > > comments.
> > > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > CatalogStore
> > > > > > should be immutable once TableEnv is created. Otherwise, there
> > might
> > > be
> > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > > not an interface method?
> > > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> deprecated
> > > > > > `registerCatalog`.
> > > > > >
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> other
> > > > > > persisted catalog store) is used.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <jinfeng1...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > >
> > > > > > > @Ron
> > > > > > >
> > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore method,
> > do
> > > we
> > > > > > need
> > > > > > > to provide a default implementation?
> > > > > > >
> > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > > create
> > > > an
> > > > > > > InMemoryCatalogStore.
> > > > > > >
> > > > > > > > If we get a Catalog from CatalogStore, after initializing
> it,
> > > > whether
> > > > > > we
> > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > >
> > > > > > > Yes, in the current design, catalogs are stored as snapshots,
> and
> > > > once
> > > > > > > initialized, the Catalog will be placed in the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > primary
> > > > > > source
> > > > > > > for obtaining the corresponding Catalog.
> > > > > > >
> > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > `catalog.store.path`?
> > > > > > >
> > > > > > > I think it is okay. Adding "sql" at the beginning may seem a
> bit
> > > > > > strange. I
> > > > > > > will update the FLIP.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > @Shammon
> > > > > > >
> > > > > > > Thank you for the review. I have made the necessary
> corrections.
> > > > > > > Regarding the modifications made to the Public Interface, I
> have
> > > also
> > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zjur...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi feng,
> > > > > > > >
> > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > >
> > > > > > > > 1. The modification of `CatalogManager` should not be in
> > `Public
> > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > >
> > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > > `CatalogStoreFactory`
> > > > > > > >
> > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> should
> > be
> > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);`
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ron9....@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Feng
> > > > > > > > >
> > > > > > > > > Thanks for driving this FLIP, this proposal is very useful
> > for
> > > > > > catalog
> > > > > > > > > management.
> > > > > > > > > I have some small questions:
> > > > > > > > >
> > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > method,
> > > > do we
> > > > > > > > need
> > > > > > > > > to provide a default implementation?
> > > > > > > > > 2. If we get Catalog from CatalogStore, after initializing
> > it,
> > > > > > whether
> > > > > > > we
> > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Feng Jin <jinfeng1...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > > >
> > > > > > > > > > Hi yuxia
> > > > > > > > > >
> > > > > > > > > > > But from the code in Proposed Changes, once we
> register
> > > the
> > > > > > > Catalog,
> > > > > > > > > we
> > > > > > > > > > initialize it and open it. right?
> > > > > > > > > >
> > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > original
> > > > > > CREATE
> > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > registerCatalog
> > > > > > > so
> > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > >
> > > > > > > > > > In the current design, lazy initialization is mainly
> > > reflected
> > > > in
> > > > > > > > > > getCatalog. If CatalogStore has already saved some
> catalog
> > > > > > > > > configurations,
> > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > initialization*.
> > > > > > > If
> > > > > > > > > i'm
> > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > >
> > > > > > > > > > > IIUC, lazy initialization means only you need to access
> > the
> > > > > > > catalog,
> > > > > > > > > then
> > > > > > > > > > > you initialize it. But from the code in Proposed
> Changes,
> > > > once we
> > > > > > > > > > register
> > > > > > > > > > > the Catalog,
> > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Yuxia
> > > > > > > > > > >
> > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > 发件人: "Jing Ge" <j...@ververica.com.INVALID>
> > > > > > > > > > > 收件人: "dev" <dev@flink.apache.org>
> > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > Catalog
> > > > > > > > > configuration
> > > > > > > > > > > and asynchronous registration
> > > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > >
> > > > > > > > > > > One of the major changes is that current design will
> > > provide
> > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > > > cache,
> > > > > > > which
> > > > > > > > > > means
> > > > > > > > > > > once it has been initialized, any changes done by other
> > > > sessions
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > affect it. Point 6 described follow-up options for
> > further
> > > > > > > > improvement.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > jinfeng1...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all, I would like to update you on the latest
> > progress
> > > > of
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > > > ShengKai
> > > > > > > Fang
> > > > > > > > > > and I
> > > > > > > > > > > > had an offline discussion regarding the overall
> > solution
> > > > for
> > > > > > > Flink
> > > > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > > > updated
> > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > configuration
> > > > > > > > similar
> > > > > > > > > > to
> > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > storeCatalog(String
> > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > CatalogDescriptor
> > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > > > instance.
> > > > > > > > This
> > > > > > > > > > way,
> > > > > > > > > > > > CatalogStore will only be responsible for saving
> and
> > > > > > > retrieving
> > > > > > > > > > > catalog
> > > > > > > > > > > > configurations without having to initialize
> > catalogs.
> > > > > > > > > > > > 3.
> > > > > > > > > > > >
> > > > > > > > > > > > The default registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > > function in CatalogManager will be marked as
> > > deprecated.
> > > > > > > > > > > > 4.
> > > > > > > > > > > >
> > > > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > catalog) will be added to serve as the default
> > > > registration
> > > > > > > > > function
> > > > > > > > > > > for
> > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > 5.
> > > > > > > > > > > >
> > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > > > remain
> > > > > > > > > unchanged
> > > > > > > > > > > and
> > > > > > > > > > > > save initialized catalogs.This means that deletion
> > > > > > operations
> > > > > > > > from
> > > > > > > > > > one
> > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > 6.
> > > > > > > > > > > >
> > > > > > > > > > > > To support multi-session synchronization scenarios
> > for
> > > > > > > deletions
> > > > > > > > > > later
> > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > configurable.There
> > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > > three possible situations:
> > > > > > > > > > > >
> > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > >
> > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > automatically
> > > > > > > clear
> > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > c.No caching of any instances; each call to
> > getCatalog
> > > > > > > creates a
> > > > > > > > > new
> > > > > > > > > > > > instance.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > >
> > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you very much for your attention and
> suggestions
> > on
> > > > this
> > > > > > > > > FLIP. A
> > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on
> the
> > > > entire
> > > > > > > > design
> > > > > > > > > > and
> > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > If you have any further suggestions or feedback about
> > > this
> > > > FLIP
> > > > > > > > > please
> > > > > > > > > > > feel
> > > > > > > > > > > > free to share.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > <j...@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to me.
> > We
> > > > could
> > > > > > > > still
> > > > > > > > > > > > > reconsider in the future how to provide more common
> > > > built-in
> > > > > > > > cache
> > > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > > CatalogSotre
> > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > jinfeng1...@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am very sorry for my carelessness and at the
> same
> > > > time, I
> > > > > > > > > greatly
> > > > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > > > checked
> > > > > > the
> > > > > > > > > > entire
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > > design,
> > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > could work without Optional CatalogStore being
> > > > configured.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > > > necessary
> > > > > > > > > because
> > > > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs
> > to
> > > > store
> > > > > > > > > catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > Therefore, I
> > > > > > > > modified
> > > > > > > > > > this
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > > catalogs
> > > > > > from
> > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve as
> > the
> > > > > > default
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > functionality of
> > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > The previous plan that kept
> > > > Map<String,Catalog>catalogs has
> > > > > > > > been
> > > > > > > > > > > moved
> > > > > > > > > > > > to
> > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > <j...@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are still many places contain
> inconsistent
> > > > content,
> > > > > > > > e.g.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > > 2. The java comment of the method
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> tell
> > > > what the
> > > > > > > > > method
> > > > > > > > > > is
> > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> that
> > > you
> > > > will
> > > > > > > > build
> > > > > > > > > > the
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > functionality in the CatalogStore? This is a
> very
> > > > > > different
> > > > > > > > > > design
> > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > from what the current FLIP described. If I am
> not
> > > > > > mistaken,
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > > > without
> > > > > > > > Optional
> > > > > > > > > > > > > > > CatalogStore being configured. That is the
> reason
> > > > why I
> > > > > > > > > mentioned
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > last email that the example code wrt the
> Optional
> > > > > > > > CatalogStore
> > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > jinfeng1...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you very much for the careful review. I
> > > have
> > > > > > > already
> > > > > > > > > made
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > multi-instance
> > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > > scenarios
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > design. I am thinking whether we should
> remove
> > > > > > > 'Map<String,
> > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > InMemoryCatalogStore.
> > > > > > > > This
> > > > > > > > > > > way,
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > can implement their own CatalogStore to
> achieve
> > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > <j...@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> > still
> > > > some
> > > > > > > NIT
> > > > > > > > > > issues
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has
> > been
> > > > used
> > > > > > as
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > instead of Optional in the code example. It
> > > > should be
> > > > > > > > fine
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > pseudo code for now and update it after you
> > > > submit
> > > > > > the
> > > > > > > > PR.
> > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere
> in
> > > the
> > > > > > > > rejected
> > > > > > > > > > > > section
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > > consistent.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > multi-instance
> > > > > > > > > > > scenarios,
> > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > used as a cache, it should be invalid, once
> > the
> > > > > > related
> > > > > > > > > > catalog
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > > instance.
> > > > > > Did
> > > > > > > I
> > > > > > > > > miss
> > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > > > jinfeng1...@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > I think this is a good function name, I
> > have
> > > > > > updated
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Some common cache features should be
> > > > > > implemented
> > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > alternative
> > > 1
> > > > > > turns
> > > > > > > > out
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > appropriate later, I will improve this
> part
> > > of
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As the above discussion moves forward,
> > the
> > > > > > option 2
> > > > > > > > > > > solution
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> offline,
> > > we
> > > > > > think
> > > > > > > > that
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > inconsistency
> > > > > > > > issues.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are some inconsistent
> descriptions
> > in
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > I will do my best to improve the document
> > and
> > > > > > > > appreciate
> > > > > > > > > > your
> > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > > `Rejected
> > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon
> FY
> > <
> > > > > > > > > > > zjur...@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I found there are two options in
> > `Proposed
> > > > > > > Changes`,
> > > > > > > > > can
> > > > > > > > > > > you
> > > > > > > > > > > > > put
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > Alternatives`? I
> > > > > > > think
> > > > > > > > > > this
> > > > > > > > > > > > may
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > > > <j...@ververica.com.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> > still
> > > > > > confused
> > > > > > > > > after
> > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> registerCatalog()
> > > and
> > > > > > > > > > addCatalog()
> > > > > > > > > > > > have
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > difference based on their names. One
> of
> > > > them is
> > > > > > > > > > > responsible
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > persistence. How about
> > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > is
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > persistence.
> > > > > > I
> > > > > > > > > would
> > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> clearly
> > in
> > > > the
> > > > > > > FLIP.
> > > > > > > > > > Some
> > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > features should be implemented, i.e.
> > data
> > > > in
> > > > > > the
> > > > > > > > > cache
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > > > should be
> > > > > > > > found
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > initialized
> > > > or it
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > consistency
> > > > will
> > > > > > > be
> > > > > > > > > > taken
> > > > > > > > > > > > care
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > forward,
> > > > the
> > > > > > > > option
> > > > > > > > > 2
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > because,
> > > > afaiu,
> > > > > > > > > issues
> > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> solved
> > > > yet. Do
> > > > > > > you
> > > > > > > > > > still
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > both options and ask for suggestions
> > for
> > > > both
> > > > > > of
> > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there
> > are
> > > > some
> > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the content. Would you like to clean
> > > them
> > > > up?
> > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> > Jin <
> > > > > > > > > > > > > jinfeng1...@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > > > completely
> > > > > > > > agree
> > > > > > > > > > > with
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > complicated
> > > > the
> > > > > > > whole
> > > > > > > > > > issue.
> > > > > > > > > > > > > First
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > we need to consider the persistence
> > of
> > > > the
> > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > persistence
> > > > for
> > > > > > > > Catalog
> > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > > > Catalog
> > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > > instance
> > > > > > to a
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > implementation
> > > > > > will
> > > > > > > > > throw
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > * @throws
> > > > UnsupportedOperationException if
> > > > > > > the
> > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > * the default implementation of
> > this
> > > > > > method
> > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > toProperties()
> > > > > > {
> > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > > CatalogStore,
> > > > > > > > the
> > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will be called when registering a
> > > Catalog
> > > > > > > > instance
> > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog).
> > > > > > > > > > > The
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > > > String>
> > > > > > > > > properties
> > > > > > > > > > > and
> > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > instances
> > > > do
> > > > > > not
> > > > > > > > > > > implement
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > CatalogStore,
> > > > > > > the
> > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > consistency
> > > > with
> > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > conflicts
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> itself
> > > and
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > implementation for lazy loading,
> such
> > > as
> > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > > implements
> > > > > > > > > Catalog
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > > String>
> > > > > > > > > properties)
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have added this design to the
> FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > Shammon
> > > > FY <
> > > > > > > > > > > > > > zjur...@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > functionality
> > > > > > > of
> > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > > > `registerCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > implementation
> > > in
> > > > > > Trino
> > > > > > > > is
> > > > > > > > > > also
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only
> has
> > > one
> > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > another
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > > Catalog>
> > > > > > > > catalogs`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > > > before,
> > > > > > the
> > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > `CatalogManager`.
> > > > > > How
> > > > > > > > > about
> > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs`
> and
> > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM
> Feng
> > > > Jin <
> > > > > > > > > > > > > > > jinfeng1...@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am
> > very
> > > > sorry
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> original
> > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > > discussion
> > > > > > > > before.
> > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that
> the
> > > > logic of
> > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> > > only
> > > > > > > persist
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > > > because
> > > > > > in
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > we
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > the configurations
> corresponding
> > to
> > > > the
> > > > > > > > Catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > CatalogStore
> > > > > > > > interface
> > > > > > > > > > for
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > part, I
> > > > also
> > > > > > > took
> > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > > catalog
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> > > will
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > code examples for this part,
> and
> > > add
> > > > it
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > still
> > > > > > > > > > looks
> > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > catalog
> > > to
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > personal
> > > > > > > > opinion,
> > > > > > > > > > you
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> intuitive
> > > to
> > > > me.
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > see
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> statement,
> > > > which
> > > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > `addCatalog`?
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > stement.
> > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > > > design
> > > > > > was
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > it
> > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > > disrupt
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > configuration to
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> same
> > > time
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > > > provide a
> > > > > > > > > default
> > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > interface
> `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > otherwise
> > > we
> > > > > > would
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > Map to store the configuration
> > for
> > > > lazy
> > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > registration
> > > > for
> > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > interface
> > > > > > because
> > > > > > > > > > Catalog
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > initialization
> > > > > > > for
> > > > > > > > > > > catalog
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > inaccurate. It
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > registration. I
> > > > > > > have
> > > > > > > > > > > already
> > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > > > Shammon
> > > > > > FY <
> > > > > > > > > > > > > > > zjur...@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > `registerCatalog`
> > > > > > > > > method
> > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> method
> > > > named
> > > > > > > > > > `addCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > memory
> > > > > > catalog
> > > > > > > > is a
> > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > exposed
> > > to
> > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> conclusion
> > > of
> > > > that
> > > > > > > > > > > discussion
> > > > > > > > > > > > is
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > interface
> > > > and
> > > > > > > > > implement
> > > > > > > > > > it
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > > > system.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> between
> > > the
> > > > > > > current
> > > > > > > > > FLIP
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of
> the
> > > > > > discussion
> > > > > > > > in
> > > > > > > > > > > thread
> > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 3:41 PM
> > > > yuxia <
> > > > > > > > > > > > > > > > > > > luoyu...@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> this
> > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> handling
> > > > catalog
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> it'll
> > be
> > > > > > better
> > > > > > > to
> > > > > > > > > > > explain
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> > brief
> > > > > > example
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > > still
> > > > > > > > > > > looks
> > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > > catalog to
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> very
> > > > personal
> > > > > > > > > > opinion,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > statement,
> > > > > > which
> > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > `addCatalog`?
> > > > > > I'm
> > > > > > > > > > > wondering
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary
> to
> > > > > > provide a
> > > > > > > > > > default
> > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > registration
> > > > > > for
> > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> > with
> > > > CREATE
> > > > > > > > > > CATALOG,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> registration
> > is
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > synchronous
> > > > > > > > > > > registration
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> `addCatalog`
> > > and
> > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > registration,
> > > > > > it
> > > > > > > > may
> > > > > > > > > > well
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > executes successfully, but
> > then
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail
> to
> > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> > > which
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > > > registration, I
> > > > > > > > > > > would
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to open
> > or
> > > > > > register
> > > > > > > > > > catalog
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will
> be
> > > > > > > initialized
> > > > > > > > > if
> > > > > > > > > > it
> > > > > > > > > > > > has
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > > > process
> > > > > > > fails,
> > > > > > > > > > these
> > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > initialization
> > > > > > > > for
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > jinfeng1...@gmail.com
> > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > dev@flink.apache.org
> > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > > > Support
> > > > > > > > > > persistence
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> registration
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all
> > for
> > > > your
> > > > > > > > > interest
> > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> > with
> > > > the
> > > > > > same
> > > > > > > > > name
> > > > > > > > > > > that
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > > > point.
> > > > > > > Based
> > > > > > > > on
> > > > > > > > > > my
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs
> and
> > > > have
> > > > > > the
> > > > > > > > > > highest
> > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> that
> > > can
> > > > > > affect
> > > > > > > > the
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> catalog
> > > > instance
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > catalog
> > > > > > > instance
> > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method first retrieves
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > > catalogs. If
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > > > configuration
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts
> the
> > > > > > > initialized
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> > only
> > > > modify
> > > > > > > the
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > properties
> > > > to
> > > > > > the
> > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > catalogName
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > specified
> > > > > > > > > > configuration
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > > > conflict
> > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > > > Map<String,
> > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> saved
> > > to
> > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > Map<String,
> > > > > > > > Catalog>,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> catalog)
> > > > method
> > > > > > > will
> > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties) method
> > can
> > > > > > directly
> > > > > > > > > save
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > getCatalog(String
> > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > > > initialization
> > > > > > > > > > > because
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> > exists
> > > > in
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > corresponding to
> > > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> > users
> > > > that
> > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> catalog)`
> > > in
> > > > the
> > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This could potentially lead
> > to
> > > > > > > > confusion. I
> > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > addCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > previously
> > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > add
> `registerCatalog(String
> > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already
> an
> > > > > > instance,
> > > > > > > > > > adding
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > > > result in
> > > > > > > lazy
> > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > necessary,
> > > > but
> > > > > > > > > perhaps
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in
> the
> > > > > > > CatalogStore
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > > FileCatalogStore
> > > > > > > > > > could
> > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> while
> > > the
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> > > have a
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > > > Catalog
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > > implement
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > additional
> > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > registration
> > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > concern.
> > > > > > When
> > > > > > > > > > > executing
> > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > > > specific
> > > > > > > > > Catalog,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it
> has
> > > not
> > > > > > been
> > > > > > > > > > > > initialized
> > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> fails,
> > > > these
> > > > > > > > > > statements
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > Catalog>
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have
> > the
> > > > same
> > > > > > > > > catalog
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved
> by
> > > > > > referring
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > > Catalog
> > > > > > that
> > > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > > > subsequent
> > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > 10:58 PM
> > > > > > Xianxun
> > > > > > > > Ye
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > yesorno828...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> > Flip,
> > > > I do
> > > > > > > > > believe
> > > > > > > > > > > this
> > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work
> > for
> > > > also
> > > > > > > > > manages a
> > > > > > > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> > > slow
> > > > > > > because
> > > > > > > > of
> > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in
> the
> > > > user's
> > > > > > > > Flink
> > > > > > > > > > SQL
> > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> catalogs
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > > > speed
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> > have
> > > > some
> > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> catalog
> > > with
> > > > > > > CREATE
> > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > What
> > > > > > > > > > if
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> implement
> > > > > > > asynchronous
> > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > > question
> > > > > > with
> > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> catalogs
> > > and
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> > > same
> > > > > > > catalog
> > > > > > > > > > name,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > For
> > > > > > > example:
> > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > > hiveCatalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > define
> > > > > > > `catalogs`
> > > > > > > > > and
> > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> > Ruan
> > > <
> > > > > > > > > > > > > > ruanhang1...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
