Hi Jacques,

Thanks for picking this up!

Coming from the Hive side I prefer solution #3.
Knowing Hive there will be plenty of surprising new properties which we either 
handle specifically (hard-coded omission) or will litter the quite neat list of 
Iceberg table properties. This will become even more pronounced if we implement 
HiveMetaHook.preAlterTable 
<https://hive.apache.org/javadocs/r3.1.2/api/org/apache/hadoop/hive/metastore/HiveMetaHook.html#preAlterTable-org.apache.hadoop.hive.metastore.api.Table-org.apache.hadoop.hive.metastore.api.EnvironmentContext->
 for propagating Hive ALTER TABLE changes to IcebergTables. Alter table is 
called by Hive every time for updating statistics on the table - if we forget 
to remove anything changing continuously we would end up creating 2 
modifications on the Iceberg table every time when we insert a single row of 
data.

Thanks,
Peter

> On Dec 7, 2020, at 18:58, Jacques Nadeau <jacq...@dremio.com> wrote:
> 
> Hey Peter, thanks for updating the doc and your heads up in the other thread 
> on your capacity to look at this before EOY.
> 
> I'm going to try to create a specification document based on the discussion 
> document you put together. I think there is general consensus around what you 
> call "Spark-like catalog configuration" so I'd like to formalize that more. 
> 
> It seems like there is less consensus around the whitelist/blacklist side of 
> things. You outline four approaches: 
> Hard coded HMS only property list
> Hard coded Iceberg only property list
> Prefix for Iceberg properties
> Prefix for HMS only properties
> I generally think #2 is a no-go as it creates too much coupling between 
> catalog implementations and core iceberg. It seems like Ryan Blue would 
> prefer #4 (correct?). Any other strong opinions?
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> 
> On Thu, Dec 3, 2020 at 9:27 AM Peter Vary <pv...@cloudera.com.invalid> wrote:
> As Jacques suggested (with the help of Zoltan) I have collected the current 
> state and the proposed solutions in a document:
> https://docs.google.com/document/d/1KumHM9IKbQyleBEUHZDbeoMjd7n6feUPJ5zK8NQb-Qw/edit?usp=sharing
>  
> <https://docs.google.com/document/d/1KumHM9IKbQyleBEUHZDbeoMjd7n6feUPJ5zK8NQb-Qw/edit?usp=sharing>
> 
> My feeling is that we do not have a final decision, so tried to list all the 
> possible solutions. 
> Please comment!
> 
> Thanks,
> Peter
> 
>> On Dec 2, 2020, at 18:10, Peter Vary <pv...@cloudera.com 
>> <mailto:pv...@cloudera.com>> wrote:
>> 
>> When I was working on the CREATE TABLE patch I found the following 
>> TBLPROPERTIES on newly created tables:
>> external.table.purge
>> EXTERNAL
>> bucketing_version
>> numRows
>> rawDataSize
>> totalSize
>> numFiles
>> numFileErasureCoded
>> 
>> I am afraid that we can not change the name of most of these properties, and 
>> might not be useful to have most of them along with Iceberg statistics 
>> already there. Also my feeling is that this is only the top of the Iceberg 
>> (pun intended :)) so this is why I think we should be more targeted way to 
>> push properties to the Iceberg tables.
>> 
>>> On Dec 2, 2020, at 18:04, Ryan Blue <rb...@netflix.com 
>>> <mailto:rb...@netflix.com>> wrote:
>>> 
>>> Sorry, I accidentally didn’t copy the dev list on this reply. Resending:
>>> 
>>> Also I expect that we want to add Hive write specific configs to table 
>>> level when the general engine independent configuration is not ideal for 
>>> Hive, but every Hive query for a given table should use some specific 
>>> config.
>>> 
>>> Hive may need configuration, but I think these should still be kept in the 
>>> Iceberg table. There is no reason to make Hive config inaccessible from 
>>> other engines. If someone wants to view all of the config for a table from 
>>> Spark, the Hive config should also be included right?
>>> 
>>> 
>>> On Tue, Dec 1, 2020 at 10:36 AM Peter Vary <pv...@cloudera.com 
>>> <mailto:pv...@cloudera.com>> wrote:
>>> I will ask Laszlo if he wants to update his doc.
>>> 
>>> I see both pros and cons of catalog definition in config files. If there is 
>>> an easy default then I do not mind any of the proposed solutions.
>>> 
>>> OTOH I am in favor of the "use prefix for Iceberg table properties" 
>>> solution, because in Hive it is common to add new keys to the property list 
>>> - no restriction is in place (I am not even sure that the currently 
>>> implemented blacklist for preventing to propagate properties to Iceberg 
>>> tables is complete). Also I expect that we want to add Hive write specific 
>>> configs to table level when the general engine independent configuration is 
>>> not ideal for Hive, but every Hive query for a given table should use some 
>>> specific config.
>>> 
>>> Thanks, Peter 
>>> 
>>> Jacques Nadeau <jacq...@dremio.com <mailto:jacq...@dremio.com>> ezt írta 
>>> (időpont: 2020. dec. 1., Ke 17:06):
>>> Would someone be willing to create a document that states the current 
>>> proposal? 
>>> 
>>> It is becoming somewhat difficult to follow this thread. I also worry that 
>>> without a complete statement of the current shape that people may be 
>>> incorrectly thinking they are in alignment.
>>> 
>>> 
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> 
>>> On Tue, Dec 1, 2020 at 5:32 AM Zoltán Borók-Nagy <borokna...@cloudera.com 
>>> <mailto:borokna...@cloudera.com>> wrote:
>>> Thanks, Ryan. I answered inline.
>>> 
>>> On Mon, Nov 30, 2020 at 8:26 PM Ryan Blue <rb...@netflix.com 
>>> <mailto:rb...@netflix.com>> wrote:
>>> This sounds like a good plan overall, but I have a couple of notes:
>>> 
>>> We need to keep in mind that users plug in their own catalogs, so 
>>> iceberg.catalog could be a Glue or Nessie catalog, not just Hive or Hadoop. 
>>> I don’t think it makes much sense to use separate hadoop.catalog and 
>>> hive.catalog values. Those should just be names for catalogs configured in 
>>> Configuration, i.e., via hive-site.xml. We then only need a special value 
>>> for loading Hadoop tables from paths.
>>> About extensibility, I think the usual Hive way is to use Java class names. 
>>> So this way the value for 'iceberg.catalog' could be e.g. 
>>> 'org.apache.iceberg.hive.HiveCatalog'. Then each catalog implementation 
>>> would need to have a factory method that constructs the catalog object from 
>>> a properties object (Map<String, String>). E.g. 
>>> 'org.apache.iceberg.hadoop.HadoopCatalog' would require 
>>> 'iceberg.catalog_location' to be present in properties.
>>> I don’t think that catalog configuration should be kept in table 
>>> properties. A catalog should not be loaded for each table. So I don’t think 
>>> we need iceberg.catalog_location. Instead, we should have a way to define 
>>> catalogs in the Configuration for tables in the metastore to reference.
>>>  I think it makes sense, on the other hand it would make adding new 
>>> catalogs more heavy-weight, i.e. now you'd need to edit configuration files 
>>> and restart/reinit services. Maybe it can be cumbersome in some 
>>> environments.
>>> I’d rather use a prefix to exclude properties from being passed to Iceberg 
>>> than to include them. Otherwise, users don’t know what to do to pass table 
>>> properties from Hive or Impala. If we exclude a prefix or specific 
>>> properties, then everything but the properties reserved for locating the 
>>> table are passed as the user would expect.
>>> I don't have a strong opinion about this, but yeah, maybe this behavior 
>>> would cause the least surprises.
>>> 
>>> On Mon, Nov 30, 2020 at 7:51 AM Zoltán Borók-Nagy <borokna...@apache.org 
>>> <mailto:borokna...@apache.org>> wrote:
>>> Thanks, Peter. I answered inline.
>>> 
>>> On Mon, Nov 30, 2020 at 3:13 PM Peter Vary <pv...@cloudera.com.invalid 
>>> <mailto:pv...@cloudera.com.invalid>> wrote:
>>> Hi Zoltan,
>>> 
>>> Answers below:
>>> 
>>>> On Nov 30, 2020, at 14:19, Zoltán Borók-Nagy 
>>>> <borokna...@cloudera.com.INVALID <mailto:borokna...@cloudera.com.INVALID>> 
>>>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Thanks for the replies. My take for the above questions are as follows
>>>> Should 'iceberg.catalog' be a required property?
>>>> Yeah, I think it would be nice if this would be required to avoid any 
>>>> implicit behavior
>>> Currently we have a Catalogs class to get/initialize/use the different 
>>> Catalogs. At that time the decision was to use HadoopTables as a default 
>>> catalog.
>>> It might be worthwhile to use the same class in Impala as well, so the 
>>> behavior is consistent.
>>>  
>>> Yeah, I think it'd be beneficial for us to use the Iceberg classes whenever 
>>> possible. The Catalogs class is very similar to what we have currently in 
>>> Impala.  
>>>> 'hadoop.catalog' LOCATION and catalog_location
>>>> In Impala we don't allow setting LOCATION for tables stored in 
>>>> 'hadoop.catalog'. But Impala internally sets LOCATION to the Iceberg 
>>>> table's actual location. We were also thinking about using only the table 
>>>> LOCATION, and set it to the catalog location, but we also found it 
>>>> confusing.
>>> It could definitely work, but it is somewhat strange that we have an 
>>> external table location set to an arbitrary path, and we have a different 
>>> location generated by other configs. It would be nice to have the real 
>>> location set in the external table location as well.
>>> 
>>> Impala sets the real Iceberg table location for external tables. E.g. if 
>>> the user issues
>>> 
>>> CREATE EXTERNAL TABLE my_hive_db.iceberg_table_hadoop_catalog
>>> STORED AS ICEBERG
>>> TBLPROPERTIES('iceberg.catalog'='hadoop.catalog',
>>>               'iceberg.catalog_location'='/path/to/hadoop/catalog',
>>>               'iceberg.table_identifier'='namespace1.namespace2.ice_t');
>>> 
>>> If the end user had specified LOCATION, then Impala would have raised an 
>>> error. But the above DDL statement is correct, so Impala loads the iceberg 
>>> table via Iceberg API, then creates the HMS table and sets LOCATION to the 
>>> Iceberg table location (something like 
>>> /path/to/hadoop/catalog/namespace1/namespace2/ice_t).
>>> 
>>> 
>>> I like the flexibility of setting the table_identifier on table level, 
>>> which could help removing naming conflicts. We might want to have this in 
>>> the Iceberg Catalog implementation.
>>>> 'iceberg.table_identifier' for HiveCatalog
>>>> Yeah, it doesn't add much if we only allow using the current HMS. I think 
>>>> it can be only useful if we are allowing external HMSes.
>>>> Moving properties to SERDEPROPERTIES
>>>> I see that these properties are used by the SerDe classes in Hive, but I 
>>>> feel that these properties are just not about serialization and 
>>>> deserialization. And as I see the current SERDEPROPERTIES are things like 
>>>> 'field.delim', 'separatorChar', 'quoteChar', etc. So properties about 
>>>> table loading more naturally belong to TBLPROPERTIES in my opinion.
>>> I have seen it used both ways for HBaseSerDe. (even the wiki page uses both 
>>> :) ). Since Impala prefers TBLPROPERTIES and if we start using prefix for 
>>> separating real Iceberg table properties from other properties, then we can 
>>> keep it at TBLPROPERTIES.
>>> 
>>> In the google doc I also had a comment about prefixing iceberg table 
>>> properties. We could use a prefix like 'iceberg.tblproperties.', and pass 
>>> every property with this prefix to the Iceberg table. Currently Impala 
>>> passes every table property to the Iceberg table.
>>> 
>>> 
>>> 
>>>> Thanks,
>>>>     Zoltan
>>>> 
>>>> 
>>>> On Mon, Nov 30, 2020 at 1:33 PM Peter Vary <pv...@cloudera.com.invalid 
>>>> <mailto:pv...@cloudera.com.invalid>> wrote:
>>>> Hi,
>>>> 
>>>> Based on the discussion below I understand we have the following kinds of 
>>>> properties:
>>>> Iceberg table properties - Engine independent, storage related parameters
>>>> "how to get to" - I think these are mostly Hive table specific properties, 
>>>> since for Spark, the Spark catalog configuration serves for the same 
>>>> purpose. I think the best place for storing these would be the Hive 
>>>> SERDEPROPERTIES, as this describes the access information for the SerDe. 
>>>> Sidenote: I think we should decide if we allow HiveCatalogs pointing to a 
>>>> different HMS and the 'iceberg.table_identifier' would make sense only if 
>>>> we allow having multiple catalogs.
>>>> Query specific properties - These are engine specific and might be mapped 
>>>> to / even override the Iceberg table properties on the engine specific 
>>>> code paths, but currently these properties have independent names and 
>>>> mapped on a case-by-case basis. 
>>>> 
>>>> Based on this:
>>>> Shall we move the "how to get to" properties to SERDEPROPERTIES?
>>>> Shall we define a prefix for setting Iceberg table properties from Hive 
>>>> queries and omitting other engine specific properties?
>>>> 
>>>> Thanks,
>>>> Peter
>>>> 
>>>> 
>>>>> On Nov 27, 2020, at 17:45, Mass Dosage <massdos...@gmail.com 
>>>>> <mailto:massdos...@gmail.com>> wrote:
>>>>> 
>>>>> I like these suggestions, comments inline below on the last round...
>>>>> 
>>>>> On Thu, 26 Nov 2020 at 09:45, Zoltán Borók-Nagy <borokna...@apache.org 
>>>>> <mailto:borokna...@apache.org>> wrote:
>>>>> Hi,
>>>>> 
>>>>> The above aligns with what we did in Impala, i.e. we store information 
>>>>> about table loading in HMS table properties. We are just a bit more 
>>>>> explicit about which catalog to use.
>>>>> We have table property 'iceberg.catalog' to determine the catalog type, 
>>>>> right now the supported values are 'hadoop.tables', 'hadoop.catalog', and 
>>>>> 'hive.catalog'. Additional table properties can be set based on the 
>>>>> catalog type.
>>>>> 
>>>>> So, if the value of 'iceberg.catalog' is
>>>>> 
>>>>> I'm all for renaming this, having "mr" in the property name is confusing.
>>>>>  
>>>>> hadoop.tables
>>>>> the table location is used to load the table
>>>>> The only question I have is should we have this as the default? i.e. if 
>>>>> you don't set a catalog it will assume its HadoopTables and use the 
>>>>> location? Or should we require this property to be here to be consistent 
>>>>> and avoid any "magic"?
>>>>>  
>>>>> hadoop.catalog
>>>>> Required table property 'iceberg.catalog_location' specifies the location 
>>>>> of the hadoop catalog in the file system
>>>>> Optional table property 'iceberg.table_identifier' specifies the table 
>>>>> id. If it's not set, then <database_name>.<table_name> is used as table 
>>>>> identifier
>>>>> I like this as it would allow you to use a different database and table 
>>>>> name in Hive as opposed to the Hadoop Catalog - at the moment they have 
>>>>> to match. The only thing here is that I think Hive requires a table 
>>>>> LOCATION to be set and it's then confusing as there are now two locations 
>>>>> on the table. I'm not sure whether in the Hive storage handler or SerDe 
>>>>> etc. we can get Hive to not require that and maybe even disallow it from 
>>>>> being set. That would probably be best in conjunction with this. Another 
>>>>> solution would be to not have the 'iceberg.catalog_location' property but 
>>>>> instead use the table LOCATION for this but that's a bit confusing from a 
>>>>> Hive point of view.
>>>>> 
>>>>> hive.catalog
>>>>> Optional table property 'iceberg.table_identifier' specifies the table 
>>>>> id. If it's not set, then <database_name>.<table_name> is used as table 
>>>>> identifier
>>>>> We have the assumption that the current Hive metastore stores the table, 
>>>>> i.e. we don't support external Hive metastores currently
>>>>> These sound fine for Hive catalog tables that are created outside of the 
>>>>> automatic Hive table creation (see https://iceberg.apache.org/hive/ 
>>>>> <https://iceberg.apache.org/hive/> -> Using Hive Catalog) we'd just need 
>>>>> to document how you can create these yourself and that one could use a 
>>>>> different Hive database and table etc.
>>>>>  
>>>>> Independent of catalog implementations, but we also have table property 
>>>>> 'iceberg.file_format' to specify the file format for the data files.
>>>>> 
>>>>> OK, I don't think we need that for Hive?
>>>>>  
>>>>> We haven't released it yet, so we are open to changes, but I think these 
>>>>> properties are reasonable and it would be great if we could standardize 
>>>>> the properties across engines that use HMS as the primary metastore of 
>>>>> tables.
>>>>> 
>>>>> 
>>>>> If others agree I think we should create an issue where we document the 
>>>>> above changes so it's very clear what we're doing and can then go an 
>>>>> implement them and update the docs etc.
>>>>>  
>>>>> Cheers,
>>>>>     Zoltan
>>>>> 
>>>>> 
>>>>> On Thu, Nov 26, 2020 at 2:20 AM Ryan Blue <rb...@netflix.com.invalid 
>>>>> <mailto:rb...@netflix.com.invalid>> wrote:
>>>>> Yes, I think that is a good summary of the principles.
>>>>> 
>>>>> #4 is correct because we provide some information that is informational 
>>>>> (Hive schema) or tracked only by the metastore (best-effort current 
>>>>> user). I also agree that it would be good to have a table identifier in 
>>>>> HMS table metadata when loading from an external table. That gives us a 
>>>>> way to handle name conflicts.
>>>>> 
>>>>> On Wed, Nov 25, 2020 at 5:14 PM Jacques Nadeau <jacq...@dremio.com 
>>>>> <mailto:jacq...@dremio.com>> wrote:
>>>>> Minor error, my last example should have been:
>>>>> 
>>>>> db1.table1_etl_branch => nessie.folder1.folder2.folder3.table1@etl_branch
>>>>> 
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>> 
>>>>> 
>>>>> On Wed, Nov 25, 2020 at 4:56 PM Jacques Nadeau <jacq...@dremio.com 
>>>>> <mailto:jacq...@dremio.com>> wrote:
>>>>> I agree with Ryan on the core principles here. As I understand them:
>>>>> Iceberg metadata describes all properties of a table
>>>>> Hive table properties describe "how to get to" Iceberg metadata (which 
>>>>> catalog + possibly ptr, path, token, etc)
>>>>> There could be default "how to get to" information set at a global level
>>>>> Best-effort schema should stored be in the table properties in HMS. This 
>>>>> should be done for information schema retrieval purposes within Hive but 
>>>>> should be ignored during Hive/other tool execution.
>>>>> Is that a fair summary of your statements Ryan (except 4, which I just 
>>>>> added)?
>>>>> 
>>>>> One comment I have on #2 is that for different catalogs and use cases, I 
>>>>> think it can be somewhat more complex where it would be desirable for a 
>>>>> table that initially existed without Hive that was later exposed in Hive 
>>>>> to support a ptr/path/token for how the table is named externally. For 
>>>>> example, in a Nessie context we support arbitrary paths for an Iceberg 
>>>>> table (such as folder1.folder2.folder3.table1). If you then want to 
>>>>> expose that table to Hive, you might have this mapping for #2
>>>>> 
>>>>> db1.table1 => nessie:folder1.folder2.folder3.table1
>>>>> 
>>>>> Similarly, you might want to expose a particular branch version of a 
>>>>> table. So it might say:
>>>>> 
>>>>> db1.table1_etl_branch => nessie.folder1@etl_branch
>>>>> 
>>>>> Just saying that the address to the table in the catalog could itself 
>>>>> have several properties. The key being that no matter what those are, we 
>>>>> should follow #1 and only store properties that are about the ptr, not 
>>>>> the content/metadata.
>>>>> 
>>>>> Lastly, I believe #4 is the case but haven't tested it. Can someone 
>>>>> confirm that it is true? And that it is possible/not problematic?
>>>>> 
>>>>> 
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>> 
>>>>> 
>>>>> On Wed, Nov 25, 2020 at 4:28 PM Ryan Blue <rb...@netflix.com.invalid 
>>>>> <mailto:rb...@netflix.com.invalid>> wrote:
>>>>> Thanks for working on this, Laszlo. I’ve been thinking about these 
>>>>> problems as well, so this is a good time to have a discussion about Hive 
>>>>> config.
>>>>> 
>>>>> I think that Hive configuration should work mostly like other engines, 
>>>>> where different configurations are used for different purposes. Different 
>>>>> purposes means that there is not a global configuration priority. 
>>>>> Hopefully, I can explain how we use the different config sources 
>>>>> elsewhere to clarify.
>>>>> 
>>>>> Let’s take Spark as an example. Spark uses Hadoop, so it has a Hadoop 
>>>>> Configuration, but it also has its own global configuration. There are 
>>>>> also Iceberg table properties, and all of the various Hive properties if 
>>>>> you’re tracking tables with a Hive MetaStore.
>>>>> 
>>>>> The first step is to simplify where we can, so we effectively eliminate 2 
>>>>> sources of config:
>>>>> 
>>>>> The Hadoop Configuration is only used to instantiate Hadoop classes, like 
>>>>> FileSystem. Iceberg should not use it for any other config.
>>>>> Config in the Hive MetaStore is only used to identify that a table is 
>>>>> Iceberg and point to its metadata location. All other config in HMS is 
>>>>> informational. For example, the input format is FileInputFormat so that 
>>>>> non-Iceberg readers cannot actually instantiate the format (it’s 
>>>>> abstract) but it is available so they also don’t fail trying to load the 
>>>>> class. Table-specific config should not be stored in table or serde 
>>>>> properties.
>>>>> That leaves Spark configuration and Iceberg table configuration.
>>>>> 
>>>>> Iceberg differs from other tables because it is opinionated: data 
>>>>> configuration should be maintained at the table level. This is cleaner 
>>>>> for users because config is standardized across engines and in one place. 
>>>>> And it also enables services that analyze a table and update its 
>>>>> configuration to tune options that users almost never do, like row group 
>>>>> or stripe size in the columnar formats. Iceberg table configuration is 
>>>>> used to configure table-specific concerns and behavior.
>>>>> 
>>>>> Spark configuration is used for engine-specific concerns, and runtime 
>>>>> overrides. A good example of an engine-specific concern is the catalogs 
>>>>> that are available to load Iceberg tables. Spark has a way to load and 
>>>>> configure catalog implementations and Iceberg uses that for all 
>>>>> catalog-level config. Runtime overrides are things like target split 
>>>>> size. Iceberg has a table-level default split size in table properties, 
>>>>> but this can be overridden by a Spark option for each table, as well as 
>>>>> an option passed to the individual read. Note that these necessarily have 
>>>>> different config names for how they are used: Iceberg uses 
>>>>> read.split.target-size and the read-specific option is target-size.
>>>>> 
>>>>> Applying this to Hive is a little strange for a couple reasons. First, 
>>>>> Hive’s engine configuration is a Hadoop Configuration. As a result, I 
>>>>> think the right place to store engine-specific config is there, including 
>>>>> Iceberg catalogs using a strategy similar to what Spark does: what 
>>>>> external Iceberg catalogs are available and their configuration should 
>>>>> come from the HiveConf.
>>>>> 
>>>>> The second way Hive is strange is that Hive needs to use its own 
>>>>> MetaStore to track Hive table concerns. The MetaStore may have tables 
>>>>> created by an Iceberg HiveCatalog, and Hive also needs to be able to load 
>>>>> tables from other Iceberg catalogs by creating table entries for them.
>>>>> 
>>>>> Here’s how I think Hive should work:
>>>>> 
>>>>> There should be a default HiveCatalog that uses the current MetaStore URI 
>>>>> to be used for HiveCatalog tables tracked in the MetaStore
>>>>> Other catalogs should be defined in HiveConf
>>>>> HMS table properties should be used to determine how to load a table: 
>>>>> using a Hadoop location, using the default metastore catalog, or using an 
>>>>> external Iceberg catalog
>>>>> If there is a metadata_location, then use the HiveCatalog for this 
>>>>> metastore (where it is tracked)
>>>>> If there is a catalog property, then load that catalog and use it to load 
>>>>> the table identifier, or maybe an identifier from HMS table properties
>>>>> If there is no catalog or metadata_location, then use HadoopTables to 
>>>>> load the table location as an Iceberg table
>>>>> This would make it possible to access all types of Iceberg tables in the 
>>>>> same query, and would match how Spark and Flink configure catalogs. Other 
>>>>> than the configuration above, I don’t think that config in HMS should be 
>>>>> used at all, like how the other engines work. Iceberg is the source of 
>>>>> truth for table metadata, HMS stores how to load the Iceberg table, and 
>>>>> HiveConf defines the catalogs (or runtime overrides).
>>>>> 
>>>>> This isn’t quite how configuration works right now. Currently, the 
>>>>> catalog is controlled by a HiveConf property, iceberg.mr.catalog. If that 
>>>>> isn’t set, HadoopTables will be used to load table locations. If it is 
>>>>> set, then that catalog will be used to load all tables by name. This 
>>>>> makes it impossible to load tables from different catalogs at the same 
>>>>> time. That’s why I think the Iceberg catalog for a table should be stored 
>>>>> in HMS table properties.
>>>>> 
>>>>> I should also explain iceberg.hive.engine.enabled flag, but I think this 
>>>>> is long enough for now.
>>>>> 
>>>>> rb
>>>>> 
>>>>> 
>>>>> On Wed, Nov 25, 2020 at 1:41 AM Laszlo Pinter 
>>>>> <lpin...@cloudera.com.invalid <mailto:lpin...@cloudera.com.invalid>> 
>>>>> wrote:
>>>>> Hi All,
>>>>> 
>>>>> I would like to start a discussion, how should we handle properties from 
>>>>> various sources like Iceberg, Hive or global configuration. I've put 
>>>>> together a short document 
>>>>> <https://docs.google.com/document/d/1tyD7mGp_hh0dx9N_Ax9kj5INkg7Wzpj9XQ5t2-7AwNs/edit?usp=sharing>,
>>>>>  please have a look and let me know what you think.
>>>>> 
>>>>> Thanks,
>>>>> Laszlo
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>> 
> 

Reply via email to