[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931588#comment-15931588
]
Hive QA commented on HIVE-12274:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859441/HIVE-12274.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10358 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=11)
[skewjoinopt15.q,udf_elt.q,join44.q,index_auto_partitioned.q,inputddl2.q,drop_table.q,udf_testlength.q,auto_join33.q,parquet_ppd_varchar.q,udf_sha2.q,groupby5_map_skew.q,merge4.q,storage_format_descriptor.q,masking_1_newdb.q,vector_leftsemi_mapjoin.q,multi_column_in_single.q,cbo_rp_semijoin.q,char_2.q,repl_1_drop.q,non_ascii_literal2.q,ppd_clusterby.q,load_part_authsuccess.q,auto_sortmerge_join_15.q,explain_rearrange.q,varchar_union1.q,vector_udf2.q,groupby_constcolval.q,annotate_stats_limit.q,explainanalyze_5.q,distinct_windowing.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=19)
[cp_mj_rc.q,order.q,udf_bitwise_shiftleft.q,insert_values_non_partitioned.q,skewjoinopt10.q,extrapolate_part_stats_date.q,udf_sin.q,vectorized_math_funcs.q,join45.q,join14.q,query_result_fileformat.q,orc_merge1.q,cbo_union_view.q,tez_union_decimal.q,nonmr_fetch.q,order_null.q,cbo_rp_views.q,lvj_mapjoin.q,insert_acid_dynamic_partition.q,skewjoinopt6.q,cbo_rp_cross_product_check_2.q,union_remove_19.q,mapreduce7.q,spark_use_op_stats.q,update_two_cols.q,update_where_no_match.q,correlationoptimizer11.q,stats_empty_partition.q,outer_join_ppr.q,authorization_2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=22)
[multi_insert_mixed.q,bucketcontext_5.q,specialChar.q,union_remove_10.q,skewjoinopt21.q,bucketmapjoin_negative.q,input_part0.q,vector_join_nulls.q,udf_hex.q,reducesink_dedup.q,masking_acid_no_masking.q,smb_mapjoin_4.q,mapreduce2.q,input16.q,udf_in_file.q,vector_empty_where.q,udf_variance.q,join42.q,limit_join_transpose.q,auto_join12.q,skewjoin.q,cte_mat_3.q,parenthesis_star_by.q,vector_decimal_round_2.q,udf_conv.q,column_names_with_leading_and_trailing_spaces.q,vectorized_mapjoin2.q,union_stats.q,nullgroup4.q,authorization_view_disable_cbo_2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=6)
[ptf_general_queries.q,correlationoptimizer9.q,cross_join_merge.q,parquet_timestamp_conversion.q,sample2.q,parquet_decimal.q,join1.q,bucket_if_with_path_filter.q,join32_lessSize.q,combine2.q,escape3.q,windowing_range_multiorder.q,cte_mat_4.q,udf_weekofyear.q,masking_disablecbo_4.q,char_pad_convert.q,groupby9.q,udaf_covar_samp.q,column_table_stats_orc.q,parquet_columnar.q,skewjoinopt18.q,colstats_all_nulls.q,union_remove_18.q,groupby_duplicate_key.q,pointlookup3.q,orc_remove_cols.q,udf_classloader.q,subq2.q,ctas.q,setop_subq.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=154)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=171)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4238/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859441 - PreCommit-HIVE-Build
> Increase width of columns used for general configuration in the metastore.
> --------------------------------------------------------------------------
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 2.0.0
> Reporter: Elliot West
> Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
