Hi Community, Hive Version: 3.1.2
We are working on building a Hive Avro table on a few Avro files. I am able to succesfully create the table and query it when I have the Avro schema definition(avsc) file on hdfs with no issues. However, when trying to load the same schema from a rest API(as mentioned in https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-CreatingAvro-backedHivetables), hive throws an exception and fails to create the table. *Sample table:* CREATE EXTERNAL TABLE ed_avro_1 STORED AS AVRO LOCATION '/tmp/sample/yyyymmdd=20200206' TBLPROPERTIES ('avro.schema.literal'='http:// <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing'); When launched hive in INFO mode below is the trace of the problem which looks like Hive is interpreting the URL as a file name and throwing out a FileNotFoundException. I have tried using avro.schema.literal instead of avro.schema.url, however it turns out that hive is interpreting URL as a String and throwing a jackson parsing error. Can anyone help look into this? Is this a bug in Hive-3.1.2? Any details will be of great help. Thanks, Ravi StackTrace: > 2020-06-26T00:06:03,283 INFO [main] org.apache.hadoop.hive.conf.HiveConf - >> Using the default value passed in for log id: >> 646da35b-84b0-43aa-9b68-5d668ebbfc36 > > 2020-06-26T00:06:03,283 INFO [main] >> org.apache.hadoop.hive.ql.session.SessionState - Updating thread name to >> 646da35b-84b0-43aa-9b68-5d668ebbfc36 main > > 2020-06-26T00:06:03,286 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Compiling >> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2): >> CREATE EXTERNAL TABLE ed_avro_1 > > STORED AS AVRO > > LOCATION '/tmp/event_detail/yyyymmdd=20200206' > > TBLPROPERTIES ('avro.schema.url'='http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing') > > 2020-06-26T00:06:03,630 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Concurrency mode is disabled, not creating a lock manager > > 2020-06-26T00:06:03,638 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> parse.CalcitePlanner: Starting Semantic Analysis > > 2020-06-26T00:06:03,669 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for >> session context : HiveAuthzSessionContext >> [sessionString=646da35b-84b0-43aa-9b68-5d668ebbfc36, clientType=HIVECLI] > > 2020-06-26T00:06:03,673 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will >> be ignored, since hive.security.authorization.manager is set to instance of >> HiveAuthorizerFactory. > > 2020-06-26T00:06:03,673 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStoreClient: Mestastore configuration >> metastore.filter.hook changed from >> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to >> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook > > 2020-06-26T00:06:03,675 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... > > 2020-06-26T00:06:03,675 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Cleaning up thread >> local RawStore... > > 2020-06-26T00:06:03,676 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore > > 2020-06-26T00:06:03,676 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Done cleaning up >> thread local RawStore > > 2020-06-26T00:06:03,680 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: Opening raw store with implementation >> class:org.apache.hadoop.hive.metastore.ObjectStore > > 2020-06-26T00:06:03,680 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to >> unsupported value null . Setting it to value: ignored > > 2020-06-26T00:06:03,681 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: ObjectStore, initialize called > > 2020-06-26T00:06:03,691 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL > > 2020-06-26T00:06:03,691 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: Initialized ObjectStore > > 2020-06-26T00:06:03,692 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class >> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hdfs >> (auth:SIMPLE) retries=1 delay=1 lifetime=0 > > 2020-06-26T00:06:03,704 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> parse.CalcitePlanner: Creating table scratch.ed_avro_1 position=22 > > 2020-06-26T00:06:03,719 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to >> unsupported value null . Setting it to value: ignored > > 2020-06-26T00:06:03,719 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: ObjectStore, initialize called > > 2020-06-26T00:06:03,730 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL > > 2020-06-26T00:06:03,730 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.ObjectStore: Initialized ObjectStore > > 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class >> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hdfs >> (auth:SIMPLE) retries=1 delay=1 lifetime=0 > > 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: get_database: @hive#scratch > > 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=get_database: >> @hive#scratch > > 2020-06-26T00:06:03,754 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Semantic Analysis Completed (retrial = false) > > 2020-06-26T00:06:03,762 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) > > 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Completed compiling >> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2); >> Time taken: 0.477 seconds > > 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> reexec.ReExecDriver: Execution #1 of query > > 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Concurrency mode is disabled, not creating a lock manager > > 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Executing >> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2): >> CREATE EXTERNAL TABLE ed_avro_1 > > STORED AS AVRO > > LOCATION '/tmp/event_detail/yyyymmdd=20200206' > > TBLPROPERTIES ('avro.schema.url'='http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing') > > 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> ql.Driver: Starting task [Stage-0:DDL] in serial mode > > 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStoreClient: Mestastore configuration >> metastore.filter.hook changed from >> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook >> to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl > > 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... > > 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Cleaning up thread >> local RawStore... > > 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore > > 2020-06-26T00:06:03,766 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Done cleaning up >> thread local RawStore > > 2020-06-26T00:06:03,790 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> avro.AvroSerDe: AvroSerde::initialize(): Preset value of >> avro.schema.literal == null > > 2020-06-26T00:06:03,809 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> avro.AvroSerDe: Encountered AvroSerdeException determining schema. >> Returning signal schema to indicate problem > > org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read >> schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:146) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:197) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:111) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:84) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[?:1.8.0_252] > > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[?:1.8.0_252] > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:1.8.0_252] > > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] > > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >> ~[hadoop-common-3.1.2.jar:?] > > Caused by: java.io.FileNotFoundException: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896) >> ~[?:1.8.0_252] > > at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) >> ~[?:1.8.0_252] > > at >> org.apache.hadoop.fs.http.AbstractHttpFileSystem.open(AbstractHttpFileSystem.java:61) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.fs.http.HttpFileSystem.open(HttpFileSystem.java:23) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899) >> ~[hadoop-common-3.1.2.jar:?] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:169) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139) >> ~[hive-exec-3.1.2.jar:3.1.2] > > ... 35 more > > 2020-06-26T00:06:03,871 ERROR [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> hive.log: error in initSerDe: org.apache.hadoop.hive.serde2.SerDeException >> Encountered AvroSerdeException determining schema. Returning signal schema >> to indicate problem: Unable to read schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > org.apache.hadoop.hive.serde2.SerDeException: Encountered >> AvroSerdeException determining schema. Returning signal schema to indicate >> problem: Unable to read schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:543) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[?:1.8.0_252] > > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[?:1.8.0_252] > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:1.8.0_252] > > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] > > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >> ~[hadoop-common-3.1.2.jar:?] > > 2020-06-26T00:06:03,874 ERROR [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> metadata.Table: Unable to get field from serde: >> org.apache.hadoop.hive.serde2.avro.AvroSerDe > > java.lang.RuntimeException: >> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException >> Encountered AvroSerdeException determining schema. Returning signal schema >> to indicate problem: Unable to read schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing) > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:291) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[?:1.8.0_252] > > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[?:1.8.0_252] > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:1.8.0_252] > > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] > > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >> ~[hadoop-common-3.1.2.jar:?] > > Caused by: org.apache.hadoop.hive.metastore.api.MetaException: >> org.apache.hadoop.hive.serde2.SerDeException Encountered AvroSerdeException >> determining schema. Returning signal schema to indicate problem: Unable to >> read schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >> ~[hive-exec-3.1.2.jar:3.1.2] > > ... 28 more > > 2020-06-26T00:06:03,874 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> avro.AvroSerDe: AvroSerde::initialize(): Preset value of >> avro.schema.literal == null > > 2020-06-26T00:06:03,878 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >> avro.AvroSerDe: Encountered AvroSerdeException determining schema. >> Returning signal schema to indicate problem > > org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read >> schema from given path: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:146) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:197) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:111) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:84) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:900) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >> ~[hive-cli-3.1.2.jar:3.1.2] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[?:1.8.0_252] > > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[?:1.8.0_252] > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[?:1.8.0_252] > > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] > > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >> ~[hadoop-common-3.1.2.jar:?] > > Caused by: java.io.FileNotFoundException: http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing > > at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896) >> ~[?:1.8.0_252] > > at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) >> ~[?:1.8.0_252] > > at >> org.apache.hadoop.fs.http.AbstractHttpFileSystem.open(AbstractHttpFileSystem.java:61) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.fs.http.HttpFileSystem.open(HttpFileSystem.java:23) >> ~[hadoop-common-3.1.2.jar:?] > > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899) >> ~[hadoop-common-3.1.2.jar:?] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:169) >> ~[hive-exec-3.1.2.jar:3.1.2] > > at >> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139) >> ~[hive-exec-3.1.2.jar:3.1.2] > > ... 33 more > > Thanks, Ravi
