Hi,
Could you please clarify, if you run all actions using IGFS, but instaed of
fs.appedn use Hive, like:
insert into table stocks PARTITION (years=2004,months=12,days=3)
values('AAPL',1501236980,120.34);
Does select work this time?
Thanks,
Mikhail.
2017-08-04 12:56 GMT+03:00 csumi <[email protected]>:
> Let me try to clear here with the sequence of steps performed.
> - Created table with partition through hive using below query. It
> creates a
> directory in hdfs.
> create table stocks3 (stock string, time timestamp, price float)
> PARTITIONED BY (years bigint, months bigint, days bigint) ROW FORMAT
> DELIMITED FIELDS TERMINATED BY ',';
> - Then I get streaming data and using IgniteFileSystem's
> append/create
> method, it gets saved to ignited hadoop.
> - Run below select query. No result returned
> select * from stocks3;
> - Stop ignite and run the select again on hive. No result with below
> logs
>
> hive> select * from stocks3;
> 17/08/04 14:59:08 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:08 INFO session.SessionState: Updating thread name to
> b5e3e924-e46a-481c-8aef-30d48605a2da main
> 17/08/04 14:59:08 WARN operation.Operation: Unable to create operation log
> file:
> D:\tmp\hive\<user>\operation_logs\b5e3e924-e46a-481c-8aef-
> 30d48605a2da\137adad6-ea23-462c-a414-6ce260e5bd49
> java.io.IOException: The system cannot find the path specified
> at java.io.WinNTFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hive.service.cli.operation.Operation.
> createOperationLog(Operation.java:237)
> at
> org.apache.hive.service.cli.operation.Operation.beforeRun(
> Operation.java:279)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:314)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.
> executeStatementInternal(HiveSessionImpl.java:499)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(
> HiveSessionImpl.java:486)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(
> CLIService.java:295)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(
> ThriftCLIService.java:506)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:491)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(
> HiveConnection.java:1412)
> at com.sun.proxy.$Proxy21.ExecuteStatement(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(
> HiveStatement.java:308)
> at
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
> at
> org.apache.hive.beeline.Commands.executeInternal(Commands.java:988)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1160)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1074)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1148)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:976)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
> at org.apache.hive.beeline.cli.HiveCli.runWithArgs(HiveCli.
> java:35)
> at org.apache.hive.beeline.cli.HiveCli.main(HiveCli.java:29)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:491)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> 17/08/04 14:59:08 INFO ql.Driver: Compiling
> command(queryId=<user>_20170804145908_b270c978-ab00-
> 4160-a2a6-c19b42eab676):
> select * from stocks3
> 17/08/04 14:59:08 INFO parse.CalcitePlanner: Starting Semantic Analysis
> 17/08/04 14:59:08 INFO parse.CalcitePlanner: Completed phase 1 of Semantic
> Analysis
> 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for source tables
> 17/08/04 14:59:08 INFO metastore.HiveMetaStore: 0: get_table : db=yt
> tbl=stocks3
> 17/08/04 14:59:08 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
> cmd=get_table : db=yt tbl=stocks3
> 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for subqueries
> 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for destination
> tables
> 17/08/04 14:59:09 INFO ql.Context: New scratch dir is
> hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-
> 481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_8316159022041430928-1
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: Completed getting MetaData in
> Semantic Analysis
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for source tables
> 17/08/04 14:59:09 INFO metastore.HiveMetaStore: 0: get_table : db=yt
> tbl=stocks3
> 17/08/04 14:59:09 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
> cmd=get_table : db=yt tbl=stocks3
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for subqueries
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for destination
> tables
> 17/08/04 14:59:09 INFO ql.Context: New scratch dir is
> hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-
> 481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_8316159022041430928-1
> 17/08/04 14:59:09 INFO common.FileUtils: Creating directory if it doesn't
> exist:
> hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-
> 481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_831615902204143
> 0928-1/-mr-10001/.hive-staging_hive_2017-08-04_14-59-
> 08_935_8316159022041430928-1
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: CBO Succeeded; optimized
> logical plan.
> 17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for FS(2)
> 17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for SEL(1)
> 17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for TS(0)
> 17/08/04 14:59:09 INFO metastore.HiveMetaStore: 0: get_partitions : db=yt
> tbl=stocks3
> 17/08/04 14:59:09 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
> cmd=get_partitions : db=yt tbl=stocks3
> 17/08/04 14:59:09 INFO parse.CalcitePlanner: Completed plan generation
> 17/08/04 14:59:09 INFO ql.Driver: Semantic Analysis Completed
> 17/08/04 14:59:09 INFO ql.Driver: Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:stocks3.stock, type:string,
> comment:null), FieldSchema(name:stocks3.time, type:timestamp,
> comment:null),
> FieldSchema(name:stocks3.price, type:float, comment:null),
> FieldSchema(name:stocks3.years, type:bigint, comment:null),
> FieldSchema(name:stocks3.months, type:bigint, comment:null),
> FieldSchema(name:sto
> cks3.days, type:bigint, comment:null)], properties:null)
> 17/08/04 14:59:09 INFO exec.TableScanOperator: Initializing operator TS[0]
> 17/08/04 14:59:09 INFO exec.SelectOperator: Initializing operator SEL[1]
> 17/08/04 14:59:09 INFO exec.SelectOperator: SELECT
> struct<stock:string,time:timestamp,price:float,years:
> bigint,months:bigint,days:bigint>
> 17/08/04 14:59:09 INFO exec.ListSinkOperator: Initializing operator
> LIST_SINK[3]
> 17/08/04 14:59:09 INFO ql.Driver: EXPLAIN output for queryid
> <user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676 : STAGE
> DEPENDENCIES:
> Stage-0 is a root stage [FETCH]
>
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: stocks3
> GatherStats: false
> Select Operator
> expressions: stock (type: string), time (type: timestamp),
> price
> (type: float), years (type: bigint), months (type: bigint), days (type:
> bigint)
> outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
> ListSink
>
>
> 17/08/04 14:59:09 INFO ql.Driver: Completed compiling
> command(queryId=<user>_20170804145908_b270c978-ab00-
> 4160-a2a6-c19b42eab676);
> Time taken: 0.586 seconds
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
> 17/08/04 14:59:09 INFO ql.Driver: Concurrency mode is disabled, not
> creating
> a lock manager
> 17/08/04 14:59:09 INFO ql.Driver: Executing
> command(queryId=<user>_20170804145908_b270c978-ab00-
> 4160-a2a6-c19b42eab676):
> select * from stocks3
> 17/08/04 14:59:09 INFO ql.Driver: Completed executing
> command(queryId=<user>_20170804145908_b270c978-ab00-
> 4160-a2a6-c19b42eab676);
> Time taken: 0.002 seconds
> OK
> 17/08/04 14:59:09 INFO ql.Driver: OK
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
> b5e3e924-e46a-481c-8aef-30d48605a2da main
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
> b5e3e924-e46a-481c-8aef-30d48605a2da Thread-52
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to
> Thread-52
> 17/08/04 14:59:09 WARN thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated
> with operation handle: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=137adad6-ea23-462c-a414-6ce260e5bd49]
>
> at
> org.apache.hive.service.cli.operation.OperationManager.
> getOperationLogRowSet(OperationManager.java:324)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.
> fetchResults(HiveSessionImpl.java:849)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:505)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.
> FetchResults(ThriftCLIService.java:698)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:491)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(
> HiveConnection.java:1412)
> at com.sun.proxy.$Proxy21.FetchResults(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:871)
> at
> org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:842)
> at
> org.apache.hive.beeline.Commands.showRemainingLogsIfAny(
> Commands.java:1211)
> at org.apache.hive.beeline.Commands.access$200(Commands.java:68)
> at org.apache.hive.beeline.Commands$2.run(Commands.java:1187)
> at java.lang.Thread.run(Thread.java:724)
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
> b5e3e924-e46a-481c-8aef-30d48605a2da main
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
> 17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
> No rows selected (0.612 seconds)
> 17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
> log id: b5e3e924-e46a-481c-8aef-30d48605a2da
>
> - Data created in HDFS
> (http://localhost:50070/explorer.html#/usr/hive/
> warehouse/yt.db/stocks3/years=2017/months=7/days=4)
> is as follows:
> -rw-r--r-- <user> supergroup 44 B Aug 04 14:48 3
> 128 MB 1
>
> - Start ignite
> - Run insert query as below
> insert into table stocks3 PARTITION (years=2004,months=12,days=3)
> values('AAPL',1501236980,120.34);
> - New partition created
> http://localhost:50070/explorer.html#/usr/hive/
> warehouse/yt.db/stocks3/years=2004/months=12/days=3
> -rwxr-xr-x <user> supergroup 15 B Aug 04 15:16 1
> 128 MB 000000_0
> - Run below select query which is returning the row inserting using
> the aboe
> insert.
> select * from stocks3;
> - Now insert new row in the table to the partition created through
> code
> earlier
> insert into table stocks3 PARTITION (years=2017,months=7,days=4)
> values('AAPL',1501236980,120.34);
> - Run select query again. Now it gives 3 rows. Two of which were
> inserted
> using insert command and one through code which was not coming in select
> query earlier.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-
> data-gets-saved-to-Hive-partitioned-table-tp15725p15991.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
