Let me try to clear here with the sequence of steps performed.
- Created table with partition through hive using below query. It creates
a
directory in hdfs.
create table stocks3 (stock string, time timestamp, price float)
PARTITIONED BY (years bigint, months bigint, days bigint) ROW FORMAT
DELIMITED FIELDS TERMINATED BY ',';
- Then I get streaming data and using IgniteFileSystem's append/create
method, it gets saved to ignited hadoop.
- Run below select query. No result returned
select * from stocks3;
- Stop ignite and run the select again on hive. No result with below logs
hive> select * from stocks3;
17/08/04 14:59:08 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:08 INFO session.SessionState: Updating thread name to
b5e3e924-e46a-481c-8aef-30d48605a2da main
17/08/04 14:59:08 WARN operation.Operation: Unable to create operation log
file:
D:\tmp\hive\<user>\operation_logs\b5e3e924-e46a-481c-8aef-30d48605a2da\137adad6-ea23-462c-a414-6ce260e5bd49
java.io.IOException: The system cannot find the path specified
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:237)
at
org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:279)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:314)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:499)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:486)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at
org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1412)
at com.sun.proxy.$Proxy21.ExecuteStatement(Unknown Source)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:308)
at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
at
org.apache.hive.beeline.Commands.executeInternal(Commands.java:988)
at org.apache.hive.beeline.Commands.execute(Commands.java:1160)
at org.apache.hive.beeline.Commands.sql(Commands.java:1074)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1148)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:976)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at org.apache.hive.beeline.cli.HiveCli.runWithArgs(HiveCli.java:35)
at org.apache.hive.beeline.cli.HiveCli.main(HiveCli.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/08/04 14:59:08 INFO ql.Driver: Compiling
command(queryId=<user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676):
select * from stocks3
17/08/04 14:59:08 INFO parse.CalcitePlanner: Starting Semantic Analysis
17/08/04 14:59:08 INFO parse.CalcitePlanner: Completed phase 1 of Semantic
Analysis
17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for source tables
17/08/04 14:59:08 INFO metastore.HiveMetaStore: 0: get_table : db=yt
tbl=stocks3
17/08/04 14:59:08 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
cmd=get_table : db=yt tbl=stocks3
17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for subqueries
17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for destination
tables
17/08/04 14:59:09 INFO ql.Context: New scratch dir is
hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_8316159022041430928-1
17/08/04 14:59:09 INFO parse.CalcitePlanner: Completed getting MetaData in
Semantic Analysis
17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for source tables
17/08/04 14:59:09 INFO metastore.HiveMetaStore: 0: get_table : db=yt
tbl=stocks3
17/08/04 14:59:09 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
cmd=get_table : db=yt tbl=stocks3
17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for subqueries
17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for destination
tables
17/08/04 14:59:09 INFO ql.Context: New scratch dir is
hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_8316159022041430928-1
17/08/04 14:59:09 INFO common.FileUtils: Creating directory if it doesn't
exist:
hdfs://localhost:9000/tmp/hive/<user>/b5e3e924-e46a-481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_831615902204143
0928-1/-mr-10001/.hive-staging_hive_2017-08-04_14-59-08_935_8316159022041430928-1
17/08/04 14:59:09 INFO parse.CalcitePlanner: CBO Succeeded; optimized
logical plan.
17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for FS(2)
17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for SEL(1)
17/08/04 14:59:09 INFO ppd.OpProcFactory: Processing for TS(0)
17/08/04 14:59:09 INFO metastore.HiveMetaStore: 0: get_partitions : db=yt
tbl=stocks3
17/08/04 14:59:09 INFO HiveMetaStore.audit: ugi=<user> ip=unknown-ip-addr
cmd=get_partitions : db=yt tbl=stocks3
17/08/04 14:59:09 INFO parse.CalcitePlanner: Completed plan generation
17/08/04 14:59:09 INFO ql.Driver: Semantic Analysis Completed
17/08/04 14:59:09 INFO ql.Driver: Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:stocks3.stock, type:string,
comment:null), FieldSchema(name:stocks3.time, type:timestamp, comment:null),
FieldSchema(name:stocks3.price, type:float, comment:null),
FieldSchema(name:stocks3.years, type:bigint, comment:null),
FieldSchema(name:stocks3.months, type:bigint, comment:null),
FieldSchema(name:sto
cks3.days, type:bigint, comment:null)], properties:null)
17/08/04 14:59:09 INFO exec.TableScanOperator: Initializing operator TS[0]
17/08/04 14:59:09 INFO exec.SelectOperator: Initializing operator SEL[1]
17/08/04 14:59:09 INFO exec.SelectOperator: SELECT
struct<stock:string,time:timestamp,price:float,years:bigint,months:bigint,days:bigint>
17/08/04 14:59:09 INFO exec.ListSinkOperator: Initializing operator
LIST_SINK[3]
17/08/04 14:59:09 INFO ql.Driver: EXPLAIN output for queryid
<user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676 : STAGE
DEPENDENCIES:
Stage-0 is a root stage [FETCH]
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: stocks3
GatherStats: false
Select Operator
expressions: stock (type: string), time (type: timestamp), price
(type: float), years (type: bigint), months (type: bigint), days (type:
bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
ListSink
17/08/04 14:59:09 INFO ql.Driver: Completed compiling
command(queryId=<user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676);
Time taken: 0.586 seconds
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
17/08/04 14:59:09 INFO ql.Driver: Concurrency mode is disabled, not creating
a lock manager
17/08/04 14:59:09 INFO ql.Driver: Executing
command(queryId=<user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676):
select * from stocks3
17/08/04 14:59:09 INFO ql.Driver: Completed executing
command(queryId=<user>_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676);
Time taken: 0.002 seconds
OK
17/08/04 14:59:09 INFO ql.Driver: OK
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
b5e3e924-e46a-481c-8aef-30d48605a2da main
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
b5e3e924-e46a-481c-8aef-30d48605a2da Thread-52
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to
Thread-52
17/08/04 14:59:09 WARN thrift.ThriftCLIService: Error fetching results:
org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated
with operation handle: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=137adad6-ea23-462c-a414-6ce260e5bd49]
at
org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:324)
at
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:849)
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:505)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:698)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at
org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1412)
at com.sun.proxy.$Proxy21.FetchResults(Unknown Source)
at
org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:871)
at
org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:842)
at
org.apache.hive.beeline.Commands.showRemainingLogsIfAny(Commands.java:1211)
at org.apache.hive.beeline.Commands.access$200(Commands.java:68)
at org.apache.hive.beeline.Commands$2.run(Commands.java:1187)
at java.lang.Thread.run(Thread.java:724)
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Updating thread name to
b5e3e924-e46a-481c-8aef-30d48605a2da main
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
17/08/04 14:59:09 INFO session.SessionState: Resetting thread name to main
No rows selected (0.612 seconds)
17/08/04 14:59:09 INFO conf.HiveConf: Using the default value passed in for
log id: b5e3e924-e46a-481c-8aef-30d48605a2da
- Data created in HDFS
(http://localhost:50070/explorer.html#/usr/hive/warehouse/yt.db/stocks3/years=2017/months=7/days=4)
is as follows:
-rw-r--r-- <user> supergroup 44 B Aug 04 14:48 3 128 MB
1
- Start ignite
- Run insert query as below
insert into table stocks3 PARTITION (years=2004,months=12,days=3)
values('AAPL',1501236980,120.34);
- New partition created
http://localhost:50070/explorer.html#/usr/hive/warehouse/yt.db/stocks3/years=2004/months=12/days=3
-rwxr-xr-x <user> supergroup 15 B Aug 04 15:16 1
128 MB 000000_0
- Run below select query which is returning the row inserting using the
aboe
insert.
select * from stocks3;
- Now insert new row in the table to the partition created through code
earlier
insert into table stocks3 PARTITION (years=2017,months=7,days=4)
values('AAPL',1501236980,120.34);
- Run select query again. Now it gives 3 rows. Two of which were inserted
using insert command and one through code which was not coming in select
query earlier.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p15991.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
