kylin log file is to large
Hi: I run kylin instance in mine test environment.I find that the kylin log increases so fast. and my hard disk is filled so I kill the kylin.sh progress.But it's not stopped. I checkd the machine and find the diag.sh is running. I never start this script. I kill this and it seems OK.why? 446463...@qq.com
I delete my table in hive,but the file in HDFS not be deleted
I meet a problem in hive. I drop a table in hive and the table name ' user_info_20161206' --- hive> show tables; OK kylin_cal_dt kylin_category_groupings kylin_intermediate_dmp_cube_fb5904cf_a4d3_4815_802d_c31afe9119e9 kylin_intermediate_test_cube_08677652_0f84_4322_a2a5_0a963723579e kylin_intermediate_test_cube_a37beebf_d7da_4956_8e25_d563dd834364 kylin_intermediate_test_cube_aa9ee162_0d45_4ea6_853d_6df127799edf kylin_sales Time taken: 0.045 seconds, Fetched: 7 row(s) hive> -- but I find the user_info_20161206 file is exist in HDFS file drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 /user/hive/warehouse/dm.db drwxrwxrwt - hadoop hadoop 0 2016-11-21 15:39 /user/hive/warehouse/dw.db drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 /user/hive/warehouse/kylin_cal_dt drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 /user/hive/warehouse/kylin_category_groupings drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 /user/hive/warehouse/kylin_sales drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 /user/hive/warehouse/ods.db drwxrwxrwt - hadoop hadoop 0 2016-11-30 17:53 /user/hive/warehouse/raw.db drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 /user/hive/warehouse/rpt.db drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 /user/hive/warehouse/temp.db drwxrwxrwt - hadoop hadoop 0 2016-11-24 13:17 /user/hive/warehouse/test.db drwxrwxrwt - hive hadoop 0 2016-12-06 21:15 /user/hive/warehouse/user_info_20161206 -- I don't kown why I drop table in hive but the same file not delete in HDFS . I test in mine test environment,it's work the owner of file is 'hive' not 'hadoop'? 446463...@qq.com
Re: Re: I delete my table in hive,but the file in HDFS not be deleted
not external this is create table statement: - create table if not exists user_info_20161207 (rowKey string,timestamp string,sex string ,birthday string ,native_place string ) row format delimited fields terminated by ',' -- I kown the owner is hive not hadoop I specified the hive2 jdbc username 'hive' not hadoop I just use 'hadoop' to create table and I drop table in hive .I find it's not delete in HDFS file system.sad :( 446463...@qq.com From: Alan Gates Date: 2016-12-07 10:34 To: user Subject: Re: I delete my table in hive,but the file in HDFS not be deleted Is the table external or managed? External tables do not remove their data when dropped, managed tables do. Alan. > On Dec 6, 2016, at 18:08, 446463...@qq.com wrote: > > I meet a problem in hive. > > I drop a table in hive and the table name ' user_info_20161206' > --- > hive> show tables; > OK > kylin_cal_dt > kylin_category_groupings > kylin_intermediate_dmp_cube_fb5904cf_a4d3_4815_802d_c31afe9119e9 > kylin_intermediate_test_cube_08677652_0f84_4322_a2a5_0a963723579e > kylin_intermediate_test_cube_a37beebf_d7da_4956_8e25_d563dd834364 > kylin_intermediate_test_cube_aa9ee162_0d45_4ea6_853d_6df127799edf > kylin_sales > Time taken: 0.045 seconds, Fetched: 7 row(s) > hive> > -- > but I find the user_info_20161206 file is exist in HDFS file > > drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 > /user/hive/warehouse/dm.db > drwxrwxrwt - hadoop hadoop 0 2016-11-21 15:39 > /user/hive/warehouse/dw.db > drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 > /user/hive/warehouse/kylin_cal_dt > drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 > /user/hive/warehouse/kylin_category_groupings > drwxrwxrwt - hadoop hadoop 0 2016-11-03 12:44 > /user/hive/warehouse/kylin_sales > drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 > /user/hive/warehouse/ods.db > drwxrwxrwt - hadoop hadoop 0 2016-11-30 17:53 > /user/hive/warehouse/raw.db > drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 > /user/hive/warehouse/rpt.db > drwxrwxrwt - hadoop hadoop 0 2016-09-26 11:50 > /user/hive/warehouse/temp.db > drwxrwxrwt - hadoop hadoop 0 2016-11-24 13:17 > /user/hive/warehouse/test.db > drwxrwxrwt - hive hadoop 0 2016-12-06 21:15 > /user/hive/warehouse/user_info_20161206 > ------ > I don't kown why I drop table in hive but the same file not delete in HDFS . > I test in mine test environment,it's work > the owner of file is 'hive' not 'hadoop'? > 446463...@qq.com
how to load ORC file into hive orc table
Hi All: I create a orc table in hive create table if not exists user_tag ( rowkey STRING , cate1 STRING , cate2 STRING , cate3 STRING , cate4 STRING ) PARTITIONED BY (timestamp STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS orc LOCATION '/user/hive/warehouse/kylinlabel.db/user_tag'; and I generate a orc file in spark and I put this file into path /user/hive/warehouse/kylinlabel.db/user_tag /user/hive/warehouse/kylinlabel.db/user_tag/part-r-0-920282f9-4d68-4af8-81c5-69522df3d374.orc this is the file path. I find there is no data in user_tag table Why? 446463...@qq.com
where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
---
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "2016100100_2016122000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 147528000,
"date_range_end": 148219200,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX":
"/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY":
"/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE":
"/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION":
"/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-
I find all thing begin with /cube or /cube_desc or /model_desc or /table or
/table_ext related to my cube and I can't find anything with Global Dictionary
with my cube
Help me !
446463...@qq.com
Re: Re: where kylin global dict store?
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find
anything with this Global Dictionary infomation related with my cube in HBase
table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys
prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my
model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find
nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the
job to build this cube
in kylin webUI job throw Error tall me ' No resource found at --
/dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict
has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module
which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw
Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList dumpList = new ArrayList();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables())
{
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List dependentResources =
SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so
I try to understand how dumpList be filled with data. thanks to your
annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter
out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or
'/table' and so on .but finally I could not find any infomation with my cube
used Global Dictionary
I don't kown whether or not it clearly with my question?
446463...@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it
will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a
search there.
2017-01-06 8:52 GMT+08:00 zy <346453...@qq.com>:
I think you could find the more content about kylin from the internet..i give
you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
-- Original --
From: "446463...@qq.com"<446463...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"; "user";
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
---
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "2016100100_2016122000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 147528000,
"date_range_end": 148219200,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"
Re: Re: where kylin global dict store?
I find the Globel Dictionary infomation in HBase table ,sorry for disturb
446463...@qq.com
From: 446463...@qq.com
Date: 2017-01-06 10:48
To: user; dev
CC: user
Subject: Re: Re: where kylin global dict store?
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find
anything with this Global Dictionary infomation related with my cube in HBase
table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys
prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my
model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find
nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the
job to build this cube
in kylin webUI job throw Error tall me ' No resource found at --
/dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict
has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module
which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw
Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList dumpList = new ArrayList();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables())
{
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List dependentResources =
SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so
I try to understand how dumpList be filled with data. thanks to your
annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter
out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or
'/table' and so on .but finally I could not find any infomation with my cube
used Global Dictionary
I don't kown whether or not it clearly with my question?
446463...@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it
will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a
search there.
2017-01-06 8:52 GMT+08:00 zy <346453...@qq.com>:
I think you could find the more content about kylin from the internet..i give
you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
-- Original --
From: "446463...@qq.com"<446463...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"; "user";
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
---
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "2016100100_2016122000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 147528000,
"date_range_end": 148219200,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
