Re: Hive Pulsar Integration

2019-04-13 Thread Jörn Franke
I think you need to develop a custom hiveserde + custom Hadoopinputformat + custom Hiveoutputformat > Am 12.04.2019 um 17:35 schrieb 李鹏辉gmail : > > Hi guys, > > I’m working on integration of hive and pulsar recently. But now i have > encountered some problems and hope to get help here. > > Fi

Re: How to Load Data From a CSV to a parquet table

2018-03-01 Thread Jörn Franke
You have defined a parquet only table. It interprets your CSV file as parquet. You can for instance define 2 tables: * one external for the CSV file * one table for the parquet file Afterwards you select from the first table and insert in the second table. > On 1. Mar 2018, at 08:31, Anubhav T

Re: Hive Custom Inputformat mapred.* vs. mapreduce.*

2017-09-13 Thread Jörn Franke
etc) and file formats transparently. > On 13. Sep 2017, at 20:53, Alan Gates wrote: > > I’m not aware of any plans in Hive to do any more work that uses Map Reduce > as the execution engine, so I expect Hive will continue to use mapred. > > Alan. > >> On Wed, Sep 13, 2

Hive Custom Inputformat mapred.* vs. mapreduce.*

2017-09-13 Thread Jörn Franke
Dear all, I have developed several custom input formats (e.g. for the Bitcoin blockchain) including a HiveSerde, which are open source. I plan to develop for my HadoopOffice inputformat also a HiveSerde, but I wonder if I should continue to use mapred.* apis or if i should use mapreduce.* My inpu

Re: TBLPROPERTIES appears to be ignored by custom inputformats

2016-12-19 Thread Jörn Franke
rstand how my select * is correctly reading my table property > then if I'm just using default serde. >> On Mon., 19 Dec. 2016 at 5:36 pm, Jörn Franke wrote: >> >> You have to write a custom hiveserde format to pass tblproperties as >> inputformat properties, but che

Re: TBLPROPERTIES appears to be ignored by custom inputformats

2016-12-18 Thread Jörn Franke
You have to write a custom hiveserde format to pass tblproperties as inputformat properties, but check the source code of the serde you used. > On 19 Dec 2016, at 07:22, Chris Teoh wrote: > > rows.

Re: TBLPROPERTIES appears to be ignored by custom inputformats

2016-12-18 Thread Jörn Franke
What is the create table statement? Do you parse the tblproperties in the HiveSerde? Do you have exceptions in the log? > On 19 Dec 2016, at 07:02, Chris Teoh wrote: > > Hi there, > > Can anyone confirm whether TBLPROPERTIES in DDLs are ignored by custom > inputformats in the context of a UDAF

Re: Load performance with partitioned table

2016-09-15 Thread Jörn Franke
What is your hardware setup? Are the bloom filters necessary on all columns? Usually they make only sense for non-numeric columns. Updating bloom filters take time and should be avoided where they do not make sense. Can you provide an example of the data and the select queries that you execute o

Analyzing Bitcoin blockchain data with Hive

2016-04-29 Thread Jörn Franke
Dear all, I prepared a small Serde to analyze Bitcoin blockchain data with Hive: https://snippetessay.wordpress.com/2016/04/28/hive-bitcoin-analytics-on-blockchain-data-with-sql/ There are some example queries, but I will add some in the future. Additionally, more unit tests will be added. Let m

Re: Reviews & commits (RTC/CTR), contributions, bylaws

2016-04-11 Thread Jörn Franke
Hi Lars, I think this is a valid concern and your proposal sounds good to me. Best regards > On 11 Apr 2016, at 15:38, Lars Francke wrote: > > Hi, > > I've been a long-time contributor to Hive (5 or so years) and have been > voted in as a committer and I'm very grateful for that. I also unde

Re: analyse command not working on decimal(38,0) datatype

2016-04-06 Thread Jörn Franke
Please provide exact log messages , create table statements, insert statements > On 06 Apr 2016, at 12:05, Ashim Sinha wrote: > > Hi Team > Need help for the issue > Steps followed > table created > Loaded the data of lenght 38 in decimal type > Analyse table - for columns gives error like zero

Re: Run hive in debug mode

2015-09-06 Thread Jörn Franke
There must be some log4j configuration file in the hive configuration folder where you can do this. Le lun. 7 sept. 2015 à 5:09, Chetna C a écrit : > Hi All, >I am new to hive community, and recently started going through code. > There are some points when I would like to see the executi

Re: HIVE:1.2, Query taking huge time

2015-08-20 Thread Jörn Franke
Additionally, although it is a PoC you should have a realistic data model. Furthermore, following good data modeling practices should be taken into account. Joining on a double is not one of them. It should be int. Furthermore, double is a type that is in most scenarios rarely used. In the business

Re: [jira] [Created] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows

2015-07-19 Thread Jörn Franke
Is this also for varchar ? Le lun. 20 juil. 2015 à 1:14, Thomas Friedrich (JIRA) a écrit : > Thomas Friedrich created HIVE-11312: > --- > > Summary: ORC format: where clause with CHAR data type not > returning any rows > Key: HIVE

Re: run tests failed when building hive query language on hadoop 2.7.0

2015-05-25 Thread Jörn Franke
Hi 韦, You must be a little bit more precise. What is the error message? What is your setup (OS, JDK version..) ? Thank you. Best regards Le 25 mai 2015 09:11, "煜 韦" a écrit : > Hi,When I'm building hive 1.2.0 on hadoop 2.7.0. During building hive > query language and running tests, it failed.W