loader? Or am I misunderstanding the lifecycle/purpose of
UDFClassLoader instances? Is there a more robust way to cast Writable
instances in a custom SerDe implementation? Thanks in advance for any
clarification you can give.
Best,
Jason
On Mon, Sep 10, 2018 at 10:37 PM Gopal Vijayaraghavan
>query the external table using HiveCLI (e.g. SELECT * FROM
>my_external_table), HiveCLI prints out a table with the correct
If the error is always on a "select *", then the issue might be the SerDe's
handling of included columns.
Check what you get for
colNames =
Arrays.asList(tblPr
Hi all,
Hive Version: 3.0.0
Hadoop Version: 3.1.0
Tez Version: 0.9.1
I help to maintain a Hive SerDe connecting Hive to Apache Solr. We've
let our custom SerDe lag quite a bit behind in Hive releases (1.2.1),
but we've recently started updating the code to work with Hive 3.0.0.
Afte
)
. 18 more
-
I know that, it is custome storageHandler about hive, but, now, my
questions are about how the two(tez&hive) to work together:
There, NBHBaseSerde is a custom S
>>
>> On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv <
>> daniel.ha...@veracity-group.com> wrote:
>>
>>> Hi,
>>> We have a custom SerDe we would like to use with Hive on Spark but I'm
>>> not sure how to.
>>> The error messa
e your jar is in ""hive.aux.jars.path".
>
> Thanks,
> Jimmy
>
> On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv <
> daniel.ha...@veracity-group.com> wrote:
>
>> Hi,
>> We have a custom SerDe we would like to use with Hive on Spark but I'm
>
Have you add your class to "spark.kryo.classesToRegister"? You also need to
make sure your jar is in ""hive.aux.jars.path".
Thanks,
Jimmy
On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> Hi,
> We have a custom Se
Hi,
We have a custom SerDe we would like to use with Hive on Spark but I'm not
sure how to.
The error messages are pretty clear about the fact that it can't find my
SerDE's class:
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
Unabl
Edward Capriolo wrote:
> Have you seen?
>
> https://github.com/edwardcapriolo/hive-protobuf/
>
>
> On Mon, Oct 14, 2013 at 8:16 AM, Rui Martins wrote:
>
>> Hi hive users,
>>
>> I am writing a custom SerDe that loads any protocol buffer generated
>> class.
Have you seen?
https://github.com/edwardcapriolo/hive-protobuf/
On Mon, Oct 14, 2013 at 8:16 AM, Rui Martins wrote:
> Hi hive users,
>
> I am writing a custom SerDe that loads any protocol buffer generated
> class.
> For flexibility this class can live in a jar external to
ks,
>
> Yin
>
>
> On Mon, Oct 14, 2013 at 8:20 AM, Rui Martins wrote:
>
> > +dev hive mailing list that I should've mailed in the first place.
> >
> > (apologies for the spam)
> >
> >
> > On Mon, Oct 14, 2013 at 1:16 PM, Rui Martins wrote:
&g
wrote:
> +dev hive mailing list that I should've mailed in the first place.
>
> (apologies for the spam)
>
>
> On Mon, Oct 14, 2013 at 1:16 PM, Rui Martins wrote:
>
>> Hi hive users,
>>
>> I am writing a custom SerDe that loads any protocol buffer generated
+dev hive mailing list that I should've mailed in the first place.
(apologies for the spam)
On Mon, Oct 14, 2013 at 1:16 PM, Rui Martins wrote:
> Hi hive users,
>
> I am writing a custom SerDe that loads any protocol buffer generated
> class.
> For flexibility this cla
Hi hive users,
I am writing a custom SerDe that loads any protocol buffer generated class.
For flexibility this class can live in a jar external to the SerDe's jar
and then I just use the Hive Configuration class passed in the initiliaze
to dynamically load it and set the schema for the
t; ** **
>
> *From:* Connell, Chuck [mailto:chuck.conn...@nuance.com]
> *Sent:* Thursday, November 29, 2012 9:51 AM
> *To:* user@hive.apache.org
> *Subject:* RE: Starting with Hive - writing custom SerDe
>
> ** **
>
> You might save yourself a lot of work by pre-processing the data, be
I meant PLAIN tab-separated text.
From: Connell, Chuck [mailto:chuck.conn...@nuance.com]
Sent: Thursday, November 29, 2012 9:51 AM
To: user@hive.apache.org
Subject: RE: Starting with Hive - writing custom SerDe
You might save yourself a lot of work by pre-processing the data, before
putting it
You might save yourself a lot of work by pre-processing the data, before
putting it into Hive. A Python script should be able to find all the fields,
and change the data to plan tab-separated text. This will load directly into
Hive, and removes the need to write a custom SerDe.
Chuck Connell
es used as
separators)...
So after looking around for a while, I found that I have to write a custom
SerDe in order to tell Hive how to parse those lines.
I've also found that I can only write them using Java (unlike UDFs for pig
for instance, which can be written using other languages), is this corre
Greetings all. I am not sure if this is a hive issue or a custom serde
issue. So I will ask in both places. I am trying to use the mongodb
connection written by yc-huang. This could have great potential with our
data. THe link is here.
https://github.com/yc-huang/Hive-mongo
I followed
ently based on the query plan?
From: Evan Pollan
mailto:evan.pol...@bazaarvoice.com>>
Reply-To: mailto:user@hive.apache.org>>
Date: Wed, 22 Feb 2012 12:28:26 +
To: "user@hive.apache.org<mailto:user@hive.apache.org>"
mailto:user@hive.apache.org>>
Subject: Re: Cu
/tmp it is! My bad — it was the one obvious place I omitted from my find/grep
statement. Thanks!
From: Matthew Byrd mailto:mb...@acunu.com>>
Reply-To: mailto:user@hive.apache.org>>
Date: Wed, 22 Feb 2012 11:54:09 +
To: mailto:user@hive.apache.org>>
Subject: Re: Custom
ble where ArrivalDate="2012-02-01"'
>
> But, this fails: 'select * from table where ArrivalDate="test"'
>
> Does this make sense to anybody?
>
>
>
> From: Evan Pollan
> Reply-To:
> Date: Tue, 21 Feb 2012 20:56:07 +
> To: &qu
an
mailto:evan.pol...@bazaarvoice.com>>
Reply-To: mailto:user@hive.apache.org>>
Date: Tue, 21 Feb 2012 20:56:07 +
To: "user@hive.apache.org<mailto:user@hive.apache.org>"
mailto:user@hive.apache.org>>
Subject: Custom SerDe -- tracking down stack trace
I hav
I have a custom SerDe that's initializing properly and works on one data set.
I built it to adapt to a couple of different data formats, though, and it's
choking on a different data set (different partitions in the same table).
A null pointer exception is being thrown on deseriali
have you tried?
2011/5/9 Jasper Knulst
> Hi Ankit,
>
>
> I got this in my java mapper code
>
> String oldSeperator = "�"; //the thorn as java sees it
> String newSeperator = "~";
>
> In Eclipse it shows as �, which is the standard java way of saying "I don't
> know this multibyte character".
>
>
Hi Ankit,
I got this in my java mapper code
String oldSeperator = "�"; //the thorn as java sees it
String newSeperator = "~";
In Eclipse it shows as �, which is the standard java way of saying "I don't
know this multibyte character".
When you copy paste this � to the linux shell it depicts as
Hi Jasper,
could you please share your MR program.
I am not able to grab this character
Ankit
Hi Ankit,
It all depends on your environment and locale en encoding. This proved to
work in my case, but I believe to have seen your characters as well, but
after all it is not your browser that has to do the work and interpret the
multibyte character. That is the main problem with the thorn; ever
Hi Jasper,
How did you find - 'þ'
My browser shows this - �
Ankit
Hi Ankit,
I know your problem because I had to deal with a thorn 'þ' separated file
too. Hive ,so far, cannot handle multibyte separators so I turned to the
custom SerDe option myself. If you manage to capture the 'þ' in the regex
you could try
I trie
Hi
I am facing a weird issue with the file parsing. My log files have a thorn
'þ' as separator.
I tried writing a test case for deserializer and kind of confused by the
fact it works fine as I pass the line to the deserializer, however when i
run it on hive the line is not split into columns and
Hi
I am facing a weird issue with the file parsing. My log files have a thorn
'þ' as separator.
I tried writing a test case for deserializer and kind of confused by the
fact it works fine as I pass the line to the deserializer, however when i
run it on hive the line is not split into columns and
textfile and using my custom serde so it had a format line
> of:
>
>
>
> ROW FORMAT SERDE ‘org.myorg.hadoop.hive.udf.MySerDe’ STORED AS textfile
>
>
>
> Then I loaded a gzipped file using LOAD DATA LOCAL INPATH ‘path.gz’ INTO
> TABLE mytable and it worked as expected, ie
Not sure what I did wrong the first time but I tried to create a table with
stored type of textfile and using my custom serde so it had a format line of:
ROW FORMAT SERDE 'org.myorg.hadoop.hive.udf.MySerDe' STORED AS textfile
Then I loaded a gzipped file using LOAD DATA LOCAL INPAT
I found the source code is very helpful for this.
There's a custom serde in the source, with a test case you can review, which
really speeds up development of your SerDe.
org.apache.hadoop.hive.contrib.serde2.TestRegexSerDe
One thing to watch out for though, is that the framework will
bhatnagar wrote:
> Hi All,
>
> Whats the best way to add custom serde jar.
>
> looks like add jar says added successfully and I am able to list that jar
> in hive interface
>
> However Create table fails to find the class.
>
> Thanks
> Ankit
>
Hi,
>>
>> I’ve written a SerDe and I’d like it to be able handle compressed data
>> (gzip). Hadoop detects and decompresses on the fly so if you have a
>> compressed data set and you don’t need to perform any custom interpretation
>> of it as you go, hadoop and hive
adoop and hive will handle it. Is there a way to get Hive
> to notice the data is compressed, decompress it then push it through the
> custom SerDe? Or will I have to either
>
> a. add some decompression logic to my SerDe (possibly impossible)
>
> b. decompress the da
re a way to get Hive to notice the data is
compressed, decompress it then push it through the custom SerDe? Or will I
have to either
a. add some decompression logic to my SerDe (possibly impossible)
b. decompress the data before pushing it into a table with my SerDe
Thanks!
Pat
Hi All,
Whats the best way to add custom serde jar.
looks like add jar says added successfully and I am able to list that jar in
hive interface
However Create table fails to find the class.
Thanks
Ankit
info - but my question will be how is
this initialize() method getting invoked with Custom SerDe while executing
query.
create table my_table(a string, b string, ...)
row format serde '..CustomSerde'
stored as textfile
;
Thanks
-Ankit
41 matches
Mail list logo
