Re: Hive on Spark engine

2016-03-26 Thread Ted Yu
According to: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf Spark 1.5.2 comes out of box. Suggest moving questions on HDP to Hortonworks forum. Cheers On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh wrote: > Thanks Jorn. > > Just to be

Re: binary file deserialization

2016-03-09 Thread Ted Yu
bq. there is a varying number of items for that record If the combination of items is very large, using case class would be tedious. On Wed, Mar 9, 2016 at 9:57 AM, Saurabh Bajaj wrote: > You can load that binary up as a String RDD, then map over that RDD and > convert each row to your case cla

Re: error while defining custom schema in Spark 1.5.0

2015-12-25 Thread Ted Yu
The error was due to blank field being defined twice. On Tue, Dec 22, 2015 at 12:03 AM, Divya Gehlot wrote: > Hi, > I am new bee to Apache Spark ,using CDH 5.5 Quick start VM.having spark > 1.5.0. > I working on custom schema and getting error > > import org.apache.spark.sql.hive.HiveContext >>

Re: Intermittent BindException during long MR jobs

2015-02-28 Thread Ted Yu
Krishna: Please take a look at: http://wiki.apache.org/hadoop/BindException Cheers On Thu, Feb 26, 2015 at 10:30 PM, wrote: > Hello Krishna, > > > > Exception seems to be IP specific. It might be occurred due to > unavailability of IP address in the system to assign. Double check the IP > addre

Re: unsubscribe

2013-07-18 Thread Ted Yu
You have to send a mail to user-unsubscr...@hive.apache.org On Thu, Jul 18, 2013 at 1:30 PM, Beau Rothrock wrote: >

Re: getRegion method is missing from RegionCoprocessorEnvironment class in version 0.94

2013-07-11 Thread Ted Yu
Which release of 0.94 do you use ? For tip of 0.94, I see: public interface RegionCoprocessorEnvironment extends CoprocessorEnvironment { /** @return the region associated with this coprocessor */ public HRegion getRegion(); On Thu, Jul 11, 2013 at 6:40 PM, ch huang wrote: > how can i do ,

Re: Variable resolution Fails

2013-05-01 Thread Ted Yu
Naidu: Please don't hijack existing thread. Your questions are not directly related to Hive. Cheers On May 1, 2013, at 12:53 AM, Naidu MS wrote: > Hi i have two questions regarding hdfs and jps utility > > I am new to Hadoop and started leraning hadoop from the past week > > 1.when ever i s

Re: no data in external table

2012-10-04 Thread Ted Yu
Can you tell us how you created mapping for the existing table ? In task log, do you see any connection attempt to HBase ? Cheers On Thu, Oct 4, 2012 at 11:30 AM, wrote: > Hello, > > I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created > external table, mapping it to an existi

Re: HBase aggregate query

2012-09-10 Thread Ted Yu
Hi, Are you able to get the number you want through hive log ? Thanks On Mon, Sep 10, 2012 at 7:03 AM, iwannaplay games < funnlearnfork...@gmail.com> wrote: > Hi , > > I want to run query like > > select month(eventdate),scene,count(1),sum(timespent) from eventlog > group by month(eventdate),sce

Re: HBaseSerDe

2012-07-25 Thread Ted Yu
The ctor is used in TestHBaseSerDe.java So maybe change it to package private ? On Wed, Jul 25, 2012 at 12:43 PM, kulkarni.swar...@gmail.com < kulkarni.swar...@gmail.com> wrote: > While going through some code for HBase/Hive Integration, I came across > this constructor: > > public HBaseSerDe()

Re: Hive and CDH4 GA

2012-07-16 Thread Ted Yu
ot;ant clean tar") so that it works with hadoop 2.0.0 on its classpath? > > Any help is appreciated. > > Thanks, > > [1] https://issues.apache.org/jira/browse/HIVE-3029 > > > On Fri, Jul 13, 2012 at 10:27 PM, Ted Yu wrote: > >> See >> http

Re: Hive and CDH4 GA

2012-07-13 Thread Ted Yu
See https://issues.apache.org/jira/browse/HADOOP-8350?focusedCommentId=13414276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13414276 Cheers On Fri, Jul 13, 2012 at 12:38 PM, kulkarni.swar...@gmail.com < kulkarni.swar...@gmail.com> wrote: > Has anyone being using

Re: What virtual column does hive.exec.rowoffset add?

2012-03-25 Thread Ted Yu
>From VirtualColumn.java: public static VirtualColumn ROWOFFSET = new VirtualColumn("ROW__OFFSET__INSIDE__BLOCK", (PrimitiveTypeInfo)TypeInfoFactory.longTypeInfo); Cheers On Sun, Mar 25, 2012 at 2:30 PM, Edward Capriolo wrote: > > hive.exec.rowoffset > false > Whether to provide the row of

Re: Running Hive Queries in Parallel?

2011-07-08 Thread Ted Yu
Set it in conf/hive-site.xml On Thu, Jul 7, 2011 at 10:59 PM, hadoop n00b wrote: > Found *hive*.*exec*.*parallel!!!* > How can I set to to be 'true' by default? > > Thanks! > > > On Fri, Jul 8, 2011 at 11:14 AM, hadoop n00b wrote: > >> Hello, >> >> When I execute multiple queries in Hive, the m

Re: URGENT: I need the Hive Server setup Wiki

2011-06-27 Thread Ted Yu
There're 3 links on that page. On Mon, Jun 27, 2011 at 2:39 PM, Ted Yu wrote: > wiki has moved. > See > https://cwiki.apache.org/confluence/display/Hive/AdminManual+SettingUpHiveServer > > > On Mon, Jun 27, 2011 at 2:31 PM, Ayon Sinha wrote: > >> >> https

Re: URGENT: I need the Hive Server setup Wiki

2011-06-27 Thread Ted Yu
wiki has moved. See https://cwiki.apache.org/confluence/display/Hive/AdminManual+SettingUpHiveServer On Mon, Jun 27, 2011 at 2:31 PM, Ayon Sinha wrote: > > https://cwiki.apache.org/confluence/display/Hive/AdminManual+SettingUpHiveServeris > empty > and the old link is gone. > > -Ayon > See My P

Re: How to change the hive.metastore.warehouse.dir ?

2011-05-17 Thread Ted Yu
Can you try as user hadoop ? Cheers On May 17, 2011, at 9:53 PM, jinhang du wrote: > hi, > The default value is "/user/hive/warehouse" in hive.site.xml. After I changed > the directory to a path on HDFS, I got the exception. > > FAILED: Error in metadata: MetaException(message:Got exception

Re: Hive/hbase integration - Rebuild the Storage Handler

2011-03-22 Thread Ted Yu
Look at this line in ivy/libraries.properties: hbase.version=0.89.0-SNAPSHOT The jars would be downloaded to: ./build/ivy/lib/default/hbase-0.89.0-SNAPSHOT-tests.jar ./build/ivy/lib/default/hbase-0.89.0-SNAPSHOT.jar After changing hbase.version, I saw: [ivy:resolve] :: problems summary :: [ivy:r

Re: Hi,all.Is it possible to generate mutiple records in one SerDe?

2011-03-21 Thread Ted Yu
I don't think so: Object deserialize(Writable blob) throws SerDeException; On Mon, Mar 21, 2011 at 4:55 AM, 幻 wrote: > Hi,all.Is it possible to generate mutiple records in one SerDe? I mean if I > can return more than one rows in deserialize? > > Thanks! >

Re: skew join optimization

2011-03-20 Thread Ted Yu
How about link to http://imageshack.us/ or TinyPic ? Thanks On Sun, Mar 20, 2011 at 7:56 AM, Edward Capriolo wrote: > On Sun, Mar 20, 2011 at 10:30 AM, Ted Yu wrote: > > Can someone re-attach the missing figures for that wiki ? > > > > Thanks > > > > On Sun,

Re: skew join optimization

2011-03-20 Thread Ted Yu
Can someone re-attach the missing figures for that wiki ? Thanks On Sun, Mar 20, 2011 at 7:15 AM, bharath vissapragada < bharathvissapragada1...@gmail.com> wrote: > Hi Igor, > > See http://wiki.apache.org/hadoop/Hive/JoinOptimization and see the > jira 1642 which automatically converts a normal

Re: does hive support Sequence File format ?

2011-02-17 Thread Ted Yu
Look under http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL#Create_Table On Thu, Feb 17, 2011 at 12:00 PM, Mapred Learn wrote: > Hi, > I was wondering if hive supports Sequence File format. If yes, could me > point me to some documentation about how to use Seq files in hive. > > Thanks, > -J

Re: Partitioning External table

2010-12-29 Thread Ted Yu
Can you try using: location 'dt=1/engine' Cheers On Wed, Dec 29, 2010 at 1:12 AM, David Ginzburg wrote: > Hi, > Thank you for the reply. > I tried ALTER TABLE tpartitions ADD PARTITION (dt='1') LOCATION > '/user/training/partitions/'; > SHOW PARTITIONS > tpartitions; > > OK > dt=1 > > > but

Re: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

2010-12-20 Thread Ted Yu
Have you found anything interesting from Hive history file (/tmp/hadoop/hive_job_log_hadoop_201012210353_775358406.txt) ? Thanks On Mon, Dec 20, 2010 at 6:11 PM, Sean Curtis wrote: > just running a simple select count(1) from a table (using movielens as an > example) doesnt seem to work for me.

Re: Hive produces very small files despite hive.merge...=true settings

2010-11-18 Thread Ted Yu
Leo: You may find this helpful: http://indoos.wordpress.com/2010/06/24/hive-remote-debugging/ On Thu, Nov 18, 2010 at 2:57 PM, Leo Alekseyev wrote: > Hi Ning, > For the dataset I'm experimenting with, the total size of the output > is 2mb, and the files are at most a few kb in size. My > hive.i

Re: EXTERNAL:Re: unable to create table

2010-11-16 Thread Ted Yu
> > > > When I run /bin/hive, it starts fine. The problem arises when > I try to create a table once hive is running. > > > > Best, > > Hannah > > > > *From:* Ted Yu [mailto:yuzhih...@gmail.com] > *Sent:* Tuesday, November 16, 2010 2:16 PM >

Re: unable to create table

2010-11-16 Thread Ted Yu
See 'Exception in hive startup' discussion - especially Edward's response on Oct 13th. On Tue, Nov 16, 2010 at 9:40 AM, Gerlach, Hannah L (IS) < hannah.gerl...@ngc.com> wrote: > Hello, > > > > I am completely new to hive and I need some help. Hive starts fine, but > when I try to create a table,

Re: Hive Getting Started Wiki assumes $CLASSPATH at end of HADOOP_CLASSPATH

2010-11-08 Thread Ted Yu
Please see Edward's reply to 'Exception in hive startup' on Oct 13th. Try running with /bin/hive On Mon, Nov 8, 2010 at 7:02 PM, Stuart Smith wrote: > Hello, > > I'm just starting with hive, and I ran into a newbie problem that didn't > have a solution via google. So I thought I'd record the s

Re: Exception in hive startup

2010-10-13 Thread Ted Yu
This should be documented in README.txt On Wed, Oct 13, 2010 at 6:14 PM, Steven Wong wrote: > You need to run hive_root/build/dist/bin/hive, not hive_root/bin/hive. > > > > > > *From:* hdev ml [mailto:hde...@gmail.com] > *Sent:* Wednesday, October 13, 2010 2:18 PM > *To:* hive-u...@hadoop.apach

Re: Exception in hive startup

2010-10-13 Thread Ted Yu
Make sure hive-exec-0.7.0.jar is under lib/ dir. Found: HiveConf Class: org.apache.hadoop.hive.conf.HiveConf Package: org.apache.hadoop.hive.conf Library Name: hive-exec-0.7.0.jar Library Path: /Users/tyu/hive/lib/hive-exec-0.7.0.jar On Wed, Oct 13, 2010 at 2:17 PM, hdev ml wrote: > Hi all, > >

LzoCodec for hadoop 0.20.x

2010-10-08 Thread Ted Yu
Hi, I am using hive trunk with default setting along with cdh3b2. I encounter the following exception: Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Co

exception in admin_list_jobs.jsp

2010-10-07 Thread Ted Yu
Hi, I compiled hive from trunk. ciq.com:/hwi/admin_list_jobs.jsp gave me: org.apache.jasper.JasperException: Unable to compile class for JSP An error occurred at line: 24 in the jsp file: /admin_list_jobs.jsp Generated servlet error: The field ExecDriver.runningJobKillURIs is not visible Has