The underFS is HDFS btw. On Thu, Aug 21, 2014 at 12:22 PM, Evan Chan <[email protected]> wrote: > Spark 1.0.2, Tachyon 0.4.1, Hadoop 1.0 (standard EC2 config) > > scala> val gdeltT = > sqlContext.parquetFile("tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/") > 14/08/21 19:07:14 INFO : > initialize(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005, > Configuration: core-default.xml, core-site.xml, mapred-default.xml, > mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to > Tachyon: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005 > 14/08/21 19:07:14 INFO : Trying to connect master @ /172.31.42.40:19998 > 14/08/21 19:07:14 INFO : User registered at the master > ip-172-31-42-40.us-west-2.compute.internal/172.31.42.40:19998 got > UserId 14 > 14/08/21 19:07:14 INFO : Trying to get local worker host : > ip-172-31-42-40.us-west-2.compute.internal > 14/08/21 19:07:14 INFO : No local worker on > ip-172-31-42-40.us-west-2.compute.internal > 14/08/21 19:07:14 INFO : Connecting remote worker @ > ip-172-31-47-74/172.31.47.74:29998 > 14/08/21 19:07:14 INFO : tachyon://172.31.42.40:19998 > tachyon://172.31.42.40:19998 > hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000 > 14/08/21 19:07:14 INFO : > getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005): > HDFS Path: > hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005 > TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005 > 14/08/21 19:07:14 INFO : tachyon.client.TachyonFS@4b05b3ff > hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000 > /gdelt-parquet/1979-2005 tachyon.PrefixList@636c50d3 > 14/08/21 19:07:14 WARN : tachyon.home is not set. Using > /mnt/tachyon_default_home as the default value. > 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_SUCCESS > 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/_metadata > 14/08/21 19:07:14 INFO : Get: /gdelt-parquet/1979-2005/part-r-1.parquet > > .... > > 14/08/21 19:07:14 INFO : > getFileStatus(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata): > HDFS Path: > hdfs://ec2-54-213-113-173.us-west-2.compute.amazonaws.com:9000/gdelt-parquet/1979-2005/_metadata > TPath: tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata > 14/08/21 19:07:14 INFO : > open(tachyon://172.31.42.40:19998/gdelt-parquet/1979-2005/_metadata, > 65536) > 14/08/21 19:07:14 ERROR : The machine does not have any local worker. > 14/08/21 19:07:14 ERROR : Reading from HDFS directly > 14/08/21 19:07:14 ERROR : Reading from HDFS directly > java.io.IOException: can not read class parquet.format.FileMetaData: null > at parquet.format.Util.read(Util.java:50) > at parquet.format.Util.readFileMetaData(Util.java:34) > at > parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:310) > at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:296) > > > I'm not sure why this is saying that, as the Tachyon UI reports all 8 > nodes being up?
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
