Re: Problem with Amazon S3

2015-05-20 Thread pietro
Dears, I am still having problem retriving data from the S3. I followed all you indication in the previous posts, but now I get this error: 15/05/20 10:47:05 INFO s3.S3FileSystem: Creating new S3 file system binding with Reduced Redundancy Storage enabled 15/05/20 10:47:13 WARN io.DelimitedInputF

Re: Problem with Amazon S3

2015-04-01 Thread pietro
Hi Stephan, for sure I can: this is how I try to read from Flink: env.readFile(new DefaultReader(), "s3://genomic/flink/input/meta/1.txt" ).map(parser(_))} I use the same format I was used to use in Pig. Thanks, -- View this message in context: http://apache-flink-incubator-user-mailing-l

Re: Problem with Amazon S3

2015-03-31 Thread pietro
Thank you Ufuk! That helped a lot. But I have an other problem now. Am I missing something? Caused by: java.net.UnknownHostException: MYBUCKETNAME at java.net.InetAddress.getAllByName0(InetAddress.java:1250) at java.net.InetAddress.getAllByName(InetAddress.java:1162) at

Problem with Amazon S3

2015-03-31 Thread pietro
Dear all, I have been developing a Flink application that has to run on Amazon Elastic Map Reduce. For convenience the data that the application has to read and write are on the S3. But, I have not been able to access S3 .This is the error I got: org.apache.flink.client.program.ProgramInvocationE

Re: Most convenient data structure for unspecified length objects

2015-03-16 Thread pietro
Hi Stephan, thanks for the reply! My problem is that I cannot know whether I will have 0, 1,2,..or more strings. Then, Option is not gonna help in my case :( -- View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Most-convenient-data-stru

Most convenient data structure for unspecified length objects

2015-03-16 Thread pietro
I have to implement a program based on Flink that process some records. The peculiarity of those records is that it is not possible to know at compile time how many fields they contain. Therefore, I cannot use a simple TupleN data type. The solution I came up with, is to use a tuple with this str

Getting the name of a file in a directory

2015-03-02 Thread pietro
I am reading files from a directory with this statement: /val text = env.readFile(new MyInputFormat(), "/path/to/input/dir/")/ /MyInputFormat/ extends /DelimitedInputFormat/ which extends /DelimitedInputFormat/. In the output Record, I need to add a field that stores the name (or full path) of t