Great, Punit, I'm glad I've been of some help.
If you have similar issues, feel free to write to the user mailing list, I
believe you'll find help more easily there as you approach Flink.
Happy hacking! :)
On Thu, Apr 28, 2016 at 9:10 PM, Punit Naik wrote:
> I am so sorry. Please ignore my previ
I am so sorry. Please ignore my previous reply. Actually my input was too
big so it hung. So stupid of me. Thanks a lot! Your example worked!
On Fri, Apr 29, 2016 at 12:35 AM, Punit Naik wrote:
> I tried exactly what you told me. But when I execute this code, first of
> all it gives me a warning
I tried exactly what you told me. But when I execute this code, first of
all it gives me a warning saying "Type Any has no fields that are visible
from Scala Type analysis. Falling back to Java Type Analysis
(TypeExtractor)." in eclipse, and when I run it, the code just hangs and
does not print a t
Hi Punit,
what you want to do is something like this:
val env = ExecutionEnvironment.getExecutionEnvironment
env.
readTextFile("path/to/test.json").
flatMap(line => JSON.parseFull(line)).
print
The JSON.parseFull function in the Scala standard library takes a string (a
I had one more request though. I have been struggling with JSONs and Flink
for the past two days since I started using it. I have a JSON file which
has one JSON object per line and I want to read it and store it as maps in
another flink Dataset. In my JSON the values might be anything, for e.g.
int
I managed to fix this error. I basically had to do val j=data.map { x => (
x.replaceAll("\"","\\\"")) } instead of val j=data.map { x => ("\"\"\""+x+
"\"\"\"") }
On Wed, Apr 27, 2016 at 4:05 PM, Punit Naik wrote:
> I have my Apache Flink program:
>
> import org.apache.flink.api.scala._import sca
I have my Apache Flink program:
import org.apache.flink.api.scala._import scala.util.parsing.json._
object numHits extends App {
val env = ExecutionEnvironment.getExecutionEnvironment
val data=env.readTextFile("file:///path/to/json/file")
val j=data.map { x => ("\"\"\""+x+"\"\"\"") }
I just tried it and it still cannot parse it. It still takes the input as a
dataset object rather than a string.
On Wed, Apr 27, 2016 at 12:36 PM, Punit Naik wrote:
> Okay Thanks a lot Fabian!
>
> On Wed, Apr 27, 2016 at 12:34 PM, Fabian Hueske wrote:
>
>> You should do the parsing in a Map ope
Okay Thanks a lot Fabian!
On Wed, Apr 27, 2016 at 12:34 PM, Fabian Hueske wrote:
> You should do the parsing in a Map operator. Map applies the MapFunction to
> each element in the DataSet.
> So you can either implement another MapFunction or extend the one you have
> to call the JSON parser.
>
You should do the parsing in a Map operator. Map applies the MapFunction to
each element in the DataSet.
So you can either implement another MapFunction or extend the one you have
to call the JSON parser.
2016-04-27 6:40 GMT+02:00 Punit Naik :
> Hi
>
> So I managed to do the map part. I stuc with
Hi
So I managed to do the map part. I stuc with the "import
scala.util.parsing.json._" library for parsing.
First I read my JSON:
val data=env.readTextFile("file:///home/punit/vik-in")
Then I transformed it so that it can be parsed to a map:
val j=data.map { x => ("\"\"\"").+(x).+("\"\"\"") }
Hi,
you need to implement the MapFunction interface [1].
Inside the MapFunction you can use any JSON parser library such as Jackson
to parse the String.
The exact logic depends on your use case.
However, you should be careful to not initialize a new parser in each map()
call, because that would b
Hi Fabian
Thanks for the reply. Yes my json is separated by new lines. It would have
been great if you had explained the function that goes inside the map. I
tried to use the 'scala.util.parsing.json._' library but got no luck.
On Tue, Apr 26, 2016 at 1:11 PM, Fabian Hueske wrote:
> Hi Punit,
>
Hi Punit,
JSON can be hard to parse in parallel due to its nested structure. It
depends on the schema and (textual) representation of the JSON whether and
how it can be done. The problem is that a parallel input format needs to be
able to identify record boundaries without context information. Thi
Hi
I am new to Flink. I was experimenting with the Dataset API and found out
that there is no explicit method for loading a JSON file as input. Can
anyone please suggest me a workaround?
--
Thank You
Regards
Punit Naik
15 matches
Mail list logo
