Hi to all.
Thank you for your quick response. I will try both of your solutions in
order to find which one I the best.
Thanks,
Stefanos
On 7 Aug 2015 20:40, "Robert Metzger" wrote:
> Hi Stefanos,
>
> you can also write yourself a little script/tool which is periodically
> requesting the followi
Hi Stefanos,
you can also write yourself a little script/tool which is periodically
requesting the following JSON from the JobManager:
http://localhost:8081/setupInfo?get=taskmanagers&_=1438972693441
It returns a JSON string like this:
{"taskmanagers":[{"path":"akka:\/\/flink\/user\/taskmanager
Hi,
You can use something like iostat to extract the CPU usage.
For instance, I call this script on the JM node:
#!/usr/bin/env bash
lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s`
echo "${lines}"
IFS="," read -ra hostnames <<<"${lines}"
for line in "${hostnames[@]}"; do
ssh ${line}
Hi to all.
I am working an a research project using flink and i would like to extract the
CPU + RAM resources consumed on each worker in order to include the stats in my
paper. Can anyone advice me on how could i extract them?
Thanks in advance.
Stefanos
Hey Zhijiang Wang,
I will update the docs next week with more information. The short version is
that flow control happens via the buffer pools that Flink uses for produced and
consumed intermediate results.
The slightly ;) longer version:
Each task has buffer pools. The size of these buffer po
Sorry Fabian but I don't understand what I should do :(
Could you provide me a simple snippet of code to achieve this?
On Fri, Aug 7, 2015 at 1:30 PM, Fabian Hueske wrote:
> Enumeration of nested files is a feature of the FileInputFormat.
> If you implement your own IF based on FileInputFormat a
Enumeration of nested files is a feature of the FileInputFormat.
If you implement your own IF based on FileInputFormat as I suggested
before, you can use that feature.
2015-08-07 12:29 GMT+02:00 Flavio Pompermaier :
> I have a directory containing a list of files, each one containing a
> kryo-ser
I have a directory containing a list of files, each one containing a
kryo-serialized object.
With json serialized objects I don't have that problem (but there I use
env.readTextFile(path.withParameters(parameters)
where parameters has the ENUMERATE_NESTED_FILES_FLAG set to true).
On Fri, Aug 7, 2
I don't know your use case.
The InputFormat interface is very flexible. Directories can be recursively
read. A file can contain one or more objects. You can also make a smarter
IF and put multiple (small) files into one split...
It is up to your use case what you need to implement.
2015-08-07 12
Should this be the case just reading recursively an entire directory
containing one object per file?
On Fri, Aug 7, 2015 at 12:04 PM, Fabian Hueske wrote:
> You could implement your own InputFormat based on FileInputFormat and
> overwrite the createInputSplits method to just create a single spli
You could implement your own InputFormat based on FileInputFormat and
overwrite the createInputSplits method to just create a single split per
file.
2015-08-07 12:02 GMT+02:00 Flavio Pompermaier :
> So what should I do?
>
> On Fri, Aug 7, 2015 at 12:01 PM, Fabian Hueske wrote:
>
>> Ah, I checked
So what should I do?
On Fri, Aug 7, 2015 at 12:01 PM, Fabian Hueske wrote:
> Ah, I checked the code.
>
> The BinaryInputFormat expects metadata which is written be the
> BinaryOutputFormat.
> So you cannot use the BinaryInputFormat to read a file which does not
> provide the metadata.
>
> 2015-0
Ah, I checked the code.
The BinaryInputFormat expects metadata which is written be the
BinaryOutputFormat.
So you cannot use the BinaryInputFormat to read a file which does not
provide the metadata.
2015-08-07 11:53 GMT+02:00 Flavio Pompermaier :
> The file containing the serialized object is 7
The file containing the serialized object is 7 bytes
On Fri, Aug 7, 2015 at 11:49 AM, Fabian Hueske wrote:
> This might be an issue with the blockSize parameter of the
> BinaryInputFormat.
> How large is the file with the single object?
>
> 2015-08-07 11:37 GMT+02:00 Flavio Pompermaier :
>
>> I
This might be an issue with the blockSize parameter of the
BinaryInputFormat.
How large is the file with the single object?
2015-08-07 11:37 GMT+02:00 Flavio Pompermaier :
> I also tried with
>
> DataSet ds = env.createInput(inputFormat).setParallelism(1);
>
> but I get the same error :(
>
> More
I also tried with
DataSet ds = env.createInput(inputFormat).setParallelism(1);
but I get the same error :(
Moreover, in this example I put exactly one object per file so it should be
able to deserialize it, right?
On Fri, Aug 7, 2015 at 11:33 AM, Fabian Hueske wrote:
> If you create your file
If you create your file by just sequentially writing all objects to the
file using Kryo, you can only read it with a parallelism of 1.
Writing binary files in a way that they can be read in parallel is a bit
tricky (and not specific to Flink).
2015-08-07 11:28 GMT+02:00 Flavio Pompermaier :
> Hi
Hi to all,
I;m trying to read a file serialized with kryo but I get this exception
(due to the fact that the createInputSplits creates 8 inputsplits, where
just one is not empty..).
Caused by: java.io.IOException: Invalid argument
at sun.nio.ch.FileChannelImpl.position0(Native Method)
at sun.nio.c
18 matches
Mail list logo
