look into the code for this. Most of the relevant implementations
> > are found in the "org.apache.flink.runtime.iterative.task" package.
> >
> > Hope this helps...
> >
> > Ufuk
> >
> >
> > On Sun, Jul 17, 2016 at 9:36 PM, Saliya Ekanayake
> wrote:
> >> Hi,
>
Hi,
I am trying to understand what's the intermediate caching support in Flink.
For example, when there's an iterative dataset what's being cached between
iterations. Is there some documentation on this?
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assi
ow’s operators, not sure how to do that.
>
> Best,
> Ovidiu
>
> On 12 Jul 2016, at 21:18, Saliya Ekanayake wrote:
>
> Hi Ovidiu,
>
> Checking the /var/log/messages based on Greg's response revealed TMs were
> killed due to out of memory. Here's the node
i,
>
> Can you post your configuration parameters (exclude default settings) and
> cluster description?
>
> Best,
> Ovidiu
>
> On 11 Jul 2016, at 17:49, Saliya Ekanayake wrote:
>
> Thank you Greg, I'll check if this was the cause for my TMs to disappear.
>
>
config.sh:318 in rotateLogFile. The way we
> generate the taskmanager index assumes that taskmanagers are started
> sequentially (flink-daemon.sh:108).
>
> On Mon, Jul 11, 2016 at 2:59 PM, Saliya Ekanayake
> wrote:
>
>> Looking at what happens with pdsh, there are two things that go wr
-1.0.3/log/flink-sekanaya-taskmanager-26-j-020.out.1':
No such file or directory
On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake
wrote:
> I meant, I'll check when current jobs are done and will let you know.
>
> On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake
> wrote:
I meant, I'll check when current jobs are done and will let you know.
On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake
wrote:
> I am running some jobs now. I'll stop and restart using pdsh to see what
> was the issue again
>
> On Mon, Jul 11, 2016 at 12:15 PM, Greg
, or fallback to
> standard ssh, a parse error on the slaves file?
>
> I'm wondering if we need to escape
> PDSH_SSH_ARGS_APPEND=$FLINK_SSH_OPTS
> as
> PDSH_SSH_ARGS_APPEND="${FLINK_SSH_OPTS}"
>
> On Mon, Jul 11, 2016 at 12:02 AM, Saliya Ekanayake
> w
lavors
> may use /var/log/syslog.
>
> http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer
>
> On Sun, Jul 10, 2016 at 11:55 PM, Saliya Ekanayake
> wrote:
>
>> Greg,
>>
>> where did you see the OOM log as shown in
Thank you, Ufuk!
On Mon, Jul 11, 2016 at 10:46 AM, Ufuk Celebi wrote:
> Yes, via jobmanager.web.history
> (
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#jobmanager-web-frontend
> )
>
> On Mon, Jul 11, 2016 at 4:45 PM, Saliya Ekanayak
Hi,
It seems by default the completed job list only shows 5 entries. Is there a
way to increase this?
Thank you,
saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
inding
> TaskManagers, both memory and CPU. I think we can improve configuration of
> task slots as we do with memory, where the latter can be a fixed measure or
> a fraction relative to total memory.
>
> Greg
>
> On Sat, Jul 9, 2016 at 3:44 AM, Saliya Ekanayake
> wrote:
&
nk can have some unexpected memory usage which can result in an
> OOM kill by the kernel, and this becomes more pronounced as the cluster
> size grows.
> https://www.mail-archive.com/dev@flink.apache.org/msg06346.html
>
> On Fri, Jul 8, 2016 at 12:46 PM, Saliya Ekanayake
> wrote:
g on big machines where you might want to run multiple task managers
> just to split the memory between multiple java processes.
>
> In any case the previous config format should also be supported as the
> default.
>
> I am curious what other developers/users think about this.
>
this seems usable to
others. For now, I assume slaves file will indicate the number of TMs per
slave in "IP N" format.
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
controlling processes?
>
>
> On Thu, Jul 7, 2016 at 5:46 PM, Saliya Ekanayake
> wrote:
>
>> I see two logs (attached), but there's only 1 TaskManger process. Also,
>> the Web console says it can find only 1 TM.
>>
>> However, I see this part in JM log,
and 1mil points.
Number of centers were 100 to 1000. None of these sizes completed.
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
econd TM on that machine
> (taskmanager-X-j-011.log where X is the TM number)? If yes, the task
> manager process does start up and there is another problem. If not,
> the task managers seems not to start even.
>
> – Ufuk
>
> On Thu, Jul 7, 2016 at 7:34 AM, Saliya Ekanayake
>
need to do?
On Sun, Jul 3, 2016 at 11:28 AM, Ufuk Celebi wrote:
> Yes, exactly.
>
> On Sat, Jul 2, 2016 at 6:28 PM, Saliya Ekanayake
> wrote:
> > Thank you, yes, it can be done externally, if not supported within Flink.
> >
> > So the way to spawn multiple task mana
Thank you!
On Sun, Jul 3, 2016 at 11:28 AM, Ufuk Celebi wrote:
> Yes, exactly.
>
> On Sat, Jul 2, 2016 at 6:28 PM, Saliya Ekanayake
> wrote:
> > Thank you, yes, it can be done externally, if not supported within Flink.
> >
> > So the way to spawn multiple task
ike something like the OS or
> resource manager should handle.
>
> On Sat, Jul 2, 2016 at 5:12 PM, Saliya Ekanayake
> wrote:
> > That's great, so is there support to pin task managers to sockets as
> well?
> >
> > On Sat, Jul 2, 2016 at 11:08 AM, Ufuk Celebi w
t;
>
> On Sat, Jul 2, 2016 at 3:26 AM, Saliya Ekanayake
> wrote:
> > Hi Ufuk,
> >
> > Looking at the document you sent it seems only 1 task manager per node
> exist
> > and within that you have multiple slots. Is it possible to run more than
> 1
> > task m
30, 2016 at 5:27 PM, Saliya Ekanayake wrote:
> Thank you, I'll check these.
>
> In 2.) you said they are likely to exchange through memory. Is there a
> case why they wouldn't?
>
> On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi wrote:
>
>> On Thu, Jun
Thank you, I'll check these.
In 2.) you said they are likely to exchange through memory. Is there a case
why they wouldn't?
On Thu, Jun 30, 2016 at 5:03 AM, Ufuk Celebi wrote:
> On Thu, Jun 30, 2016 at 1:44 AM, Saliya Ekanayake
> wrote:
> > 1. What parameters ar
there support for Infiniband interconnect?
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
ink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/utils/DataSetUtils.java#L77
>
> On Thu, Feb 25, 2016 at 5:52 PM, Saliya Ekanayake
> wrote:
>
>> Thank you, Marton. That seems doable.
>>
>> However, is there a way I can create a dummy indexed data set?
n you can join the two DataSets on the field containing 'i' and
> do the mapping on the joined result.
>
> Hope this helps,
>
> Marton
>
> On Thu, Feb 25, 2016 at 5:38 PM, Saliya Ekanayake
> wrote:
>
>> Hi,
>>
>> I've two data sets lik
ay to do this?
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
Cell 812-391-4914
http://saliya.org
On Tue, Feb 16, 2016 at 10:28 PM, Saliya Ekanayake
> wrote:
>
>> Thank you. I'll check this
>>
>> On Tue, Feb 16, 2016 at 4:01 PM, Fabian Hueske wrote:
>>
>>> Broadcasted DataSets are stored on the JVM heap of each task manager
>>> (but sh
roadcast sets as described above.
>
> 2016-02-16 21:54 GMT+01:00 Saliya Ekanayake :
>
>> Thank you, yes, this makes sense. The broadcasted data in my case would a
>> large array of 3D coordinates,
>>
>> On a side note, how can I take the output from a reduce function? I can
; In your case it might be even better to read the data twice instead of
> reading, writing, and reading it.
>
> Fabian
>
> 2016-02-16 21:15 GMT+01:00 Saliya Ekanayake :
>
>> I looked at the samples and I think what you meant is clear, but I didn't
>> find a so
he same dataset?
Thank you,
saliya
On Tue, Feb 16, 2016 at 12:03 PM, Fabian Hueske wrote:
> Yes, if you implement both maps in a single job, data is read once.
>
> 2016-02-16 15:53 GMT+01:00 Saliya Ekanayake :
>
>> Fabian,
>>
>> I've a quick follow-up question on w
\---> Map 2 --> SInk2
>
> and execute it at once.
> For that you define you data flow and call execute once after all sinks
> have been created.
>
> Best, Fabian
>
> 2016-02-15 21:32 GMT+01:00 Saliya Ekanayake :
>
>> Fabian,
>>
>> count() was j
les/java
>
> Best, Fabian
>
> 2016-02-15 22:03 GMT+01:00 Saliya Ekanayake :
>
>> Thank you, Fabian.
>>
>> Any chance you might have an example on how to define a data flow with
>> Flink?
>>
>>
>>
>> On Mon, Feb 15, 2016 at 3:58 PM, F
r all sinks
> have been created.
>
> Best, Fabian
>
> 2016-02-15 21:32 GMT+01:00 Saliya Ekanayake :
>
>> Fabian,
>>
>> count() was just an example. What I would like to do is say run two map
>> operations on the dataset (ds). Each map will have it's ow
unction.
>
> Best, Fabian
>
>
>
> 2016-02-15 21:05 GMT+01:00 Saliya Ekanayake :
>
>> Hi,
>>
>> I see that an InputFormat's open() and nextRecord() methods get called
>> for each terminal operation on a given dataset using that particular
>>
Input(smif,
BasicArrayTypeInfo.SHORT_ARRAY_TYPE_INFO);
MapOperator op = ds.map(...)
*op.reduce(...)*
*op.count(...)*
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
Cell 812-391
as a file with 64
> short values.
> I suggest to implement a custom input format based on FileInputFormat.
>
> Best, Fabian
>
> 2016-02-08 22:05 GMT+01:00 Saliya Ekanayake :
>
>> Thank you, Fabian. It solved the compilation error, but at runtime I get
>> an end-of-file
n Hueske wrote:
> Hi,
>
> please try to replace
> DataSet ds = env.createInput(sif);
> by
> DataSet ds = env.createInput(sif,
> ValueTypeInfo.SHORT_VALUE_TYPE_INFO);
>
> Best, Fabian
>
> 2016-02-08 19:33 GMT+01:00 Saliya Ekanayake :
>
>> Till,
>>
>
erializedInputFormat();
> inputFormat.setFilePath(PATH_TO_FILE);
>
> env.createInput(inputFormat, myTypeInfo);
>
> Cheers,
> Till
>
>
> On Mon, Feb 8, 2016 at 7:00 AM, Saliya Ekanayake
> wrote:
>
>> Hi,
>>
>> I was trying to read a simple binary file using
sultTypeQueryable interface
> in your custom source. That way you can manually specify the correct
> type. If that doesn't help you, could you please share more of the
> stack trace?
>
> Thanks,
> Max
>
> On Mon, Feb 8, 2016 at 7:00 AM, Saliya Ekanayake
> w
at could not be automatically determined. Please specify the
TypeInformation of the produced type explicitly by using the
'createInput(InputFormat, TypeInformation)' method instead.
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Co
be careful. Each node must have an exact copy of the data at
> exactly the same location. Otherwise, it won't work.
>
> Best, Fabian
>
> 2016-01-25 16:46 GMT+01:00 Saliya Ekanayake :
>
>> Hi Fabian,
>>
>> Thank you for the information.
>>
>> So, is
;
> Cheers, Fabian
>
> 2016-01-24 19:10 GMT+01:00 Suneel Marthi :
>
>> There should be a env.readbinaryfile() IIRC, check that
>>
>> Sent from my iPhone
>>
>> On Jan 24, 2016, at 12:44 PM, Saliya Ekanayake wrote:
>>
>> Thank you for the response on
Hadoop’s FileInputFormats and thus
> you can also do everything with Flink, what you can do with Hadoop. Simply
> take the same Hadoop FileInputFormat which you would take for your
> MapReduce job.
>
> Cheers,
> Till
>
>
> On Wed, Jan 20, 2016 at 3:16 PM, Saliya Ekanayake
utFormat to be able to read
> blocks of data from HDFS
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/api/common/io/BinaryInputFormat.html
> >
> > On Wed, Jan 20, 2016 at 12:45 AM, Saliya Ekanayake
> wrote:
> &g
block
and emit result to the reducer.
How is this done in Flink? I am new to Flink and couldn't find a binary
reader so far. Any help is greatly appreciated.
Thank you,
Saliya
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science C
47 matches
Mail list logo
