Looking at what happens with pdsh, there are two things that go wrong.
1. pdsh is installed in a node other than where the job manager would run,
so invoking *start-cluster *from there does not spawn a job manager. Only
if I do start-cluster from the node I specify as the job manager's node
that it'll be created.
2. If the slaves file has the same IP more than once, then the following
error happens trying to move log files. For example I had node j-029
specified twice in my slaves file.
j-020: mv: cannot move
`/N/u/sekanaya/sali/software/flink-1.0.3/log/flink-sekanaya-taskmanager-26-j-020.log'
to
`/N/u/sekanaya/sali/software/flink-1.0.3/log/flink-sekanaya-taskmanager-26-j-020.log.1':
No such file or directory
j-020: mv: cannot move
`/N/u/sekanaya/sali/software/flink-1.0.3/log/flink-sekanaya-taskmanager-26-j-020.out'
to
`/N/u/sekanaya/sali/software/flink-1.0.3/log/flink-sekanaya-taskmanager-26-j-020.out.1':
No such file or directory
On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake <esal...@gmail.com>
wrote:
> I meant, I'll check when current jobs are done and will let you know.
>
> On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake <esal...@gmail.com>
> wrote:
>
>> I am running some jobs now. I'll stop and restart using pdsh to see what
>> was the issue again
>>
>> On Mon, Jul 11, 2016 at 12:15 PM, Greg Hogan <c...@greghogan.com> wrote:
>>
>>> I'd definitely be interested to hear any insight into what failed when
>>> starting the taskmanagers with pdsh. Did the command fail, or fallback to
>>> standard ssh, a parse error on the slaves file?
>>>
>>> I'm wondering if we need to escape
>>> PDSH_SSH_ARGS_APPEND=$FLINK_SSH_OPTS
>>> as
>>> PDSH_SSH_ARGS_APPEND="${FLINK_SSH_OPTS}"
>>>
>>> On Mon, Jul 11, 2016 at 12:02 AM, Saliya Ekanayake <esal...@gmail.com>
>>> wrote:
>>>
>>>> pdsh is available in head node only, but when I tried to do *start-cluster
>>>> *from head node (note Job manager node is not head node) it didn't
>>>> work, which is why I modified the scripts.
>>>>
>>>> Yes, exactly, this is what I was trying to do. My research area has
>>>> been on these NUMA related issues and binding a process to a socket (CPU)
>>>> and then its thread to individual cores have shown great advantage. I
>>>> actually have Java code that automatically (user configurable as well) bind
>>>> processes and threads. For Flink, I've manually done this using shell
>>>> script that scans TMs in a node and pin them appropriately. This approach
>>>> is OK, but it's better if the support is integrated to Flink.
>>>>
>>>> On Sun, Jul 10, 2016 at 8:33 PM, Greg Hogan <c...@greghogan.com> wrote:
>>>>
>>>>> Hi Saliya,
>>>>>
>>>>> Would you happen to have pdsh (parallel distributed shell) installed?
>>>>> If so the TaskManager startup in start-cluster.sh will run in parallel.
>>>>>
>>>>> As to running 24 TaskManagers together, are these running across
>>>>> multiple NUMA nodes? I had filed FLINK-3163 (
>>>>> https://issues.apache.org/jira/browse/FLINK-3163) last year as I have
>>>>> seen that even with only two NUMA nodes performance is improved by binding
>>>>> TaskManagers, both memory and CPU. I think we can improve configuration of
>>>>> task slots as we do with memory, where the latter can be a fixed measure
>>>>> or
>>>>> a fraction relative to total memory.
>>>>>
>>>>> Greg
>>>>>
>>>>> On Sat, Jul 9, 2016 at 3:44 AM, Saliya Ekanayake <esal...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The current start/stop scripts SSH worker nodes each time they appear
>>>>>> in the slaves file. When spawning multiple TMs (like 24 per node), this
>>>>>> is
>>>>>> very inefficient.
>>>>>>
>>>>>> I've changed the scripts to do one SSH per node and spawn a given N
>>>>>> number of TMs afterwards. I can make a pull request if this seems usable
>>>>>> to
>>>>>> others. For now, I assume slaves file will indicate the number of TMs per
>>>>>> slave in "IP N" format.
>>>>>>
>>>>>> Thank you,
>>>>>> Saliya
>>>>>>
>>>>>> --
>>>>>> Saliya Ekanayake
>>>>>> Ph.D. Candidate | Research Assistant
>>>>>> School of Informatics and Computing | Digital Science Center
>>>>>> Indiana University, Bloomington
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Saliya Ekanayake
>>>> Ph.D. Candidate | Research Assistant
>>>> School of Informatics and Computing | Digital Science Center
>>>> Indiana University, Bloomington
>>>>
>>>>
>>>
>>
>>
>> --
>> Saliya Ekanayake
>> Ph.D. Candidate | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>>
>>
>
>
> --
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
>
>
--
Saliya Ekanayake
Ph.D. Candidate | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
