Hi Maxim,
Both Yarn per-job and session cluster should work. Since before the
JobManager and TaskManager
launcher, Yarn NodeManager could guarantee that all the local resources
have been localized and
accessible.
If do not want to use the getResource to read the file and use File
interface instead, you should use
"lib/job.properties" as your file path. Because the PWD of JobManager and
TaskManager process is
the Yarn container workdir.
If it still could not work, do you mind to share your codes and TaskManager
logs?
Best,
Yang
Maxim Parkachov <lazy.gop...@gmail.com> 于2020年2月18日周二 上午12:31写道：
> Hi Yang,
>
> I've just tried your suggestions, but, unfortunately, in yarn per job mode
> it doesn't work, both commands return null.
> I double checked that file is shipped to yarn container, but I feel that
> it happens later in process.
> At the moment I'm reading file with File interface, instead of getting it
> as resource, which I do in local mode.
>
> Regards,
> Maxim.
>
>
> On Mon, Feb 17, 2020 at 3:03 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi Maxim,
>>
>> I have verified that the following two ways could both work.
>>
>> getClass().getClassLoader().getResource("lib/job.properties")
>> getClass().getClassLoader().getResource("job.properties")
>>
>>
>> Best,
>> Yang
>>
>> Maxim Parkachov <lazy.gop...@gmail.com> 于2020年2月17日周一 下午6:47写道：
>>
>>> Hi Yang,
>>>
>>> thanks, this explains why classpath behavior changed, but now I struggle
>>> to
>>> understand how I could overwrite resource, which is already shipped in
>>> job jar.
>>>
>>> Before I had job.properties files in JAR in under
>>> resources/lib/job.properties
>>> for local development and deploying on cluster it was overwritten
>>> with environment specific settings in lib/job.properties of flink
>>> distribution.
>>> Now this doesn't seem to work. I'm using:
>>>
>>> getClass.getClassLoader.getResource("lib/job.properties")
>>>
>>> to get file. Could it be the problem ?
>>>
>>> Thanks,
>>> Maxim.
>>>
>>> On Mon, Feb 17, 2020 at 4:12 AM Yang Wang <danrtsey...@gmail.com> wrote:
>>>
>>>> Hi Maxim Parkachov,
>>>>
>>>> The users files also have been shipped to JobManager and TaskManager.
>>>> However, it
>>>> is not directly added to the classpath. Instead, the parent directory
>>>> is added to the
>>>> classpath. This changes are to make resource classloading work. You
>>>> could check more
>>>> information here[1].
>>>>
>>>>
>>>> [1]. https://issues.apache.org/jira/browse/FLINK-13127
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> Maxim Parkachov <lazy.gop...@gmail.com> 于2020年2月15日周六 上午12:58写道：
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I'm trying to run my job with flink 1.10 with YARN cluster per-job
>>>>> mode. In the previous versions all files in lib/ folder were automatically
>>>>> included in classpath. Now, with 1.10 I see only *.jar files are included
>>>>> in classpath. but not "other" files. Is this deliberate change or bug ?
>>>>>
>>>>> Generally, what is recommended way to include custom files in
>>>>> classpath and ship it during start to all containers ?
>>>>>
>>>>> Thank
>>>>>
>>>>
