I understood how I can assign labels to yarn nodes. But it is still not clear to me how can I specify the node label for a samza application. I am referring to the section "Specifying node label for application" in the link <https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/NodeLabel.html> you shared in your last email.
On Wed, Dec 18, 2019 at 11:17 PM Yang Zhang <zhan3...@umn.edu> wrote: > Hi Debraj Manna, > > The app-def in previous email is just an example where you can configure > node labels. Yarn node labels > < > https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/NodeLabel.html > > > is > a general feature (not specific to Samza), and it depends on the > configuration system your system uses. The example uses xml format to > configure Samza job, but Samza as a framework, it does not restrict > configuration format. Please let us know if you have further questions, and > we should detail the documents in OSS to describe the usage of certain > features. > > Best, > Yang > > On Tue, Dec 17, 2019 at 9:58 PM Debraj Manna <subharaj.ma...@gmail.com> > wrote: > > > Thanks, Yang for replaying. > > > > Yes, my use case is almost similar. > > > > Can you let me know which app-def you are referring to? I am not able to > > locate yarn.am.container.label in samza-configurations > > < > > > http://samza.apache.org/learn/documentation/latest/jobs/samza-configurations.html > > > > > . > > Is there any samza project whose code I can refer to regarding the usage > of > > these configurations? > > > > On Wed, Dec 18, 2019 at 7:42 AM Yang Zhang <zhan3...@umn.edu> wrote: > > > > > Hello Debraj, > > > > > > We do not have a formal documentation in open source to describe how > yarn > > > node label is used in general. In contrast, we have an example of using > > > yarn node label to specify Samza container to run over "HDD" rather > than > > > default "SSD" nodes. Please take a look at the following guide and let > us > > > know whether it can be applied for your use case. Thank you for > reporting > > > this issue! > > > =================================================Step-by-step guide > > > > > > > > > 1. > > > > > > Add the *yarn.container.label *and* yarn.am.container.label* to the > > > job's *app-def* if not already present. The default of an empty > string > > > will keep the current default behavior of using SSD nodes. > > > <?xml version="1.0" encoding="UTF-8"?> > > > <application > xmlns="urn:com:linkedin:ns:configuration:definition:1.0" > > > name="my-application" version=""> > > > <configuration-definition> > > > <property name="yarn.am.container.label" default="" /> <!-- > > the > > > label used for launching the application master --> > > > <property name="yarn.container.label" default="" /> <!-- the > > > label used for other containers --> > > > </configuration-definition> > > > </application> > > > 2. If you had to modify your *app-def* in step 1, you will need to > do > > a > > > trigger-build to get the change to take effect. > > > 3. > > > > > > Add the label to *application.src* for your job. The *hdd* label > will > > > assign your containers to machines with spinning disks instead of > > solid > > > state disks. > > > <?xml version="1.0" encoding="UTF-8"?> > > > <application xmlns="urn:com:linkedin:ns:configuration:source:1.0" > > > name="my-application"> > > > <configuration-source> > > > <property name="yarn.container.label" value="hdd" /> > > > <property name="yarn.am.container.label" value="hdd" /> > > > </configuration-source> > > > </application> > > > > > > > > > 4. > > > > > > Deploy. > > > > > > ================================================= > > > > > > > > > Best, > > > > > > Yang > > > > > > On Tue, Dec 17, 2019 at 10:13 AM Debraj Manna < > subharaj.ma...@gmail.com> > > > wrote: > > > > > > > Hi > > > > > > > > I am seeing running samza with yarn node label is resolved in 0.12. > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/SAMZA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel > > > > > > > > But I am not able to locate the relevant documentation in samza-yarn > > > > documentation > > > > < > > > > https://samza.apache.org/learn/documentation/latest/deployment/yarn.html > > > > > > > > > > > Can someone point me to the relevant documentation? > > > > > > > > > >
