Hello,
I have attached the container log and the job coordinator logs for the two
samples I ran, and the .properties configuration files used. The
wikipedia-feed and wikipedia-application code are exactly the same as in
the master branch on hello-samza. The only change is that I added the
metrics config to the wikipedia-feed.properties file.
ubuntu@ubuntu-xenial /vagrant/hello-samza % git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: src/main/config/wikipedia-feed.properties
no changes added to commit (use "git add" and/or "git commit -a")
ubuntu@ubuntu-xenial /vagrant/hello-samza % git rev-parse HEAD
f48892747dea196ab80eb0c76081efa86f0ba00c
I scanned through the log files but I do not see anything alarming. The
MetricsSnapshotReporter is being invoked in both places, but I only see
messages in the Kafka metrics stream for the wikipedia-feed job, not
wikipedia-application.
Is there anything else I can try? Thank you!
On Fri, Jan 26, 2018 at 6:32 PM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:
> Hi Srujun,
>
> Metrics reporting should be supported independent of the deployment model.
> Can you please attach your container log file?
>
> Are you seeing log lines from MetricsSnapshotReporter:
> https://github.com/apache/samza/blob/master/samza-core/
> src/main/scala/org/apache/samza/metrics/reporter/
> MetricsSnapshotReporter.scala
> ?
> Alternately, you could also set your log-level to DEBUG and verify if the
> snapshot reporter is being invoked.
>
> Best,
> Jagdish
>
--
Srujun Thanmay Gupta
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Application / Job
app.class=samza.examples.wikipedia.application.WikipediaApplication
app.runner.class=org.apache.samza.runtime.RemoteApplicationRunner
job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
job.name=wikipedia-application
job.default.system=kafka
# YARN
yarn.package.path=file:///vagrant/hello-samza/target/hello-samza-0.14.0-dist.tar.gz
# Wikipedia System
systems.wikipedia.samza.factory=samza.examples.wikipedia.system.WikipediaSystemFactory
systems.wikipedia.host=irc.wikimedia.org
systems.wikipedia.port=6667
# Kafka System
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.consumer.zookeeper.connect=localhost:2181
systems.kafka.producer.bootstrap.servers=localhost:9092
systems.kafka.default.stream.replication.factor=1
# Serializers
serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory
serializers.registry.integer.class=org.apache.samza.serializers.IntegerSerdeFactory
# Streams which are not on default system or have special characters in the
physical name.
streams.en-wikipedia.samza.system=wikipedia
streams.en-wikipedia.samza.physical.name=#en.wikipedia
streams.en-wiktionary.samza.system=wikipedia
streams.en-wiktionary.samza.physical.name=#en.wiktionary
streams.en-wikinews.samza.system=wikipedia
streams.en-wikinews.samza.physical.name=#en.wikinews
# Key-value storage
stores.wikipedia-stats.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.wikipedia-stats.changelog=kafka.wikipedia-stats-changelog
stores.wikipedia-stats.key.serde=string
stores.wikipedia-stats.msg.serde=integer
# Metrics
metrics.reporters=snapshot,jmx
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
metrics.reporter.snapshot.stream=kafka.metrics
metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Job
job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
job.name=wikipedia-feed
# YARN
yarn.package.path=file:///vagrant/hello-samza/target/hello-samza-0.14.0-dist.tar.gz
# Task
task.class=samza.examples.wikipedia.task.WikipediaFeedStreamTask
task.inputs=wikipedia.#en.wikipedia,wikipedia.#en.wiktionary,wikipedia.#en.wikinews
# Serializers
serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory
# Wikipedia System
systems.wikipedia.samza.factory=samza.examples.wikipedia.system.WikipediaSystemFactory
systems.wikipedia.host=irc.wikimedia.org
systems.wikipedia.port=6667
# Kafka System
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.samza.msg.serde=json
systems.kafka.consumer.zookeeper.connect=localhost:2181/
systems.kafka.producer.bootstrap.servers=localhost:9092
# Job Coordinator
job.coordinator.system=kafka
# Add configuration to disable checkpointing for this job once it is available
in the Coordinator Stream model
# See
https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14533346&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14533346
for more details
job.coordinator.replication.factor=1
# Metrics
metrics.reporters=snapshot,jmx
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
metrics.reporter.snapshot.stream=kafka.metrics
metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory
