Re: Review Request 34011: Add support for a Graphite Metrics Reporter

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34011/#review83727 --- Thanks for the comments Yan! I will address them and will follow up

Re: Review Request 34011: Add support for a Graphite Metrics Reporter

> On May 14, 2015, 7:09 a.m., Luis De Pombo wrote: > > Thanks for the comments Yan! I will address them and will follow up on the > > issue around org.apache.samza.metrics.Snapshot.getMax() I will also add to the documentation! - Luis

Re: Review Request 34011: Add support for a Graphite Metrics Reporter

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34011/ --- (Updated May 14, 2015, 7:20 a.m.) Review request for samza. Repository: samza

Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34206/ --- Review request for samza, Yan Fang, Chris Riccomini, Guozhang Wang, Milinda Path

Review Request 34207: SAMZA-552: use OperatorCallback to allow implementation of callbacks w/o inheriting and creating many sub-classes from operators

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34207/ --- Review request for samza, Yan Fang, Chris Riccomini, Guozhang Wang, Milinda Path

Re: tkafka.common.ReplicaNotAvailableException on application logs

Here is the complete log: http://pastebin.com/nX7twETm Interesting, I see a leader not available exception instead of the earlier one. ./container_1431601903660_0001_01_02/samza-container-0.log:2015-05-14 04:53:41 BrokerPartitionInfo [WARN] Error while fetching metadata partition 0 leader: n

Re: tkafka.common.ReplicaNotAvailableException on application logs

Here is what I see on Kafka log: [2015-05-14 04:11:27,752] ERROR Closing socket for /10.180.195.32 because of error (kafka.network.Processor) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(S

Access config properties inside Job

Folks, We are using an external component in one of our tasks, the external component persists some info to MySQL DB, the component reads database connection info from a specific property file that should be available in classpath. Is there a way I can specify custom property files as part of

Logback logger

Hello, I’m using logback in a Samza project I’m working on. In the dist.tar.gz package slf4j-log4j12 is not included. But logback requires the configuration file to be in the class path and the run-class.sh script only includes .*jar packages from lib folder. Do you know how I can sort that issu

Re: Logback logger

We use logback as well and simply include logback.xml in the task's jar file. On 05/14/2015 08:48 AM, Davide Simoncelli wrote: Hello, I’m using logback in a Samza project I’m working on. In the dist.tar.gz package slf4j-log4j12 is not included. But logback requires the configuration file to be

Re: Logback logger

It is included and the root level is set to info. But it prints debug logs as well. Also looking how class path is built in run-class.sh, it doesn’t include the lib folder where the logback.xml is stored in. Davide > On 14 May 2015, at 3:55 pm, Tommy Becker wrote: > > We use logback as well

Re: Logback logger

Sorry for not being clear, I'm saying we include logback.xml in the tasks *jar* file, not just the Samza job's tarball. On 05/14/2015 11:19 AM, Davide Simoncelli wrote: It is included and the root level is set to info. But it prints debug logs as well. Also looking how class path is built in

Re: tkafka.common.ReplicaNotAvailableException on application logs

Hi Shekar, It seems the incoming / outgoing topics are not the root of the problem here, but the checkpoint topic "__samza_checkpoint_ver_1_for_Argos". From the error logs this topic only has one replica 1018019532, which was down and hence not available. Guozhang On Thu, May 14, 2015 at 5:16 AM

Log rotation on Samza/yarn logs

Hello, How are the Samza/yarn logs rotated as a best practice? We seem to be filling our disk. Should I resort to a Linux logrotate utility or is there any log4j config we can leverage? - Shekar

Review Request 34226: SAMZA-643: Improve error message for null stream in OutgoingMessageEnvelope

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34226/ --- Review request for samza. Repository: samza Description --- Improve erro

Re: Access config properties inside Job

Hi Elango, Currently Samza does not support having two config files, like --classpath-config-file. But there are a few workarounds: 1. you already mentioned putting the properties in the samza properties file 2. use key=value pair. so it would look like deploy/samza/bin/run-job.sh --config-facto

Re: tkafka.common.ReplicaNotAvailableException on application logs

Wang, That did it. Thanks a lot. - Shekar On Thu, May 14, 2015 at 10:38 AM, Guozhang Wang wrote: > Hi Shekar, > > It seems the incoming / outgoing topics are not the root of the problem > here, but the checkpoint topic "__samza_checkpoint_ver_1_for_Argos". From > the error logs this topic onl

Yarn redundancy

Hello, I have setup redundancy on resource manager based on this doc https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html I then shut down server 1 and was expecting that 02 server would take over. Instead I see this error. I am not sure if I am missing someth

Re: Yarn redundancy

Other observation I forgot to mention is that if I kill the rm and nm process, samza job seem to run properly. Only when 01 server is rebooted, I seem to encounter this error and as a result, no jobs get processed. - Shekar On Thu, May 14, 2015 at 12:14 PM, Shekar Tippur wrote: > Hello, > > I h

Re: Review Request 33735: RocksDB TTL support

> On May 12, 2015, 6:10 p.m., Mohamed Mahmoud (El-Geish) wrote: > > samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala, > > line 77 > > > > > > This kind of polymorphism should be ac

Re: Input stream partition to task mapping

Hi Yan, Thanks for your quick reply. Since we don't need to change our rules, we decided to use the consumer to read the rules in init(), as you suggested. Works great - thanks so much for your help! Susan On Tue, May 12, 2015 at 5:32 PM, Yan Fang wrote: > Hi Susan, > > -- " I have a custom Sy

Re: Yarn redundancy

Is the HA set correctly? The log looks like it's in the YARN setting side. Fang, Yan yanfang...@gmail.com On Thu, May 14, 2015 at 12:29 PM, Shekar Tippur wrote: > Other observation I forgot to mention is that if I kill the rm and nm > process, samza job seem to run properly. Only when 01 server

Re: Review Request 33735: RocksDB TTL support

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33735/#review83847 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On May 13, 2015

Re: Yarn redundancy

Yan, I have followed the doc. Here is what was done ... 1. Setup the yarn-site.xml yarn.resourcemanager.ha.enabled true yarn.resourcemanager.cluster-id cluster1 yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 sprdargas402.

Re: Log rotation on Samza/yarn logs

Hi Shekar, Currently by default, the log4j is using DailyRollingFileAppender. You can change the log4j.xml to config as you want. Usually, daily base is good enough. Thanks, Fang, Yan yanfang...@gmail.com On Thu, May 14, 2015 at 10:56 AM, Shekar Tippur wrote: > Hello, > > How are the Samza/ya

Re: Review Request 33735: RocksDB TTL support

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33735/#review83856 --- Ship it! Ship It! - Mohamed Mahmoud (El-Geish) On May 13, 2015,

Re: Review Request 33735: RocksDB TTL support

> On May 12, 2015, 6:10 p.m., Mohamed Mahmoud (El-Geish) wrote: > > samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala, > > line 77 > > > > > > This kind of polymorphism should be ac

Re: Log rotation on Samza/yarn logs

Hi, Shekar, Are you having a problem w/ retention of too many old log files on disk? I did a quick search online to see whether there is any configuration for DailyRollingFileAppender and couldn't find any. The closest thing is this one: http://wiki.apache.org/logging-log4j/DailyRollingFileAppende

Re: Log rotation on Samza/yarn logs

Thanks Yan. - Shekar On Thu, May 14, 2015 at 5:07 PM, Yi Pan wrote: > Hi, Shekar, > > Are you having a problem w/ retention of too many old log files on disk? I > did a quick search online to see whether there is any configuration for > DailyRollingFileAppender and couldn't find any. The closes

Re: Log rotation on Samza/yarn logs

Hey Shekar, You can also use the RollingFileAppender. You can both specify the maximum number of files and the maximum number of files you want to keep. https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxBackupIndex On May 14, 2015, at

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34206/ --- (Updated May 15, 2015, 12:59 a.m.) Review request for samza, Yan Fang, Chris Ri

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34206/ --- (Updated May 15, 2015, 2:16 a.m.) Review request for samza, Yan Fang, Chris Ric

Re: Logback logger

Thank you for your help. It worked. On 14 May 2015 5:17 pm, "Tommy Becker" wrote: > Sorry for not being clear, I'm saying we include logback.xml in the tasks > *jar* file, not just the Samza job's tarball. > > On 05/14/2015 11:19 AM, Davide Simoncelli wrote: > > It is included and the root level