[GitHub] samza pull request #13: removing git pull unsupported line from README

2016-06-23 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/13 removing git pull unsupported line from README Testing pull requests! You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza test-branch

[GitHub] samza pull request #13: removing git pull unsupported line from README

2016-06-23 Thread navina
Github user navina closed the pull request at: https://github.com/apache/samza/pull/13 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] samza pull request #16: SAMZA-1030 : Add documentation for change in the con...

2016-09-29 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/16 SAMZA-1030 : Add documentation for change in the contribution process I re-organized some of the website files and created a "Contributor's Corner" that collates all info related to n

[GitHub] samza pull request #17: SAMZA-1033: Remove import-control from checkstyle

2016-10-05 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/17 SAMZA-1033: Remove import-control from checkstyle You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza SAMZA-1033 Alternatively you can

[GitHub] samza pull request #27: Adding Build Status to README.md

2016-12-02 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/27 Adding Build Status to README.md I setup a Jenkins Build Job to get triggered every time changes are pushed to samza repository. https://builds.apache.org/view/S-Z/view/Samza/job/samza-freestyle

[GitHub] samza pull request #44: Initial Standalone JobCoordinator and StreamProcesso...

2017-01-24 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/44 Initial Standalone JobCoordinator and StreamProcessor API This patch contains changes associated with the Standalone StreamProcessor, where there is no coordination. This will work for load-balanced

[GitHub] samza pull request #48: SAMZA-1082 : Implement Leader Election using ZK

2017-02-06 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/48 SAMZA-1082 : Implement Leader Election using ZK Simple implementation of leader election recipe along with unit tests You can merge this pull request into a Git repository by running: $ git pull

[GitHub] samza pull request #54: SAMZA-1084 - User thread does not see errors from th...

2017-02-15 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/54 SAMZA-1084 - User thread does not see errors from the processor thread Adding a ProcessorLifecycleCallback for specifying user-defined error handling You can merge this pull request into a Git

[GitHub] samza pull request #102: SAMZA-1175 - Removing CoordinationService from JobC...

2017-03-29 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/102 SAMZA-1175 - Removing CoordinationService from JobCoordinatorFactory interface @sborya I have made the changes. Please review. Thanks! You can merge this pull request into a Git repository by

[GitHub] samza pull request #103: SAMZA-1126 - Semantics of processorId in Samza

2017-03-29 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/103 SAMZA-1126 - Semantics of processorId in Samza Implementation based on [SEP-1](https://cwiki.apache.org/confluence/display/SAMZA/SEP-1%3A+Semantics+of+ProcessorId+in+Samza) You can merge this

[GitHub] samza pull request #107: SAMZA-1182 - Commenting out some of the flaky tests

2017-03-31 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/107 SAMZA-1182 - Commenting out some of the flaky tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza SAMZA-1182 Alternatively you

[GitHub] samza pull request #112: Samza-1187 : TestZkProcessorLatch tests share state...

2017-04-05 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/112 Samza-1187 : TestZkProcessorLatch tests share state causing transient failures in CI builds * Removed testSingleCountdown - didn't understand what was being tested * The timeout behavio

[GitHub] samza pull request #121: SAMZA-1208 - IllegalFormatConversionException in Lo...

2017-04-12 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/121 SAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza SAMZA-1208

[GitHub] samza pull request #121: SAMZA-1208 - IllegalFormatConversionException in Lo...

2017-04-12 Thread navina
Github user navina closed the pull request at: https://github.com/apache/samza/pull/121 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] samza pull request #121: SAMZA-1208 - IllegalFormatConversionException in Lo...

2017-04-12 Thread navina
GitHub user navina reopened a pull request: https://github.com/apache/samza/pull/121 SAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza SAMZA-1208

[GitHub] samza pull request #125: SAMZA-1213 - StreamProcessorLifeCycleAware interfac...

2017-04-17 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/125 SAMZA-1213 - StreamProcessorLifeCycleAware interface should not use processorId Refactoring LocalApplicationRunner s.t. each processor has its own listener instance, instead of a single listener

[GitHub] samza pull request #139: SAMZA-1220 : Add thread name to SamzaContainer shut...

2017-04-25 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/139 SAMZA-1220 : Add thread name to SamzaContainer shutdown hook and prevent shutdown deadlock * SamzaContainerExceptionHandler is written in Java and used by LocalContainerRunner.java You can merge

[GitHub] samza pull request #146: SAMZA-1224 : Revert job coordinator factory config ...

2017-04-27 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/146 SAMZA-1224 : Revert job coordinator factory config to the old format We didn't release since adding this config. So, it is ok to change the format now. You can merge this pull request into

[GitHub] samza pull request #148: SAMZA-1212 - Refactor interaction between StreamPro...

2017-04-28 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/148 SAMZA-1212 - Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer See SAMZA-1212 for motivation toward this refactoring. Changes here are: * Removed awaitStart

[GitHub] samza pull request #153: SAMZA-1251 - Remove DebounceTimer dependency from Z...

2017-05-01 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/153 SAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector Fyi: This PR depends on PR #148 You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] samza pull request #54: SAMZA-1084 - User thread does not see errors from th...

2017-05-03 Thread navina
Github user navina closed the pull request at: https://github.com/apache/samza/pull/54 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] samza pull request #162: SAMZA-1228 : StreamProcessor should stop JmxServer

2017-05-03 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/162 SAMZA-1228 : StreamProcessor should stop JmxServer This is not the solution posted in SAMZA-1228. For now, we are moving jmxserver lifecycle to be within the container. Ideally, it should be within

[GitHub] samza pull request #166: SAMZA-1150 : Handling Error propagation between ZkJ...

2017-05-05 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/166 SAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer * Treats all errors in jobcoordinator as FATAL and shuts-down the streamprocessor * [Bug] Fixed bug reporte

[GitHub] samza pull request #173: SAMZA-1272 : ZkCoordinationUtils deletes the entire...

2017-05-08 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/173 SAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset * ZkCoordinationUtils has a reset interface that deletes the entire Zk tree. This is not desirable. * Also, fixed flakiness in

[GitHub] samza pull request #179: SAMZA-1276 : Adding customReporters passed by Strea...

2017-05-09 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/179 SAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer SamzaContainer is not adding custom reporters along side class-loaded reporters from config. This was missed in

[GitHub] samza pull request #195: SAMZA-1128 : Remove dependency of debounce timer fr...

2017-05-18 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/195 SAMZA-1128 : Remove dependency of debounce timer from the CoordinationUtils This patch addresses the following: * Removes CoordinationUtils#getBarrier, BarrierForVersionUpgrade interface

[GitHub] samza pull request #201: Updating committer page on samza.apache.org

2017-05-23 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/201 Updating committer page on samza.apache.org Giving more screen time for currently active Apache Samza committers (no JIRA) You can merge this pull request into a Git repository by running

[GitHub] samza pull request #202: SAMZA-1307 - Fix ZkKeyBuilder null checks for pathP...

2017-05-23 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/202 SAMZA-1307 - Fix ZkKeyBuilder null checks for pathPrefix You can merge this pull request into a Git repository by running: $ git pull https://github.com/navina/samza SAMZA-1307 Alternatively

[GitHub] samza pull request #230: SAMZA-1340 - StreamProcessor does not propagate con...

2017-06-21 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/230 SAMZA-1340 - StreamProcessor does not propagate container failures from StreamTask Storing the exception seen from the container in the `SamzaContainerListener#onFailure(Throwable)` in the

[GitHub] samza pull request #274: SAMZA-1396 TestZkLocalApplicationRunner tests fails...

2017-08-16 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/274 SAMZA-1396 TestZkLocalApplicationRunner tests fails after SAMZA-1385 * Fixes ZkPath issues * Fixes appname / jobname mismatch You can merge this pull request into a Git repository by running

[GitHub] samza pull request #301: Samza versioned Release Notes

2017-09-25 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/301 Samza versioned Release Notes Adding a versioned page for release/upgrade notes. We can start this process from the next major version release, aka 0.14.0. Please update this page as and

[GitHub] samza pull request #302: Fixing broken link between Yarn Host Affinity and R...

2017-09-25 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/302 Fixing broken link between Yarn Host Affinity and Resource Localizati… Fixing broken link between Yarn Host Affinity and Resource Localization pages under Documentation Patch needs to be

[GitHub] samza pull request #379: SAMZA-1523 Cleanup table entries before shutting do...

2017-12-05 Thread navina
GitHub user navina opened a pull request: https://github.com/apache/samza/pull/379 SAMZA-1523 Cleanup table entries before shutting down the processor Modified the `TableUtils#deleteProcessorEntity` to provide an option to disable optimistic locking during a call to Azure Table

Re: Testing dev@samza.apache.org

2015-01-23 Thread Navina Ramesh
Yep! On 1/23/15, 9:56 AM, "Yi Pan" wrote: >Yes! > >On Fri, Jan 23, 2015 at 9:49 AM, Chris Riccomini >wrote: > >> Hey all, >> >> Could you please confirm that you're seeing this? I'm trying to verify >>the >> TLP migration for: >> >> https://issues.apache.org/jira/browse/INFRA-9055 >> >> Cheer

Re: Review Request 29899: Changes for Kafka producer upgrade

2015-01-23 Thread Navina Ramesh
ile831263line26> > > > > Would org.apache.kafka.clients.producer.MockProducer be configured to > > satisfy the requests here? I have addresssed the same question in https://reviews.apache.org/r/29899/#comment113812 - Navina -

Re: Review Request 29899: Changes for Kafka producer upgrade

2015-01-23 Thread Navina Ramesh
ni wrote: > > samza-test/src/main/config/negate-number.properties, line 47 > > <https://reviews.apache.org/r/29899/diff/4/?file=831268#file831268line47> > > > > Can you do a quick grep through the code base, adn make

Re: Review Request 29899: Changes for Kafka producer upgrade

2015-01-23 Thread Navina Ramesh
clean build - SUCCESSFUL Thanks, Navina Ramesh

Re: Review Request 29899: Changes for Kafka producer upgrade

2015-01-23 Thread Navina Ramesh
.kafka.clients.producer.MockProducer be configured to > > satisfy the requests here? > > Navina Ramesh wrote: > I have addresssed the same question in > https://reviews.apache.org/r/29899/#comment113812 Created a follow-up task here - ht

Re: Review Request 29899: Changes for Kafka producer upgrade

2015-01-27 Thread Navina Ramesh
negate-number.properties 4989b279a22ffd8acdb2c31ca4813f6768edc75b samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ca25258217e5ebc44b34fbc4d69ecb28c81df618 Diff: https://reviews.apache.org/r/29899/diff/ Testing --- ./gradlew clean build - SUCCESSFUL Thanks, Navina Ramesh

Review Request 30634: Serde for Stream Tuples

2015-02-04 Thread Navina Ramesh
/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION settings.gradle 3a01fd66359b8c79954ae8f34eeaf4b2e3fdc0b4 Diff: https://reviews.apache.org/r/30634/diff/ Testing --- Thanks, Navina Ramesh

Re: Review Request 30634: Serde for Stream Tuples

2015-02-04 Thread Navina Ramesh
://reviews.apache.org/r/30627/ . Main changes are: SqlStringSerde, SqlStringSerdeFactory, StringData and StringSchema - Navina Ramesh On Feb. 4, 2015, 7:39 p.m., Navina Ramesh wrote: > > --- > This is an automatically generated e-mail.

Re: [DISCUSS] SQL workflow

2015-02-04 Thread Navina Ramesh
I have updates SAMZA-484 and RB is here - https://reviews.apache.org/r/30634/diff/# Navina On 2/4/15, 11:13 AM, "Yi Pan" wrote: >Just did the update w/ SAMZA-482. > >On Wed, Feb 4, 2015 at 10:31 AM, Chris Riccomini >wrote: > >> I think so. There was some RB down

Re: Review Request 30634: Serde for Stream Tuples

2015-02-04 Thread Navina Ramesh
ine 141). Anything with "serializers.registry." substring is parsed and used to generate a map of form "serdeType => serdeFactory instance". This is used by SerdeManager for lookups. For example, if a job defines "serializers.registry.string.class=org.apache.samza.se

Re: Review Request 30634: Serde for Stream Tuples

2015-02-05 Thread Navina Ramesh
hanks, Navina Ramesh

Re: Review Request 30634: Serde for Stream Tuples

2015-02-05 Thread Navina Ramesh
ATION settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb Diff: https://reviews.apache.org/r/30634/diff/ Testing --- Thanks, Navina Ramesh

Re: Review Request 30781: SAMZA-548

2015-02-09 Thread Navina Ramesh
tps://reviews.apache.org/r/30781/#comment117547> What are the final job states that is returned by Yarn? samza-test/src/main/python/tests/performance_tests.py <https://reviews.apache.org/r/30781/#comment117545> What is the purpose of this if statement ? - Navina Ramesh On Feb. 9

Re: Terminology: Tumbling and sliding windows

2015-02-17 Thread Navina Ramesh
+1 on using the right terminology. Navina On 2/17/15, 9:33 AM, "Yi Pan" wrote: >+1 on consolidating the terminology as well. Azure's definition looks good >to me. > >On Tue, Feb 17, 2015 at 9:13 AM, Chris Riccomini >wrote: > >> Hey Julian, >> >

Re: Samza commit guarantees

2015-02-19 Thread Navina Ramesh
). Checkpointing is synchronous, if you use KafkaCheckpointManager. In that implementation, you can take a look at the ³writeLog² method that makes a synchronous call to the KafkaProducer. Hope this helps! Cheers, Navina On 2/19/15, 3:59 PM, "Richard Lee" wrote: > > >> On Feb 19, 201

Re: Review Request 31405: Serde Implementation or AvroData (SAMZA-575)

2015-02-25 Thread Navina Ramesh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31405/#review74065 --- Looks good! - Navina Ramesh On Feb. 25, 2015, 5:54 a.m., Milinda

Review Request 32188: Disable WAL in RocksDB KV store

2015-03-17 Thread Navina Ramesh
/ Testing --- Thanks, Navina Ramesh

Re: Review Request 32155: SAMZA-458: Close in KafkaSystemProducer should flush all source buffers

2015-03-18 Thread Navina Ramesh
;t you do the failedSources empty check here instead of Line 153 ? - Navina Ramesh On March 17, 2015, 9:37 a.m., Yan Fang wrote: > > --- > This is an automatically generated e-mail. To reply, visit:

Re: Review Request 32188: Disable WAL in RocksDB KV store

2015-03-18 Thread Navina Ramesh
://reviews.apache.org/r/32188/diff/ Testing (updated) --- ./bin/check-all.sh ran successfully on MAC Thanks, Navina Ramesh

Re: Review Request 32188: Disable WAL in RocksDB KV store

2015-03-18 Thread Navina Ramesh
> > nit: do we need multiple instances of ByteSerde here? Ah..not really. Will change that. - Navina --- This is an automatically generated e-mail. To reply, visit: https://

Re: Review Request 32188: Disable WAL in RocksDB KV store

2015-03-18 Thread Navina Ramesh
Diff: https://reviews.apache.org/r/32188/diff/ Testing --- ./bin/check-all.sh ran successfully on MAC Thanks, Navina Ramesh

Re: Review Request 32155: SAMZA-458: Close in KafkaSystemProducer should flush all source buffers

2015-03-20 Thread Navina Ramesh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32155/#review77293 --- Ship it! Ship It! - Navina Ramesh On March 20, 2015, 9:40 p.m

Re: New to Samza/Yarn and having Kafka issues

2015-03-23 Thread Navina Ramesh
org.apache.samza.container.TaskInstance$$anonfun$process$1.apply$mcV$sp(Tas kInstance.scala:133) Did you make the changes to fix this error? Your patch doesn¹t seem to have that. Line 38 Map jsonObject = (Map) envelope.getMessage(); Lmk so I can investigate further. Cheers! Navina On 3/23/15, 6:43 PM, "Ash W Mat

Tags for Samza in stack overflow

2015-03-23 Thread Navina Ramesh
s a reputation of 1500 and my reputation is just 1 :) If any stackoverflow enthusiast here wants to earn some good karma, please help us in creating a tag - “apache-samza” in stack overflow. Thanks! Navina

Re: New to Samza/Yarn and having Kafka issues

2015-03-23 Thread Navina Ramesh
Did you remove those lines as well? Navina On 3/23/15, 8:31 PM, "Ash W Matheson" wrote: >Just looking at the diff I posted and it's: > > > 1. try { > 2. - Map parsedJsonObject = >parse(event.getRawEvent( > )); > 3. + S

Re: Tags for Samza in stack overflow

2015-03-24 Thread Navina Ramesh
a new Samza user, please let us know if it is hard to find any information. Thanks! Navina On 3/24/15, 12:38 AM, "Jordi Blasi Uribarri" wrote: >I tried to do it in the question you (kindly) answered and found that >same problem. I only have 150 reputation (I have some d

Re: Review Request 33305: [SAMZA-655] Add EnvironmentConfigRewriter to samza-api.

2015-04-20 Thread Navina Ramesh
://samza.apache.org/learn/documentation/latest/jobs/configuration-table.html) with info about this re-writer ? - Navina Ramesh On April 17, 2015, 1:12 p.m., Dan Harvey wrote: > > --- > This is an automatically generated e-mail. To reply, visit

Re: Review Request 33297: [SAMZA-654] Added ElasticsearchSystemProducer and Factory to output messages into Elasticseach indexes.

2015-04-22 Thread Navina Ramesh
/elasticsearch/ElasticsearchSystemProducer.java <https://reviews.apache.org/r/33297/#comment131450> Is this flush a blocking call in elasticsearch? - Navina Ramesh On April 22, 2015, 5:12 p.m., Dan Harvey wrote: > > ---

Re: Review Request 33297: [SAMZA-654] Added ElasticsearchSystemProducer and Factory to output messages into Elasticseach indexes.

2015-04-22 Thread Navina Ramesh
> > > If the previous send has failed, i.e., sendFailed is true here, > > shouldn't we throw an exception here ? Otherwise it might result in message > > develivered out of order. I think the messages are only buffered and not sent until we i

Review Request 33453: SAMZA-557 initial working draft - needs refactoring

2015-04-22 Thread Navina Ramesh
/RocksDbKeyValueStore.scala 66c2a0dc2e38e21f951727a30f0987776ac52fe2 Diff: https://reviews.apache.org/r/33453/diff/ Testing --- Thanks, Navina Ramesh

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-04-22 Thread Navina Ramesh
. Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state. Thanks, Navina Ramesh

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-04-22 Thread Navina Ramesh
environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state. Thanks, Navina Ramesh

Re: Review Request 33297: [SAMZA-654] Added ElasticsearchSystemProducer and Factory to output messages into Elasticseach indexes.

2015-04-23 Thread Navina Ramesh
s here? - Navina Ramesh On April 23, 2015, 6:18 p.m., Dan Harvey wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-04-28 Thread Navina Ramesh
> > > > From the store initiation code, it seems that storeBaseDir won't have > > any logged store paths if logged store base dir is configured. Hence, here > > we may be deleting empty/non-existing storagePar

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-04-29 Thread Navina Ramesh
Question: shouldn't the base dir to be determined by container ID, not > > jobId? > > Navina Ramesh wrote: > I think containerId is not really relevant because the store partition > directories are uniquely named across the containers. The ownership of > containers

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-04 Thread Navina Ramesh
hanks, Navina Ramesh

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-04 Thread Navina Ramesh
s there any checksum in the offset file to make sure that we don't > > read in a corrupted value? > > Navina Ramesh wrote: > No. As of now, we don't have any checksum. Do you think it is better to > add it now than later ? Yi, I created SAMZA

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
--- Tested locally using hello-samza. Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state. Thanks, Navina Ramesh

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
/org/apache/samza/util/Util.scala, line 274 > > <https://reviews.apache.org/r/33453/diff/4/?file=950517#file950517line274> > > > > nit: log ("CRC doesn't match, data corrupted, skipping file") or > > something along those lines. Ok. Makes sense. - Navina

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
r changelogged over logged, as it seems consistent with other > > usages in our code. I don't see any "changelogged" usage. It actually makes the variable name really long. Since loggedStore can't really mean anything other than a

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
za/container/SamzaContainer.scala, > > lines 508-509 > > <https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line508> > > > > why do we need 2 storeDir? Is it because we want to delete the default > > state Dir? Yes > On

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
locally using hello-samza. Note: you have to set an environment variable LOGGED_STORE_BASE_DIR pointing to the new location to persist the changelog attached stores. Otherwise, it will default to YARN's cwd and will not re-use local state. Thanks, Navina Ramesh

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-05 Thread Navina Ramesh
ould this logic be another method, such as > > readOffsetFromFile, not "cleanBaseDir"? A little misleading. > > Navina Ramesh wrote: > I can modularize it further if it helps. Since we need to load offsets > (if available) before cleaning directories, it became a p

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-06 Thread Navina Ramesh
ngeLogSystemStreamPartition matter here? > > 2. prefer the storePartitionDir, because it's the dir for each > > partition/task, right? > > Navina Ramesh wrote: > 1. why does the changeLogSystemStreamPartition matter here? > >> changeLogSystemStreamParti

Re: Review Request 34009: SAMZA-552 window store implementation

2015-05-13 Thread Navina Ramesh
sk/sql/RandomWindowOperatorTask.java <https://reviews.apache.org/r/34009/#comment134678> Shouldn't the window id be system generated? - Navina Ramesh On May 13, 2015, 5:36 p.m., Yi Pan (Data Infrastructure) wrote: > > --- > This is an automati

Re: Review Request 34009: SAMZA-552 window store implementation

2015-05-13 Thread Navina Ramesh
> On May 13, 2015, 9:56 p.m., Navina Ramesh wrote: > > samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java, > > line 96 > > <https://reviews.apache.org/r/34009/diff/1/?file=954320#file954320line96> > > > > Shouldn'

Re: Yarn jobs in accepted state

2015-05-21 Thread Navina Ramesh
Hi Shekar, Actually, I am not able to view even after signing in. Like Yan suggested, please try to share it elsewhere on the internet. Should G drive work? Thanks! Navina On 5/21/15, 11:21 AM, "Yan Fang" wrote: >Hi Shekar, > >This website requires signed in. Could you p

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-05-21 Thread Navina Ramesh
names, irrespective of whether it is storename or taskname. Just my two cents. - Navina Ramesh On May 20, 2015, 10:04 p.m., Yan Fang wrote: > > --- > This is an automaticall

Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-27 Thread Navina Ramesh
g up YARN on 3 machines. Verified that the message is correctly written and consumed from the Coordinator Stream Thanks, Navina Ramesh

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-28 Thread Navina Ramesh
omment137337> Why do we need 2 instances of the TopologyBuilder here? I think this occurs because stream() and sink() method return OperatorSource type rather than a TopologyBuilder instance. How differently does the TopologyBuilder handle the OperatorSource and the OperatorS

Re: Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-29 Thread Navina Ramesh
ttps://reviews.apache.org/r/34746/diff/ Testing --- Used a sample job to test it locally and also, by setting up YARN on 3 machines. Verified that the message is correctly written and consumed from the Coordinator Stream Thanks, Navina Ramesh

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-29 Thread Navina Ramesh
> On May 29, 2015, 1:28 a.m., Navina Ramesh wrote: > > samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java, > > line 120 > > <https://reviews.apache.org/r/34500/diff/1/?file=965740#file965740line120> > > > > I thought T

Re: Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-29 Thread Navina Ramesh
73286#file973286line629> > > > > catch the UnknownHostException ? Mmm.. makes sense. It could be thrown. I will change it to handle the exception with a no-op. - Navina --- This is an automatically generated e-m

Re: Reprocessing old events no longer in Kafka

2015-05-29 Thread Navina Ramesh
Hi Zach, It sounds like you are asking for a SystemConsumer for hdfs. Does SAMZA-263 match your requirements? Thanks! Navina On 5/29/15, 2:23 PM, "Zach Cox" wrote: >(continuing from previous email) in addition to not wanting to duplicate >code, say that some of the Samza jobs

Re: Reprocessing old events no longer in Kafka

2015-05-29 Thread Navina Ramesh
messages from the other Kafka topic. Thanks! Navina On 5/29/15, 2:26 PM, "Navina Ramesh" wrote: >Hi Zach, > >It sounds like you are asking for a SystemConsumer for hdfs. Does >SAMZA-263 match your requirements? > >Thanks! >Navina > >On 5/29/15, 2:23 PM, &q

Re: Reprocessing old events no longer in Kafka

2015-05-29 Thread Navina Ramesh
Hi Zach, Regarding the JIRA, it is assigned to Jakob Homan. He will be the right person to comment on that. Thanks! Navina On 5/29/15, 2:33 PM, "Zach Cox" wrote: >Hi Navina, > >I did see that jira and it would definitely be useful. I was thinking of >maybe trying to bu

Re: Reprocessing old events no longer in Kafka

2015-05-29 Thread Navina Ramesh
would be to consume/bootstrap directly from HDFS :) Cheers! Navina On 5/29/15, 2:44 PM, "Zach Cox" wrote: >Hi Navina, > >A similar approach I considered was using an infinite/very large retention >period on those event kafka topics, so they would always contain all &g

Re: Reprocessing old events no longer in Kafka

2015-05-29 Thread Navina Ramesh
streams. Cheers! Navina [1] https://github.com/apache/samza/blob/master/samza-api/src/main/java/org/apa che/samza/system/chooser/MessageChooser.java#L26 On 5/29/15, 3:07 PM, "Zach Cox" wrote: >Hi Navina, > >Do you mean bootstrapping from hdfs as in [1]? That is an interes

Re: Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-29 Thread Navina Ramesh
Diff: https://reviews.apache.org/r/34746/diff/ Testing --- Used a sample job to test it locally and also, by setting up YARN on 3 machines. Verified that the message is correctly written and consumed from the Coordinator Stream Thanks, Navina Ramesh

Re: Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-30 Thread Navina Ramesh
a > > single CoordinatorStreamManager? And > > CoordinatorStreamManager.getCheckpointManager()/getChangelogManager()/getLocalityManager() > > would return the specific management function handler? I guess this is also a pattern we can follow during the refactoring. As men

Re: Review Request 34746: Adding new CoordinatorStreamMessage "SetContainerHostMapping" and LocalityManager (SAMZA-618)

2015-05-30 Thread Navina Ramesh
Diff: https://reviews.apache.org/r/34746/diff/ Testing --- Used a sample job to test it locally and also, by setting up YARN on 3 machines. Verified that the message is correctly written and consumed from the Coordinator Stream Thanks, Navina Ramesh

Re: [2/2] samza git commit: Yi's TopologyBuilder RB 34500

2015-06-01 Thread Navina Ramesh
Milinda Pathirage >wrote: > >> Hi Navina, >> >> Did we decided to push this patch to samza-sql branch. I thought Yi is >> still working on this. Some Git conflict related texts are still there >>in >> this commit. >> >> +<<<<<<

Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

2015-06-07 Thread Navina Ramesh
rg/apache/samza/config/KafkaConfig.scala <https://reviews.apache.org/r/35067/#comment139191> We need a log message when this value is overriden (to < INT.MAX) and that Samza cannot handle producer failures. Hence, the onus of retries is left up to the kafka producer. - Navina Ramesh

Re: Issues with certain characters in config values

2015-06-09 Thread Navina Ramesh
While Util.envVarEscape() is not exactly smart in the escaping characters, it is a vital minimum to have. Thanks! Navina On 6/9/15, 5:22 AM, "Tommy Becker" wrote: >Ok, the issue with spaces appears to have been addressed by >https://issues.apache.org/jira/browse/SAMZA-530. But quo

Re: Missing artifact? org.apache.samza:samza-serializers_2.10:jar:0.9.0

2015-06-10 Thread Navina Ramesh
Hi Mart, Tommy is right. We moved the serializers into samza-core. I think that documentation requires an update. Please remove the dependency section for samza_serializers. Thanks! Navina On 6/10/15, 4:36 AM, "Tommy Becker" wrote: >I'm sure others will chime in, but I

Re: Missing artifact? org.apache.samza:samza-serializers_2.10:jar:0.9.0

2015-06-10 Thread Navina Ramesh
Hey Mart, I created a task for this -> https://issues.apache.org/jira/browse/SAMZA-707 I will try to update it today. Thanks! Navina From: Mart Haitjema mailto:mhaitj...@skyportsystems.com>> Reply-To: mailto:dev@samza.apache.org>> Date: Wednesday, June 10, 2015 at 1:36 PM

Re: Review Request 35071: SAMZA-572 SamzaContainer checkpoints and windows immediately on startup

2015-06-10 Thread Navina Ramesh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35071/#review87507 --- Ship it! lgtm! - Navina Ramesh On June 4, 2015, 2:42 p.m

  1   2   3   4   5   6   >