Re: Got Error Produce Respons with Correlation Id.

Yes, I explicitly commit via code for this job as an effort to ensure only once processing. Thanks for taking the time to look into our concerns. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Yi Pan

Re: Got Error Produce Respons with Correlation Id.

-1 Is the Samza container still running after you see those WARN logs? Yes. I am thinking this is a timeout issue. We've never seen the issue before. The warning first appeared after testing Confluent's Cloud kafka offering. We had no issues when running our own kafka clusters

Re: Got Error Produce Respons with Correlation Id.

Can anyone take a look at the message below? We are trying to gauge our risk before moving forward. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jeremiah Adams Sent: Wednesday, March 4, 2020 2:28 PM

Got Error Produce Response iwth Correlation Id.

sconnected before a response was received.. Going to request metadata update now Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook&l

Re: Nullpointer After Upgrade from samza 1.0.0 to 1.3.0

Thanks Brett. In the short term, can I fix this by specifying an alternate directory? It will still be empty on initial startup. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Brett Konold Sent

Re: Nullpointer After Upgrade from samza 1.0.0 to 1.3.0

RocksDbTableDescriptor(tableName, KVSerde.of(new StringSerde(), new StringSerde(.withTtl(this.rocksDbTtl); Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Brett Konold Sent: Thursday, January 23, 2020

Nullpointer After Upgrade from samza 1.0.0 to 1.3.0

r.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook&l

Issues with Samza-Shell and Exectuable Jar

I unpack the tgz and copy it into the executable jar? All of my attempts to do this have failed. Thanks for any advice. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.co

Re: Samza tasks aren't starting in YARN containers

rn is concerned. What exactly do you mean by 'submitting multiple jobs at once'? I start ours one at a time though do not verify they are 'Running' until I've submitted all three instances of the jar. Jeremiah Adams Software Engineer www.helixeducation.com Blog |

Re: Context when Converting from Yarn Task to Standalone LocalApplicationRunner

Thanks for the clarification. I completely missed that there LocalApplicationRunner could accept a TaskApplication. The javadocs are actually hard to find. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn

Context when Converting from Yarn Task to Standalone LocalApplicationRunner

r(app, config); localApplicationRunner.run(); localApplicationRunner.waitForFinish(); }? Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducatio

Re: Empty Kafka topic partition Warning

Thanks for the input Jagadish. This makes sense now. I'm seeing one of our input topic partitions empty. One of our upstream jobs is likely configured incorrectly. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | Lin

Re: Empty Kafka topic partition Warning

=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory stores.redelivery-store.changelog=kafka.delivery-changelog stores.default.changelog.replication.factor=3 Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Friday

Empty Kafka topic partition Warning

offset to 0 to consume from beginning Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> |

Re: Backing Kafka/Yarn/Zookeeper version for Samza 1.0.0?

Is zookeeper required for the tech stack? If samza does all of its checkpointing on kafka, I may be able to save us some money by eliminating the ZK cluster. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation

Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Prateek, Migrating to v1.0.0 went off without a hitch. Thanks. Jeremiah Adams Software Engineer www.helixeducation.com <http://www.helixeducation.com/> Blog <http://www.helixeducation.com/blog/> | Twitter <https://twitter.com/HelixEducation> | Facebook <htt

Backing Kafka/Yarn/Zookeeper version for Samza 1.0.0?

to containers. Also, does Samza have any hard requirements for Zookeeper since checkpointing was implemented? Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com

Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

Thanks for checking Prateek. There’s a similar thread in the group. I decided to follow advice there and am working on migrating to V1.0.0 instead of 0.14. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation

Re: [POSSIBLE PHISHING] Task Partition Commit Failed After Upgrade

MethodException: org.apache.samza.container.grouper.task.GroupByContainerCount.() at java.lang.Class.getConstructor0(Class.java:3082) at java.lang.Class.newInstance(Class.java:412) ... 8 more Jeremiah Adams Software Engineer www.helixeducation.com <http://www.helixeducatio

Task Partition Commit Failed After Upgrade

] Caught throwable and stopping run loop Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducat

Re: Looking for YARN Alternative

Thanks you Jagadish! This is exactly what was looking for. I've been behind samza versions due to long dev time on a product so I often miss the v.1 documentation. Jeremiah Adams Software Engineer www.helixeducation.com <http://www.helixeducation.com/> Blog <http://www.heli

Looking for YARN Alternative

partitions I can get away with? Any advice on determining the appropriate number of partitions? Kafka, Zookeeper and Secor are also costs we could potentially reduce via partition count. Thanks for any input. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.

Re: Alerting and Monitoring Samza Checkpointing?

Thanks for the information Tom and Jagadish. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Tuesday, November 27, 2018 4:43 PM To: dev@samza.apache.org Subject: Re: Alerting

Re: Alerting and Monitoring Samza Checkpointing?

nkedin/Burrow) for a test drive locally, hoping it would solve my problem due to it looking at the internal consumers. However, I have the same problem - can't get data on a consumer group that doesn't exist. Jeremiah Adams Software Engineer www.helixeducation.com Blog |

Alerting and Monitoring Samza Checkpointing?

s for examining a topic's checkpoint aren't readily useful for application consumption. Can some of you share your approaches to monitoring and alerting on consumer lag? Regards. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http:

Re: Need Some Help w/ Gradle Build on OpenJDK 11

No luck Prateek. Same problem. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Prateek Maheshwari Sent: Thursday, October 11, 2018 10:54 AM To: dev@samza.apache.org Subject: Re: Need Some Help w/ Gradle

Re: Need Some Help w/ Gradle Build on OpenJDK 11

Anyone have a few cycles to help me out with Apache Rat failing the build on OpenJDK 11? Some things I’ve tried: Bumped Gradle to 4.10.2 (required for Java11). Bumped the Rat version to 0.12. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn

Re: Open JDK 11 Early Results

Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jeremiah Adams Sent: Tuesday, October 2, 2018 12:55 PM To: dev@samza.apache.org Subject: [POSSIBLE PHISHING] Re: Open JDK 11 Early Results I am trying both building

Re: Open JDK 11 Early Results

264?jql=project%20%3D%20KAFKA%20AND%20text%20~%20%22java%2011%22 Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Yi Pan Sent: Tuesday, October 2, 2018 11:16 AM To: dev@samza.apache.org Subject: Re: Op

Need Some Help w/ Gradle Build on OpenJDK 11

I know very little about Gradle. I got a response on the issue I opened regarding Gradle builds failing on OpenJDK 11. I've since upgraded the Gradle version in gradle-wrapper.properties to 4.10.2. Build now gets past the java11 issue but is dying on the Apache rat task. The build/plugin can

Open JDK 11 Early Results

- Does not start on Open JDK 11 Kafka: - 2.11-2.0.0 - Starts on Open JDK 11 Zookeeper: 3.4.13 - Starts on Open JDK 11 I can’t really test anything interesting until Yarn can play nice on JDK 11. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook

Re: Java 11, Oracle Java and OpenJDK Release Cadence

Yes it does. Thanks Yi. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Yi Pan Sent: Tuesday, September 25, 2018 9:09 AM To: dev@samza.apache.org Subject: Re: Java 11, Oracle Java and OpenJDK Release

Re: Java 11, Oracle Java and OpenJDK Release Cadence

Has anyone had a chance to digest this information regarding Java11 and OpenJDK release cadences? What would be next steps to drive adoption of the OpenJDK release cadence (never participated in an apache proj) ? Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter

Java 11, Oracle Java and OpenJDK Release Cadence

cription http://www.oracle.com/technetwork/java/javaseproducts/javasesubscription-data-sheet-4891969.pdf https://www.infoq.com/presentations/java-10-11 Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> |

Re: Path to Java 11

e.com/java-platform-group/update-and-faq-on-the-java-se-release-cadence Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Tuesday, September 11, 2018 1:09 AM To: dev@samza.apache

Path to Java 11

How does the java 11 changes to Oracle Java and Open JDK affect Samza's release schedules? When can I upgrade to java 11? Are there any docs for this information yet? Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.heli

Re: MessageCollector and WindowableTask

as a side-effect of processing a normal message. Keep in mind our data is nowhere near 'big'. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Tuesday, June 5, 2018 1

MessageCollector and WindowableTask

or processing the window()? Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | LinkedIn<http://www.linkedin.com/company/3609946>

Re: Noise in Logs After Upgrade to 0.13

Thanks Prateek. That got me pointed in the right direction. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: prateek...@gmail.com on behalf of Prateek Maheshwari Sent: Monday, August 21, 2017 2:30 PM

Noise in Logs After Upgrade to 0.13

ICS] DEBUG org.apache.samza.metrics.JvmMetrics - updating jvm metrics? What log4j file do I need to edit to correct this? Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEdu

Re: Issue with TopicExistsException in 0.13.0

Jacob, please ignore previous message. It looks like this is caused by a config on our end. Thanks for your input in resolving our issues. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jeremiah Adams

Re: Issue with TopicExistsException in 0.13.0

from 10.201.10.222:9092 Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jacob Maes Sent: Tuesday, August 15, 2017 9:23 AM To: dev@samza.apache.org Subject: Re: Issue with TopicExistsException in 0.13.0

Re: Issue with TopicExistsException in 0.13.0

cture, the directory is empty at appcache/ and filecache/ both: [centos@porter-yarn-slave-1 container_1502753192195_0007_02_01]$ ls /tmp/hadoop-centos/nm-local-dir/usercache/centos/appcache/ [centos@porter-yarn-slave-1 container_1502753192195_0007_02_000001]$ Jeremiah Adams Softwar

Issue with TopicExistsException in 0.13.0

session 0x25de16b1f500013 : Unable to read additional data from server sessionid 0x25de16b1f500013, likely server has closed socket [10.201.9.105] out: 17:18:49.349 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down? Jeremiah Adams Software Engineer ww

Re: Question Concerning Custom Metrics and Timers

Thanks for the info Jagdish. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Friday, July 07, 2017 6:49 PM To: dev@samza.apache.org Subject: Re: Question Concerning Custom

Question Concerning Custom Metrics and Timers

am instrumenting a method via Timer.update() which is called n-number of times in 60 seconds am I seeing an aggregation of the time spent in said method during a 60 second period or am I seeing the last value sent through Timer.update()? Jeremiah Adams Software Engineer www.helixeducation.com

Re: How to Report Timer Snapshot via Snapshot Reporter

tasks? If not it seems better to implement the Histogram? Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn From: Jagadish Venkatraman Sent: Friday, June 30, 2017 9:08 AM To: dev@samza.apache.org Subject:

How to Report Timer Snapshot via Snapshot Reporter

lliseconds into timer.update() Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | Linked

Samza Jobs Not Stopping after Application --kill command

ntImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91) ... 16 more Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https:

Unit and Integration Testing Samza Tasks

with docker compose but this approach seems heavy handed. I am hoping someone has already plowed this ground. "No matter how far you have gone on the wrong road, turn back." - Turkish proverb Jeremiah Adams So

Re: Manually Commit Offsets?

Thank you for the info. Is there any way to 'pause' the job or stop processing kafka from inside a StreamTask.process() method? That would work for me too. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook

Re: Manually Commit Offsets?

they were not handled at all when the remote system is unavailable. I have been looking at the CheckpointManager to do this but cannot see how to wire it into my StreamTask. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn

Manually Commit Offsets?

me in the right direction? Thanks in advance. Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> |

Re: ElasticsearchSystemProducer Crashes Samza Job

csearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemProducer.java#L101 > ). > > > > On Mon, Feb 15, 2016 at 2:30 PM, jeremiah adams > wrote: > > > We have a samza job configured to run in a yarn cluster. This job > consumes > > multiple kafka

ElasticsearchSystemProducer Crashes Samza Job

We have a samza job configured to run in a yarn cluster. This job consumes multiple kafka topics and routes the messages to elasticsearch for indexing. When enough batch-updates to elasticsearch fail using the ElasticsearchSystemProducer, the entire samza job dies. Due to checkpointing + yarn, the