Glad that helped!
Honestly I can't say I recognize either that exception but I'm fairly
confident it's
not directly related to rocksdb or Streams. It sounds like a connection
somewhere
got screwed up, which would be more of a configuration issue I probably
can't help with.
Of course it's always p
Hi Sophie,
Thanks for the information! After setting cache.max.bytes.buffering to
zero, commenting out my customized rocksDB config and using default
rocksDB config, I do see the repartition topic bytesout goes up. But I
noticed that some nodes have IO exception as: An I/O error has
occurred while
Ah ok, 2.3.0 has a known performance issue in the caching layer which tends
to get worse the larger the cache size. That might explain what you're
seeing with
regards to the traffic correlation.
It's fixed in 2.3.1 which should be released very soon, but until then you
might want
to try turning of
Hi Sophie,
Thanks for the help!
I'm using version 2.3.0.
The repartition topic with huge lag is the created during the first
reduce method, named "XX-KSTREAM-STATE-STORE-0030-repartition". All
other internal topics have almost zero lags. For my case, how could I
find out if rocksDB causes the lag
Honestly I can't say whether 256 partitions is enough to trigger the
performance issues
in 2.3.0 but I'd definitely recommend upgrading as soon as you can, just in
case. On a
related note, how many instances, with how many threads, are you running?
256 partitions
with several subtopologies will res
Out of curiosity, which version are you using?
There's nothing that really jumps out at me as problematic in your
RocksDBConfigSetter, but note that I think you may need to increase
the number of threads in the "LOW priority thread pool" in addition to
setting the maxBackgroundCompactions -- this
> timed out
while waiting for producer to flush outstanding
Yeah, that's what I'd expect to see if Connect was unable to send records
to the downstream remote topics, e.g. if min.in-sync.replicas were
misconfigured. Given some data seems to arrive, it's possible that
everything is configured corr
You can disable ACL sync to avoid that error, but it doesn't step on
anything else. That can't be it. Consider enabling TRACE or DEBUG logging
within Connect to see what it's doing. Otherwise Connect tends to swallow
exceptions and retry silently.
Ryanne
On Mon, Oct 14, 2019, 2:16 PM Vishal Santo
Aah no.. this is more to it. Note sure if related to the above.
https://github.com/axbaretto/kafka/blob/master/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceTaskOffsetCommitter.java#L114
Is timing out based on
https://github.com/axbaretto/kafka/blob/master/connect/runtime
I'm not sure you really need to/ should be sending data back to the table
with .to(table) -- that was kind of the original issue, since it introduces
a cycle*
into your topology (which should be a DAG).
*Of course it's still technically a cycle if you implement the store
updates manually
with a tr
I think this might be it.. Could you confirm. It seems to be on the path to
commit the offsets.. but not sure...
[2019-10-14 15:29:14,531] ERROR Scheduler for MirrorSourceConnector caught
exception in scheduled task: syncing topic ACLs
(org.apache.kafka.connect.mirror.Scheduler:102)
java.util.con
Hi David,
Probably not a blocker, but I noticed the authorization section in the
security docs was not updated in 2.3.0 to include authorization
details for ElectPreferredLeaders and IncrementalAlterConfigs.
I just opened a PR to address that in 2.3:
https://github.com/apache/kafka/pull/7508 (and
> I do not have a single record in the offsets topic
That's definitely not normal. You are correct that without records in that
topic, MM2 will restart from EARLIEST. The offsets should be stored
periodically and whenever the connectors gracefully shutdown or restart.
Is it possible the topics do
David,
Why do we have two site-doc packages, one for each Scala version? It
is just HTML, right? IIRC, in previous releases we only packaged the
docs once?
Gwen
On Fri, Oct 4, 2019 at 6:52 PM David Arthur wrote:
>
> Hello all, we identified a few bugs and a dependency update we wanted to
> get
Hi all,
Can we integrate Kafka with Monolithic Architecture? Is this bad practice?
2nd/restore issue ( I think I need to solve the offsets topic issue before
I go with the scale up and down issue )
As you had indicated, I went ahead and created the offsets topic. The
status of the cluster ( destination ) is thus
opic# Partitions# BrokersBrokers Spread %Brokers Skew %Brokers L
Hi,
We have a KStream and a KTable that we are left-joining. The KTable has a
"backlog" of records that we want to consume before any of the entries in
the KStream is processed. To guarantee that, we have played with the
timestamp extraction, setting the time for those records in the "distant"
pas
Hi,
I'm looking for a good way to log some broker API requests which might be
considered important. Topic deletion request is quite good an example of
such important requests.
This piece [1] is the only thing I found that remotely is what I need.
However, it logs all the requests and does it on t
18 matches
Mail list logo
