Hi,
Thanks for your prompt reply. Consumers have different group.id in our
case, as we just need all the data in the topic, so every time the
application starts, the consumer will be in a new group. For current
testing environment, we just one partition for one topic. though we have
tried with mul
What is your `auto.offset.reset` policy?
On 12/10/19 12:02 AM, Frank Zhou wrote:
> Hi,
>
> Thanks for your prompt reply. Consumers have different group.id in our
> case, as we just need all the data in the topic, so every time the
> application starts, the consumer will be in a new group. For cur
"earliest", as we need everything from the start
On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax
wrote:
> What is your `auto.offset.reset` policy?
>
> On 12/10/19 12:02 AM, Frank Zhou wrote:
> > Hi,
> >
> > Thanks for your prompt reply. Consumers have different group.id in our
> > case, as we j
Hard to say why the second consumer does not fetch data.
I would try to inspect the logs at DEBUG level to get more insight what
going on.
-Matthias
On 12/10/19 12:27 AM, Frank Zhou wrote:
> "earliest", as we need everything from the start
>
> On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax
>
> Am 10.12.2019 um 06:36 schrieb Valentin :
>
> Hi Chao,
>
> I suppose, you would like to know:
> within a consumer group which message is coming from which partition, since
> partitions corresponds to broker and broker = ip, right?
>
> Well, if you really want to know this, then you have to
Hello,
running Kafka as a single Broker or a cluster of one node voids the whole
purpose you are trying to achieve "...robustness of our system by using
kafka..."
Just think that is that single node goes down, what's gonna happen? Kafka
besides the performance is well known by the availability.
C
Hi, thanks for the swift response. Does this mean that even with the
promise of kubernetes' self healing and with the acceptable downtime of a
few minutes or seconds deploying kafka on a single node is still not
possible??
On Tue, Dec 10, 2019, 18:23 Jonathan Santilli
wrote:
> Hello,
>
> running
Yes, technically is possible, no problem with that,
Kafka offers high availability (when well provision and configured) and
tolerance to failure, like, hard drive failure, your data could be lost if
the disk fails and you do not have replicas right?
--
Jonathan
On Tue, Dec 10, 2019 at 10:30 AM
Backing up and syncing to the clous should be handled by our internal
services since this servers are deployed in an environment with little to
no internet connection.
On Tue, Dec 10, 2019, 18:35 Jonathan Santilli
wrote:
> Yes, technically is possible, no problem with that,
>
> Kafka offers high
Hi,
I am using streams and I get messages like: (K, V)
(A, a), (B, b), (C, c), (A, a), (C, c), (A, a) .
I wanted to define a topology which would filter out duplicate messages
from upstream.
I want to know if this is possible?
The code I have written to do this is something like this:
source.
Hi Sachin, is your goal to prevent any records with a duplicate key from
ever getting sent downstream? The KTable you have in your example will of
course have the most recent record for a given key, but it will still emit
updates. So if key "A" arrives a second time (with no change to the
value),
I have two Kafka consumers, subscribed to different topics and belonging
to the same consumer group, having different consumer ids, running in the
same thread. They are executing poll sequentially but after the first is
done second seems to be stuck in poll. I tried using proper Kafka broker
and e
Hi Alex,
Thanks for the quick response.
What I have is around 8 streams branched from a single stream, that down
the line again gets joined into 1.
Now each branched stream can have duplicates and when joining all this data
I just have kind of endless tuples of data.
So what I was thinking what if
- All PGP signatures are good
- All md5, sha1sums and sha512sums pass
Initial test results:
1310 tests completed, 2 failed, 17 skipped
> Task :core:integrationTest FAILED
The failed tests:
SaslSslAdminClientIntegrationTest. testElectPreferredLeaders
SslAdminClientIntegrationTest.
testSynchronous
How do I set up the AdminClient for the kafka cluster so that I don't need to
pass "--command-config" each time I run a describe command with the parameter
"bootstrap-server" (e.g: kafka-topics.sh --describe --bootstrap-server
localhost:9093 --topic my-topic --command-config )?
I've never used that dedup transformer before, but what you've got looks
right. (though if there's a way to hash your message value, or somehow get
a guid out of it that might be preferable) As you probably noticed it's
state is Windowed - so if your use-case depends on being able to remove
duplic
Just an update since it has been happening again now and I have some more
metrics to show, the topology is this:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-00 (topics: [sensors])
--> KSTREAM-TRANSFORMVALUES-01
Processor: KSTREAM-TRANSFORMVALUES-01 (
First of all, from a performance point of view, it does not make sense
to have two consumers of the same group in one thread. They will share
the same network and same CPU resource (but are not able to utilize them
more efficiently as both are on the same thread). Each consumer within a
group will
18 matches
Mail list logo
