d on
a Red Hat 6 server.
I don't routinely kill this setup, but when I do, it never seems to
exit cleanly. I always wind up having to forcibly kill the remaining
two processes. Am I doing something wrong? How long should shutdown
typically take?
Thanks,
Skip Montanaro
> I'm porting some Kafka-using code from Python 2.7 to 3.6. My values
> are Python dicts mapping strings to strings which are serialized with
> json.dumps(). My keys are simple strings. I haven't been defining a
> key serializer function. I'm using kafka_2.11-1.0.0 and kafka-python
> 1.3.4 installe
a ecosystem.
Thanks,
Skip Montanaro
> Unfortunately, Kafka's careful approach to shutdown means my script is
> probably not given enough time for a proper shutdown, and upon reboot, I
> often find my brokers won't accept connections.
Never mind. I believe I shot myself in the foot elsewhere. Not directly a
Kafka problem.
Skip
my
kafka-logs-N and zookeeper directories.
The latest reboot was early Saturday. Two days later, my dev clients still
can't connect to my dev brokers. Are there tools available which allow me
to view logs and perhaps recover state? None of the shell scripts in
.../bin looked promising.
Thanks,
Skip Montanaro
simple guy exactly what {TopicPartition: OffsetAndMetadata}
means. If I am subscibed to two topics, A and B with group_id G, and
want to commit offsets a1 and b5, Is that dictionary defined as
{
"A": a1,
"B": b5,
}
or something more complex? (I've clearly not men
ttp://www.jesse-anderson.com/2016/11/unit-testing-kafka-consumers/
but that appears to be Java-specific.
Maybe it's not entirely necessary? Do people just unit test through a
development broker setup using fake messages sent to test-specific
topics?
Thx,
Skip Montanaro
Skip:Can you pastebin the stack trace of the stuck broker ?
Will try to later, when the system is quiescent. I forgot to save the
gstack output yesterday.
S
Did you stop the broker before stoping zookeeper?
Yes. My stop script executes the server stop scripts in reverse order from
my start script. Should I have stuck in a couple second sleep between
stopping the brokers and stopping zookeeper?
I was actually running two brokers. The one my stop scri
Have you tried a normal kill (sigterm) against the java process?
Yes. Nothing worked until kill -9.
Skip
(short of kill -9)?
Thx,
Skip Montanaro
ot;chintan mavawala" wrote:
> Exactly. This is what I want to achieve.
>
> Is it good practice to keep on updating subscription list of same kafka
> consumer?
>
> Any concerns in multi threaded environment?
>
> Regards,
> Chintan
>
> On 07-Jan-2018 6:32 PM,
y
> subscribed topics.
>
> I hope it makes sense.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 11:30 PM, "Skip Montanaro"
> wrote:
>
> I'm a bit confused. How do you discover that new topics have been created?
> If you discover a new topic somehow, h
eory pattern "." would work. But that would mean subscribing on all
> topics which I don't want.
>
> I share same pain in using pattern which is why I would like to know if it
> is good practice to subscribe to topics after kafka consumer has started.
>
> Rega
In theory, wouldn't
consumer.subscribe(pattern=".")
work? I say "in theory" because my experience with subscribing by pattern
hasn't been great. I suspect my mental model of how it's implemented isn't
a close approximation of reality.
Skip
On Jan 6, 2018 4:07 AM, "chintan mavawala" wrote:
I w
sn't sufficient:
^topic.*
I needed to add the $ as well so the pattern was anchored at both ends:
^topic.*$
I don't remember seeing any sort of pattern restrictions in my reading
of the docs. Shouldn't any old regular expression do the trick?
Any clues/documentation pointers appreciated.
Skip Montanaro
r more details and suggestions.
____
From: Skip Montanaro
Sent: Friday, December 29, 2017 11:21:52 AM
To: users@kafka.apache.org
Subject: Can I commit from a thread?
I subscribe to topics and receive input in my main thread (this
happens to be in Python):
consumer = kafka.Kafk
or do I need
to do the complete lock dance for that as well?
Thanks,
Skip Montanaro
> Sometimes it seems to receive messages from all partitions (0, 1, 2).
> At other times, it seems to start up only listening to just one
> partition.
Eh, never mind. Looks like pilot error. I accidentally had a stalled
client still running. ("Stalled", as in had been doing something
useful, but w
dge. Looking in kafka.partitioner, I
didn't see a "GreedyPartitioner" object, or anything else which might
obviously be used to grab everything.
Thanks,
Skip Montanaro
art my brokers with those
configuration changes, will existing topics acquire the desired
replication factor, or will I need to do something else to tweak
existing topics?
Skip
> Ben
>
> On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro
> wrote:
>
>> Are dynamically created topics thu
topic my-replicated-topic
>
> -Dave
>
> -----Original Message-
> From: Skip Montanaro [mailto:skip.montan...@gmail.com]
> Sent: Thursday, November 30, 2017 4:20 PM
> To: users@kafka.apache.org
> Subject: Re: Multiple brokers - do they share the load?
>
> > If you
> If you create a partitioned topic with at least 3 partitions then you will
> see your client connect to all of the brokers. The client decides which
> partition a message should go to and then sends it directly to the broker
> that is the leader for that partition. If you have replicated top
onsumer connects to host1:9092 will the brokers
listening to the other ports starve for attention, or does the
connection process somehow redirect clients to the other brokers so
the three (or more) brokers get fairly equitable loads?
Thanks,
Skip Montanaro
leftovers, though if you restart at just
the right (wrong?) time, you run the risk of discarding a pending - but
stale - value which would have eventually be matched.
Is there a better solution than my discard-after-awhile heuristic?
Thx,
Skip Montanaro
> For my own nefarious reasons, I generally run with enable_auto_commit
> == False in my consumers. I'm just using Kafka's default commit
> capability, nothing more sophisticated so far. Is it okay to commit
> out-of-sequence, effectively allowing me to back up and replay
> messages? For example, i
eout gods
happy?
Thanks,
Skip Montanaro
27 matches
Mail list logo
