Hello,
I am interested in learning more about Kafka. I just sent a
subscription request to kafka-users-subscr...@incubator.apache.org (as
indicated at http://incubator.apache.org/kafka/contact.html) but this
email address seems to be incorrect.
Could someone help to add me to the mail list?
tha
Hello,
Is it possible to fetch messages from the Kafka message queue since a
specific time? For example, a user may subscribe to a topic and the
producer will continuously publish messages related to this topic. The
first time this user logs in, we will fetch all the messages from the
beginning. H
unless you're
> using the ConsumerConnector?
>
> [1]: http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/
>
>
> On 17 December 2012 19:23, Jason Huang wrote:
>
>> Hello,
>>
>> Is it possible to fetch messages from the Kafka message queue since
Haven't really used the method to anything besides getting
>> first and last offset (using -1 and -2 as timestamps IIRC) of a
>> topic+partition combination.
>>
>> Maybe someone else can shed some light on this?
>>
>> Cheers,
>> Mathias
>>
>>
&g
Hello,
I am writing a simple java client that uses SimpleConsumer to fetch
messages from a Kafka Server:
..
KafkaConnectionParams connection = new KafkaConnectionParams();
SimpleConsumer consumer = new SimpleConsumer(connection.getKafkaServerURL(),
connection.
, Jason Huang wrote:
> Hello,
>
> I am writing a simple java client that uses SimpleConsumer to fetch
> messages from a Kafka Server:
>
> ..
> KafkaConnectionParams connection = new KafkaConnectionParams();
> SimpleConsumer consu
Hello,
I did some search on the web but couldn't find any documentation for
0.8 so I am trying to ask here:
KeyedMessage is introduced in 0.8.0:
class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
Does the parameter "key" = "partition key"?
If I build a KeyedMessage with a s
The default partitioner
> ignores the key and selects a partition at random.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang wrote:
>
>> Hello,
>>
>> I did some search on the web but couldn't find any documentation for
>> 0.8
ecause we use a default partitioner that does a simple
> hash(key) % num_partitions.
>
> Thanks,
> Neha
>
>
> On Mon, Jan 7, 2013 at 9:30 AM, Jason Huang wrote:
>
>> Jun,
>>
>> Thanks for the response. If I understand you correctly, messages with
>> the
Hello,
I am trying out Kafka 0.8 using only one broker and I am unable to
start the server.
With the instruction from this link -
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.8+Quick+Start,
I was able to download and install 0.8. Since I only have one
machine, I did the following co
Never mind - I was able to start the server after removing the
previous installed 0.7.2 instance of Kafka.
Jason
On Mon, Jan 7, 2013 at 2:56 PM, Jason Huang wrote:
> Hello,
>
> I am trying out Kafka 0.8 using only one broker and I am unable to
> start the server.
>
> With th
; On Mon, Jan 7, 2013 at 12:28 PM, Jason Huang wrote:
>
>> Never mind - I was able to start the server after removing the
>> previous installed 0.7.2 instance of Kafka.
>>
>> Jason
>>
>> On Mon, Jan 7, 2013 at 2:56 PM, Jason Huang wrote:
>> > Hello
Hello,
I am testing the producer to send a message from my local PC to a
remote Kafka server with 0.8.0 installed. However, I was only able to
create the log file with that specific topic but not able to write any
message to the log file.
Here is what the log from the kafka server:
[2013-01-11 1
should send messages
> to the broker at address kafka.icare.com:9092, which was unsuccessful. The
> problem may be item #2 listed in http://kafka.apache.org/faq.html
>
> Thanks,
>
> Jun
>
> On Fri, Jan 11, 2013 at 8:15 AM, Jason Huang wrote:
>
>> Hello,
>>
&
em may be item #2 listed in http://kafka.apache.org/faq.html
>
> Thanks,
>
> Jun
>
> On Fri, Jan 11, 2013 at 8:15 AM, Jason Huang wrote:
>
>> Hello,
>>
>> I am testing the producer to send a message from my local PC to a
>> remote Kafka server with 0.8.0
Hello,
Do you know if there is any support to delete topics with Kafka 0.8?
Since I couldn't find any, I tried to manually delete the log folders
(topic name + partition). However, if I restart the broker (we are
testing with one single broker), it automatically regenerates those
log folders agai
zookeeper data, and then restart everything.
> Understand that it is cumbersome, but until the feature is in, I can't
> think of an easier way to delete a topic
>
> Thanks,
> Neha
>
>
> On Fri, Feb 1, 2013 at 7:25 AM, Jason Huang wrote:
>
>> Hello,
>>
&
Hello,
I am confused about "log file flush". In my naive understanding, once
a message is produced and sent to the kafka server, it will be written
to the hard drive at the log file. Since it is in the hard drive
already, what exactly do you mean by "log file flush"?
I asked because we found that
Thanks for response.
My confusion is that - once I see the message content in the .log
file, doesn't that mean the message has already been flushed to the
hard drive? Why would those messages still get lost if someone
manually kill the process (or if the server crashes unexpectedly)?
Jason
On Tu
y be lost. The data that you saw in the .log file can be
>> just in the pagecache.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Feb 19, 2013 at 4:05 AM, Jason Huang wrote:
>>
>>> Thanks for response.
>>>
>>> My confusion is that - once
th the new leader who
> would have the full log and there would be no loss of committed
> messages.
>
> -Jay
>
>
> On Tue, Feb 19, 2013 at 8:03 PM, Jason Huang wrote:
>> This starts to make sense to me.
>>
>> So a log segment file (0.log) may have some
Hello,
Is there a way to migrate kafka data from server A to another server B?
Can I just copy the data from $log.dir in server.properties and
zookeeper file from $dataDir in zookeeper.properties from server A to
server B, and then run server B's kafka instance?
thanks,
Jason
gt; Neha
> On Feb 27, 2013 2:11 AM, "Jason Huang" wrote:
>
>> Hello,
>>
>> Is there a way to migrate kafka data from server A to another server B?
>>
>> Can I just copy the data from $log.dir in server.properties and
>> zookeeper file from $dataD
er without changes. I believe this should work in 0.8
> as well. However, we've never tried it so not sure if there are bugs. Let
> us know how it goes.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 27, 2013 at 1:31 PM, Jason Huang wrote:
>
>> Thanks for the response.
>
rokerid of the new broker is not the same as the old one
> maybe ? This will work only if you copy the data over and maintain the same
> broker id.
> If not, then this could be a bug.
>
> Thanks,
> Neha
>
>
> On Thu, Feb 28, 2013 at 3:21 AM, Jason Huang wrote:
>
>>
Hello,
I am reading the "Kafka data structures in Zookeeper"
(https://cwiki.apache.org/KAFKA/kafka-data-structures-in-zookeeper.html)
document and trying to figure out how to view topic registration
information from zookeeper. Are these information stored in the
zookeeper snapshot file in the har
Thanks - I will definitely check that out.
Any other options to suggest?
thanks,
Jason
On Fri, Mar 1, 2013 at 5:29 AM, Matan Safriel wrote:
> You can use the eclipse zookeeper plugin, if you only want to
> sporadically view state..
>
> On Mar 1, 2013, at 11:49 AM, Jason
> hierarchy respectively. The data structures described in the wiki can be
>> > viewed using get.
>> >
>> > Thanks,
>> > Swapnil
>> >
>> >
>> >
>> > On 3/1/13 3:51 AM, "Jason Huang" wrote:
>> >
>> >
zookeeper
> 4. Start zookeeper and Kafka
> 5. Consume all messages
>
> Do you mind sending around the entire Kafka and Consumer logs ?
>
> Thanks,
> Neha
>
>
> On Thu, Feb 28, 2013 at 5:10 PM, Jason Huang wrote:
>
>> Hello,
>>
>> I actually tried to l
Hello,
My kafka (0.8) server went down today for unknown reason and when I
restarted both zookeeper and kafka server I got the following error at
the kafka server log:
[2013-03-19 13:39:16,131] INFO [Partition state machine on Controller
1]: Invoking state change to OnlinePartition for partitions
13 at 12:30 AM, Jun Rao wrote:
> Did the broker get restarted with the same broker id?
>
> Thanks,
>
> Jun
>
> On Tue, Mar 19, 2013 at 1:34 PM, Jason Huang wrote:
>
>> Hello,
>>
>> My kafka (0.8) server went down today for unknown reason and when I
>>
ipe
> out all ZK and Kafka data first since some ZK data structures have been
> rename a few weeks ago.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 20, 2013 at 6:57 AM, Jason Huang wrote:
>
>> I restarted the zookeeper server first, then broker. It's the same
>&g
nk.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 20, 2013 at 7:47 AM, Jason Huang wrote:
>
>> The 0.8 version I use was built from trunk last Dec. Since then, this
>> error happened 3 times. Each time we had to remove all the ZK and
>> Kafka log data and restart the serv
?
thanks,
Jason
On Fri, Mar 22, 2013 at 11:07 AM, Jun Rao wrote:
> The easiest way is to wipe out both ZK and kafka data and start from
> scratch.
>
> Thanks,
>
> Jun
>
> On Fri, Mar 22, 2013 at 6:51 AM, Jason Huang wrote:
>
>> Thanks Jun.
>>
>> I h
uch changes are expected until the final
> release. Once it is released, we don't expect to make such big changes.
>
> Thanks,
> Neha
>
>
> On Fri, Mar 22, 2013 at 8:09 AM, Jason Huang wrote:
>
>> I see.
>>
>> Since I am not running anything
Just curious - I don't see any limit in the # of topics that you can
have in a Kafka cluster.
So in principle, you could have as many topics as you want, so long as
your hardware can keep up, right?
Jason
On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao wrote:
> At LinkedIn, our largest cluster has mo
Just curious - given the situation above, if you have two sets of .log and
.index files under the same topic and same partition, is there any way to
merge these two sets of log files and index files?
On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike wrote:
> Otherwise if you are using the high lev
gt; do that ? To reduce the number of open file handles ?
>
> Thanks,
> Neha
>
> On Thursday, April 18, 2013, Jason Huang wrote:
>
> > Just curious - given the situation above, if you have two sets of .log
> and
> > .index files under the same topic and same partiti
fewer log files change the settings so that the log files
> are bigger. However the only impact of this is fewer file handles, it
> won't impact the consumer in any way at all.
>
> Not sure if that helps...
> -Jay
>
> On Thu, Apr 18, 2013 at 8:36 AM, Jason Huang
> wr
Hello,
We've been playing around with kafka 0.8 for a few months now and decided
to install kafka on a small cluster for further testing. I tried to search
online but couldn't find any setup documentation for a kafka 0.8 cluster.
Does anyone know if such documents exist? If they don't exist, what
:32 PM, Neha Narkhede >wrote:
>
> > Hi Jason,
> >
> > We are in the process of updating the documentation. Hoping to finish
> > it by this week. Stay tuned.
> >
> > Thanks,
> > Neha
> >
> > On Mon, Apr 22, 2013 at 2:12 PM, Jason Huang
> &
ich should be stored
> in a file
> Named /tmp/zookeeper/myid
>
> Echo "1" > /tmp/zookeeper/myid`
>
>
> Cheers,
> Eric Sites
>
>
> On 4/23/13 9:50 AM, "Jason Huang" wrote:
>
> >Thanks Chris and Neha.
> >
> >Chris - I'
o be changed at it. That was fun to find :)
>
> We use /var/zookeeper for our storage.
>
> Thanks,
>
> Chris
>
>
> On Tue, Apr 23, 2013 at 10:30 AM, Jason Huang
> wrote:
>
> > Thanks Eric - this helps quite a bit.
> >
> > I will play a
43 matches
Mail list logo
