Aw: Re: Re: Split Kafka JSON String

[Daniela S]

Henry,

 

thank you for your help. I did not use any github link for my project. I tried to build the project on my own, with the help of some books and available code parts from the Internet. I try to explain what I am doing:

 

1. I have generated random JSON data with a JSON data generator from the Internet

    the JSON data looks as follows:

    {"timestamp": "nowTimestamp()",

      "ID1": "integer(1,1000)",

      "ID2": "integer(1,100)",

      "command": "random('START','END')"

   }

   The JSON generator simulates a data stream, sending the data to a specific Kafka topic as String

 

2. I would like to have a KafkaSpout to fetch the data from the Kafka topic.

 

3. Then I would like to store all the events containing "START" into Redis and to delete all the events containing "END" from Redis. The key should be a combination of ID1 and ID2.

 

4. In the next step I would like to pull all the entries stored in Redis and to add a specific value for every entry (this values is located in a database and I have to look it up for every entry). Then I would like to use these values to build a sum over all entries. This should be repeated every minute, as there might have been new start or end events. I thought I could use a sliding window for that.

 

But unfortunately I have no idea how to proceed with the JSON String, I receive from Kafka. I already read a lot about the possibility to store JSON Strings in Redis, but I have no idea how to build the key with both IDs and how to access the fields in the JSON String.

 

Hope you can help me.

 

Thank you so much in advance!

 

Regards,

Daniela 

 

 

 

Gesendet: Sonntag, 15. Mai 2016 um 22:07 Uhr
Von: "Henry Hottelet" <hotte...@gmail.com>
An: user@storm.apache.org
Betreff: Re: Aw: Re: Split Kafka JSON String

Daniella,

 

I am curious about your architecture with Kafka and storm.  Can you send the github link you used for that project?

 

I would like to be able to see your example. I use IntelliJ.

 

Henry

Sent from my iPad

On May 15, 2016, at 2:14 PM, Daniela S <daniela_4...@gmx.at> wrote:
 

Hi,

 

thank you very much for your answers. But unfortunately I do not have a JSON object yet, I only have a String, containing a JSON object. I receive the Strings from Kafka but I do not know how to get the information out of the String.

 

example: {"timestamp": 1463335828342, "ID":570,"command":"START"}

 

I would like to extract the three fields out of the String. Then I would like to store it into Redis.

Afterwards I would like to pull all the stored (active) devices out of Redis and to build a sum over all the devices every minute. For building the sum I have to add another value for every device. I would like to use Redis as a kind of cache.

 

Regards,

Daniela

 

Gesendet: Sonntag, 15. Mai 2016 um 20:06 Uhr
Von: "Henry Hottelet" <hotte...@gmail.com>
An: user@storm.apache.org
Betreff: Re: Split Kafka JSON String

Try using intellij plug in that converts json to pojo classes

On May 15, 2016 2:04 PM, "Nathan Leung" <ncle...@gmail.com> wrote:

Easiest way is to map your job to a pojo and use Jackson or gson to convert the json.

On May 15, 2016 1:48 PM, "Daniela S" <daniela_4...@gmx.at> wrote:

Hi

 

I am receiving Strings of JSON from Kafka. I would like to split the string to get each field from the JSON object to store it into Redis.

How can I split the string and assign the parts to variables?

 

Thank you in advance.

 

Regards,

Daniela