Hi Satendra,
in what order did you start your connector and the console consumer and did
you specify "--from-beginning"?
>From your connector config it looks like the connector will only poll for
data every 15 minutes (900000 ms). So if you started the connector first
and then your consumer it might have started reading the topic at the end
and you'd habe to wait for 15 minutes to see new data.
Might it be as simple as that?
Best regards,
Sönke
Am Fr., 30. Nov. 2018, 11:48 hat jan <rtm4...@googlemail.com.invalid>
geschrieben:
> I may have missed this (I'm missing the first few messages), so sorry
> in advance if I have, but what OS are you using?
> Kafka does not work well on windows, I had problems using it that
> sounds a little like this (just a little though) when on win.
>
> jan
>
> On 30/11/2018, Satendra Pratap Singh <sameerp...@gmail.com> wrote:
> > Hi Sönke,
> >
> > when topic got created so i tried to read the topic data using console
> > consumer but it didn't worked out consumer didn't consumer a single
> > message. i read the log and got this every time. i don't understand
> where m
> > i making mistake.
> >
> >
> >
> [\!�qg_z��g_z�����������������(�"{"error":{"name":"Error","status":404,"message":"Shared
> > class \"Dia\" has no method handling GET
> > /getDia","statusCode":404,"stack":"Error: Shared class \"Dia\" has no
> > method handling GET /getDia\n at restRemoteMethodNotFound
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/strong-remoting/lib/rest-adapter.js:371:17)\n
> > at Layer.handle [as handle_request]
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> > at trim_prefix
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)\n
> > at
> >
> /Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:284:7\n
> > at Function.process_params
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:335:12)\n
> > at next
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:275:10)\n
> > at Function.handle
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:174:3)\n
> > at router
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:47:12)\n
> > at Layer.handle [as handle_request]
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> > at trim_prefix
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)\n
> > at
> >
> /Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:284:7\n
> > at Function.process_params
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:335:12)\n
> > at next
> >
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:275:10)\n
> > at jsonParser
> >
> (/Users/satendra/analyticsApi/node_modules/body-parser/lib/types/json.js:109:7)\n
> > at Layer.handle [as handle_request]
> >
> (/Users/satendra/analyticsApi/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> > at trim_prefix
> >
> (/Users/satendra/analyticsApi/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)"}}@access-control-allow-credentials["true"]connection["keep-alive"]content-typeF["application/json;
> > charset=utf-8"dateB["Thu, 29 Nov 2018 12:38:59
> >
> GMT"etagR["W/\"89a-6BIHHr4YrwjSfEHOpFjIThnXbCY\""]"transfer-encoding["chunked"vary6["Origin,
> >
> Accept-Encoding"],x-content-type-options["nosniff"]$x-download-options["noopen"]x-frame-options["DENY"]
> > x-xss-protection"["1; mode=block"]
> >
> > On Fri, Nov 30, 2018 at 2:30 AM Sönke Liebau
> > <soenke.lie...@opencore.com.invalid> wrote:
> >
> >> Hi Satendra,
> >>
> >> if I understand correctly you are using the RestSourceConnector to pull
> >> data into a Kafka topic and that seems to work - at least as far as the
> >> topic getting created.
> >>
> >> But are you saying that you tried to read data with 'cat filename.log'
> on
> >> a
> >> file within your kafka data directory? I assume I misunderstood this
> >> part,
> >> can you please add some detail around what you did there?
> >>
> >> Best regards,
> >> Sönke
> >>
> >> On Thu, Nov 29, 2018 at 7:27 PM Satendra Pratap Singh <
> >> sameerp...@gmail.com>
> >> wrote:
> >>
> >> > Hi Sönke,
> >> >
> >> > I have single node kafka cluster, i have setup web server locally,
> many
> >> > apps are sending logs to this server. I have Rest API which
> >> > send/receive
> >> > client request to/from server.
> >> >
> >> > here is my connector configuration : curl -i -X POST -H
> >> > "Accept:application/json" -H "Content-Type: application/json"
> >> > http://$CONNECT_HOST_SOURCE:8083/connectors/
> >> > -d '{ "name": "source_rest_telemetry_data”, "config": {
> >> > "key.converter":"org.apache.kafka.connect.storage.StringConverter",
> >> > "value.converter":"org.apache.kafka.connect.storage.StringConverter",
> >> > "connector.class": "com.tm.kafka.connect.rest.RestSourceConnector",
> >> > "tasks.max": "1", "rest.source.poll.interval.ms": "900000",
> >> > "rest.source.method": "GET", "rest.source.url": "
> >> > http://abc.server.in:0000/api/Dia/getDiaData";,
> >> > "rest.source.payload.converter.class":
> >> > "com.tm.kafka.connect.rest.converter.StringPayloadConverter",
> >> > "rest.source.properties":
> >> > "Content-Type:application/json,Accept::application/json",
> >> > "rest.source.topic.selector":
> >> > "com.tm.kafka.connect.rest.selector.SimpleTopicSelector",
> >> > "rest.source.destination.topics": “telemetry_data” } }'
> >> >
> >> > After running this command connector runs and and create a topic
> called
> >> > telemetry_data. Inside telemetry data topic got 4 files with extension
> >> > called .log, .index, .timeindex and leader-epoch-checkpoint.
> >> >
> >> > i used cat command and open .log file and got following error :
> >> >
> >> >
> >> >
> >>
> [\!�qg_z��g_z�����������������(�"{"error":{"name":"Error","status":404,"message":"Shared
> >> > class \"Dia\" has no method handling GET
> >> > /getDia","statusCode":404,"stack":"Error: Shared class \"Dia\" has no
> >> > method handling GET /getDia\n at restRemoteMethodNotFound
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/strong-remoting/lib/rest-adapter.js:371:17)\n
> >> > at Layer.handle [as handle_request]
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> >> > at trim_prefix
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)\n
> >> > at
> >> >
> >> >
> >>
> /Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:284:7\n
> >> > at Function.process_params
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:335:12)\n
> >> > at next
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:275:10)\n
> >> > at Function.handle
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:174:3)\n
> >> > at router
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:47:12)\n
> >> > at Layer.handle [as handle_request]
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> >> > at trim_prefix
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)\n
> >> > at
> >> >
> >> >
> >>
> /Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:284:7\n
> >> > at Function.process_params
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:335:12)\n
> >> > at next
> >> >
> >> >
> >>
> (/Users/satendra/Api/node_modules/loopback/node_modules/express/lib/router/index.js:275:10)\n
> >> > at jsonParser
> >> >
> >> >
> >>
> (/Users/satendra/analyticsApi/node_modules/body-parser/lib/types/json.js:109:7)\n
> >> > at Layer.handle [as handle_request]
> >> >
> >> >
> >>
> (/Users/satendra/analyticsApi/node_modules/loopback/node_modules/express/lib/router/layer.js:95:5)\n
> >> > at trim_prefix
> >> >
> >> >
> >>
> (/Users/satendra/analyticsApi/node_modules/loopback/node_modules/express/lib/router/index.js:317:13)"}}@access-control-allow-credentials["true"]connection["keep-alive"]content-typeF["application/json;
> >> > charset=utf-8"dateB["Thu, 29 Nov 2018 12:38:59
> >> >
> >> >
> >>
> GMT"etagR["W/\"89a-6BIHHr4YrwjSfEHOpFjIThnXbCY\""]"transfer-encoding["chunked"vary6["Origin,
> >> >
> >> >
> >>
> Accept-Encoding"],x-content-type-options["nosniff"]$x-download-options["noopen"]x-frame-options["DENY"]
> >> > x-xss-protection"["1; mode=block"]
> >> >
> >> > but couldn't get data what looking for. but when i run curl -X POST
> >> > http://localhost:0000/api/dia/diadata by terminal/postman it's
> fetching
> >> > desired result.
> >> >
> >> > Please help me.
> >> >
> >> > looking
> >> >
> >> > On Thu, Nov 29, 2018 at 8:42 PM Sönke Liebau
> >> > <soenke.lie...@opencore.com.invalid> wrote:
> >> >
> >> > > Hi Satendra,
> >> > >
> >> > > feel free to share the error and I'll see if I can help.
> >> > >
> >> > > Best regards,
> >> > > Sönke
> >> > >
> >> > >
> >> > > Am Do., 29. Nov. 2018, 14:59 hat Satendra Pratap Singh <
> >> > > sameerp...@gmail.com>
> >> > > geschrieben:
> >> > >
> >> > > > Hi Sönke,
> >> > > >
> >> > > > Thanks for your response. The reason for asking about rest bcoz I
> m
> >> > stack
> >> > > > while pulling pulling request from rest api. I have been using
> >> > > > Kafka
> >> > rest
> >> > > > source connect to pull the request, connected is pulling some
> error
> >> > > instead
> >> > > > actual data.
> >> > > >
> >> > > > Would you like to help me. I will share that err once you respond.
> >> > > >
> >> > > > Sent from my iPhone
> >> > > >
> >> > > > > On 29-Nov-2018, at
> >> > > >
> >> > > >
> >> > > > > 18:11, Sönke Liebau <soenke.lie...@opencore.com.INVALID> wrote:
> >> > > > >
> >> > > > > Hi Satendra,
> >> > > > >
> >> > > > > in addition to what Ryanne said, maybe a graphical REST tool
> >> > > > > would
> >> > > better
> >> > > > > suit your needs, there is a couple of good ones out there:
> >> > > > > Postman - https://www.getpostman.com/
> >> > > > > Insomnia - https://insomnia.rest/
> >> > > > >
> >> > > > > are two that I have used myself, but just googling for "rest
> gui"
> >> > > should
> >> > > > > turn up many alternatives.
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Sönke
> >> > > > >
> >> > > > >> On Wed, Nov 28, 2018 at 5:06 PM Ryanne Dolan <
> >> ryannedo...@gmail.com
> >> > >
> >> > > > wrote:
> >> > > > >>
> >> > > > >> I think you might be looking for cURL?
> >> > > > >>
> >> > > > >> On Fri, Nov 23, 2018 at 10:28 AM Satendra Pratap Singh <
> >> > > > >> sameerp...@gmail.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >>> Hi Ryan,
> >> > > > >>>
> >> > > > >>> Thanks. Since I am new to Kafka don’t understand how to
> >> configures
> >> > > rest
> >> > > > >>> api and how to reconfigure connectors. In general I don’t know
> >> how
> >> > to
> >> > > > run
> >> > > > >>> GET /connector command and where to run.
> >> > > > >>>
> >> > > > >>> Please elaborate.
> >> > > > >>>
> >> > > > >>> Sent from my iPhone
> >> > > > >>>
> >> > > > >>>> On 23-Nov-2018, at 21:43, Ryanne Dolan <
> ryannedo...@gmail.com>
> >> > > wrote:
> >> > > > >>>>
> >> > > > >>>> Satendra, the Connect REST API enables starting, stopping,
> and
> >> > > > >>>> reconfiguring Connecters in a distributed Connect cluster.
> >> > > > >>>> Kafka
> >> > > > >> Connect,
> >> > > > >>>> more generally, is a framework for connecting Kafka to
> >> > > > >>>> external
> >> > data
> >> > > > >>>> sources, e.g. Change Data Capture from relational databases,
> >> etc.
> >> > > > >>>>
> >> > > > >>>> Ryanne
> >> > > > >>>>
> >> > > > >>>> On Fri, Nov 23, 2018, 6:48 AM Satendra Pratap Singh <
> >> > > > >>> sameerp...@gmail.com>
> >> > > > >>>> wrote:
> >> > > > >>>>
> >> > > > >>>>> Hi Team,
> >> > > > >>>>>
> >> > > > >>>>> Please click on this link :
> >> > > > >>>>> https://kafka.apache.org/documentation/#connect_rest
> >> > > > >>>>>
> >> > > > >>>>> I am kafka beginner recently started learning kafka. i want
> >> > > > >>>>> to
> >> > get
> >> > > my
> >> > > > >>>>> hands dirty in kafka. most of the topics have explained well
> >> but
> >> > i
> >> > > > >>>>> couldn't understand Rest API. what is used for? how to use
> >> > > > >>>>> it?
> >> > > > >>>>> all commands were listed under documentation i couldn't get
> >> that
> >> > > how
> >> > > > >>>>> to run them. if it's itself framework how to work on it.
> >> > > > >>>>>
> >> > > > >>>>> Please explain me. I am keen to learn about this.
> >> > > > >>>>>
> >> > > > >>>>> Looking forward to hear from you.
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>> Sent from my iPhone
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Sönke Liebau
> >> > > > > Partner
> >> > > > > Tel. +49 179 7940878
> >> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> >> Germany
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Sönke Liebau
> >> Partner
> >> Tel. +49 179 7940878
> >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >>
> >
>
