Hi! Another solution would be to locally install kafka+zookeeper and push your dumped json (from the production server) data in a topic(you create a Kafka producer). Then you configure your code to point to this local broker. Consume your data from topic from either strategy you need (earliest offset, latest). The advantage is that you can repeat your tests multiple times as in real scenario.
Depending on your use case, there can be different behaviour of your processing pipeline when you consume from a file (batch) or from a stream (kafka). I had this kind of issue when some CEP functionalities. I hope it helps. Regards, Florin On Tue, Oct 29, 2019 at 12:00 PM Anyang Hu <[email protected]> wrote: > Hi guys, > > In flink1.9, we can set `connector.type` to `kafka` and `format.type` to > json to read/write json data from kafka or write json data to kafka. > > In my scenario, I wish to read local json data as a souce table, since I > need to do local debug and don't consume online kafka data. > > For example: > >> create table source ( >> first varchar, >> id int >> ) with ( >> 'connector.type' = 'filesystem', >> 'connector.path' = '/path/to/json', >> 'format.type' = 'json' >> ) > > > In addition, writing local json data is also needed. > > Does anyone have similar needs? > > Best regards, > Anyang >
