Hello Ezmlm,
I have gone through flink documentation and found it quit interesting but I
am stuck in one task i.e. mongodb streaming oplog using flink. Can you help
me to figure out this?
---
*Amol Suryawanshi*
Java Developer
am...@iprogrammer.com
*iP
Hello Ezmlm,
I have implemented code to read mongodb oplog and stream this oplog in
flink, But I need all the records in the order the are coming from oplog.
first of all clarify me is flink maintains order of insertion? if yes then
give me some source document where I can find how flink does thi
Hi,
I have used flink streaming API in my application where the source of
streaming is kafka. My kafka producer will publish data in ascending order
of time in different partitions of kafka and consumer will read data from
these partitions. However some kafka partitions may be slow due to some
ope
s maybe impossible in practice, because
> you need to store the all the incoming records and order the all data for
> every incoming records, also you need to send retracted message for the
> previous result(because every incoming record might change the global order
> of the records).
&
lism(1);
> {code}
>
>
> In all the case, you need to fix the parallelism of the OrderTheRecord
> operate to 1, which makes your job non-scale-able and becomes the
> bottleneck. So a global ordering maybe not practical on production (but if
> the source's TPS is very low, then ma
uent operator, sorted by time if they were originally sorted in its
> Kafka partition. This is more scalable approach than total global ordering.
>
> Cheers,
> Andrey
>
> On 20 Jun 2018, at 13:17, Amol S - iProgrammer
> wrote:
>
> Hello Andrey,
>
> In above code als
Hello,
I wrote an streaming programme using kafka and flink to stream mongodb
oplog. I need to maintain an order of streaming within different kafka
partitions. As global ordering of records not possible throughout all
partitions I need N consumers for N different partitions. Is it possible to
con
25, 2018 at 2:09 PM, Amol S - iProgrammer wrote:
> Hello,
>
> I wrote an streaming programme using kafka and flink to stream mongodb
> oplog. I need to maintain an order of streaming within different kafka
> partitions. As global ordering of records not possible throughout all
>
I think maybe @Gordon(CC)
> could give you more useful information.
>
> Best, Sihua
>
>
>
> On 06/25/2018 17:19,Amol S - iProgrammer
> wrote:
>
> Same kind of question I have asked on stack overflow also.
>
> Please answer it ASAP
&
Hello,
I am streaming mongodb oplog using kafka and flink and want to join
multiple tables using flink table api but i have some concerns like is it
possible to join streamed tables in flink and if yes then please provide me
some example of stream join using table API.
I gone through your dynamic
ated docs can be found here https://ci.apache.org/projects
> /flink/flink-docs-release-1.5/dev/table/tableApi.html#joins <
> https://ci.apache.org/projects/flink/flink-docs-release-1.
> 5/dev/table/tableApi.html#joins>.
>
> Best,
> Xingcan
>
>
> > On Jul 2, 201
r.com
On Mon, Jul 2, 2018 at 3:03 PM, Amol S - iProgrammer
wrote:
> Hello Xingcan
>
> DataStream streamSource = env
> .addSource(kafkaConsumer)
> .setParallelism(4);
>
> StreamTableEnvironment tableEnv = TableEnvironme
er is not known to Flink.
> What's the output of
>
> customerMISMaster.printSchema(); ?
>
> Best, Fabian
>
>
>
> 2018-07-02 11:33 GMT+02:00 Amol S - iProgrammer :
>
> > Hello Xingcan
> >
> > DataStream streamSource = env
>
flink-docs-
> release-1.5/dev/api_concepts.html#pojos
>
> 2018-07-02 12:19 GMT+02:00 Amol S - iProgrammer :
>
> > Hello Xingcan
> >
> > As mentioned in above mail thread I am streaming mongodb oplog to join
> > multiple mongo tables based on some unique key (Primary key). T
:
> You can also use Row, but then you cannot rely on automatic type extraction
> and provide TypeInformation.
>
> Amol S - iProgrammer schrieb am Mo., 2. Juli 2018,
> 12:37:
>
> > Hello Fabian,
> >
> > According to my requirement I can not create static pojo'
ammer*
www.iprogrammer.com
On Mon, Jul 2, 2018 at 4:43 PM, Amol S - iProgrammer
wrote:
> Hello Fabian,
>
> Can you please tell me hot to convert Table back into DataStream? I just
> want to print the table result.
>
> ---
Hello folks,
I am using flink table api to join multiple tables and create a single
table from them. I have some doubts in my mind.
1. How long the query will maintain partial results per key and how it
maintains state of each key?
2. If it is maintains state in memory then the memory will conti
r query and the distribution of your data.
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/streaming.html#idle-state-retention-time
>
> 2018-07-04 7:46 GMT+02:00 Amol S - iProgrammer :
>
>> Hello folks,
>>
>> I
Hello folks,
I am trying to write my streaming result into mongodb using
RIchSinkFunction as below.
gonogoCustomerApplicationStream.addSink(mongoSink)
where mongoSink is Autowired i.e. injected object and it is giving me below
error.
The implementation of the RichSinkFunction is not serializabl
Hello,
I am using flink with kafka and getting below exception.
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
helloworld.t-7: 30525 ms has passed since last append
---
*Amol Suryawanshi*
Java Developer
am...@iprogrammer.com
My flink job is failed with below exception
java.lang.RuntimeException: Buffer pool is destroyed.
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitLatencyMarker(RecordWriterOutput.java:147)
~[flink-streaming-java_2.11-1.5.0.jar:1.5.0]
at
org.apache.flink.streaming.api.operators.Abs
*
www.iprogrammer.com
On Thu, Jul 5, 2018 at 7:37 PM, Amol S - iProgrammer
wrote:
> My flink job is failed with below exception
>
>
> java.lang.RuntimeException: Buffer pool is destroyed.
> at org.apache.flink.streaming.runtime.io.Rec
Hello Fabian,
I am streaming my mongodb oplog using flink and want to use flink table API
to join multiple tables. My code looks like
DataStream streamSource = env
.addSource(kafkaConsumer)
.setParallelism(4);
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironmen
onId")`
> operation. See also [1] under "Value access functions".
>
> Regards,
> Timo
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/
> dev/table/tableApi.html#built-in-functions
>
>
> Am 27.07.18 um 09:10 schrieb Amol S - iProgrammer:
>
plaining. BasicDBObject is not recognized
> as a POJO by Flink. A POJO is required such that the Table API knows the
> types of fields for following operations.
>
> The easiest way is to implement your own scalar function. E.g. a
> `accessBasicDBObject(obj, key)`.
>
> Regards,
> Timo
>
>
but everything worked fine. Which Flink
> version are you using?
>
> Inner joins are a Flink 1.5 feature.
>
>
> Am 27.07.18 um 13:28 schrieb Amol S - iProgrammer:
>
> Table master = table1.filter("ns === 'Master'").select("o as master,
>> '
26 matches
Mail list logo