I would use flume to import these sources to HDFS and then use flink or Hadoop
or whatever to process them. While it is possible to do it in flink, you do not
want that your processing fails because the web service is not available etc.
Via flume which is suitable for this kind of tasks it is mor
Hi all,
I have a Streaming pipeline as follows:
1 - read a folder continuousely from HDFS
2 - filter duplicates (using keyby(x->x) and keeping a state per key
indicating whether its is seen)
3 - schedule some future actions on the stream using ProcessFunction and
processing time timers (elements a
Thanks for filing the JIRA!
Would you also be up to open a PR to for the change? That would be very very
helpful :)
Cheers,
Gordon
On 24 April 2017 at 3:27:48 AM, Steffen Hausmann (stef...@hausmann-family.de)
wrote:
Hi Gordon,
thanks for looking into this and sorry it took me so long to fi
Hi Sathi,
Here, in the producer-side log, it says:
2017-04-22 19:46:53,608 INFO [main] DLKnesisPublisher: Successfully published
record, of bytes :162810 partition key
:fb043b73-1d9d-447a-a593-b9c1bf3d4974-1492915604228 ShardId:
shardId-Sequence
number49572539577762637793132019619
Hi Gordon,
thanks for looking into this and sorry it took me so long to file the
issue: https://issues.apache.org/jira/browse/FLINK-6365.
Really appreciate your contributions for the Kinesis connector!
Cheers,
Steffen
On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote:
Hi Steffan,
I have to ad
+ user@flink (somehow I didn't reply to the list)
Hi Georg,
there are two classes "MultipleProgramsTestBase" and
"StreamingMultipeProgramsTestBase", which your integration tests can
extend from.
This will spin up a local Flink Mini-Cluster before the test execution
and tear it down afterwards.
Hi Georg,
Have a look at (Streaming)MultipleProgramsTestBase
(https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/StreamingMultipleProgramsTestBase.java
|
https://github.com/apache/flink/blob/master/flink-test-utils-
Since it is something special to Kafka 0.8, it could be implemented in a
simple addition to the ZooKeeperOffsetHandler used by the
FlinkKafkaConsumer08.
Would you be willing to contribute this? That would certainly help speeding
up the resolution of the issue...
On Fri, Apr 21, 2017 at 2:33 AM,
Hi Flink Dev,
I thought something will work easily with flink and it is simple enough ,yer I
am struggling to make it work.
I am using flink kinesis connector ..using 1.3-SNAPSHOT version.
Basically I am trying to bootstrap a stream with one event pushed into it as a
warmup inside flink job’s ma
New to flink I would like to do a small project to get a better feeling for
flink. I am thinking of getting some stats from several REST api (i.e.
Bitcoin course values from different exchanges) and comparing prices over
different exchanges in real time.
Are there already some REST api sources for
Hi Marc,
You can do such a map on your edges dataset that switches the
direction of edges if the source id is bigger than the target id. So
your MapFunction will have an if, which checks whether the source id
is bigger than the target id, and if it is bigger, than returns the
edge reversed, else r
Spark testing base https://github.com/holdenk/spark-testing-base offers
some Base classes to use when writing tests with Spark which make it
simpler to write unit tests for functions in spark i.e. you do not manually
need to instantiate a spark context / flink execution environment for each
test ca
Hi Gábor and anyone else ;-) ,
I need your help again.
My goal is a graph without self-loops and sum all edge values between two
vertices into one single edge (both direction).
e.g. my input graph is described by:
1 2 10
1 2 10
2 1 10
1 1 1
1 3 5
The result has to be:
1 2 30 (sum and reduce all
Please give more context by describing what spark-test-base does :-)
> On Apr 22, 2017, at 10:57 PM, Georg Heiler wrote:
>
> Hi,
>
> is there something like spark-testing-base for flink as well?
>
> Cheers,
> Georg
15 matches
Mail list logo
