Can you post the shading section of your pom please?
On 19 Jan 2017 08:01, "Giuliano Caliari" wrote:
> Hey guys,
>
> I have tried the shading trick but I can't get it to work.
> I followed the documented steps and it builds but when I try to run the
> newly built version it fails when trying to
Hello,
Yuri's description of the issue is spot on. We are running our cluster on
YARN and using Avro for serialization, exactly as described.
@Ufuk, I'm running my Cluster on YARN, 4 Task Managers with 2 slots each but
this particular job has parallelism 1.
@Yuri, I'll test your fix as soon as
Hey guys,
I have tried the shading trick but I can't get it to work.
I followed the documented steps and it builds but when I try to run the
newly built version it fails when trying to connect to the Resource Manager:
2017-01-17 00:42:05,872 INFO org.apache.flink.yarn.YarnClusterDescriptor
Hi team,
Is there any way that I can register a metric in a window reduce function?
As per the flink doc getRuntimecontext is only available in RichFunction
but window operator doesn't allow RichFunction to be chained. Any way to
workaround this?
Hi there,
We are evaluating flink SQL to understand if it would be a better fit
instead of Spark. So far we loved how natural it is to consume streams on
Flink.
We do read bunch of Kafka topics and like to join those streams and
eventually run some SQL queries. We've used Kafka tables yet if I'm n
Hi all,
We bootstrap data from some DB and then like to keep it updated with
updates coming through Kafka. At spark it was kind of easy by utilizing
through UpdateStateByKey yet I'm kind of stuck to figure out how to do it
with Flink. I've taken a look into iterate yet I don't think it's meant to
Thank you for your reply.
If I were to use a keyed stream with a count-based window of 2, would
Flink keep the last state persistently until the next state is
received? Would this be another way of having Flink keep this
information persistently without having to implement it manually?
Thanks,
Ra
Hallo Fabian,
that IS the error message. The job continues to run without restarting.
There is not really more to see from the logs.
-- Jonas
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Fetch-Failed-DisconnectException-tp11142p111
I have a use case where I need to start a stream replaying historical data, and
then have it continue processing on a live kafka source, and am looking for
guidance / best practices for implementation.
Basically, I want to start up a new “version” of the stream job, and have it
process each ele
Hello, fellow Apache enthusiast. Thanks for your participation, and
interest in, the projects of the Apache Software Foundation.
I wanted to remind you that the Call For Papers (CFP) for ApacheCon
North America, and Apache: Big Data North America, closes in less than a
month. If you've been puttin
Hi Jonas,
your mail did not include the error message. Can you send it again?
Thanks, Fabian
2017-01-18 17:37 GMT+01:00 Jonas :
> Hi!
>
> According to the output, I'm having some problems with the KafkaConsumer09.
> It reports the following on stdout:
>
>
>
> Is that something I should worry abo
Hi!
According to the output, I'm having some problems with the KafkaConsumer09.
It reports the following on stdout:
Is that something I should worry about?
-- Jonas
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Fetch-Failed-Disco
Hi Niels,
I was more talking from a theoretical point of view.
Flink does not have a hook to inject a custom hash function (yet). I'm not
familiar with the details of the implementation to make an assessment
whether this would be possible or how much work it would be. However,
several users have a
Hi Raman,
I would approach this issues as follows.
You key the input stream on the sourceId and apply a stateful
FlatMapFunction.
The FlatMapFunction has a key-partioned state and stores for each key
(sourceId) the latest event as state.
When a new event arrives, you can compute the time spend in
Hi,
> However, if you would like to keyBy the original key attribute, Flink
would need to have access to the hash function that was used to assign
events to partitions.
So if my producing application and my consuming application use the same
source attributes AND the same hashing function to dete
I am investigating Flink. I am considering a relatively simple use
case -- I want to ingest streams of events that are essentially
timestamped state changes. These events may look something like:
{
sourceId: 111,
state: OPEN,
timestamp:
}
I want to apply various processing to these state c
Hi Kat,
thanks for the clarification about cases and traces.
Regarding the aggregation of traces: You can either do that in the same job
that constructs the cases or in a job which is decoupled by for instance
Kafka.
If I got your requirements right, you need a mechanism for retraction.
A case (i
Hi,
The answer to question one is clearly yes, and you can configure RocksDB
through the DBOptions.
Question two is obviously more tricky with the given information. But it is
surely possible that some resources are not properly released. All classes from
the RocksDB Java API have a safety net
For my case I tracked down the culprit. It's been Avro indeed. I'm
providing details below, since I believe the pattern is pretty common for
such issues.
In YARN setup there are several sources where classes are loaded from:
Flink lib directory, YARN lib directories, user code. The first two sourc
Hi Andrew!
There’s nothing special about extending the checkpointing interfaces for the
SinkFunction; for Flink they’re essentially user functions that have user state
to be checkpointed.
So yes, you’ll just implement is as you would for a flatMap / map / etc.
function.
Fell free to let me kno
Hi,
Yes, Flink does not rely on consumer offset commits in Kafka / Zookeeper. It
manages offsets as checkpointed state with Flink, and uses those offsets for
exactly-once.
Currently the “auto.offset.reset” is passed into the internally used
KafkaConsumer as a means to define start position, but
Hi Biswajit,
We use the following Writer for Parquet using Avro conversion (using Scala):
Using this library as dependency: "org.apache.parquet" % "parquet-avro" %
"1.8.1". We use this writer in a rolling sink and seems fine so far.
Cheers,
Bruno
--
View this message in context:
http://ap
Sorry, something went wrong with the code for the Writer. Here it is again:
import org.apache.avro.Schema
import org.apache.flink.streaming.connectors.fs.Writer
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.parquet.avro.AvroParquetWriter
import org.apache.parquet.hadoop.ParquetW
Great! Let us know if you need help.
-Vasia.
On 17 January 2017 at 10:30, Miguel Coimbra
wrote:
> Hello Vasia,
>
> I am going to look into this.
> Hopefully I will contribute to the implementation and documentation.
>
> Regards,
>
> -- Forwarded message --
> From: Vasiliki Kal
Hi Biswajit,
We use the following Writer for Parquet using Avro conversion (using Scala):
Using this library as dependency: "org.apache.parquet" % "parquet-avro" %
"1.8.1". We use this writer in a rolling sink and seems fine so far.
Cheers,
Bruno
--
View this message in context:
http://ap
Ah, OK :-)
Thanks for reporting back!
Cheers, Fabian
2017-01-17 17:50 GMT+01:00 Neil Derraugh <
neil.derra...@intellifylearning.com>:
> I re-read that enough times and it finally made sense. I wasn’t paying
> attention and thought 0.10.2 was the Kafka version —which hasn’t been
> released yet ei
26 matches
Mail list logo
