Re: Serialization and kryo

ah ok gotcha, sort of like this ? ExecutionEnvironment.getExecutionEnvironment().registerTypeWithKryoSerializer(Product.class, new com.esotericsoftware.kryo.Serializer() { PS thanks, i think this should work... will update On Sun, Aug 16, 2015 at 11:01 AM, Stephan Ewen wrote: > Hi! > > The iss

Re: Flink to ingest from Kafka to HDFS?

If you are up for it, this would be a very nice addition to Flink, a great contribution :-) On Sun, Aug 16, 2015 at 7:56 PM, Stephan Ewen wrote: > Hi! > > This should definitely be possible in Flink. Pretty much exactly like you > describe it. > > You need a custom version of the HDFS sink with

Re: Flink to ingest from Kafka to HDFS?

Hi! This should definitely be possible in Flink. Pretty much exactly like you describe it. You need a custom version of the HDFS sink with some logic when to roll over to a new file. You can also make the sink "exactly once" by integrating it with the checkpointing. For that, you would probably

Flink to ingest from Kafka to HDFS?

Hi, Did anybody think of (mis-) using Flink streaming as an alternative to Apache Flume just for ingesting data from Kafka (or other streaming sources) to HDFS? Knowing that Flink can read from Kafka and write to hdfs I assume it should be possible, but Is this a good idea to do? Flume basically

Re: Serialization and kryo

Hi! The issue you refer to is: Flink automatically registers types it encounters at Kryo, if the types are not handled by Flink itself. You can turn that off in any program by "ExecutionEnvionment.getConfig().disableAutoTypeRegistration()". I am not sure that causes your problem, as this makes Kr

Re: Serialization and kryo

1. Just wanted to updated.. possibly I'm hitting ? FLINK-1417 Automatically register nested types at Kryo ... On Sun, Aug 16, 2015 at 10:38 AM, jay vyas wrote: > Hi flink. > > I just ran into the following serialization error in BigPet

Serialization and kryo

Hi flink. I just ran into the following serialization error in BigPetStore Flink. It appears to be that someone is trying to add elements to a map during serialization. I doubt this is a bug in flink because it would have surely been caught early on. Some possible explanations ... 1 - Probably

[ANNOUNCEMENT] Please add more Description and information to JIRA issues

Hi All, Apology for the cross-posting. Please remember to add detail Description in the JIRA ticket to help reproduce and prioritize the issues. Assume that some other people will try to solve and reproduce the issues. Adding information in JIRA issues is to help describe the issue and steps to