I created a JIRA ticket for this problem (
https://issues.apache.org/jira/browse/FLINK-1828) and I just made a PR that
fix that (thanks Fabian and Robert for the great support!)
Last question: is the "mapred.output.dir" parameter really necessary?
At the end of my job that writes to hbase I found o
Yes, reusing output objects is a good practice but optional. It can help to
bring down GC overhead.
You could make your function a RichFunction and initialize the output object in
open().
Switching function serialization to Kryo is on our TODO list (FLINK-1256).
Would be good to fix that so
Ok graeat!this was not prtfectly clear to me! I'll try that now.
About reuse variable instead..I use it because I saw that this is a common
practice ib the examples but I'd like to know whether there's a real
benefit in reusing it for return tuple with respect to returning a brand
new one each tim
User functions are still serialized using Java serialization, not Kryo.
Kryo is only used for data exchange at runtime between tasks.
If a function such as your MapFunction has a non-serializable member
variable, you need to declare it as transient and initialize it before it
is executed, e.g., vi
There's no way to register put with kryo for example?
On Apr 4, 2015 10:06 AM, "Robert Metzger" wrote:
> Hey Flavio,
>
> I checked out your "master" branch and started the HBaseWriteExample.
> It started without errors (there were some errors connecting to Zookeeper,
> but thats probably because
Hey Flavio,
I checked out your "master" branch and started the HBaseWriteExample.
It started without errors (there were some errors connecting to Zookeeper,
but thats probably because I don't have HBase running).
Am I using the right code (
https://github.com/fpompermaier/flink/commit/c1934da379db
Any fix for this?
On Apr 3, 2015 7:43 AM, "Flavio Pompermaier" wrote:
Which field?the Tuple2?I use it with Flink 0.8.1 without errors
On Apr 3, 2015 2:27 AM, wrote:
> If Put is not Serializable it cannot be serialized and shipped.
>
> Is it possible to make that field transient and initialize Put in
> configure()?
>
>
>
>
>
>
> From: Flavio Pompermaier
> Sent: Fri
If Put is not Serializable it cannot be serialized and shipped.
Is it possible to make that field transient and initialize Put in configure()?
From: Flavio Pompermaier
Sent: Friday, 3. April, 2015 01:42
To: dev@flink.apache.org
Now I made my fork (https://github.com/fpompermai
Now I made my fork (https://github.com/fpompermaier/flink) but when I run
the application I get this error:
java.io.NotSerializableException: org.apache.hadoop.hbase.client.Put
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.writeObject(Objec
What ever works best for you.
We can easily backport or forwardport the patch.
2015-04-01 14:12 GMT+02:00 Flavio Pompermaier :
> Ok..I'd like to have this fix in the next release. Should I branch Flink
> 0.8.1 or 0.9 or which version?
>
> On Wed, Apr 1, 2015 at 2:04 PM, Maximilian Michels wrote:
Just base your changes on the current master.
On Wed, Apr 1, 2015 at 2:12 PM, Flavio Pompermaier
wrote:
> Ok..I'd like to have this fix in the next release. Should I branch Flink
> 0.8.1 or 0.9 or which version?
>
> On Wed, Apr 1, 2015 at 2:04 PM, Maximilian Michels wrote:
>
> > Hi Flavio,
> >
Ok..I'd like to have this fix in the next release. Should I branch Flink
0.8.1 or 0.9 or which version?
On Wed, Apr 1, 2015 at 2:04 PM, Maximilian Michels wrote:
> Hi Flavio,
>
> Thanks for looking into this problem. Actually, it's a bit difficult to
> discuss your changes here because of the fo
Hi Flavio,
Thanks for looking into this problem. Actually, it's a bit difficult to
discuss your changes here because of the formatting/syntax highlighting and
missing context of the classes. Usually, we do that in a pull request. Do
you have a GitHub account? If so, push your changes to your forke
As I said before, I think the configure() method of the original
HadoopOutputFormat should be called in the configure() method of the Flink
HadoopOutputFormatBase. Flink calls configure() before open() and
finalizeOnMaster(), so that should work.
Have you checked if that fixes your problem?
If yes
Any feedback about this?
On Tue, Mar 31, 2015 at 7:07 PM, Flavio Pompermaier
wrote:
> Hi Flink devs,
> this is my final report about the HBaseOutputFormat problem (with Flink
> 0.8.1) and I hope you could suggest me the best way to make a PR:
>
> 1) The following code produce the error reported
Hi Flink devs,
this is my final report about the HBaseOutputFormat problem (with Flink
0.8.1) and I hope you could suggest me the best way to make a PR:
1) The following code produce the error reported below (this should be
fixed in 0.9 right?)
Job job = Job.getInstance();
myDataset.output
17 matches
Mail list logo
