Hi, I've been started to use Apache Beam not so long ago (so bear with
me please) and I have a question about Beam coders and how are they
related with under the hood runner serializer... Let me explain myself
better:
As far I've understood from Beam documentation, coders are being used
in order t
duced between two Beam releases.
>
> Best,
> Max
>
> PS:
> This blog post might shed some more light on the matter:
> https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
>
> On 19.05.20 13:45, Ivan San Jose wrote:
> > Hi, I've been sta
Hi, we have a Beam application running with Flink runner and we are
struggling using Flink checkpoints. Everytime we evolve the source code
modifying a Java model, an exception is thrown when trying to restore
last checkpoint taken:
Caused by: java.lang.IllegalStateException: Could not Java-deseri
when you evolve your data model, the only way you can
> maintain
> compatible is to use a serialization format which can evolve, e.g.
> KafkaIO or Protobuf.
>
> Cheers,
> Max
>
> On 03.06.20 16:47, Ivan San Jose wrote:
> > Hi, we have a Beam application running with
ckpoint is still using Java serialization mechanism...
On Thu, 2020-06-04 at 13:05 +0000, Ivan San Jose wrote:
> Thanks Max for your response. I'd try with AvroCoder then. But I
> still
> have a question, I guess AvroCoder is generating the AVRO schema
> using
> Java reflection,
e old coder
>
> From all the options (a) is the most feasible for you. It looks like
> neither ProtoCoder nor AvroCoder fall into this category.
>
> -Max
>
> On 04.06.20 16:22, Ivan San Jose wrote:
> > I've changed my Java model in order to use ProtoCoder (with
> &
By the way, I've just tried using AvroCoder, which is inferring the
schema from the Java object to be deserialized and I got same error
when restoring the checkpoint :(
On Fri, 2020-06-05 at 06:24 +0000, Ivan San Jose wrote:
> Sorry but I'm afraid I'm not understanding
in Beam, the best solution is to write
> coder which allows Schema migration, e.g.
> AvroCoder/ProtoCoder/SchemaCoder but does not keep any non-transient
> references to the type class. It has to lazily initially the schema
> from
> the current data class.
>
>
> > Could we try
);
But is not working neither. Using JODA time works though.
Does it ring any bell for you?
Thanks
On Fri, 2020-06-05 at 13:15 +, Ivan San Jose wrote:
> Thank you so much for your detailed answers Max, I will try to
> achieve
> what you've suggested about creating a custom coder w
bjectInputStream ois = new ObjectInputStream(fis);
System.out.println(coder.decode(ois));
ois.close();
}
}
On Mon, 2020-06-08 at 14:21 +, Ivan San Jose wrote:
> Finally I've managed to modify Beam's AvroCoder in order not to
> serialize any Class reference of the
update uses the new graph, so doesn't hit
> this problem.
>
> On Mon, Jun 8, 2020 at 7:21 AM Ivan San Jose <
> isanj...@theworkshop.com> wrote:
> > Finally I've managed to modify Beam's AvroCoder in order not to
> > serialize any Class reference of the
lizer, you can add fields to your POJOs and
recover from saved checkpoints containing old POJO versions.
On Mon, 2020-06-08 at 14:37 +0000, Ivan San Jose wrote:
> Hi Reuven, as far I've understood, Apache Beam coders are wrapped
> into
> Flink's TypeSerializers, so they are be
agree that serializing coders into the checkpoint creates
> problems. I'm wondering whether it is possible to serialize the coder
> URN + args instead.
>
> On Mon, Jun 22, 2020 at 11:00 PM Ivan San Jose <
> isanj...@theworkshop.com> wrote:
> > Hi again, just replyin
Hi, I'm the reporter of
https://issues.apache.org/jira/browse/BEAM-10928 . You are right, I had
same issue and did a really ugly hack but it works, I will make a PR to
Beam as soon as I have some free time, sorry.
The issue is really weird and I think is related with Google AutoValue
implementatio
14 matches
Mail list logo
