Re: Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-20 Thread Enrico Olivelli
I have filed a PR with an update to the Release notes for 2.8.0 https://github.com/apache/pulsar/pull/11392 Thank you all for your feedback Enrico Il giorno mar 20 lug 2021 alle ore 00:54 Neng Lu ha scritto: > Based on my local test, it's fine for String Schema. > > On 2021/07/19 18:47:49 Devin

Re: Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Neng Lu
Based on my local test, it's fine for String Schema. On 2021/07/19 18:47:49 Devin Bost wrote: > > This leads to an IncompatibleClassChangeError when you have a Function or > > a Connector that is using Schema.JSON(Pojo.class) > > I just noticed this detail. Do we have a sense of how often people

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Enrico Olivelli
Il Lun 19 Lug 2021, 20:48 Devin Bost ha scritto: > > This leads to an IncompatibleClassChangeError when you have a Function > or > > a Connector that is using Schema.JSON(Pojo.class) > > I just noticed this detail. Do we have a sense of how often people are > using Schema.JSON in Functions/Conne

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Devin Bost
> This leads to an IncompatibleClassChangeError when you have a Function or > a Connector that is using Schema.JSON(Pojo.class) I just noticed this detail. Do we have a sense of how often people are using Schema.JSON in Functions/Connectors? Most of our functions are using a string schema, so it'

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Devin Bost
> I think Sijie is referring to using KubernetesRuntime to deploy functions > where each function/source/sink runs as an independent statefulset in K8s. > In this scenario, it is possible to have fine grained control over which > version of the function container the function is using. Not everybo

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Devin Bost
> For example, if you are upgrading Flink from one version to the other > version, you have to make a save point in the previous version for all > the Flink jobs. > Upgrade the Flink cluster and resume jobs in a new version. > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Sijie Guo
On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng wrote: > > I agree that the best we can do right now is to just clearly document this > as a potential problem when updating 2.7 to 2.8. > > We should definitely make every attempt to not make BC breaking changes. > However, there are times when we have

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Sijie Guo
On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli wrote: > > Sijie, > Thank you for your feedback > Some additional considerations inline > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > I don't think this is a big problem. Because people can recompile the > > function and submit the func

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Devin Bost
Is the client at least backwards compatible so that we could have all of our teams upgrade functions before a broker upgrade is scheduled? We have hundreds of functions running in production that are maintained by many different teams and departments. If upgrades of the functions need be all happe

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Jerry Peng
I agree that the best we can do right now is to just clearly document this as a potential problem when updating 2.7 to 2.8. We should definitely make every attempt to not make BC breaking changes. However, there are times when we have to make these tough decisions for one reason or another. The bi

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-19 Thread Enrico Olivelli
Sijie, Thank you for your feedback Some additional considerations inline Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > I don't think this is a big problem. Because people can recompile the > function and submit the function. Most of the computing/streaming > engines ask users to recompile th

Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-18 Thread Sijie Guo
I don't think this is a big problem. Because people can recompile the function and submit the function. Most of the computing/streaming engines ask users to recompile the jobs and resubmit the jobs when it upgrades to a new version. The best approach here is to document this behavior. Also, if you

Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8

2021-07-16 Thread Enrico Olivelli
Hello, I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. More information is on the ticket, but the short version of the story is that in Pulsar 2.8 we introduced a breaking change in the Schema API, by switching SchemaInfo from a class to an interface. This leads to an Incomp