Hi Neng, Thanks for bringing this issue up. Using JSON as the default schema and wrapping it with other primitive types are counterintuitive, and +1 to make [2] align with [1] so that both Pulsar Source and Pulsar Sink will make correct support with other primitive types.
And as per the code [3], if the topic already exists, it will try to use the existing schema instead of the schema type returned by [2]. So the changes will only affect the newly deployed instances. [3] https://github.com/apache/pulsar/blob/branch-3.0/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java#L102-L122 Best, Rui Fu On Apr 28, 2023 at 13:36 +0800, Pengcheng Jiang <pengcheng.ji...@streamnative.io.invalid>, wrote: > Hello Neng, > > IMO, we should update code[2] to follow the doc, and for existing > functions, if they are in running status, they won't touch code[2]; and for > a new run, functions > will fail to start, and this will remind users to update their function > > Regards, > Pengcheng Jiang > > Neng Lu <nl...@apache.org> 于2023年4月28日周五 06:59写道： > > > Hi All, > > > > Based on [1], Pulsar has various primitive schema types and has a very > > clear mapping between java classes to primitive schema types. > > > > But in code [2], Pulsar Function Source only handles the byte and String > > java classes primitive schema mapping while default all other primitive > > types to JSON schema. Also for byte class types, the NONE schema is used > > instead of the BYTES schema. > > > > All these differences cause confusion for users trying to use Pulsar > > Functions for the first time, and also make Pulsar Function not following > > the Pulsar Schema official document. > > > > Ideally, we should change the code [2], to make it following [1]. But such > > changes may lead to breaking behaviors for existing users who adapted their > > code to run the Pulsar Functions. > > > > I would like to hear your thoughts on this and see how we should proceed. > > > > Thank you! Regards > > > > [1] > > https://pulsar.apache.org/docs/2.11.x/schema-understand/#primitive-type > > [2] > > > > https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java#L124 > >
