Artem created FLINK-34620:
-----------------------------

             Summary: Process recursive protobuf schemas in PbToRowTypeUtil
                 Key: FLINK-34620
                 URL: https://issues.apache.org/jira/browse/FLINK-34620
             Project: Flink
          Issue Type: Improvement
          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
            Reporter: Artem


The current version of _PbToRowTypeUtil.generateRowType_ doesn't support 
recursive protobuf schemas like
message RecursiveMessage \{
   int32 id = 1;
   RecursiveMessage message = 2;
 }
It throws _java.lang.StackOverflowError._


This could be implemented by trimming the recursion. Same approach is used in 
Apache Spark, for example. 
([link|[https://github.com/apache/spark/blob/master/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala#L178])]

 

then output Table schemas for the above proto-schema will be like this:
recursiveFieldMaxDepth=0:  message: ROW< id: INTEGER > 
recursiveFieldMaxDepth=1:  message: ROW< id: INTEGER, message: ROW< id: INTEGER 
> > recursiveFieldMaxDepth=2:  message: ROW< id: INTEGER, message: ROW< id: 
INTEGER, message: ROW< id: INTEGER > > >
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to