Re: [DISCUSS] [Spark SQL, PySpark] Combining StructTypes into a new StructType

2022-08-14 Thread Maciej
I have mixed feelings about this proposal. Merging or diffing schemas is a common operation, but specific requirements differ from case to case, especially when complex nested data is used. Even if we put ordering of the fields aside, data types equality semantics (StructField in particular) i

Re: [DISCUSS] [Spark SQL, PySpark] Combining StructTypes into a new StructType

2022-08-14 Thread Alexandros Biratsis
Hello Rui and Tim, Indeed this sound a good idea and quite useful. To make it more formal the list of a StructType could be treated as a Scala/Python set by providing(inheriting?) the common sets' functionality i.e add, remove, concat, intersect, diff etc. The set like functionality could be part