Re: Still incompatible schemas

2020-03-09 Thread Hamish Whittal
Yeah, thanks Zahid for the reply; but that's not it. I found two schemas that differ. So I have the sucker(s) now...but how to handle them? In this case there are two columns, one is a Double and the other is Decimal(19,5) which in Parquet seems to be represented at FIXED_LENGTH_BYTE_ARRAY price

Re: Still incompatible schemas

2020-03-09 Thread Zahid Rahman
*This issue of has been discussed resolved on this page * *https://issues.apache.org/jira/browse/SPARK-17557 * *It is suggested by one person that by simply reading the parquet file in a different way as illustrated the error may go away. It

Still incompatible schemas

2020-03-09 Thread Hamish Whittal
Hi folks, Thanks for the help thus far. I'm trying to track down the source of this error: java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary w hen doing a message.show() Basically I'm reading in a single Parquet file (to try to narrow th