manesioz commented on PR #377: URL: https://github.com/apache/doris-spark-connector/pull/377#issuecomment-5452405615
Hi @JNSimba, Thank you for your reply. We would like to keep the connector compatible with Doris versions before https://github.com/apache/doris/pull/66873. Also, #66873 currently fixes only Arrow Flight SQL. The external Thrift scanner uses `MEMORY_SCRATCH_SINK`, which calls `get_arrow_schema_from_block` with `datetime_naive=false`. It therefore still emits timezone-aware `DATETIMEV2`. This connector change handles that path while preserving existing `TIMESTAMPTZ` behaviour. Currently this behaviour is resulting in us not being able to reliably export data from Doris via Spark (Thrift interface) since all datetimes are shifted 8hrs (BE nodes have default Shanghai/UTC+8). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
