Hi Team, We have used Iceberg file format for our Data pipeline setup on AWS. We are using Glue for Data processing and output tables are then read in Quicksight for dashboard. Our pipeline was working fine and Quicksight dashboard were seeing table columns which were same as per the iceberg tables. But once we ran schema evolution commands mentioned below on the iceberg tables to change column names we observed below issues. Schema evolution commands - Alter table glue_catalog.stream_db.session_table RENAME COLUMN nextgentravelprogramidentifier to programnextgenidentifier Issues: Due to this, though we see the column name changes when we view tables in Athena but in QuickSight i see old columns and new columns both. I checked that the glue catalog for the table has both columns. Though we can change the glue catalog using boto3 to keep only new columns only, but this gets overwritten every time we write new data to iceberg table. Can we setup a quick call to investigate this or suggest support slack community to discuss on this.
Thanks & Regards, Sanket Chaure Unless indicated otherwise, the information contained in this message is privileged and confidential, and is intended only for the use of the addressee(s) named above and others who have been specifically authorized to receive it. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message and/or attachments is strictly prohibited. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses. Therefore, the company accepts no liability for any damage caused by this email. If you have received this message in error, please contact the sender and delete the message
