haripriyarhp commented on issue #6166: URL: https://github.com/apache/hudi/issues/6166#issuecomment-1191910129
Okay, let me clarify. First, i sent 100 messages. It was fine. Athena also showed 100 records Next I sent 100 new messages + 25 updates + 25 duplicates of previous 100 messages. In total there are 250 messages in Kafka but Athena showed only 247. Irrespective of inserts, duplicates or updates, I am assuming that the connector should append the messages. Later on, I continued sending several rounds of messages and found that the count did not match. Few records were missing (somewhere between 20 -50) for around 500-600 messages sent to Kafka. I made this test several times And each time, there were some missing records. I tested with CoW too, it also had missing records. The no.of records in Athena was always less than no.of messages in Kafka -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
