yanqinghe opened a new issue, #6436: URL: https://github.com/apache/hudi/issues/6436
**Describe the problem you faced** We have a partition table. The partition key of this table is hour. Becauese we write incremental data into it in real-time, partitions will be added continuously. For example: partition : hour = 1, hour = 2, hour =3, .... hour = 23 There is a real-time consumption task downstream of the table. When we use the following SQL consumption, we find that the data of the new partition is ignored. ``` sql select * from table_1 where hour is not null. ``` For example, our mission was started at 11:00. If the latest partition in the table is hour = 11 at this time, our downstream users cannot consume some data belong to partition( hour = 12 or later). A clear and concise description of the problem. **To Reproduce** Steps to reproduce the behavior: 1. create a partition table and write some data. 2. start a Flink task that continuously streaming read hudi table with partition key filters, such as 'greater', 'is not null'. 3. create a new partition of this table. 4. check output. **Expected behavior** Some filter conditions should not be applied to the latest partition A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.10.1 It seems that master branch has not been repaired * Spark version : * Hive version : * Hadoop version : * Storage (HDFS/S3/GCS..) : * Running on Docker? (yes/no) : **Additional context** Flink Version 1.13+ **Stacktrace** ```Add the stacktrace of the error.``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
