Hi, Logan. I guess your program used an inner process function which has closure whose checkSerialization is true (actually, this is a setting of most functions). When a closure object is created in Flink, it may hold references to other non-serializable objects, which may cause serialization errors when the closure object is sent over the network. ClosureCleaner.clean is responsible for recursively analyzing the closure object's fields and cleaning them up for serialization so that they can be removed or replaced with serializable versions, which is important in the distributed environment.
On Sun, Jun 11, 2023 at 9:23 AM DONOUGHE Logan via user < user@flink.apache.org> wrote: > Hi Everyone, > > Why does every recursive ClosureCleaner.clean call check if the associated > object is serializable? > > While investigating some issues getting a test with > ProcessFunctionTestHarnesses.forKeyedProcessFunction working, I noticed > that initially ClosureCleaner.clean is called with checkSerialization=false > but a related object is throwing a serialization error during the recursive > check due to checkSerialization always being set to true for recursive > checks: > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/ClosureCleaner.java#L135 > > While trying to understand why this is, I checked the tests and there > don’t appear to be any for this case: > https://raw.githubusercontent.com/apache/flink/master/flink-core/src/test/java/org/apache/flink/api/java/ClosureCleanerTest.java > > > > I do not believe I should have to modify a class incidentally used in the > test file to be serializable when the application does not require it for > compile or runtime so I would like to verify that this is the intended > behavior by flink. > > Thanks, > Logan > > This email and any attachments are intended solely for the use of the > individual or entity to whom it is addressed and may be confidential and/or > privileged. > > If you are not one of the named recipients or have received this email in > error, > > (i) you should not read, disclose, or copy it, > > (ii) please notify sender of your receipt by reply email and delete this > email and all attachments, > > (iii) Dassault Systèmes does not accept or assume any liability or > responsibility for any use of or reliance on this email. > > Please be informed that your personal data are processed according to our > data privacy policy as described on our website. Should you have any > questions related to personal data protection, please contact 3DS Data > Protection Officer https://www.3ds.com/privacy-policy/contact/ > > > -- Best, Hangxiang.
