xintongsong commented on code in PR #103: URL: https://github.com/apache/flink-agents/pull/103#discussion_r2284360447
########## runtime/src/main/java/org/apache/flink/agents/runtime/logger/FileEventLogger.java: ########## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.agents.runtime.logger; + +import org.apache.flink.agents.api.Event; +import org.apache.flink.agents.api.EventContext; +import org.apache.flink.agents.api.EventFilter; +import org.apache.flink.agents.api.logger.EventLogRecord; +import org.apache.flink.agents.api.logger.EventLogger; +import org.apache.flink.agents.api.logger.EventLoggerOpenParams; +import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper; + +import java.io.BufferedWriter; +import java.io.FileWriter; +import java.io.PrintWriter; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** + * A file-based event logger that logs events to a file specific to each subtask. + * + * <p>This logger creates a unique directory structure for each subtask based on the job name and + * subtask ID, ensuring no file conflicts in multi-TaskManager deployments. Events are appended to a + * log file named "events.log" within that directory in JSON Lines format. + * + * <h3>Thread Safety</h3> + * + * <p>This class is <strong>thread-safe at the Flink subtask level</strong>. Flink's execution model + * guarantees that each subtask instance processes events in a single-threaded manner within the + * operator's mailbox thread. This means: + * + * <ul> + * <li>No synchronization is needed for concurrent access within a subtask + * <li>Each subtask instance gets its own logger instance and unique log file + * <li>Multiple subtasks can run concurrently without file conflicts + * </ul> + * + * <h3>File Structure</h3> + * + * <p>The logger creates the following directory structure: + * + * <pre> + * {baseLogDir}/ + * └── {jobId}/ + * ├── 0/ + * │ └── events.log (subtask 0) + * ├── 1/ + * │ └── events.log (subtask 1) + * └── 2/ + * └── events.log (subtask 2) Review Comment: > IIUC, the existing implementation doesn't support multiple agents in a same Flink job: Actually, I think multiple agents are supported. You cannot call `apply()` multiple times on one AgentBuilder, but you can create multiple AgentBuilder-s by calling `fromDataStream/Table()` multiple times. > with meaningful structured names (e.g., events---.log) I just noticed that my example file name here doesn't display as expected. I meant `events-<jobid>-<taskid>-<subtaskid>.log` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
