Zhuoxi2000 opened a new issue, #1059: URL: https://github.com/apache/flink-agents/issues/1059
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description Tracking issue for the "Support invoking multimodal models" 0.4 Must item from #862. Design discussed and agreed in #1031. Agreed direction: * Replace `ChatMessage.content: String` with `blocks: List<ContentBlock>` in both Java and Python. No dual representation. Existing text factories continue to work by creating a single `TextBlock`, and `getText()` / `.text` provides the ordered text projection. * `ContentBlock` includes `TextBlock` plus `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`. Media blocks share `mime_type`, exactly one of base64 `data` or external `url`, and optional metadata such as `name`, `size_bytes`, and `sha256`. * Java and Python share the same serialized shape using a `type` discriminator (`text|image|audio|video|document`), covered by the cross-language snapshots and pemja bridge. * Block ordering is preserved end-to-end. TOOL messages use the same block model, which also provides the path to fix MCP image tool results being stringified today. * Each `ChatModelConnection` maps blocks to its provider-native representation. Unsupported block types fail explicitly rather than being silently dropped or converted. * Event Log records media metadata only at every log level, including VERBOSE. Inline payloads are omitted and URL query parameters are stripped. * `tool_calls` and `extra_args` stay unchanged here. Their cleanup remains tracked in #1056. Subtasks: **Phase 1: framework** * [ ] Add the `ContentBlock` model and migrate `ChatMessage` in Java and Python * [ ] Update factories, text projection, prompt/message merging, and repo call sites * [ ] Update chat-event serialization, cross-language snapshots, and pemja bridge * [ ] Add Event Log media redaction and URL sanitization **Phase 2: providers, 0.4 target** * [ ] OpenAI Chat Completions support in Java and Python, covering OpenAI, Azure OpenAI, and vLLM * [ ] Ollama support in Java and Python * [ ] Explicit unsupported-block errors for providers not yet migrated **Follow-ups** * [ ] OpenAI Responses API * [ ] Anthropic, replacing the existing `anthropic_content_blocks` workaround * [ ] Gemini, Bedrock, and Tongyi * [ ] MCP image tool results as TOOL message blocks * [ ] Multimodal YAML prompts See #1031 for the full design discussion and rationale. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
