GitHub user Zhuoxi2000 created a discussion: [Discuss][API] Multimodal content parts in ChatMessage
Follow-up to the 0.4 planning discussion in #862. I volunteered for the “Support invoking multimodal models” Must item there, and @wenjin272 confirmed no one is working on it yet. The proposal is to add first-class typed content parts to `ChatMessage` in both Java and Python, while keeping plain-string content fully backward compatible. ## Motivation Today `ChatMessage.content` is a plain string, even though the chat-model SDKs we already integrate with support multimodal input natively. There are also two concrete problems in the current code: 1. We already have provider-specific workarounds. For example, Python Anthropic carries native content blocks through `extra_args["anthropic_content_blocks"]`. This works, but it is untyped and provider-specific. 2. MCP tool results can already contain images, but `ChatModelAction` eventually stringifies the result before sending it back to the model. So an image currently becomes the string representation of a map containing base64 data rather than actual image content. A first-class content-part API would give us one framework-level representation for these cases. ## Proposed API Add a flat `ContentPart` type in both languages with three initial variants: * `text` * `image` * `document` Each part would carry only the fields it needs, e.g. `text`, `mime_type`, and either `data` or `url`. I’d keep it flat rather than introducing a nested provider-style `source` object. Besides being simpler across providers, this also works better with the current Event Log depth limits. `ChatMessage` would keep its existing `content: String` and gain `contentParts` / `content_parts`, defaulting to empty. The compatibility rule would be: * no parts → `content` behaves exactly as it does today * parts present → parts are authoritative, while `content` remains a plain-text projection of the text parts for existing consumers I considered making `content` itself a `String | List<ContentPart>` union, but that would be much more disruptive across Java callers, Jackson mapping, and the pemja bridge. An additive field seems safer for 0.4. This also requires one small framework fix: image-only messages must not be dropped by the current “empty text content” filtering in the prompt/message merge logic. ## Cross-language / serialization `content_parts` would be additive in the serialized Event Log format, with old records continuing to deserialize normally. The pemja bridge needs explicit support because those conversions copy fields directly rather than going through JSON. I’d leave multimodal YAML prompts out of scope for 0.4. Existing text YAML stays unchanged, and YAML `parts:` can be added separately once the API settles. ## Event Log and state At STANDARD Event Log level, inline base64 is already bounded by the existing string truncation limit, so this does not make logs unbounded by default. There is still a state-size concern for long agent/tool loops, since inline media may be carried through sensory memory. For 0.4, my preference would be to recommend URL-backed parts for large media and leave a first-class blob/reference type for follow-up work. ## Proposed 0.4 scope **Phase 1 — framework** * `ContentPart` + `ChatMessage` support in Java and Python * Event Log / cross-language serialization * pemja bridge * image-only message filtering fix * round-trip and snapshot tests **Phase 2 — initial providers** Start with OpenAI Chat Completions and Ollama. OpenAI is a good first target because the shared conversion path also covers Azure OpenAI and vLLM. Ollama already has direct image support in both SDKs. I’d start with USER-role multimodal input, which covers the primary invocation use case. **Follow-ups** Anthropic, Gemini, Bedrock, Responses API, Tongyi, MCP image tool results, YAML parts, and richer media-reference support can follow incrementally. ## Open questions 1. Should Event Log store a truncated base64 prefix, or replace media payloads with metadata such as `<image/png, 123 KB>`? In particular, should VERBOSE ever record full media bytes? 2. Is recommending URL-backed media enough for 0.4, or should we introduce a reference/blob type now? 3. For existing `extra_args` escape hatches, my inclination is to gradually subsume provider-specific ones like `anthropic_content_blocks`, while leaving generic `extra_args` available for other provider-specific options. 4. Naming: `content_parts` or simply `parts`? If this direction looks reasonable, I’ll open a tracking issue with Phase 1 / Phase 2 subtasks and start with the framework work. GitHub link: https://github.com/apache/flink-agents/discussions/1031 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
