GitHub user wenjin272 added a comment to the discussion: [Discuss][API]
Multimodal content parts in ChatMessage
Thanks for proposing this. I agree that supporting multimodal messages is
necessary for Flink Agents.
At the moment, however, I feel the proposal is still too high-level to evaluate
as an API change. It would be helpful to extend it with two concrete parts:
**1. The public data model**
Please provide the proposed Java and Python structures, including their
serialized representation. For example, one possible shape could be:
```text
ChatMessage
- role
- blocks: List<ContentBlock>
ContentBlock
- TextBlock
- ImageBlock
- AudioBlock
- VideoBlock
- DocumentBlock / FileBlock
```
The proposal should also define:
- How binary content is represented, such as URL, URI, bytes, or Base64.
- How block types are identified during serialization and cross-language
communication.
- Whether mixed blocks preserve their original order.
- Whether tool results can contain the same content block types.
Since Flink Agents is currently in Beta and we do not need to preserve this API
for compatibility, I would prefer replacing `content: String` with the block
list instead of keeping both. Having two representations would introduce two
sources of truth. Convenience APIs can still be provided for creating text-only
messages or extracting their text content.
**2. Integration with model providers**
The proposal should explain how these blocks flow into a model provider, for
example:
```text
ChatMessage blocks
-> ChatModelConnection/provider adapter
-> provider-native request content
```
Each provider uses a different representation for multimodal input. Therefore,
it would be useful to clarify:
- Whether each `ChatModelConnection` implementation is responsible for
converting `ContentBlock`s into its provider-native format.
- How unsupported block types are handled: explicit validation/error,
capability declaration, or fallback conversion.
- Whether provider responses and tool results can also return multimodal blocks.
- How this works consistently across the Java and Python implementations.
Frameworks such as LlamaIndex and AgentScope demonstrate different valid
approaches: LlamaIndex exposes modality-specific block types, while AgentScope
uses a more generic `DataBlock` routed by MIME type. I do not think Flink
Agents must copy either design, but the proposal should make an explicit choice
and provide at least one end-to-end example, such as a text-and-image user
message being converted into an OpenAI, Anthropic, or Gemini request.
GitHub link:
https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18156558
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
