GitHub user wenjin272 added a comment to the discussion: [Discuss][API]
Multimodal content parts in ChatMessage
Here are my thoughts on the four open questions.
### 1. Event Log representation
I would prefer recording media metadata rather than a truncated Base64 prefix.
A Base64 prefix has little diagnostic value, but it can still expose part of
the original content.
For example, the Event Log could contain:
```json
{
"type": "image",
"mime_type": "image/png",
"source_type": "inline",
"size_bytes": 126432,
"sha256": "...",
"data": "<omitted>"
}
```
For URL-backed content, we could record a sanitized URI or reference
identifier, avoiding signed query parameters where appropriate.
I also do not think `VERBOSE` should automatically record complete media bytes.
Binary content can be large and sensitive, and an Event Log should not
implicitly become media storage. If preserving the complete payload is needed
in the future, it would be better to introduce a separate, explicitly enabled
artifact-capture mechanism.
### 2. URL-backed content versus a Blob/Reference type
For 0.4, I think supporting inline data and externally managed URLs is
sufficient. I would not introduce a Flink Agents-managed Blob Storage service
as part of this change, since that would also require decisions about storage
backends, credentials, cleanup, lifetime, and recovery.
However, the data model should leave room for adding a managed reference later.
For example, an image block could initially allow mutually exclusive `data` and
`url` fields, together with `mime_type` and optional metadata such as size,
checksum, and name. A `blob_id` or another reference form could then be added
later without redesigning the content-block hierarchy.
The documentation should also make it clear that URL-backed content is
externally managed. The user is responsible for its accessibility and lifetime.
URLs may expire, may not be accessible to the model provider, and may no longer
be valid after recovery from a checkpoint. Therefore, recommending URLs reduces
state size, but it does not provide durable media storage.
### 3. Existing `extra_args` escape hatches
I agree with gradually replacing provider-specific content workarounds such as
`anthropic_content_blocks` once the standard content-block API can represent
the same information. Multimodal message content should have one canonical
representation and should not be duplicated in both content blocks and
`extra_args`.
Generic extensibility is still useful, so I would not remove `extra_args`
entirely. However, its current name and responsibilities are somewhat
ambiguous: it contains message metadata, framework protocol fields, provider
response information, and provider-specific extensions. If its intended
long-term role is auxiliary message metadata, `metadata` may be a clearer name,
while provider-specific request fields could use a separate mechanism.
Some stable fields currently stored there, such as the tool-call ID and token
usage, may also deserve dedicated typed fields. Usage may even belong to a
response-level object rather than `ChatMessage`. This is orthogonal to the
multimodal proposal and does not need to block it, but it may be worth tracking
separately.
### 4. Naming
Since Flink Agents is still in Beta and backward compatibility is not required
for this API, I would prefer replacing `content: String` rather than adding a
second authoritative representation.
My preferred shape would be:
```text
ChatMessage
- role
- blocks: List<ContentBlock>
- toolCalls
- metadata
```
A text-only message would simply contain one `TextBlock`. Convenience factory
methods can preserve the simple text-message experience without storing a
separate string field.
Therefore, I would prefer `blocks` with `ContentBlock` over either
`content_parts` or `parts`. If the existing string field must be retained, then
`content_parts` is clearer than the more generic `parts`, but I think a single
canonical `blocks` field would produce a cleaner API.
GitHub link:
https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18156757
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
