Jiri Ondrusek created CAMEL-25027:
-------------------------------------
Summary: camel-langchain4j-ingest - Add audio modality to embed
audio documents whole
Key: CAMEL-25027
URL: https://issues.apache.org/jira/browse/CAMEL-25027
Project: Camel
Issue Type: New Feature
Components: camel-langchain4j
Affects Versions: 4.23.0
Reporter: Jiri Ondrusek
Assignee: Jiri Ondrusek
The langchain4j-ingest producer only handles text: the body is read as a
String, split into segments and embedded with embedAll. Audio files cannot be
ingested as audio
embeddings, the way a Wav2Vec2 or CLAP model would embed them for acoustic
similarity search. Without a parser they are decoded as garbage text and
embedded anyway.
LangChain4j 1.19 has the API for this: EmbeddingRequest accepts AudioContent
and a model declares AUDIO in supportedContentTypes().
Proposal: a modality endpoint option, text (default, unchanged behaviour) or
audio.
With modality=audio:
- the body is read as bytes and embedded whole, as one vector, through
embed(EmbeddingRequest) with an AudioContent
- the vector is stored with a placeholder TextSegment whose text is the
document id, carrying the same camel_ingest_pipeline and
camel_ingest_document_id metadata as text
segments
- the endpoint fails to start when the model does not declare AUDIO
- the MIME type comes from a new contentType option or, when unset, from the
document id's file extension (wav, mp3, flac, ogg, m4a, aac); an unknown type
fails the
exchange
- maxDocumentSize and minDocumentSize count bytes; maxSegmentSize,
maxOverlapSize and embeddingBatchSize do not apply; documentSplitter is rejected
- id filters, dedup claim and release, documentFilter and IngestResult
outcomes work unchanged
Out of scope: chunking long audio into time windows, storing the MIME type as
metadata, sniffing the type from the bytes.
Note: no LangChain4j provider embeds audio out of the box yet, so the model
is typically a custom EmbeddingModel wrapping an audio embedding service. Tests
use a
deterministic audio-capable fake.
Follow-ups planned in camel-quarkus (extension option on camel-main) and
camel-kamelets (langchain4j-ingest-sink property).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
