Hey, Asimansu Happy to hear you’re interested to integrate vector database and Apache Flink, I’ve talked similar topic with milvus community[1] members, I am also very familiar with the CDC, thus I know the underlying value well that the integration of the two can bring to AI users.
If you are willing to contribute the corresponding connector, I will be very happy to take time to help review FLIP[2] and code in the community. Best, Leonard [1] https://github.com/milvus-io/milvus [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > 2024年3月17日 上午2:18，Asimansu Bera <asimansu.b...@gmail.com> 写道： > > Hello Dev, > > Vector databases like Waveiate are utilized in GenAI-based applications to > store vector data in numeric format, supporting CRUD operations. > Write-ahead logging (WAL) is employed to capture any changes made to the > vector databases. For streaming-based applications, it's uncertain if > tracking changes in vector classes and objects is significant. However, > incorporating CDC (Change Data Capture) connectors can enhance processing > capabilities for vector data in streaming applications, especially for > LLM-based applications focused on data ingestion. > > https://lnkd.in/gByTZYDY > > Thoughts? > > Note: all RDBMS are introducing support for vector data.
