Hi All, I've been working on a custom Ptransform that integrates large language models as a PTransform in an Apache beam pipeline using Langchain.
The aim of this integration is to leverage the capabilities of LLMs for data processing and transformations of unstructured data. It treats language models as PTransform block in a data pipeline. The transform takes a prompt that has instructions on what to do with the input element, The LLM applies the instruction on the element and the model's output is returned as a Pcollection. It uses langchian to interact with the models as it provides a common interface for interacting with multiple model providers. I'm excited to share langchian-beam Ptransform with the beam community. Would like to hear your thoughts and feedback. Repository link - https://github.com/Ganeshsivakumar/langchain-beam Thanks, Ganesh.
