Hi Ganesh, I'm very excited to see this new PTransform for Beam's Java SDK. Thank you for sharing this valuable contribution.
Best regards > 2024. 12. 1. 오후 11:46, Ganesh <ganeshsivakuma...@gmail.com> 작성: > > Hi All, > > I've been working on a custom Ptransform that integrates large language > models as a PTransform in an Apache beam pipeline using Langchain. > > The aim of this integration is to leverage the capabilities of LLMs for data > processing and transformations of unstructured data. > It treats language models as PTransform block in a data pipeline. The > transform takes a prompt that has instructions on what to do with the input > element, The LLM applies the instruction on the element and the model's > output is returned as a Pcollection. > > It uses langchian to interact with the models as it provides a common > interface for interacting with multiple model providers. > I'm excited to share langchian-beam Ptransform with the beam community. Would > like to hear your thoughts and feedback. > > Repository link - https://github.com/Ganeshsivakumar/langchain-beam > > > Thanks, > Ganesh.
