AlexStocks commented on code in PR #692: URL: https://github.com/apache/dubbo-go-pixiu/pull/692#discussion_r2249531497
########## docs/ai/endpoint.md: ########## @@ -0,0 +1,153 @@ + +## LLM Gateway Endpoint Configuration + +English | [中文](./endpoint_CN.md) + +This document explains how to configure upstream endpoints for Large Language Models (LLMs) within your gateway's routing configuration. + +### Endpoint Structure + +Each endpoint within a cluster is defined by an id and can contain an llm_meta block for custom behavior.clusters: + +```yaml +clusters: + - name: "my_llm_cluster" + endpoints: + - id: "provider-1-main" + llm_meta: + provider: "deepseek" + # ... other LLM-specific configuration goes here ... + - id: "provider-2-fallback" + llm_meta: + provider: "openai" + # ... other LLM-specific configuration goes here ... +``` + +### `llm_meta` Configuration Fields + +The llm_meta block holds all the configuration specific to how the gateway should treat this LLM endpoint. + +`provider` + +- Type: `string` +- Description: A name to identify the LLM provider check [here]() for all supported llm providers. This is primarily for routing for specific llm provider. + +`fallback` + +- Type: `boolean` +- Description: Determines if the gateway should proceed to the next endpoint in the cluster if all retry attempts on this endpoint fail.true: If this endpoint fails, the gateway will attempt the next available endpoint.false: If this endpoint fails, the process stops, and the last error is returned to the client. This should be set to false for the last endpoint in a fallback chain. Review Comment: When the value is "true", and if this endpoint fails, the gateway will attempt the next available endpoint. When the value is "false", and if this endpoint fails, the process stops, and the last error is returned to the client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
