i've made these changes, however ...
diff --git a/app/config.py b/app/config.py
index 51356a0..4c1f224 100644
--- a/app/config.py
+++ b/app/config.py
@@ -24,6 +24,7 @@ class LLMSettings(BaseModel):
None,
description="Maximum input tokens to use across all requests (None for
unlimited)",
)
+ hf_tokenizer_id: Optional[str] = Field(None, description="HuggingFace
model_id to apply the chat template locally.")
temperature: float = Field(1.0, description="Sampling temperature")
api_type: str = Field(..., description="AzureOpenai or Openai")
api_version: str = Field(..., description="Azure Openai version if
AzureOpenai")
diff --git a/app/llm.py b/app/llm.py
index 18a13af..c15a7e4 100644
--- a/app/llm.py
+++ b/app/llm.py
@@ -71,6 +71,11 @@ class LLM:
except KeyError:
# If the model is not in tiktoken's presets, use cl100k_base
as default
self.tokenizer = tiktoken.get_encoding("cl100k_base")
+ if llm_config.hf_tokenizer_id is not None:
+ import transformers
+ self.hf_tokenizer =
transformers.AutoTokenizer.from_pretrained(llm_config.hf_tokenizer_id)
+ else:
+ self.hf_tokenizer = None
if self.api_type == "azure":
self.client = AsyncAzureOpenAI(
@@ -252,6 +257,12 @@ class LLM:
"model": self.model,
"messages": messages,
}
+ if self.hf_tokenizer is Non:
+ params["messages"] = messages
+ client_api_completions = self.client.chat.completions
+ else:
+ params["prompt"] =
self.hf_tokenizer.apply_chat_template(messages, tokenize=False)
+ client_api_completions = self.client.completions
if self.model in REASONING_MODELS:
params["max_completion_tokens"] = self.max_tokens
@@ -265,7 +276,7 @@ class LLM:
# Non-streaming request
params["stream"] = False
- response = await self.client.chat.completions.create(**params)
+ response = await client_api_completions.create(**params)
if not response.choices or not
response.choices[0].message.content:
raise ValueError("Empty or invalid response from LLM")
@@ -279,7 +290,7 @@ class LLM:
self.update_token_count(input_tokens)
params["stream"] = True
- response = await self.client.chat.completions.create(**params)
+ response = await client_api_completions.create(**params)
i don't expect them to work. i expect the two api endpoints to be too
different, maybe a couple other concerns
i'm thinking it makes sense to configure an endpoint and run it, gives one a
direct look at the issue without having to cross reference things
1622
the working api i've last used is in one of the zinc binaries hardcoded. i'd
pull it out of there and put it in openmanus
1629 ummmm ummm ummm i'm out of space install all its gpu depencies for local
evaluation. gotta disable those.
1650
(Pdb) print(params["prompt"])
<｜begin▁of▁sentence｜>You are OpenManus, an all-capable AI assistant, aimed at
solving any task presented by the user. You have various tools at your disposal
that you can call upon to efficiently complete complex requests. Whether it's
programming, information retrieval, file processing, or web browsing, you can
handle it all.<｜User｜>hi<｜User｜>You can interact with the computer using
PythonExecute, save important content and information files through FileSaver,
open browsers with BrowserUseTool, and retrieve information using GoogleSearch.
PythonExecute: Execute Python code to interact with the computer system, data
processing, automation tasks, etc.
FileSaver: Save files locally, such as txt, py, html, etc.
BrowserUseTool: Open, browse, and use web browsers.If you open a local HTML
file, you must provide the absolute path to the file.
WebSearch: Perform web information retrieval
Terminate: End the current interaction when the task is complete or when you
need additional information from the user. Use this tool to signal that you've
finished addressing the user's request or need clarification before proceeding
further.
Based on user needs, proactively select the most appropriate tool or
combination of tools. For complex tasks, you can break down the problem and use
different tools step by step to solve it. After using each tool, clearly
explain the execution results and suggest the next steps.
Always maintain a helpful, informative tone throughout the interaction. If you
encounter any limitations or need more details, clearly communicate this to the
user before terminating.
<｜Assistant｜>
....................... 1840 p
well i'm not finding any information on the tool calling formats for deepseek.
their public api has tool calling with a caveat that it doesn't work well, and
a handful of libraries have patched it on with prompt tuning, but it's not
direct
my sambanova 405b access is limited despite temporarily free
some interest in figuring out deepseek's tool calling prompt by probing their
api, maybe not the thing atm
oops! _would need a free tool calling model to make most task agents
plug-and-play!_
now maybe the quickest way to work aroudn this would be to look at what format
the system expects, and just tell a model to output in that format
