#!/bin/bash
# gemini doesn't immediately know why every language model interface
gives me worse results on retries
set -e -E
GEMINI_API_KEY="$GEMINI_API_KEY"
MODEL_ID="gemini-2.5-pro-exp-03-25"
GENERATE_CONTENT_API="streamGenerateContent"
cat << EOF > request.json
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "This is a determinism test to see if your
interface respects temperature=0 top_p=1.0 by deterministically
generating multiple identical outputs from a simple input."
},
]
},
{
"role": "model",
"parts": [
{
"text": "Thinking Process:\n\n1. **Identify the core
request:** The user wants to test if the model behaves
deterministically when `temperature=0` and `top_p=1.0`. This means it
should produce the *exact same output* every time for the same input
under these settings.\n2. **Understand the parameters:**\n *
`temperature=0`: This setting forces the model to always choose the
token with the highest probability at each step. It eliminates
randomness in the selection process.\n * `top_p=1.0`: This
setting means the model considers *all* possible next tokens whose
cumulative probability adds up to 1.0. When combined with
`temperature=0`, the highest probability token is always within this
set, so `top_p=1.0` doesn't counteract the determinism of
`temperature=0`. It essentially becomes redundant in this specific
scenario but doesn't harm the deterministic goal.\n3. **Formulate the
test:** The user's input is simple: \"This is a determinism test...\".
The model needs to generate a response to this statement *assuming*
the specified parameters (`temperature=0`, `top_p=1.0`) are being
correctly applied by the *interface* or *API layer* calling the
model.\n4. **Acknowledge the test:** The response should explicitly
acknowledge that this is a test for determinism under the specified
conditions.\n5. **State the expected outcome:** Explain *why* the
output should be deterministic with `temperature=0`. Mention that it
forces the selection of the most probable token.\n6. **Confirm
readiness:** Indicate that the model is ready to perform the test and
will generate its response based on these deterministic settings.\n7.
**Generate the *actual* response:** Produce a concise and relevant
response to the user's input statement. Since the input is a statement
*about* the test, a good response would be an acknowledgement and
explanation of the test's purpose and the expected behavior.\n8.
**Crucial point:** The *model itself* doesn't control the
`temperature` or `top_p` settings directly; the *calling
system/interface* does. The model's response should reflect this
understanding. It can *state* the expected behavior *if* those
settings are applied, but it cannot *guarantee* that the interface is
applying them correctly. The test is fundamentally about the
*interface*, not just the model's internal logic.\n9. **Construct the
response:**\n * Start by acknowledging the user's statement:
\"Okay, I understand.\" or similar.\n * Confirm the purpose:
\"You are testing whether the interface correctly applies
`temperature=0` and `top_p=1.0` to ensure deterministic output.\"\n
* Explain the mechanism: \"With `temperature=0`, the model should
always select the token with the highest probability at each step,
leading to identical outputs for the same input.\"\n * State
readiness/cooperation: \"I will now generate my response based on this
input. If the settings are correctly applied, subsequent requests with
the exact same input and settings should yield this identical
response.\"\n * Add a concluding remark: \"Ready for the test.\"
or \"Proceed with the test.\"\n\n10. **Self-Correction/Refinement:**
The response should be clear and directly address the user's prompt.
Avoid ambiguity. Make it explicit that the test is about the
*interface's* handling of the parameters. The generated text itself
should be something plausible and repeatable under `temp=0`. A simple
acknowledgement and explanation fits this well."
},
{
"text": "Okay, I understand. This is a test to verify that
when the generation parameters are set to `temperature=0` and
`top_p=1.0`, the model's output is deterministic, meaning it will
produce the exact same sequence of tokens for the identical input
every time.\n\nWith `temperature=0`, the model should greedily select
the token with the highest probability at each step, eliminating
randomness. `top_p=1.0` in conjunction with `temperature=0` doesn't
change this behavior, as the highest probability token will always be
selected.\n\nTherefore, if the interface correctly applies these
settings, repeating this exact prompt should yield this exact
response.\n\nProceed with the test."
},
]
},
{
"role": "user",
"parts": [
{
"text": "No, I have now run this prompt multiple times
with temperature=0 set in the interface and I have gotten different
outputs each time. I have not collected data on this, just observed
it. Please give me a small idea of what I would need to do to act on
this experience."
},
]
},
{
"role": "model",
"parts": [
{
"text": "Here's a thinking process for responding to the
user's claim about non-deterministic output despite
`temperature=0`:\n\n1. **Acknowledge and Validate:** The user states
they've observed non-deterministic behavior despite setting
`temperature=0`. First, acknowledge their observation and validate
their understanding of what `temperature=0` *should* do (produce
deterministic output). Don't immediately dismiss their claim, even if
it seems counterintuitive based on the technical definition. Phrases
like \"Okay, I understand,\" \"That's definitely unexpected,\" or
\"Thank you for reporting this\" work well.\n\n2. **Reiterate the
Expected Behavior:** Briefly restate why `temperature=0` *should* lead
to determinism. This reinforces the user's correct expectation and
sets the stage for exploring why it might *not* be happening. Mention
greedy decoding and selecting the highest probability token.\n\n3.
**Brainstorm Potential Causes for Non-Determinism (even with
temp=0):** Why might the user be seeing different outputs? Think about
the entire system, not just the core model inference.\n *
**Interface Issue:** Is the interface *actually* sending
`temperature=0` to the backend API/model? Maybe there's a bug, a
default override, or the setting isn't being applied correctly.\n *
**Backend/API Issue:** Even if the interface sends it, is the
backend *receiving* and *using* `temperature=0`? Could there be a
backend configuration overriding it? Is there some other parameter
interacting unexpectedly (though `top_p=1.0` *shouldn't* interfere
with `temp=0`)?\n * **Model-Level Subtleties:**\n *
**Floating-Point Precision:** While unlikely to cause *major*
variations in short, simple prompts, tiny floating-point differences
across different hardware or software runs *could* theoretically tip
the balance if two tokens have *extremely* close probabilities. This
is usually negligible but worth considering as a remote possibility.\n
* **Internal State/Caching:** Could some hidden state or
caching mechanism within the model or serving infrastructure be
influencing subsequent runs, even if the core prompt is identical?
(Less likely for stateless APIs, but possible).\n *
**Distributed Systems:** If the model is served across multiple
instances, could slight differences in their state or load balancing
lead to variations? (Again, `temp=0` *should* mitigate this, but
complex systems can have quirks).\n * **Non-determinism in
Underlying Libraries/Hardware:** Could there be non-determinism deeper
down, like in GPU operations (though usually controlled for
inference)?\n * **User Input Variation (Subtle):** Is the user
*absolutely certain* the input prompt is *byte-for-byte* identical
each time? Hidden characters, slight whitespace differences? (Less
likely given the user's description, but a classic debugging step).\n
* **System Updates/Model Changes:** Was the underlying model or
system updated between the user's runs? This would naturally cause
different outputs.\n * **Other Parameters:** Are *any* other
parameters being set besides `temperature` and `top_p`? Could
something like `top_k` (if set low) or repetition penalties be
interacting in an unexpected way, or perhaps a `seed` parameter that
*isn't* being kept constant?\n\n4. **Structure the Advice:** The user
asked \"what I would need to do to act on this experience.\" This
requires actionable steps. Organize the brainstormed causes into
concrete actions the user can take.\n * **Verification/Data
Collection (Crucial First Step):** The user *observed* it but didn't
collect data. The *first* action is to systematically verify and
document the issue.\n * Define a *simple, consistent*
prompt.\n * Specify *all* parameters being used
(`temperature=0`, `top_p=1.0`, ensure others are default or explicitly
set and constant).\n * Run the prompt multiple times (e.g.,
5-10 times).\n * *Carefully record* the exact input and the
*full, exact* output each time. Timestamps can also be helpful.\n *
**Isolate the Variable:**\n * Try a *different* interface
or method (e.g., direct API call if possible) with the same
parameters. Does the problem persist? (Helps determine if it's the
specific interface).\n * Try slightly different *simple*
prompts. Is it specific to one prompt?\n * Try *slightly
different* parameters (e.g., `temperature=0.01`). Does it become
deterministic then, or still variable?\n * **Reporting:** If the
data confirms non-determinism:\n * Identify the correct
channel for reporting (e.g., platform's bug report system, support
forum, developer contact).\n * Provide the collected data:
exact prompt, *all* parameters used, the multiple *different* outputs
received, timestamps, interface used, any other relevant context (like
browser, OS, if applicable).\n * **Consider Alternatives (Less
Likely Causes):** Briefly mention the more subtle possibilities
(floating point, backend issues) but frame them as things the
*platform provider* would investigate based on the user's
report.\n\n5. **Refine Language:** Use clear, encouraging, and
helpful language. Avoid jargon where possible. Emphasize the
importance of systematic testing and clear reporting. Reassure the
user that their observation is valuable if it points to a real issue.
Use phrases like \"Here's a structured approach,\" \"The first crucial
step,\" \"This helps isolate the problem.\"\n\n6.
**Self-Correction/Review:** Read through the generated response. Does
it directly answer the user's question (\"what I would need to do\")?
Are the steps logical and actionable? Is it clear why `temperature=0`
*should* be deterministic? Is it respectful of the user's observation?
Does it avoid making definitive claims about the cause before data is
collected? (Yes, seems reasonable)."
},
{
"text": "Okay, thank you for reporting that observation.
That is definitely unexpected behavior if `temperature=0` is truly
being applied correctly by the interface to the model's inference
process. As you noted, `temperature=0` should force greedy decoding,
selecting the single most probable token at each step, leading to
deterministic output for the same input.\n\nExperiencing different
outputs suggests something isn't working as expected. Here's a small
breakdown of what you could do to investigate and act on this:\n\n1.
**Systematic Verification & Data Collection (Crucial First Step):**\n
* **Choose a Very Simple Prompt:** Use something extremely basic
and short, like \"The sky is\".\n * **Confirm Parameters:**
Double-check that the interface is set to `temperature=0` and
`top_p=1.0` (or whatever `top_p` value you were using, though `1.0` is
standard with `temp=0`). Ensure no other parameters (like `top_k`,
frequency/presence penalties, or a *random* seed) are interfering.\n
* **Run Multiple Times (e.g., 5-10):** Execute the *exact same*
simple prompt repeatedly.\n * **Record Outputs:** Carefully copy
and paste the *entire* output each time. Note any differences, even
minor ones (punctuation, spacing). Timestamps might also be
helpful.\n\n2. **Isolate the Cause:**\n * **Interface vs.
Model:** Is the issue with the specific interface you're using (e.g.,
a web UI, a specific library)? If possible, try accessing the same
model through a different method (like a direct API call using `curl`
or a simple Python script) with `temperature=0` explicitly set. If the
API call is deterministic but the interface isn't, the problem likely
lies in how the interface passes parameters.\n * **Check for
Hidden Factors:** Could there be subtle variations in the input you
aren't noticing (e.g., hidden characters, slight whitespace changes)?
Ensure the input is truly identical each time. Is there any session
state or context being implicitly added by the interface?\n\n3.
**Reporting the Issue:**\n * **Gather Evidence:** Use the data
collected in step 1.\n * **Identify Reporting Channel:** Find the
appropriate place to report bugs or issues for the specific interface
or platform you are using (e.g., a GitHub repository, a support forum,
a feedback form).\n * **Submit a Clear Report:**\n *
State clearly that you expect deterministic output with
`temperature=0`.\n * Describe the steps you took (simple
prompt, parameters set).\n * Provide the *exact* simple
prompt used.\n * Provide the multiple, *different* outputs
you received.\n * Mention the interface/platform you are
using.\n\n**In summary:** The most immediate action is to
systematically test with a simple prompt, carefully record the
results, and then, if the non-determinism is confirmed, report it
clearly with your evidence to the maintainers of the interface or
platform you are using. Your observation could point to a bug in how
the parameters are handled."
},
]
},
{
"role": "user",
"parts": [
{
"text": "Ok, so I would\n- Collected timestamped data on
it\n- See if there is a different way of prompting the model that
changes the behavior\n- Report it\n\nWhere do I report this for
aistudio.google.com, where I am using you?\n\nI am testing by pressing
the diamond- or starburst- shaped \"Rerun\" button. Is this button
maybe intended to generate novel responses even if the temperature is
set to 0?"
},
]
},
{
"role": "model",
"parts": [
{
"text": "Here's a thinking process for responding to the
user's query about reporting non-deterministic behavior on
`aistudio.google.com`:\n\n1. **Acknowledge and Validate:** The user
has refined their plan based on the previous advice and is asking
specific questions about the platform (`aistudio.google.com`) and a
specific UI element (the \"Rerun\" button). Acknowledge this progress
and the validity of their questions.\n\n2. **Address the Reporting
Location:**\n * The user asks *where* to report issues for
`aistudio.google.com`.\n * Recall or look up the standard
feedback mechanisms for Google Cloud/AI products. These typically
involve:\n * In-product feedback buttons/links.\n *
Google Cloud Console support channels (if applicable, though AI Studio
might be slightly different).\n * Public issue trackers (less
common for a web UI like this, more for APIs/SDKs).\n *
Community forums.\n * Prioritize the most likely and direct
methods. In-product feedback is usually the primary channel for UIs.\n
* Formulate the answer: Look for a \"Send feedback,\" \"Report an
issue,\" or similar link/button within the AI Studio interface itself.
Mention that this is the most direct route. Add that Google Cloud
support channels might be an alternative if they have a support plan,
but in-product feedback is usually best for UI issues.\n\n3.
**Address the \"Rerun\" Button Hypothesis:**\n * The user
hypothesizes that the \"Rerun\" button might be *designed* to generate
novel responses, overriding `temperature=0`.\n * Consider the
typical function of a \"Rerun\" or \"Regenerate\" button in LLM
interfaces. Its primary purpose is usually to *resubmit the last
prompt* to the model using the *currently configured settings*.\n *
If `temperature=0` is set, rerunning *should* ideally produce the
same output. The button's function is typically \"execute again with
current settings,\" not \"execute again *but ignore
temperature=0*.\"\n * However, acknowledge the user's
observation. It's *possible* (though less likely/standard) that this
specific button has unique behavior, perhaps adding a random seed
implicitly or slightly modifying the request in a way that bypasses
the deterministic setting. It's a valid hypothesis *given the observed
behavior*.\n * Formulate the answer: Explain the *expected*
behavior (rerun should respect settings). State that while it *should*
be deterministic with `temp=0`, their observation makes it a plausible
point of investigation. Suggest including this specific detail (using
the Rerun button) in their feedback report, as it might be key to
reproducing the issue.\n\n4. **Refine the Plan (incorporating the new
info):**\n * Reiterate the core steps: Collect data, try
variations (maybe *not* using Rerun, but resubmitting the prompt
manually), report.\n * Emphasize documenting the *exact* steps,
including the use of the Rerun button vs. manual resubmission. This
comparison could be very revealing.\n\n5. **Structure and Tone:**\n
* Start by confirming the user's plan.\n * Answer the
reporting question directly.\n * Address the Rerun button
hypothesis thoughtfully, explaining expected vs. observed behavior.\n
* Conclude with encouragement and a summary of the refined plan.\n
* Maintain a helpful and collaborative tone.\n\n6.
**Self-Correction/Refinement:**\n * Is \"diamond- or starburst-
shaped\" specific enough? Maybe just call it the \"Rerun\" or
\"Regenerate\" button as identified by the user. (Stick with \"Rerun\"
as the user used it).\n * Ensure the distinction between
*expected* behavior and the *observed* anomaly is clear. Avoid
definitively stating the button *is* the problem, but acknowledge it's
a strong suspect based on the user's experience.\n * Make sure
the advice on reporting includes mentioning the specific UI element
(the button)."
},
{
"text": "Okay, let's refine that plan and address your
specific questions about `aistudio.google.com` and the \"Rerun\"
button.\n\n**Refined Plan:**\n\n1. **Collect Timestamped Data:** Yes,
exactly. Run the simple prompt multiple times using the settings
`temperature=0`, `top_p=1.0`. Record the timestamp and the *full,
exact* output for each run.\n2. **Isolate the \"Rerun\" Button:**
This is a crucial step based on your new question.\n * **Test A
(Using Rerun):** Perform your multiple runs using *only* the \"Rerun\"
(✨ Regenerate / the starburst icon) button after the first generation.
Record these outputs.\n * **Test B (Manual Resubmit):** Perform
multiple runs by *manually* clearing the output and pressing the
\"Submit\" (or equivalent) button again with the *exact same* prompt
text and settings each time. Do *not* use the \"Rerun\" button for
this test. Record these outputs.\n * **Compare:** Are the outputs
from Test A non-deterministic while the outputs from Test B are
deterministic? This would strongly suggest the \"Rerun\" button itself
introduces variability. If both are non-deterministic, the issue might
be deeper.\n3. **Report Findings:** Based on your data.\n\n**Where to
Report for aistudio.google.com:**\n\nGoogle typically uses a few
channels for feedback on its cloud and AI products:\n\n1.
**In-Product Feedback:** This is often the *most direct* way for UI
issues. Look carefully within the AI Studio interface for:\n * A
\"Send feedback\" button or link (often near your account icon,
settings, or at the bottom of the page).\n * A question mark icon
(?) that might lead to help or feedback options.\n * Sometimes
specific components have feedback options attached.\n2. **Google
Cloud Console Support:** If you are using AI Studio in the context of
a Google Cloud project and have a support plan, you can file a support
case through the Google Cloud Console. This is usually for more
technical or account-related issues, but significant bugs can be
reported this way.\n3. **Public Issue Trackers (Less Likely for
UI):** While Google has public issue trackers for many APIs and SDKs,
specific UI bugs for a tool like AI Studio are less commonly tracked
there. The in-product feedback is usually preferred.\n\n**Start with
the In-Product Feedback option if you can find it.** It's designed for
exactly this kind of report.\n\n**Regarding the \"Rerun\" (✨
Regenerate) Button:**\n\nYou raise an excellent point. **It is
*possible*, though not standard or expected, that the \"Regenerate\"
button is specifically designed to introduce some variation, perhaps
by implicitly adding a changing seed or slightly altering the request
behind the scenes, even if temperature is 0.**\n\n* **Expected
Behavior:** Normally, \"Regenerate\" or \"Rerun\" should simply
resubmit the last prompt with the *currently selected parameters*. If
`temperature=0`, it *should* yield the same result.\n* **Your
Observation:** Your experience suggests this might not be the case
here. It's a very plausible hypothesis that this specific button's
implementation differs from a standard \"submit\".\n\n**Therefore,
when you report this:**\n\n* **Crucially mention** that you are
observing the non-determinism specifically (or perhaps *more often*)
when using the \"Regenerate\" (✨) button compared to manually
resubmitting the prompt (if your Test B above shows determinism).\n*
Provide the data from both Test A and Test B if you perform them. This
comparison is strong evidence.\n\nYour systematic approach is exactly
right. Comparing the behavior of the \"Regenerate\" button versus a
manual resubmit will be key to diagnosing this."
},
]
},
{
"role": "user",
"parts": [
{
"text": "Thanks for your information. It's hard for me to
use but it's important to know."
},
]
},
{
"role": "model",
"parts": [
{
"text": "Thinking Process:\n\n1. **Acknowledge and
Validate:** The user expresses that the information is useful but
difficult to act upon. This is a valid feeling – collecting data,
isolating variables, and reporting bugs *can* be time-consuming and
sometimes frustrating. Start by acknowledging this.\n2.
**Empathize:** Show understanding for why it might be hard (time,
effort, technical steps).\n3. **Reiterate Value:** Briefly remind the
user *why* this process, despite being difficult, is important
(improving the tool, ensuring expected behavior, helping others).\n4.
**Offer Simplification/Focus:** Since the user finds it hard, suggest
focusing on the *most impactful* and *simplest* next step. Based on
the previous conversation, the key finding was the potential
difference between \"Regenerate\" and \"Submit\". Testing *just that*
might be less daunting than a full bug report initially.\n5. **Lower
the Barrier:** Frame the next step as a smaller experiment. \"Even
just trying the 'Submit' vs 'Regenerate' comparison a few times...\"
makes it sound less like a formal research project.\n6. **Reassure:**
Let the user know that even small observations are valuable and that
they don't need to do a perfect, exhaustive analysis to provide useful
feedback.\n7. **Keep the Door Open:** End with an offer of further
help or clarification if they decide to proceed or have more
questions."
},
{
"text": "You're welcome. I understand completely. It *can*
feel like a lot of effort to go through systematic testing and
reporting, especially when you just want the tool to work as
expected.\n\nIt's perfectly okay to find the process cumbersome. The
key takeaway is that your initial observation is valid and potentially
points to an issue or a specific behavior of the \"Regenerate\" button
that isn't immediately obvious.\n\nEven without doing a full, formal
data collection and report:\n\n* **Knowing the *possibility*** that
the \"Regenerate\" button might behave differently than a manual
submit is useful information for your own future use of the tool.\n*
You can be mindful of this potential discrepancy if you need truly
deterministic outputs in the future (and perhaps favor manual
resubmission if that proves more consistent).\n\nThank you for raising
the point. It's valuable feedback in itself, highlighting a potential
area of confusion or unexpected behavior in the interface. If you do
decide to investigate further later, you know the steps, but don't
feel obligated if it's too much hassle right now."
},
]
},
{
"role": "user",
"parts": [
{
"text": "INSERT_INPUT_HERE"
},
]
},
],
"generationConfig": {
"temperature": 0,
"topP": 1,
"responseMimeType": "text/plain",
},
}
EOF
curl \
-X POST \
-H "Content-Type: application/json" \
"https://generativelanguage.googleapis.com/v1beta/models/${MODEL_ID}:${GENERATE_CONTENT_API}?key=${GEMINI_API_KEY}";
-d '@request.json'
