This is part of my series on Microsoft Foundry:
- Microsoft Foundry’s Evolution
- Microsoft Foundry BYO AI Gateway (BYO Model) – Part 1
- Microsoft Foundry BYO AI Gateway (BYO Model) – Part 2
- Microsoft Foundry BYO AI Gateway (BYO Model) – Part 3
- Microsoft Foundry Publishing Agent to Microsoft Teams – Part 1 – Overview
- Microsoft Foundry Publishing Agent to Microsoft Teams – Part 2 – The GUI Experience
- Microsoft Foundry Publishing Agent to Microsoft Teams – Part 3 – The programmatic experience
- Microsoft Foundry – OpenAI Stateful API Calls and APIM
Hello again geeks.
Today I’m going to give you a brain dump of a neat little feature I learned about in APIM (Azure API Management) while it’s still freshly floating around in the chaos that is my brain. We’ll be getting a little bit nerdy, so grab your favorite caffeinated beverage and prepare to talk some APIM and API calls.
The challenge
If you’ve followed my ramblings over the past two years (oh man, what is wrong with you?), you’ve read some of my posts about AI Gateways. One of the primary functions of an AI Gateway is to provide some level of load balancing across multiple model API endpoints to maximize requests and TPM (tokens per minute). Packing a whole bunch of requests and TPM behind a single endpoint is typically the first use case my customers have to talk about using an AI Gateway.
In the olden days of last year, customers would establish a single endpoint and pack a whole bunch of Foundry resources stretched across a whole bunch of subscriptions and bask in the stateless nature of the ChatCompletions API. Sure, you had your batch use cases, but they were largely handled through a single dedicated Foundry resource for applications that required it. Times changed with the introduction of the responses API and its introduction of server-side storage of messages (and whole bunch of other cool features). Suddenly, multi-turn conversations with the models no longer required maintaining the message array locally.
When you send a request to the responses API, the response is cached by default for 30 days. While you can turn this off by modifying the store property of the request to false, it’s unlikely this will occur often. First, store is set to true by default and most folks are pretty lazy about modifying defaults, and second, there is a real benefit for developers to offload storage and compaction of the conversation to the service vs doing it themselves.
What this means for you central IT folks is if you were relying on the stateless spraying methodology you had with ChatCompletions, your system breaks down. Whether you like it or not most of your user base is going to be living in the Responses API world. This means you need to figure out how to deal with state. APIM historically didn’t support sticky sessions, so what is a poor central IT peep supposed to do?
APIM Backend Session Affinity
Now I was asking myself this same question today. WTF am I supposed to do without session stickiness? Whenever I need to brainstorm around APIM challenges, I call my bestie Shaun Callighan to pool our brain power (about 1.5 brains counting Shaun’s full brain). We went back and forth and talked about what we typically see, which is the dedicated backends for stateful operations like batch. Shaun then remembered coming across a piece of documentation that held the solution to our problem.
Unbeknownst to me, the APIM PG (Product Group) slipped in a new session affinity property in APIM’s backend pool. For those of you folks who don’t have a ton of time smashing your keyboard working with APIM, a backend is logical APIM resource that points to some endpoint hosting some service. You reference these backends in your APIM policy and APIM forwards requests along to them. In the olden days, you had to incorporate your own logic into your APIM policy to load balance across multiple backends. A few years back, the PG introduced the concept of circuit breaker logic for backends and the concept of a backend pool. This allowed you to slap a bunch of backends configured with circuit breaker logic into the backend pool thus creating a load balanced pool. Below, is an example of a backend with a circuit breaker and a backend pool.
The left image above is a backend pool with two backends behind it. You’ll notice the sessionAffinity property. If I configure this property on a backend pool like above screenshot, APIM will return a session cookie to the client. If the client returns that session cookie in additional requests APIM will redirect that request to the same backend from the pool thus supporting the stateful responses API!
from openai import OpenAI, NotFoundError, DefaultHttpxClientfrom http.cookies import SimpleCookieimport jsonreceived_cookie_values = set()cookie_checks = []# Function to report if cookies are presentdef report_cookie_exchange(response): sent_cookies = SimpleCookie() sent_cookies.load(response.request.headers.get("cookie", "")) reused_names = sorted( name for name, cookie in sent_cookies.items() if (name, cookie.value) in received_cookie_values ) received_names = set() for header in response.headers.get_list("set-cookie"): response_cookies = SimpleCookie() response_cookies.load(header) for name, cookie in response_cookies.items(): received_names.add(name) received_cookie_values.add((name, cookie.value)) cookie_checks.append({ "received": bool(response.headers.get_list("set-cookie")), "reused": bool(reused_names) }) print(f"\nCookie check: {response.request.method} {response.request.url.path} ({response.status_code})") print(f" Sent cookie names: {sorted(sent_cookies) or 'none'}") print(f" Received Set-Cookie names: {sorted(received_names) or 'none'}") print(f" Previously received cookies sent back with matching values: {reused_names or 'none'}")# Create the Azure OpenAI Service client# client = OpenAI( base_url=FOUNDRY_MODEL_APIM_ENDPOINT, api_key=token_provider, http_client=DefaultHttpxClient(event_hooks={"response": [report_cookie_exchange]}))first_question = "What is the capital of New Hampshire?"first_response = client.responses.create( model=LLM_DEPLOYMENT_NAME, # Store responses server-side store=True, instructions="You are a helpful assistant.", input=[ { "role": "user", "content": [ { "type": "input_text", "text": first_question } ] } ])# Use the prior response id as context for a follow up questionfollow_up_question = "What is the population?"follow_up_response = client.responses.create( model=LLM_DEPLOYMENT_NAME, # Store responses server-side store=True, instructions="You are a helpful assistant.", input=[ { "role": "user", "content": [ { "type": "input_text", "text": follow_up_question } ] } ], previous_response_id=first_response.id)flower_question = "What is that state's official flower?"flower_response = client.responses.create( model=LLM_DEPLOYMENT_NAME, store=True, instructions="You are a helpful assistant.", input=[ { "role": "user", "content": [ { "type": "input_text", "text": flower_question } ] } ], previous_response_id=follow_up_response.id)flag_question = "What are the colors of that state's flag?"flag_response = client.responses.create( model=LLM_DEPLOYMENT_NAME, store=True, instructions="You are a helpful assistant.", input=[ { "role": "user", "content": [ { "type": "input_text", "text": flag_question } ] } ], previous_response_id=flower_response.id)print(f"User: {first_question}")print(f"Assistant: {first_response.output_text}")print(f"User: {follow_up_question}")print(f"Assistant: {follow_up_response.output_text}")print(f"User: {flower_question}")print(f"Assistant: {flower_response.output_text}")print(f"User: {flag_question}")print(f"Assistant: {flag_response.output_text}")print(f"--------------------------------------------")print("Cookie received:", any(check["received"] for check in cookie_checks))print("Cookie sent back with matching value:", any(check["reused"] for check in cookie_checks))# Dump the raw JSON of the responses for debugging purposesprint("Response API call response:")print(first_response.model_dump_json(indent=2))
If I run the code above, I get returned the output below showing the openai client receiving the cookie and passing with each multi-turn response. Each follow-up question includes the referenc eot the prior model response giving the model context to the entire conversation. Without session affinity, I’d end up getting round robin-ed across the backends since they’ve been configured with the same priority causing the API to spit back an error because the backends don’t have access to the per instance cached server-side response.
Cookie check: POST /openai-account/responses (200) Sent cookie names: none Received Set-Cookie names: ['SessionId'] Previously received cookies sent back with matching values: noneCookie check: POST /openai-account/responses (200) Sent cookie names: ['SessionId'] Received Set-Cookie names: none Previously received cookies sent back with matching values: ['SessionId']Cookie check: POST /openai-account/responses (200) Sent cookie names: ['SessionId'] Received Set-Cookie names: none Previously received cookies sent back with matching values: ['SessionId']Cookie check: POST /openai-account/responses (200) Sent cookie names: ['SessionId'] Received Set-Cookie names: none Previously received cookies sent back with matching values: ['SessionId']User: What is the capital of New Hampshire?Assistant: The capital of New Hampshire is Concord.User: What is the population?Assistant: Concord, New Hampshire has an estimated population of about **44,000–45,000 people**.
You’ll notice in the docs an additional APIM policy snippet. This snippet is specific to the assistants API (since the path includes the thread ID) and isn’t required for the responses API. The issuance of the cookie comes from the sessionAffinity property on the backend, not the APIM policy in link.
Now while this is cool for your stateful operations, what about the tried-and-true ChatCompletions? You may still be using the ChatCompletion for some use cases where you don’t need the stateful functionality or advanced features available in the Responses API. Now the rub here is the sessionAffinity property on the backend causes APIM to always return a session cookie which clients like the OpenAI client will use and reply meaning you’ll be pinning yourself for stateless operations like ChatCompletions to a specific backend losing some of the benefits of the stateless spray.
To work around this you can create another backend pool for stateless operations. In the APIM policy you could toss in some simple logic (taken from my Terraform template) like the below to send your stateful stuff to the stateful backend and the stateless stuff to the stateless backend.
<when condition='@{
var segments = context.Request.OriginalUrl.Path.ToLowerInvariant().Split('/');
return segments.Contains("responses") || segments.Contains("batches") || segments.Contains("files");
}'>
<set-backend-service backend-id="${module.backend_pool_msfoundry_instances_openai_accounts_stateful.name}" />
</when>
<otherwise>
<set-backend-service backend-id="${module.backend_pool_msfoundry_instances_openai_accounts_stateless.name}" />
</otherwise>Summing it up
So yeah, this was news to me. I don’t think I have any customers using this feature, but I’m def gonna bring it up next time I chat. If anyone has a more elegant way to handle the stateless operations within a single pool (vs. two) please comment below.
Your takeaways for today are:
- The documentation has the API version wrong in the sample. You need to use version 2025-09-01-preview.
- Enabling this session affinity property will cause APIM to return session cookies and will make your stuff sticky if your client is sending that session cookie back to APIM. Do your own testing before you enable to ensure don’t blow up your backend and piss off your user base.
- Figure out a way to keep stateless operations like ChatCompletions stateless and sprayed across backends evenly. I will put money you still have a lot of folks using ChatCompletions in your environment and are benefiting from the spray technique.
That’s it for me. Hope this helps some poor soul out there!
