Models & API

How to connect

The gateway is OpenAI-compatible. Point any OpenAI SDK at it with the API key issued from your Dashboard.

Primary models

Always-on GPU lanes (live status). Fastest first token.

On-demand models are down until an operator launches them, and a call returns an error until then. Today that is deepseek-v4-flash-abliterated (restricted research use), which needs essentially all of Isaac. Vision is now always available on qwen3.8-27b.

Restricted models (per-key permission)

Some models on the gateway are not available to a key unless an admin grants them. Calling one without permission returns 403 key not allowed to access model. Ask an admin if you need access; grants are per key and take effect immediately — your key value does not change.

Everything else on this page is available to every key. GET /v1/models lists what the gateway serves; a model appearing there does not mean your key may call it.

Model ids that changed

The coding lane is now qwen3-coder-next-80b-a3b. The old id qwen3-coder-30b-fp8 still works and points at the same lane, but it was misleading — it said “30b” while serving an 80B-A3B model. Please move to the new name; the alias will not be kept forever.

Retired 2026-08-21: qwen3.5-122b-a10b (replaced by qwen3.8-27b, which matched it on our reasoning tests at a fifth the size), plus qwen2.5-32b-fp8, qwen3.8-27b-vl and qwen3-vl-235b-awq — three ids that either named the wrong model or could never start.

Which model should I use?

Three questions. Every model here is free to you, but none is free to the cluster — picking a bigger model than you need means slower answers for you and fewer students served at once.

RAG, reranking & speech (new)

If you hit a database connection error on 8 September, it is fixed — and you did not lose anything. 192.168.1.199:5433 was unreachable for about 40 minutes after the cluster was rebooted for a driver update. The database itself shut down cleanly and its storage was never touched, so any tables and embeddings you had created are still there. Just reconnect. A boot check now starts the database explicitly and verifies the port actually answers, so a silent repeat is not possible.

The RAG recipe: embed chunks with bge-m3 → store in pgvector → fetch ~20 nearest → rerank down to 5 with /v1/rerank → answer with gemma4-26b-fp8. Adding the reranking step is the single cheapest quality improvement in that chain.

Fine-tuned adapters: the qwen3.8-27b lane runs with LoRA enabled, so an adapter you train can be served from this API next to the base models. Ask an operator to register it.

Reading documents (PDF, Word, PowerPoint, Excel — new)

Models cannot read a .docx. Send us the file and get back text you can put straight in a prompt — so your project starts at the interesting part instead of at a parsing problem. Same key as everything else.

# a file you have
curl https://abraham.cuwcs.com:14000/v1/documents/extract \
  -H "Authorization: Bearer YOUR_KEY" -F file=@lecture.pptx

# a file on the web
curl https://abraham.cuwcs.com:14000/v1/documents/extract_url \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://arxiv.org/pdf/1706.03762"}'

Formats: PDF, .docx, .pptx, .xlsx, .csv, HTML and plain text. The format is read from the file’s content, not its extension, so a misnamed upload still works.

What you get: PDF text is marked --- page N --- so a model can cite a page number instead of guessing. Word includes tables. PowerPoint includes speaker notes — often where the real argument lives. Excel comes sheet by sheet, formulas resolved to values.

Limits: 30 MB per file, 200,000 characters returned. A scanned PDF has no text to extract and returns a 415 that says so — send the page images to qwen3.8-27b (vision) instead.

Writing documents (Word, PowerPoint, Excel, PDF — new)

You write Markdown; the server writes the file. A model already produces good Markdown without being asked, so that is the input — headings, bold, italic, nested lists, tables, code blocks and quotes all convert.

curl https://abraham.cuwcs.com:14000/v1/documents/create \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" -o report.docx \
  -d '{"format":"docx","title":"Q3 Report",
       "markdown":"# Findings\n\nThroughput was **higher** than forecast."}'

Need exactness instead of prose? Send a JSON spec in place of markdown — blocks for a document, slides for a deck, sheets for a workbook. It compiles to the same Markdown internally, so the two inputs can never drift apart.

Known limit: right-to-left scripts (Arabic, Hebrew, Persian, Urdu) are not shaped correctly in PDF output. Use docx or html for those — the reader does the shaping there.

File storage (OpenAI Files-compatible — new)

Add "delivery":"file" to a document request and it is stored instead of returned, and you get a link. You can also upload your own files.

from openai import OpenAI
# the OpenAI SDK works unmodified — just point base_url one level up
c = OpenAI(api_key=KEY, base_url="https://abraham.cuwcs.com:14000/v1/documents")

c.files.create(file=open("data.csv","rb"), purpose="user_data")
c.files.list(); c.files.retrieve(fid); c.files.content(fid); c.files.delete(fid)

Your files are yours. Ownership comes from your key and is re-checked on every read and delete. Another key asking for your file id gets a plain 404 — not a 403, because the store must not become an oracle for which ids exist. Verified with two separate keys.

Two kinds of link. /files/{id}/content needs your API key. /documents/download?id=…&token=… does not — the signature is the authorisation, because a browser following a link cannot send an Authorization header. That signed link is what a model hands you after writing a document: it is good for 24 hours, covers exactly one file, and cannot be edited to reach another or to live longer.

Limits: 100 MB per file, 500 files and 2 GB per key, 7-day retention. Downloads are always attachment + nosniff, and anything HTML-like is served as opaque bytes so a stored file can never run in this site’s origin.

Why not plain /v1/files? The gateway ships its own implementation of that route which would proxy using the gateway’s credential rather than yours — putting every student’s files under one shared owner. Ours sits one level down, where your key reaches it.

Moderation (POST /v1/moderations — new)

The standard OpenAI path, answered locally by llama-guard3-8b. Your text never leaves the cluster. Returns OpenAI’s exact response shape, so existing safety code works unchanged.

curl https://abraham.cuwcs.com:14000/v1/moderations \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"model":"omni-moderation-latest","input":"..."}'

The scores are real. The classifier emits one safe/unsafe token; we read that token’s logprobs and normalise, so you get a genuine confidence instead of 1.0/0.0. Measured: 0.9969 for a weapons-construction prompt, 0.9718 for a suicidal statement, 0.0001 for a bread recipe.

What it does not measure. Llama Guard’s hazard taxonomy only partly overlaps OpenAI’s categories. These six have no signal behind them and are always false — meaning not measured, not clean: harassment, harassment/threatening, hate/threatening, self-harm/intent, self-harm/instructions, violence/graphic. Every response repeats that list in backend.unmapped_categories so your code can check it rather than your memory. Images are not classified — the model is text-only.

Cost to the cluster: the classifier runs with an 8K context rather than the fleet default 32K — 5.9 GB resident instead of 9.2 GB. It loads on demand, so the first call after a quiet period can take a minute; retry once.

Grounding tools (web_search & web_fetch)

Server-side retrieval on the same gateway, authenticated with the same key as /v1/chat/completions — your app needs no search-provider key of its own. The domain allowlist is enforced on the server: results whose host is not the allowed domain or a subdomain of it are dropped before you see them.

General web search cascades across Serper and Brave (both configured as of 2026-08-20). Serper is tried first; Brave is queried only when Serper returns nothing that survives your allowed_domains filter — which happens for real: an exact trial lookup like NCT06613100 returns ten confident Serper results and none on clinicaltrials.gov, while Brave finds them. The notes field tells you when an escalation happened, and each result’s source names the engine that found it. Use "provider":"all" to query every backend at once for maximum recall. Authoritative medical connectors (ClinicalTrials.gov, PubMed, openFDA, MedlinePlus) answer alongside it and rank first. A self-hosted SearXNG on the DGX remains configured as the automatic fallback if the Serper quota is exhausted, and as the privacy option: pin "provider":"searxng" and the query never leaves the box. No client change is needed when the backend changes — GET /v1/web_search/health reports the active one as web_search_backend.

Status codes. 200 = backends ran (an empty results really does mean nothing found) · 422 = no backend covers this request · 403 = a route ceiling excluded every requested domain · 429 = rate limited · 502/504 = an upstream search backend failed. degraded:true means general web search was unavailable for this request, not that the results are unreliable.

When does search fire? Never automatically. On :14000 your app calls /v1/web_search and puts the snippets in the prompt — and if you define your own web_search function tool, the call comes back to you to execute, the gateway does not run it. On the Anthropic router :14100 it is genuinely “as needed”: include the web_search_20250305 tool definition once and the model decides each turn, with the gateway executing. It is selective — measured, it searched for a trial’s recruiting status and did not search for “what is 17 + 26”, so leaving the tool on costs nothing when it is not needed.

Keyless connectors (authoritative, no vendor key, work today): ClinicalTrials.gov API v2 — an NCT######## anywhere in the query does an exact study lookup and returns the real eligibility criteria; PubMed (E-utilities); openFDA drug labels, cited as DailyMed pages; MedlinePlus patient-facing health topics. Results from several backends are round-robined so one source can’t consume your whole count.

web_fetch returns extracted text, never raw HTML, and refuses any URL that resolves to a private, loopback or link-local address (SSRF guard) — re-checked on every redirect hop. A clinicaltrials.gov/study/<NCT> URL is served from the API rather than scraped, because that page is a JavaScript app that would otherwise yield only navigation text. Both tools cache for 15 minutes.

Domain ceiling (operators). allowed_domains is yours to set — the server enforces what you send. An operator can additionally pin a ceiling your key cannot widen, either fleet-wide or on a dedicated route; requested domains outside it are dropped, and if that leaves none the call is refused with 403 rather than silently becoming unrestricted. Route-level settings are authenticated with a server-side secret, so they cannot be set or altered by a client.

Rate limits. Your key’s rpm_limit does apply to these routes, but it is key-wide — spending it on search also throttles that key’s /v1/chat/completions. The tool service adds a coarse global backstop to protect the free upstream APIs. A search-specific per-tenant limit needs a dedicated route; ask an operator.

curl https://abraham.cuwcs.com:14000/v1/web_search \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"query":"NCT06613100 eligibility",
       "allowed_domains":["clinicaltrials.gov"],
       "count":3}'

Also on the Anthropic router (:14100) — changed 2026-08-20. Send Anthropic’s own web_search_20250305 / web_fetch tool definitions and the gateway executes them for you, returning a grounded answer (it used to drop them and answer ungrounded with no error). Works on all three always-on lanes, streaming or not; max_uses and allowed_domains on the tool definition are honoured, and the definition’s allowlist beats whatever the model asks for. A web_search tool you define (one with an input_schema) is never shadowed. x-abraham-tool-calls reports how many calls ran; tools the fleet still cannot run (code_execution, bash) are reported in x-abraham-dropped-tools rather than vanishing silently.

curl https://abraham.cuwcs.com:14100/v1/messages \
  -H "x-api-key: YOUR_KEY" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-26b-fp8","max_tokens":600,
       "tools":[{"type":"web_search_20250305","name":"web_search",
                 "max_uses":3,"allowed_domains":["clinicaltrials.gov"]}],
       "messages":[{"role":"user","content":"Find trial NCT06613100 - phase and sponsor. Cite the URL."}]}'

Models calling models (agentic delegation — new)

On the Anthropic router (:14100), a model can hand a self-contained sub-task to a different model on the fleet and use the answer — a fast model can call the coder for code, or the 27B for a hard reasoning step, without you orchestrating it. Add the tool and it decides when to use it.

curl https://abraham.cuwcs.com:14100/v1/messages \
  -H "x-api-key: YOUR_KEY" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-26b-fp8","max_tokens":900,
       "tools":[{"type":"ask_model","name":"ask_model"},
                {"type":"doc_extract","name":"doc_extract"},
                {"type":"web_search_20250305","name":"web_search"}],
       "messages":[{"role":"user","content":"Write me a Python quicksort and explain the pivot choice."}]}'

ask_model takes model, prompt, and optionally system / max_tokens.

The sub-call runs under your key. The router forwards your credential, so a delegated call has exactly the model permissions your key already has and the tokens are billed to you. A key that cannot use a restricted model cannot reach it by asking another model to. The delegated model gets no tools and cannot see your conversation — put everything it needs in prompt. One level deep, capped at 4,096 tokens and 180 seconds.

Using this fleet with Claude Code (whose tools run where)

Claude Code runs its own tools on your machine. Read, Write, Edit, Bash, Grep and the rest are implemented by the CLI locally: it sends the model a list of tool definitions, the model replies “call Read on this path”, and the CLI runs it against your filesystem. None of that touches this server. Pointing Claude Code at :14100 needs nothing from us but a model that emits well-formed tool calls — which all three always-on lanes do.

Use qwen3-coder-next-80b-a3b for it: that lane is trained for agentic tool use (verified — 77 turns, zero malformed calls).

So what are the server-side tools above for? Everything that is not Claude Code: your own agent, any Anthropic-format client that asks for web_search (without server execution it would get a confident ungrounded answer and no error), and capabilities the client does not have — Claude Code cannot read a PDF off the web, but doc_extract here can. If your tools[] already defines a tool by one of our names, we do not touch it: your implementation wins.

How close is this to OpenAI?

Measured against OpenAI’s published OpenAPI spec (info.version 2.3.0, commit 010421dcbd04, 2026-08-20): 182 paths, ~129 of them non-deprecated and developer-facing once org/admin, Assistants and the deprecated Videos family are set aside. Here is the honest tally.

Things OpenAI does not have: /v1/rerank, /v1/web_search, /v1/web_fetch, /v1/documents/extract, /v1/documents/create, and an Anthropic Messages API on :14100 with server-executed tools and model-to-model delegation.

Examples

Python (OpenAI SDK)

from openai import OpenAI
client = OpenAI(base_url="https://abraham.cuwcs.com:14000/v1", api_key="YOUR_KEY")
r = client.chat.completions.create(
    model="qwen3-coder-next-80b-a3b",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
)
print(r.choices[0].message.content)

curl

curl https://abraham.cuwcs.com:14000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-26b-fp8","messages":[{"role":"user","content":"hello"}]}'

Vision — qwen3.8-27b (always on)

client.chat.completions.create(model="qwen3.8-27b", max_tokens=600,
  messages=[{"role":"user","content":[
    {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}},
    {"type":"text","text":"What is in this image?"}]}],
  extra_body={"chat_template_kwargs": {"enable_thinking": False}})
# tip: send images as an image_url content block with a data: URL

Model settings (sampling & thinking)

Put these at the top level of the request JSON (or the OpenAI SDK's extra_body for the non-standard ones). All verified live against the fleet.

Sampling — OpenAI gateway (:14000)

The primary FP8 lanes are vLLM-backed and validate these (an out-of-range value 400s, so acceptance = honored).

Ollama-backed models accept temperature, top_p, top_k, seed, stop, repetition_penalty but clamp out-of-range values instead of erroring; vLLM-only extras (min_p, JSON-schema) may be ignored.

Thinking / “no-think” — reasoning models (qwen3.8-27b)

• Default: thinking ON. Chain-of-thought is in message.reasoning_content, the answer in message.content. Give it generous max_tokens or the budget is spent thinking and content comes back empty.
• Disable (“no-think”): send chat_template_kwargs: {"enable_thinking": false} → direct answer, no reasoning. (The /no_think prompt suffix is not reliable on this build — use the flag.)

client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "What is 17+26?"}],
    extra_body={"top_k": 40, "min_p": 0.05,
                "chat_template_kwargs": {"enable_thinking": False}},
)

Anthropic router (:14100)

Anthropic-native params are forwarded: temperature (0 = deterministic), top_p, top_k, stop_sequences, system, tools, stream. Endpoints: /v1/messages, /count_tokens, /healthz.

Thinking is ON by default on reasoning lanes. Set effort with output_config: {"effort": "high" | "medium" | "low"} (still returns a thinking block), or disable it with chat_template_kwargs: {"enable_thinking": false} (verified: no thinking, direct answer).

curl https://abraham.cuwcs.com:14100/v1/messages \
  -H "x-api-key: YOUR_KEY" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","max_tokens":800,"temperature":0.3,
       "output_config":{"effort":"high"},
       "messages":[{"role":"user","content":"Prove sqrt(2) is irrational."}]}'

All models on the gateway (31)

Everything callable with your key. Non-primary models are Ollama-backed and load on demand: the first call reads the weights from disk (cost scales with model size), then the model stays resident for 2 minutes and is evicted — so a repeat call is fast and a cold one is slow. OLLAMA_MAX_LOADED_MODELS=4 and OLLAMA_KEEP_ALIVE=2m. Improved 2026-09-03: Ollama was upgraded 0.30.11 → 0.33.3, and the three pinned tiny models (arch-router, bge-m3, embeddinggemma, ~2.5GB together) now stay resident simultaneously as intended — previously only about one fitted at a time and each load evicted the last. Measured after the upgrade: all three pinned plus one 5.9GB on-demand model co-resident, then back to 13GB free once the on-demand one expired. Embeddings are now effectively always-warm rather than on-demand: 1,040 vectors, single and batched, returned with zero failures. Cold loads of the larger on-demand models still retry. A multimodal model (the qwen3.5 family) whose vision projector will not fit logs CUDA error: out of memory and automatically retries with the projector on CPU, or evicts another model and retries once; the call succeeds, it is just slower. That is expected on a box with ~13GB free — treat a first call to a big Ollama-backed model as slow, and keep max_tokens generous. Embeddings: embeddinggemma, bge-m3. Note: 13 large Ollama models (16.8–68GB) were de-registered on 2026-08-18 — the always-on lanes reserve ~89% of Abraham's memory, so those could not load and attempting it destabilised the host. For a big model use a primary lane instead (qwen3-coder-next-80b-a3b, gemma4-26b-fp8, qwen3.8-27b).

Added 2026-09-03 — prefer these: granite4.2-8b (IBM Granite 4.2, 128K context, tools + thinking), qwen3.5-4b and qwen3.5-0.8b (Qwen3.5, text+image). They join the already-current qwen3.5-9b, gemma4-12b and gpt-oss-20b.

Deprecated 2026-09-03 — still working, removed at the next audit. granite3.3-8b → granite4.2-8b · qwen3-4b → qwen3.5-4b · qwen3-0.6b → qwen3.5-0.8b · qwen2.5-7b / qwen2.5-14b → qwen3.5-4b / qwen3.5-9b (smaller and newer) · qwen2.5-vl-7b → vision is on qwen3.8-27b and qwen3.5-9b · deepseek-r1-8b → granite4.2-8b · aya-expanse-8b (no successor; multilingual is covered by the Qwen3.5 and Gemma-4 lines) · deepseek-coder-v2-16b → the always-on qwen3-coder-next-80b-a3b · phi4-reasoning (14B dense at only 32K context, no successor shipped). Nothing breaks today — move when convenient.

Most small models now think. They emit reasoning_content and think first, so max_tokens: 24 comes back with an empty content and finish_reason: "length". Give them ≥1000 and read content. Note that chat_template_kwargs {"enable_thinking": false} works on the three primary vLLM lanes but is ignored on these Ollama-backed ids — budget for the thinking instead of trying to turn it off.

Advanced — Anthropic-format router

For Claude Code / Anthropic-format clients, the router at https://abraham.cuwcs.com:14100 speaks the Anthropic Messages API (/v1/messages) for the primary models. Lanes (verified 2026-09-03): the three always-on models — qwen3-coder-next-80b-a3b, gemma4-26b-fp8, qwen3.8-27b — plus the deprecated alias qwen3-coder-30b-fp8, which resolves to the same coder lane. There are no on-demand VL lanes any more: vision is part of the always-on qwen3.8-27b. /healthz lists every lane, and calling a lane whose upstream is down returns HTTP 503 (not 500 — the router is fine, the backend isn't) with an Anthropic-format error body naming the lane, so clients can parse it and back off. Public calls require a portal key (Authorization: Bearer or x-api-key), validated at the gateway; keyless on the LAN.