Introducing the VLM Run Gateway: Unified API for Visual Intelligence
Since the Orion agent launch in November, our customers have run hundreds of thousands of requests spanning millions of tool calls in production every month, across healthcare claims, construction submittals, legal contracts, and clinical faxes. One question came back more than any other: which models are you running underneath, and can we call them directly?
We listened to our customers and gave them initial access to the models through our gateway. Today, we're opening access to everyone via a free public API.
The VLM Run Gateway runs open-weight VLMs, OCR VLMs, embodied VLMs, and specialized ViTs behind one OpenAI-compatible API, at a fraction of commercial API pricing. Because they all share one interface, you can compare them on your own data and pick your own point on the cost/accuracy frontier by changing a single parameter. Point base_url at https://gateway.vlm.run/v1/openai and every model takes the request shape your OpenAI SDK already uses. It's free during alpha, so try it on your own documents, images, and videos today.
You don't need a trillion parameter VLM
Parsing a document, understanding an image, summarizing a video: the best model for the job is rarely the biggest one. zai-org/glm-ocr is 0.9B parameters, rednote-hilab/dots.mocr 3B. Both are trained on page images at document-native resolutions, and it shows: over 3,000 output tokens/sec on commodity hardware, a fraction of the vision tokens a general-purpose VLM burns on the same page, and small enough to run on older GPUs, which is what makes on-prem and VPC deployment tractable.
Frontier VLMs are built for open-ended reasoning, "figure out what this means" rather than "read this," and there they remain the right call. On documents they fail repeatably: dense multi-column tables, low-DPI fax scans, overlaid stamps and handwriting, non-Latin scripts. For extraction, OCR, and layout, they're an expensive default nobody chose deliberately.
To test this, we evaluated six OCR models on the Gateway against eight commercial document parsers on ParseBench, and three open-weight VLMs on the Gateway against the frontier VLMs on MVBench. Every model is plotted at its measured cost, per thousand pages for documents and per thousand clips for video:
On documents, two models land in the optimal quadrant, score 70 or better under $3 per thousand pages, and both are open-weight models on the Gateway. infly/infinity-parser2-pro scores about 79 at roughly $2.10, and dots.mocr about 74 at roughly $0.78. No commercial parser makes it into that box.
infinity-parser2-pro also matches or beats 7 of the 8 commercial parsers on the chart, including Gemini 3 Flash, Mistral OCR 4, Azure DI and Extend 2.0, at 2-12x less. AWS Textract sits about 24 points lower and costs 7x more. The one parser above it is LlamaParse in its cost-effective tier at roughly 80.5, a point and a half of accuracy for about twice the price.
Switch to the video tab and the picture is similar. Three models clear 70 under $10 per thousand clips, and two are open-weight models on the Gateway: qwen3.8-27b at about 75.5 for roughly $5.50, and muse-glimmer-30b at about 74 for roughly $1.50. qwen3.8-27b lands above GPT-5.6 Terra, while muse-glimmer-30b matches GPT-5, at a fraction of the $22 those cost. Gemini 3.7 Flash tops the chart at about 82, eight points up and roughly 6x the price of muse-glimmer-30b.
In short: open-weight models are competitive on accuracy while often being significantly cheaper. Which point on that frontier is yours depends on your use case, and the Gateway is what makes trying all of them easy.
Choosing the model is the hard part
Why does anyone send documents to a frontier API? Because no single small model wins everywhere. Healthcare forms, construction drawings and legal contracts have different winners, and the winner changes with scan quality, language and layout density.
Evaluating candidates is expensive for reasons unrelated to the models. ONNX, GGUF, and PyTorch checkpoints need different runtimes, and some can't share a process at all: PaddleOCR ships a private CUDA runtime that conflicts with PyTorch. Every model returns a different output shape, so each candidate needs its own parser. And you run out of VRAM long before you run out of candidates. So teams do the rational thing: pick the one API that handles everything and absorb the markup.
We know because we hit it first. Orion tool calls each request to multiple underlying models because no single model leads across localization, segmentation, and video.
The Gateway's product isn't any single model. It's making the choice reversible. When every model runs on one API, changing models is as simple as changing a parameter: same endpoint, same request shape, same output contract.
- model="rednote-hilab/dots.mocr",
+ model="infly/infinity-parser2-pro",Everything else in your code stays identical.
What the Gateway is
Try the models in the Gateway model catalog today. Alongside the OCR models above, it covers layout detection with reading order, captioning and detection, visual chat over images, video and PDFs, embeddings, and audio transcription. Each model is served behind the standard chat.completions, embeddings, or transcriptions interface.
Five runtimes sit behind one router, each model an isolated plugin, so conflicting dependency stacks coexist. Models stay resident rather than cold-starting per request, which is what keeps latency predictable and prices low.
The Gateway also handles the work around the model. PDFs are rasterized page by page across parallel workers, with document_dpi, document_pages, and stream: true controlling raster quality, page range, and incremental delivery. A page that fails is retried or marked rather than killing the document. Output comes back normalized: markdown with page markers, a unified bounding-box schema, self-describing JSON. Every response carries metered usage.cost and an x-request-id. The output contract belongs to the Gateway, not the model. That's what makes the one-line swap real.
Build your own agents
An MCP server at https://gateway.vlm.run/mcp exposes read_document, read_video, and read_audio, so any MCP-compatible framework (Pydantic AI, LangChain, Mastra, OpenAI Agents SDK, Claude Code) consumes visual content through the same pipeline, auth, and billing. Everything in the catalog is open-weight: run it in your VPC, fine-tune it on your data, own the weights. Enterprise deployments support BYOK, private VPC, and BAAs for HIPAA.
Get started
The visual frontier just moved. Your AI bill should too.
All endpoints support anonymous access with IP-based rate limits, the Gateway is free during alpha, and the published prices are already live, so the cost signals you build against today are the ones you'll use in production. Point your OpenAI SDK at https://gateway.vlm.run/v1/openai, or list the catalog from your terminal:
# List the models on the gateway
uvx vlmrun gw models
# Run Paddle OCR VL, dots.mocr, Deepseek OCR 2 on a document
uvx vlmrun gw chat -m paddleocr-vl-1.6 <doc>.pdf
uvx vlmrun gw chat -m dots.mocr <doc>.pdf
uvx vlmrun gw chat -m deepseek-ocr-2 <doc>.pdf
# Run qwen3.5-0.8b on image / video
uvx vlmrun gw chat -m qwen3.5-0.8b -p "describe the image" <image>.png
uvx vlmrun gw chat -m qwen3.5-0.8b -p "describe the video" <video>.mp4For more detail, see the docs. Join the discussion on Discord.
Quickstart · Model catalog · MCP server
