OpenAI built a chip, and it actually works. That’s not a given in this industry. Custom silicon projects from big tech companies have a long history of ambitious announcements followed by quiet shelving. So when OpenAI published the first real benchmark results for Jalapeño, its custom inference accelerator, the numbers deserved a hard look rather than a press release skim.
The headline figures are striking. Tested against GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T on the InferenceX public benchmark from SemiAnalysis, Jalapeño delivered 1.5 to 1.9 times more AI work per watt at peak throughput. End-to-end latency came in 1.7 to 3.6 times lower than the comparison systems. For highly interactive workloads, the performance gap widened to 2.1 to 4.1 times. These aren’t cherry-picked internal metrics. The team used a public benchmark across three models, two of which were built outside OpenAI entirely.
What the architecture actually does differently
The core insight behind Jalapeño’s design is that inference has two very different phases. Prefill, where the system processes an input prompt, is compute-heavy. Decode, where it generates tokens one by one, is bottlenecked by memory bandwidth. Most existing hardware is optimized for one or the other, or forces engineers to make tradeoffs between throughput and latency. Jalapeño was designed to handle both without that compromise.
The key is minimizing data movement. Model state, including the KV cache generated during decode, stays local rather than bouncing between chips or memory pools. The network is built into the architecture at the rack level, so the entire workload stays within a single connected system. The result is what OpenAI describes as a fungible accelerator that can shift resources between prefill and decode as workloads demand, which matters a lot for agentic tasks where those demands shift constantly mid-run.
The chip is rated at 700 watts but ran at or below 550 watts sustained on the tested workloads. That gap between rated and actual power draw directly improves the efficiency numbers being reported, and it’s a meaningful real-world advantage for anyone paying data center power costs.
How AI helped build the chip that runs AI
One of the more interesting details here is the development process itself. OpenAI used earlier model generations to help design and bring up Jalapeño, and is now using its latest models to optimize and program it. The team went from initial design to tapeout in nine months, a fast timeline for custom silicon. AI helped explore chip implementations, shorten verification loops, and optimize arithmetic circuits to fit more compute into the design on schedule.
The programming model was built with AI in mind from the start. Engineers and AI systems alike can describe work through local tensors, explicit communication, and predictable synchronization. That structure gives AI a tractable surface for parallel programming optimization, which has historically been one of the hardest problems in chip software.
Why this matters for the market
Jalapeño is aimed squarely at the economics of inference at scale. The relevant comparison isn’t just Nvidia H100s or B200s. It’s also Google’s TPUs, which have been running internal inference workloads for years, and AWS Trainium, which is still finding its footing in serving workloads. OpenAI’s angle is vertical integration: designing models, serving software, chips, memory, networking, and systems together in a single stack. That’s the same playbook Google used to make TPUs formidable, and it works when you have enough of your own workload to tune against.
- 1.5 to 1.9x higher throughput per watt at peak load
- 1.7 to 3.6x lower end-to-end latency than existing accelerators
- 2.1 to 4.1x better performance on interactive, low-latency workloads
- Sustained power at or below 550W against a 700W rating
- Nine-month design-to-tapeout timeline with AI-assisted development
OpenAI says Jalapeño is the beginning of a multigenerational platform, not a one-off project. The internal results on frontier OpenAI models apparently show an even wider advantage than the public benchmark numbers, which suggests the chip is tuned for the workloads OpenAI actually runs. For developers and enterprises buying inference capacity through OpenAI’s API, this could mean lower costs and faster response times as Jalapeño scales. For Nvidia, it means its biggest customer is now also a serious silicon competitor.
