Top Local LLMs vs ChatGPT and Claude: The Pelican Bicycle Test
There are too many large language models out there to test them all properly. Sure, you can chat with a few, pick your favorite, and call it done. But here's the problem: a model that excels at conversation might be terrible at coding, generating structured output, or following specific instructions. What you really need is a quick way to spot the differences.
When you need an LLM for a specific task—say, ranking a list and returning clean JSON—running the same prompt through dozens of models just to find the right fit sounds like torture. That's where performance benchmarks come in. Unlike PC benchmarks you can run with a few clicks, though, most LLM benchmarks require either trusting someone else's results or building your own complex test suite.
That's why a good smoke test matters. It needs to be fast, repeatable, and revealing enough to show how a model really performs. Enter: "A pelican on a bicycle."
The "Pelican on a Bicycle" Test
AI developer and researcher Simon Willison introduced this test on October 25, 2024. The prompt is simple:
Generate an SVG of a pelican riding a bicycle.Willison initially tested it against 16 models from OpenAI, Anthropic, Google, and Meta. He chose pelicans partly because he likes them, and partly because he suspected there wouldn't be many SVG illustrations of bicycling pelicans in the training data of these models.
Here's what makes this test clever: SVG is technically a text format—it's XML markup that renders as graphics. Theoretically, any language model capable of handling text and basic programming should generate valid SVG code. You don't need a special benchmarking toolkit, code interpreter, API scripts, IDE, or complex constraints. You don't even need to install anything. Just feed the model one line of text, copy the result into an SVG file if the interface doesn't auto-render, and open it in your browser.
Evaluation is equally straightforward. There's no answer key or abstract score to interpret. You just look at the image. Is the bird clearly a pelican? Does the bicycle have two wheels? Are those wheels connected to the frame? Is the pelican actually riding, or floating somewhere in the background? Did the model follow instructions and return SVG code, or did it ignore you and invoke an image generation tool instead?
That single prompt ends up testing instruction-following, programming ability, valid syntax, object recognition, spatial reasoning, layout, and even restraint. Some models produced a functional bicycle with the pelican in the right place. Others added clouds, gradients, grass, animations, and sunny backgrounds while forgetting to connect the front wheel.
It also reveals something about the product wrapper around the model. ChatGPT, Claude, and Gemini don't always give you raw LLM access. They wrap the model in system prompts, tools, renderers, and routing decisions. The Pelican test exposes those decisions almost immediately.
Even the Most Advanced Models Struggle With the Pelican Test
The prompt was tested on Claude, Gemini, and ChatGPT. The tests used standard web interfaces tied to existing subscription plans, not APIs. That means we're evaluating the entire product, not isolated models. The interfaces don't expose exact token counts or tokens-per-second metrics, so those metrics show as N/A.
Gemini outperformed all competitors by a significant margin. ChatGPT's result was odd and disappointing—GPT rushed straight to its image generation tool instead of writing code. You could argue that's technically valid, but it broke the original instruction.
Claude models performed well overall. What's interesting here is how little the output varies across vastly different price tiers. Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens via API. Claude Opus 5 runs $5 and $25 respectively—five times more expensive. Fable 5 is even pricier at $10 and $50. That's a dramatic cost difference. You'd expect Opus 5 to produce a considerably better pelican than Haiku 4.5. Reality disagrees. There are differences, but nowhere near as dramatic as the pricing gap.
Local Models Hold Their Own
Gemini certainly delivered the most impressive output among the big players, but it added a lot of flourish that nobody asked for. It created a polished illustration, but that's not really what a simple, random prompt calls for.
Among open-source models, Qwen 3.6-27B (quantized Q4 version) delivered the best results. It took nearly 3 minutes to think through the problem, but the output was exactly what you'd expect: a recognizable pelican, a recognizable bicycle, and zero unnecessary decorative details. Qwen 3.6-27B is Alibaba's flagship model in the Qwen 3.6 line, so it's not small—but running a quantized version locally still keeps it within reach of high-end consumer hardware.
Ornith 35B Q4 also performed well with both the bicycle and pelican. The main issue: the pelican seemed to lag behind the bicycle rather than sit on it properly. The individual objects were fine, but their spatial relationship wasn't.
For reference, the test also ran Gemma 2 2B. It's one of the smallest models in the Gemma line, designed for mobile and edge hardware rather than workstations. The kind of model you could actually run on a Raspberry Pi. Initially, it refused the request. After being convinced it could actually generate SVG by writing XML, it produced output. That initial refusal itself was telling—the model had the technical capability but didn't understand its own potential until prompted to reconsider. Gemma 2 31B performed reasonably well, though the front wheel didn't connect properly to the frame.
GLM-4.7-Flash's output wasn't spectacular, but it was impressive considering it generated at 112.7 tokens per second. It's a mixture-of-experts model with 30B total parameters but only 3B active, which explains the speed relative to tiny models like Gemma 2 2B.
The most disappointing local result came from North Mini Code 1.0. Its SVG was worse than what Gemma 2 2B produced. To be fair, North Mini Code is a mixture-of-experts model with 30 billion total parameters but only 3 billion active during inference. It was specifically optimized for agentic programming and terminal work, not drawing pelicans on bicycles. Still, a model built around code generation should do better.
Local models don't charge per token when you run them—though hardware and electricity are definitely not free. What matters more: their results don't lag several generations behind expensive, closed models. In some cases, they're actually preferable.
This Benchmark Hasn't Been Gamed (Yet)
Goodhart's Law states that when a measure becomes a target, it ceases to be a good measure. That's increasingly becoming a problem for LLM benchmarks.
Once model providers know which benchmarks matter, they can optimize for them. That doesn't necessarily mean an engineer manually hands the model benchmark answer keys. Benchmark questions, solutions, code repositories, discussions, and explanations simply end up in the model's massive training dataset. The model then performs well because it's seen the same problem before—not because it developed genuine problem-solving ability.
We've already watched major benchmarks lose their teeth this way. In February 2026, OpenAI stopped reporting SWE-bench Verified scores—one of the most widely-used programming benchmarks. Investigation found both flawed test cases and evidence that advanced models had encountered some benchmark problems and human-written patches during training. OpenAI concluded that higher scores increasingly reflected benchmark exposure rather than real improvements in actual software development.
Benchmarks are also commercially important. OpenAI and Anthropic are preparing for public listings. Benchmark charts aren't just research anymore. They're marketing assets that shape investor narratives, expected valuations, and the perception that one company is developing faster than another.
When a company launches a new model and fills its presentation with charts showing it beats every competitor, the valuation story becomes far easier to sell. There's enormous incentive to make those charts look impressive.
The pelican test remains useful because, to our knowledge, model providers haven't specifically trained their systems to draw pelicans riding bicycles.
But here's the real concern: the moment the pelican test becomes important enough, it will backfire. Once providers start proudly including pelican-drawing results in product launch presentations, someone will ensure the next model draws a perfect pelican. Then we'll need to change the prompt. Maybe a flamingo on a unicycle. Or a panda driving a forklift.
Description: We compare leading open-source language models with ChatGPT and Claude using a clever real-world benchmark test.
Related Articles
- Building Android Apps Directly in Google AI Studio: No Coding Required
- Building Letter Unscramble Games on Educaplay: A Quick Tutorial
- How to Disable and Control AI Features on Your Samsung Galaxy Phone
- Expand Your Images with Generative Fill in Adobe Photoshop Online
- How to Access BIOS on Different Computer Models: A Complete Guide
No Comment to " Top Local LLMs vs ChatGPT and Claude: The Pelican Bicycle Test "