Small Models, Big Impact: The New AI Reality

Original Insights
Written By Shakie Liu

While everyone’s been obsessed with the race to build ever-larger AI models, something interesting has been happening behind the scenes. The smart money is quietly moving in the opposite direction.

I recently came across three pieces of insight that, when you connect them, paint a compelling picture of where AI is actually headed. The first came from Asha Sharma, VP of Product for Microsoft’s AI platform, in an interview on Lenny’s Podcast where she discussed how thousands of companies deploy AI at scale. The second from NVIDIA researchers who’ve been analyzing what makes economic sense. The third from an academic perspective published in ACM Communications, looking at what actually works in practice.

Their conclusion? The future belongs to small, specialized AI models, not massive generalist ones. And this shift is already happening faster than most people realize.

 

The Economics Finally Make Sense

Here’s the reality check that’s driving this change: most AI tasks don’t actually need the full power of GPT-4 or Claude.

The NVIDIA research team analyzed what AI agents do in practice. Parsing emails. Generating reports. Writing code snippets. Answering customer support questions. Processing invoices. These are specialized, repetitive tasks that a much smaller, focused model can handle just as well.

The difference? Cost. Running a 7 billion parameter model costs 10–30x less than running a 175 billion parameter model. When you’re doing thousands of operations daily, that difference compounds fast.

This isn’t just theoretical. A recent ACM Communications article makes the same point from a different angle: “Obtaining state-of-the-art results can be achieved with smarter questioning, not planetary-scale computation.” The author demonstrated building effective models with just 63 examples using active learning — a stark contrast to the billions-of-parameters approach.

Sharma sees this playing out in real time at Microsoft. “Post-training is the new pre-training,” she notes. Instead of spending billions training the next giant foundation model, smart companies are taking existing models and fine-tuning them for specific use cases. It’s more efficient, cheaper, and often works better for narrow tasks.

 

The Real World Doesn’t Need Perfect AI

The NVIDIA researchers made a crucial discovery when they analyzed popular AI agent systems: 40–70% of current large model queries could be handled by specialized small models. That’s not theoretical — that’s looking at real systems doing real work today.

Why does this make sense? A small model trained specifically for parsing medical documents will often outperform a giant general-purpose model at that exact task. It’s focused, it’s fast, and it doesn’t get distracted trying to be good at everything.

Microsoft is already seeing this pattern play out. They have over 15,000 customers running millions of specialized agents on their platform. These aren’t chatbots trying to have philosophical conversations — they’re focused systems handling specific workflows.

Take Sharma’s example of their Dragon system for physicians. When they fine-tuned a model on 600,000 real physician-patient interactions, acceptance rates jumped from 30–60% to 83%. The specialized model understood the specific context and language patterns that mattered for that exact use case — something a general model would struggle with despite being much larger.

 

You Don’t Need Billions of Examples

Here’s where things get really interesting. Research in active learning has demonstrated something that challenges everything we’ve been told about AI: you can build effective models with tiny amounts of carefully selected data.

The ACM Communications article describes building models for 63 different software engineering tasks using active learning techniques. Instead of throwing millions of examples at the problem, the researcher used smart questioning to identify which examples would be most informative.

The results? Eight carefully chosen labels achieved 62% optimal performance. Sixteen labels reached nearly 80%. Thirty-two labels approached 90%.

Read that again: you can get near-optimal results with dozens of well-chosen examples, not billions of random ones. The article makes it explicit: “Active learning provides a compelling alternative to sheer scale in AI. Its ability to deliver rapid, efficient, and transparent results fundamentally questions the ‘bigger is better’ assumption dominating current thinking about AI.”

If this is true — and the evidence suggests it is — then the entire economics of AI deployment shifts. You don’t need massive datasets. You don’t need enormous compute budgets. You need smart data selection and focused models.

This aligns perfectly with what Sharma observes at Microsoft and what the NVIDIA researchers found in their analysis. The path forward isn’t necessarily bigger models — it’s smarter approaches to building smaller, specialized ones.

 

Small Models You Can Actually Own

Here’s a practical advantage that often gets overlooked: small models can run on normal hardware. That means better privacy, lower latency, and you’re not at the mercy of API rate limits or sudden pricing changes.

The NVIDIA researchers highlight this as a key democratizing factor. When models can run locally on consumer-grade GPUs, more organizations can experiment with AI without massive infrastructure investments. You can fine-tune a small model overnight on a single machine, not over weeks in a datacenter.

The ACM article reinforces this point dramatically. The author notes that their compact models were “created without billions of parameters” and that “active learners need no vast pre-existing knowledge or massive datasets, avoiding the colossal energy demands of specialized hardware demands of large-scale AI.” For practical tasks, they generated results in three minutes on a standard laptop.

This local deployment capability is already changing how companies think about AI. Instead of sending sensitive data to external APIs, they can keep everything in-house while still getting the AI capabilities they need. And the results are often better — as the article notes, unlike opaque LLMs, smaller models trained with active learning produce “explainable results via small labeled sets” that humans can understand and verify.

 

The Data Advantage Gets Real

Both Sharma and the NVIDIA researchers emphasize something that’s becoming clear: the companies winning with AI aren’t necessarily those with the biggest models — they’re the ones with the best data loops.

Sharma describes products as “living organisms that just get better with the more interactions that happen.” The key insight is that specialized models can be retrained quickly as you gather more domain-specific data.

With a giant foundation model, incorporating new learnings is expensive and slow. With a small specialized model, you can retrain it with fresh examples in hours, not weeks. This creates a compounding advantage for companies that are good at capturing and curating the right data.

The NVIDIA researchers outline a practical playbook for this: collect usage data from your systems, identify recurring patterns, fine-tune small models for those specific tasks, then iterate based on results. It’s systematic and achievable, not some moonshot requiring hundreds of millions in funding.

 

The Platform Play

Interestingly, all three sources converge on the same prediction about market structure. Instead of one or two massive models dominating everything, we’ll see ecosystems of specialized models working together.

Microsoft’s platform already hosts millions of these specialized agents. Each handles specific tasks, but they can work together when needed. The magic happens in the orchestration — knowing when to use which model for which task.

This creates opportunities for companies to build moats around specialized AI capabilities rather than trying to compete with OpenAI or Anthropic on general intelligence. You don’t need to build GPT-5. You need to build the best AI for your specific domain, trained on your specific data.

 

The Boring Infrastructure Still Matters Most

Here’s something all three sources emphasize: the successful deployment of small models isn’t about flashy demos — it’s about solid infrastructure.

Data pipelines for continuous training. Evaluation systems to ensure quality. Deployment infrastructure that can handle multiple specialized models. Observability to understand how well each model performs. This unglamorous work determines who actually succeeds at scale.

Sharma emphasizes this repeatedly when talking about Microsoft’s platform: reliability, data residency, privacy, observability. The NVIDIA researchers stress the importance of proper fine-tuning pipelines and evaluation frameworks. The ACM article highlights explainability and transparency.

This mirrors every major technology shift. WhatsApp won on reliability — messages actually got delivered. AWS won on uptime — servers stayed running. The flashy features grab attention, but the boring infrastructure determines who wins long-term.

 

Where This Goes Next

The evidence points to a major shift already underway. The economics favor smaller, specialized models. The technology enables rapid fine-tuning and local deployment. The research shows you can achieve strong results with focused approaches rather than massive scale.

For companies, this means rethinking AI strategy. Instead of asking “How do we use GPT-4?”, the better question is “What specific tasks could we handle with our own fine-tuned models?” Instead of paying per API call indefinitely, you invest in building specialized capabilities you own and control.

For builders, this opens up opportunities that didn’t exist when AI meant paying premium prices for every query. You can create focused solutions that work well for specific problems, iterate quickly based on real usage data, and build sustainable businesses around specialized AI capabilities.

The future of AI looks less like a few giant models doing everything and more like ecosystems of focused, efficient, specialized systems working together. That future isn’t coming — it’s already here. The question is whether you’ll build with it or keep waiting for the next big foundation model.

 

The shift toward small, specialized AI models isn’t just about cost savings — it’s about building AI systems that actually work reliably for real problems. The companies that recognize this early will have a significant advantage in the years ahead.

 
Shakie Liu
Next

The Future of Work: From Role Boundaries to AI-Enhanced Collaboration