There's a new site called canirun.ai that lets you check if your hardware can run specific AI models locally. It detects your hardware via browser APIs and WebGPU, then grades each model from S to F based on whether your machine can run it — showing VRAM requirements across different quantisation levels from Q2_K through F16.
It's a useful tool, but it also signals something we've been watching with growing concern: local AI deployment is becoming an enterprise-grade complexity problem that most teams aren't prepared for.
The Hardware Reality Check
The site makes the hardware requirements brutally clear. Want to run Llama 2 70B with decent performance? You need at least 48GB of VRAM. That's multiple high-end GPUs or a single H100 that costs more than most cars. Stable Diffusion XL needs 12GB minimum for reasonable generation times.
These aren't edge cases. These are the models teams actually want to deploy.
We've seen this pattern before. In our COVID-19 research dashboard work, the difference between a proof-of-concept NLP system and one that could handle production load wasn't just scale — it was architectural complexity. The hardware requirements forced us into distributed processing, which meant solving orchestration, fault tolerance, and resource scheduling problems that had nothing to do with the core AI task.
Beyond Hardware: The Integration Nightmare
Hardware compatibility is just the surface layer. The real complexity lies underneath:
Model versioning and updates: Unlike traditional software, AI models don't have clean upgrade paths. A new version of Llama might require different quantization strategies, different inference engines, or completely different hardware profiles. Teams end up maintaining multiple model versions simultaneously.
Quantization trade-offs: The site shows quantisation options from Q2_K through F16 for each model, but a grading system doesn't capture the engineering overhead. Each quantization level requires testing, validation, and often different code paths. You're not just deploying a model — you're managing a matrix of model versions × quantization strategies × hardware configurations.
Inference optimization: Getting a model to run locally is different from getting it to run well. Batch sizes, sequence lengths, memory pooling, and GPU scheduling become critical performance variables. These optimizations are model-specific and hardware-specific.
The Enterprise Trap
Here's what we think is happening: organizations are looking at local AI deployment as a way to control costs and data privacy, but they're underestimating the operational complexity.
Cloud APIs abstract away all of this complexity. Call OpenAI's API, and you get consistent performance, automatic updates, and predictable scaling. Deploy locally, and you inherit the entire stack — from CUDA drivers to model serving infrastructure.
We're seeing teams that can build sophisticated applications on top of cloud APIs struggle with basic local deployment scenarios. The skillset is different. It's closer to DevOps than data science, but it requires deep understanding of both.
What Actually Works
Successful local AI deployments share a few characteristics:
Narrow scope: Teams that succeed pick one model for one specific task and optimize the hell out of that single use case. The COVID-19 dashboard worked because we could focus on a specific NLP task with known input patterns.
Infrastructure-first thinking: The teams that struggle treat local deployment as a model problem. The teams that succeed treat it as an infrastructure problem. They invest in containerization, monitoring, and automated deployment before they worry about model selection.
Hardware standardization: Instead of trying to support every possible hardware configuration, successful deployments standardize on specific GPU families and memory configurations. This reduces the combinatorial complexity.
The Coming Shakeout
We expect to see a bifurcation in the local AI market:
Turnkey solutions will emerge for common use cases. Think Ollama or LocalAI, but with enterprise-grade deployment and management tools. These will work well for standard applications but offer limited customization.
Infrastructure platforms will develop for teams that need full control. These will look more like Kubernetes than model serving platforms — focused on orchestrating complex, multi-model workloads across heterogeneous hardware.
Most teams will end up in the middle, trying to build custom solutions that are too complex for their infrastructure capabilities but too specific for turnkey platforms.
The Skills Gap
This is ultimately a talent problem. The engineers who understand distributed systems and hardware optimization aren't necessarily the same people building AI applications. The data scientists who understand model behavior aren't necessarily comfortable with CUDA programming and memory management.
Tools like canirun.ai are helpful for understanding hardware requirements, but they don't bridge the skills gap. They might even make it worse by making local deployment look simpler than it actually is.
What We're Watching
The next 12 months will show whether the local AI tooling ecosystem can catch up to the complexity problem. We're specifically watching:
- Model compilation tools that can optimize models for specific hardware configurations automatically
- Deployment platforms that can abstract hardware heterogeneity without sacrificing performance
- Monitoring and observability tools built specifically for local AI workloads
The organizations that figure out local AI deployment will have significant competitive advantages — better cost control, data privacy, and customization capabilities. But the path there is much more complex than most teams realize.
Hardware compatibility checkers are a good start, but they're solving the easy part of the problem. The hard part is everything that comes after you know your GPU can theoretically run the model.