Local AI for Business: Why AI Pilots Fail, and How Local AI Pods Fix Them

If you lived through an ERP rollout, the reasons AI pilots stall will sound familiar. Same trap, new tech.

In Plugged In, a three-part conversation, Bernie Thompson (Founder and CTO, Plugable) and Carm Taglienti (CTO Public Sector, Insight) work through the problem, the fix, and where local AI is headed. Carm has been in the industry for about 30 years and teaches data warehousing and organizational change at several universities. Bernie founded the company building the hardware. Here's the whole conversation, start to finish, with the videos if you'd rather watch.

Part 1: Why AI pilots fail

Bernie opened with the numbers. A study updated in June found about 35% of AI pilots were resulting in a clear failure, and about 43% were stuck in pilot mode: achieving some level of success, but unable to get to full deployment across the company.

Two blockers, and neither is the model

Both men see the same two causes: data and data preparation, and organizational adoption. "Both of those things, we probably don't talk about it enough, are very similar to the issues that organizations faced in doing ERP rollouts," Bernie said.

Carm has watched this cycle before. Twenty-five years ago, business intelligence had organizations promising to get their data in order. "Now we're saying, 'Oh, AI needs good data too,' and we never really did anything about it." Finance was the exception, he noted, because the books had to balance.

The 4 Vs, and the data swamp

Carm went back to a framework from the early days of data warehousing: the four Vs of data.

Add the half-life of the data, the data life cycle and security, and most organizations still haven't decided what their most important data is. That's how a data lakehouse becomes a data swamp. "It's kind of like your attic," Carm said. "You put stuff up there and it's like, 'Yeah, I'll get back to it someday.'"

You don't need perfect data to start

Bernie shared Plugable's own experience. In 2025 he expected a lot of data cleanup before AI could work as a universal analyst for the business. Instead, newer techniques made it easier than he expected. Plugable has all of its data in one system, which many organizations don't, and it layers skills on top of the model: instructions, scripts and metadata that help the model understand the swamp. "We've had a lot of success," he said, "of not forcing ourselves to make our data perfect, but rather covering over some of that imperfection, or explaining that imperfection to the model, through skills."

Fit for purpose

Carm has seen both failure and success, and the difference is fit for purpose.

A classic ERP system captures so many fields that organizations probably populate only 20% of them. "You may never use them," Carm said. Going after the data a use case actually needs means "you're not boiling the ocean."

And because there's no instruction manual for AI ("chatbots were dropped on our desks"), the organizations getting results build capability in small steps, the way his grandkids learn: core concepts first, then more. That only works if the organization adopts it. Carm's doctoral dissertation studied why technology succeeds or fails, and the biggest problem wasn't bad technology. "The organization can't adopt." For more on the organizational side, see our AI adoption framework.

Part 2: The fix, AI pods at the desk

If big organizations stall, what's working? Bernie pointed to the other end of the spectrum. Small, entrepreneurial companies, with systems designed together instead of assembled over decades, are seeing remarkable AI adoption and success rates. The question for a larger organization is how to get some of those results. "One of the main tricks, of course, is breaking the problem down."

The 2-trillion-parameter trap

Carm described the mismatch. A small or medium-sized company plugs its internal knowledge into a two-trillion-parameter frontier model, "but I don't need all that in order to be able to accomplish this. And oh, by the way, it doesn't even include my particular implementation techniques." Centralized models are good for some things, like pushing toward AGI. But if you're trying to get your ERP system working the way you want, "you don't need all that capacity and power."

The alternative is local: learning in your own private space, shared across your team. It isn't either-or. Carm sees local models working "in concert with large language models when you need them."

The hidden cost: running out of tokens

Then there's cost. "I can't tell you how many times I've been in the middle of doing something and it's like, oops, out of tokens," Carm said. He expects token costs to start limiting experimentation. Local hardware has its own math: some of the latest Blackwell cards run about $14,000 to $15,000, and it might take a year to break even. "There is no free lunch," he said, "but I think you can be smart about it."

A token factory at your desk

That's why Plugable and Insight work together, Bernie explained. Plugable delivers "a token factory at your desk," infrastructure that adds a lot of compute right where you work. Insight brings the services and complete solutions, from hardware to software, that a larger organization needs to roll it out.

The biggest win is how it changes a pilot. Pilots bog down when they run inside the company's main systems and pull in a lot of departments and stakeholders. "Whereas if you can get compute and AI on that one thought leader's desk, or lead engineer's desk, and have them pilot and prototype things, that's a great way to get started." Prove the value at a small scale, then build on it.

How fast is Thunderbolt 5 for local AI?

Bernie, by his own description a hardware geek, couldn't resist explaining the bandwidth.

After the model loads, very little crosses the cable: "pretty much just your prompts and the return data." The heavy work happens between the NVIDIA GPU and its own high-speed memory. Plugable sells local hardware, Bernie said, that gets "up to 96 gigabytes of RAM and a really powerful NVIDIA GPU, the whole way up to the RTX 6000 Pro."

What Carm ran on it

Carm built a large Jupyter notebook to show customers what a desk-side GPU can do, step by step. He started with computer vision, since "these started off as gaming chips." Then classical machine learning: at a cybersecurity conference, he created synthetic deepfakes on the setup ("I would be arrested if I was doing real deepfakes") and wrote the detection to catch them. Then large language models: Mistral, Gemma and Llama, run in a CUDA-enabled container pinned to the GPU in the Plugable enclosure. "Start the container, finds the device, starts running." From there, chatbots, agents and coding.

He was also at an Air Force conference where the message was clear: "We need air-gap solutions." Teams want a local environment for writing code and reverse-engineering legacy code. In Carm's view, a well-equipped enclosure could support a team of five developers.

Plug in, do the work, leave nothing behind

The other ways to get local AI today, Bernie noted, are a dedicated machine such as an NVIDIA DGX Spark, or a high-end Mac. Both can mean changing the laptop you use. If your laptops are hardened and provisioned for security, that's unworkable in a lot of situations. The Plugable TBT5-AI connects to the Windows or Linux laptop you already have.

It's modular in other ways too. You can choose a very high-end card (about a $15,000 solution right now, Bernie said) or go down to the low thousands for smaller workloads. And the enclosure stores nothing: "The compute device actually has no hard drive, no state." A commander in the field can plug it in, run a heavy AI workload, unplug and hand it to the next person. "You've left absolutely no data behind because there's nowhere to leave it."

He was honest about the limits. For an embedded, all-in-one appliance, "something like a DGX Spark or something is better." But for a modular, Thunderbolt-connected token factory at the desk, "it really, really shines."

Part 3: Why AI is moving back to the desktop

Data sovereignty, from banks to classrooms

When Carm worked at major banks, secure environments had to prove that nothing was leaving the system. The same holds in the military, and in the commercial world, where data leakage and data poisoning are real problems. "These are things that I think people do worry about, and that prevents the adoption."

He sees the same need in state and local government, critical infrastructure and education. A community college that can't afford token costs for every student could give them hands-on access instead of "looking at a slide deck." And in K-12, privacy rules like FERPA matter. Bernie had just spoken with a K-12 school system "really worried about student information making its way out to the cloud." Carm's reply: "Here you go, solution."

More than one model, and a card you can swap

Carm ran three or four models at once on the same device, sending commands to them synchronously or asynchronously over Thunderbolt. Inside the enclosure is a standard PCIe Gen 4 x4 slot, so the card can change: a GPU today, and for some workloads, FPGA capacity.

That matters because laptops and AI compute age differently. "Laptops can last four years," Carm said, but GPU advances will move faster than that. Bernie agreed: a year from now, instead of NVIDIA's Blackwell generation, "we're going to be running Vera Rubin," which he put at about 10 times the tokens per watt. "You don't want to be turning over your laptop or your primary compute system every year the way the GPU is going to be driving you to."

Local models caught up

This, Bernie said, is the big story of the past year. Agentic and coding results that needed a cloud frontier model, paid by the token, in mid-2025 are now achievable with models that run on a desk in 2026. "This has never happened before in my lifetime." Usually hardware improves and software eats the gains. Right now it's flipped: model innovation keeps cutting the compute needed, while hardware has stayed relatively static because of shortages and rising memory costs.

He's been running a 27-billion-parameter model in an open-source coding harness on his desk. Once you've paid for the hardware, there's no cost per token. "If I come up with a workflow that fits on that today, I could be running that 5 years from now with no connection to the cloud." For an organization, that may matter more than the frontier: "what can I do securely and consistently."

Carm sees the same shift. Large providers now let you plug a local model into their coding tools, and the Air Force conference had a talk about local models specifically. "Gone are the days of 'I have to use the latest frontier model to be able to do this.'"

Mainframe to desktop, again

Bernie sees a familiar arc. Computing went from mainframes in the 1950s and '60s to minicomputers to the desktop. "We're in this centralized kind of mainframe era of AI right now, but we're quickly transitioning to the desktop." Hardware shortages are slowing that move, but data centers face limits too: a new data center that wants to connect to the grid in the United States is waiting into the 2030s for permission from the electric utilities.

Carm's closing point was about education. Most organizations aren't frontier labs. "The vast majority of what we're trying to accomplish today doesn't fit into that realm." Part of the job, he said, is helping people understand what they really need, and why.

Cloud AI or a local AI pod?

Neither Bernie nor Carm argued for local AI everywhere. Here's how the trade-offs came up across the three episodes.

Want to see the hardware? Start with Plugable's local AI products, or go straight to the TBT5-AI. For larger rollouts, Bernie's suggestion is to talk with Insight.

Frequently asked questions