How does edge AI work?
Most edge AI splits the lifecycle in two. A model is trained in a data center, compressed, converted for an on-device runtime, and shipped to the device, where inference runs on the local CPU, GPU or NPU. Training usually stays central; the edge runs the finished model. NIST describes levels: at the base, edge nodes use models created elsewhere, for example in the cloud; at the top, they learn from their own local data, which NIST calls edge learning (NIST Edge AI project, updated 12 August 2026).
What is the difference between edge AI and cloud AI?
Edge and cloud AI trade the same properties in opposite directions. The cloud wins on model size and update speed; the edge wins on latency, privacy and offline operation.
| Edge AI | Cloud AI | |
|---|---|---|
| Where inference runs | On the device or a nearby gateway | In a remote data center |
| Latency | No network round trip | Network travel on every request |
| Model size | Limited by device memory and power | The largest models available |
| Data movement | Raw data can stay on site | Inputs leave the device |
| Works offline | Yes | No |
| Updating the model | Push to every device in the fleet | Change one endpoint |
Many systems are hybrid: a small local model handles the fast, frequent or private cases and escalates hard ones to a cloud model, while data sovereignty rules decide what may leave the site.
Which tools run AI models on edge devices?
A trained model is converted for an on-device runtime, and the runtime decides which accelerators it can reach. On a device, the runtime’s size and hardware reach count as much as the model. Vendor pages read 16 November 2026:
| Runtime | From | Typical targets |
|---|---|---|
| LiteRT (formerly TensorFlow Lite) | Android, iOS, web, desktop, embedded and IoT | |
| ExecuTorch | PyTorch | Mobile phones down to microcontrollers |
| ONNX Runtime | ONNX Runtime project | Mobile, IoT and desktop, via execution providers such as NNAPI and Core ML |
| Core ML | Apple | Apps on Apple devices |
| OpenVINO | Intel | AI PCs and edge devices on Intel hardware |
Binary size matters on a phone: ONNX Runtime’s mobile guide lists its prebuilt 1.18.0 Android package at 24,415,212 bytes and a custom build with only the operators a ResNet50 model needs at 7,532,309 bytes.
Can a language model run at the edge?
A language model can run at the edge if it is small and compressed. Memory is the first limit. A 3 billion parameter model at 16 bits per weight needs about 6 GB for the weights alone; at 4 bits, about 1.5 GB, before the KV cache and runtime overhead. That is why edge language models are small language models, often made by model distillation and then quantization. ExecuTorch’s documentation includes a guide to running a 3 billion parameter Llama 3 instruct model on Android. Test the compressed model on your own hard inputs, not only on its publisher’s benchmark.
When should you choose edge AI?
Choose edge AI when the network is the problem: a latency budget too tight for a round trip, sites with patchy connectivity, or data that should not leave the building. Edge AI moves the operational burden from one endpoint to a fleet of devices, so plan how you will ship model updates, roll them back and monitor quality on hardware you cannot easily log into. Robots and vehicles in physical AI depend on local reaction times, and a model shipped inside a product’s firmware is also embedded AI. Without those constraints, a cloud endpoint is simpler to run and update.