Selfhosted

Do you want to run TensorFlow Lite / LiteRT models? PyTorch Mobile? TensorRT? onnx? YOLO? vLLM? Something else? The recommendations will vary based on your use case.

Google Coral was decent for TensorFlow Lite, but it's EOL (end of life) now. I've got the dual TPU Mini PCIe version in my home server, via a PCIe adapter board. I use it for object detection with Blue Iris + CodeProject AI and it works pretty well for that use case.

Hailo-8 is supposed to be like a more powerful version of the Coral, but I don't have experience with it. It supports a bunch of frameworks: TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX. I'd be interested in hearing other people's thoughts on it.

I don't know if any of these work over USB though. They're usually internal devices. Google marketed the Coral USB as being for development and testing only, pointing people to the M.2 and PCIe versions for production usage.

As for something totally different... There's the Nvidia Jetson single board computer which supports TensorRT, but I don't have experience with it either. I also think it's a bit older too. You could also consider getting a newer mini PC with a AMD Ryzen AI processor in it, or an Nvidia DGX Spark.

Google's latest TPUs are only available in their cloud - they're not selling the hardware to end users any more.

if all you need is a tiny NPU for things like image object identification or OCR the IGPU you already have will be better.

if you need slightly more than that, but not as much power as a large dedicated gpu, then you would have to look at a newer mini-pc with a more modern chip.

I do immich face/object detection and Birdnet audio analysis on the igpu of an intel 14600 and it's fine, tiny amount of usage and plenty fast. my other node has a core ultra 235 and the igpu is faster than the on-chip npu in every regard, the npu is also a bit harder to actually get working and put to use.

Can you tell us what atually will you need? Don't forget that most of the time applications will use your bare CPU to perform AI whereas you can use the integrated GPU for better performances.

google coral is what you need, there's also an m.2 pcie version, also intel igpu also has limited AI capabilities through openvino, i use it to accelerate AI features in Immich (face and object recognition) on my N100 based NAS, but it's a bit too slow for Frigate realtime object recognition for example

You should probably have a look at the software you intend to run and see what kind of external NPU they support. I'm aware of Frigate for example which leverages hardware acceleration to classify camera surveillance footage. Guess their community knows which stick to get. Or you'd have a look at the frameworks which package machine learning models for inference on some NPU and have a look at their table of supported hardware.

if you don't have oculink, you can get a nvme to oculink adapter (if you have 2nd nvme port).