The era of artificial intelligence inference is enabling applications such as real-time data analysis that accelerates health research and assistants that resolve thousands of customer requests simultaneously. Underpinning these services is infrastructure that supports continuously operating intelligence and the smart endpoints of IoT and consumer devices. Latency, bottlenecks and wasted energy affect human outcomes and operating costs.
The fact that inference workloads are continuous, geographically distributed and sensitive to response times requires performance, memory bandwidth, storage throughput and networking to be designed together. According to Jim McGregor, founder and chief analyst at Tirias Research, artificial intelligence is not a single workload but consists of thousands, millions and billions of different workloads. Optimization depends not only on computing power but also on coordinating memory, storage and networking.
Adding modern artificial intelligence systems to legacy infrastructure is insufficient, while scalable, resilient and efficient architectures are required. Memory and storage are moving to the center of the system; the data pipeline must receive, clean, transform, store, transport and deliver data. Organizations must balance performance against cost, flexibility, scalability, efficiency per watt, environmental footprint and future readiness, and eliminate bottlenecks without limiting growth.
Term: inference
Inference is the generation of a response to a new input by a trained model; unlike training, it incurs a new cost with every use.