The high cost of specialized hardware (e.g. GPUs, TPUs, NPUs, and FPGAs) that enables the rapid training, tuning and inference of LLMs, combined with high electricity consumption, demanding cooling requirements, and complex compliance standards and regulations, prevent application development teams from harnessing the advantages of AI at scale. In a nutshell, instead of being able to harness AI capabilities to ensure optimal user experiences and the best possible application functionality, organizations are often deterred by the large price tags of obtaining the required infrastructure resources.
The quickly increasing popularity of agentic AI adds cost and performance pressure, due to these agents solving problems collaboratively, while at the same time consuming large numbers of costly LLM tokens for communication and coordination. Different agents can leverage different open source LLMs to contribute to this iterative problem-solving approach, each LLM with its own set of performance characteristics. These challenges are often compounded by a lack of developer skill and experience in configuring and optimizing complex AI workflows, leading to increased operational costs and delays.
Red Hat acquired Neural Magic to make it more cost efficient and easier for development teams to add LLM capabilities to their applications. Adding an “AI accelerator” to their Red Hat AI deployments could significantly improve Red Hat’s position in the marketplace for cloud-native application platforms.