Unleashing the Power of Deep Learning on Standard CPUs with ThirdAI and Ray

Table of Contents

1. Introduction
2. The Importance of Training Machine Learning on CPUs
3. Overcoming GPU Shortages with CPU Training
4. Introducing Bolt 2.5 B: The World's First Generative Large Language Model Trained on CPUs
   1. Comparing Bolt 2.5 B with GPT2 XL
   2. The Efficiency of Fine-Tuning on CPUs
5. The Advantages of Training AI on CPUs
   1. Latency Benefits
   2. Cost Savings
   3. Building an AI Ecosystem
6. The Dynamic Sparity Approach to More Efficient Deep Learning Training
   1. The Problem of Calculating Zero Values
   2. Third Eye's Solution: Dynamics Per Stack
   3. The Benefits of Dynamic Sparse Training
7. Introducing Neural DB: A Database for Retrieval Augmented Generation
   1. The Challenges of Augmenting Generation with Retrieval
   2. The Friction of Heterogeneous Hardware and Software Stacks
   3. The Solution: Neural DB and Learning-to-Index
8. Case Studies: Third Eye's AI Ecosystem in Action
   1. Reduction of Latency in Wayfair's Retrieval Systems
   2. Improved Accuracy with Domain-Specialized Pre-Training
9. Technical Aspects of Scaling Sparse Training on CPUs with Ray
   1. Eliminating the Need for Model Parallel Training
   2. The Benefits of Data Parallel Training
   3. Using Ray for Distributed Data Parallel Training
   4. A Simplified Developer Experience with Bold Trainer
10. Experimental Evaluation of Third Eye's CPU Training Approach
    1. Scaling up Sparse Training with Bolt AI Models
    2. Training Time Improvements with Increased Node Size

Training Machine Learning on CPUs: Overcoming GPU Shortages and Enhancing Efficiency

Machine learning has revolutionized various industries, but the reliance on GPUs for training large models has become a bottleneck due to ongoing GPU shortages. Third Eye aims to address this issue by developing software libraries that enable the training and fine-tuning of billion-parameter neural networks and large language models on CPUs. In this article, we will explore the benefits of training AI on CPUs, introduce Bolt 2.5 B as the world's first generative large language model trained on CPUs, discuss the advantages of CPU training, and Delve into the technical aspects of scaling sparse training on CPUs using Ray.

Introduction

Training machine learning models has traditionally relied heavily on GPUs due to their high computational power and parallel processing capabilities. However, the recent surge in demand for GPUs, fueled by the AI boom, has resulted in widespread GPU shortages. As companies increasingly adopt Generative AI models and require extensive training and fine-tuning, the need for alternative training methods arises. Third Eye presents a solution by focusing on CPU training, enabling the development of AI models without relying on scarce GPU resources.

The Importance of Training Machine Learning on CPUs

When discussing AI on CPUs, the common Perception mainly revolves around inference. However, Third Eye emphasizes that training, retraining, fine-tuning, and reinforcement learning can all be efficiently performed on CPUs. By leveraging readily available CPUs, organizations can achieve better latency, eliminate the need for data transfer between CPU and GPU, reduce costs, and foster an AI ecosystem centered around fine-tuning and daily ownership.

Overcoming GPU Shortages with CPU Training

The ongoing GPU shortages have disrupted the AI market, leading to delays and increased demand for GPU instances. However, Third Eye proposes an alternative solution by training and fine-tuning the world's first generative large language model, Bolt 2.5 B, entirely on CPUs. In contrast to previous practices that heavily relied on GPU instances, this breakthrough demonstrates the feasibility of utilizing CPUs for generative AI workloads. By doing so, organizations can overcome the challenges posed by GPU shortages and achieve efficient training and fine-tuning on readily available CPUs.

Introducing Bolt 2.5 B: The World's First Generative Large Language Model Trained on CPUs

Bolt 2.5 B serves as a testament to the capabilities of training large language models on CPUs. Powered by Third Eye's Bolt deep learning engine, this model utilizes dynamic sparsity to selectively train only the Relevant neurons for each training sample. This Novel approach enables the training of a 2.5 billion parameter model on 40 billion tokens using 10 Sapphire Rapids machines in just 20 days. The efficiency achieved through dynamic sparsity offers unprecedented accessibility to fine-tuning. Case studies comparing Bolt 2.5 B with the popular 1.5 billion parameter GPT2 XL model demonstrate its comparable capabilities.

The Advantages of Training AI on CPUs

CPU training offers several advantages that make it a compelling alternative to GPU-Based training:

Latency Benefits

Training AI models on CPUs eliminates the need for data transfer between CPU and GPU, resulting in significantly reduced latency. By leveraging CPUs for AI training, organizations can achieve faster results and enhance real-time decision-making processes.

Cost Savings

While GPUs have been the go-to choice for AI training, the cost associated with GPU instances can be a significant barrier, especially in the face of GPU shortages. CPU training, on the other HAND, leverages readily available and cost-effective resources, making it an affordable alternative that democratizes AI training across organizations and individuals.

Building an AI Ecosystem

Third Eye's approach to AI training on CPUs is centered around an AI ecosystem that emphasizes ownership. Organizations and individuals are encouraged to fine-tune models daily, enabling a continuous improvement cycle. This approach fosters innovation, creativity, and collaboration within the AI community.

The Dynamic Sparity Approach to More Efficient Deep Learning Training

Dynamic sparsity lies at the Core of Third Eye's efficient deep learning training on CPUs. Traditional deep learning models often waste computational resources by calculating operations involving zero values. Third Eye's solution, known as Dynamics Per Stack, addresses this inefficiency by dynamically determining the relevant parameters for each training sample. By utilizing techniques inspired by information retrieval, the relevant parameters are queried from the large neural network, significantly reducing computation and energy costs. This approach allows for massive Scale deep learning training on CPUs without compromising accuracy.

Introducing Neural DB: A Database for Retrieval Augmented Generation

To further enhance the capabilities of AI models, Third Eye introduces Neural DB, a revolutionary database designed for retrieval augmented generation. Retrieval augmented generation involves combining generative AI models with retrieval systems to provide more accurate and contextually relevant answers. However, Current software stacks face challenges such as data movement, complicated hardware requirements, and dependency on external models. Neural DB addresses these challenges by offering a seamless solution that eliminates data movement, simplifies software stacks, and provides efficient retrieval augmented generation capabilities.

The Challenges of Augmenting Generation with Retrieval

Retrieval augmented generation faces several challenges, including data movement, complex software stacks, and the need for external models such as embedding models. Data movement between GPUs for generating embeddings and storing/retrieving text in Vector DB creates friction and inefficiencies. Additionally, the reliance on specific software stacks for different hardware types further complicates the integration of retrieval augmented generation systems.

The Friction of Heterogeneous Hardware and Software Stacks

The heterogeneity of different hardware types, such as GPUs and CPUs, leads to complexities in software stacks. To leverage the benefits of both GPUs and CPUs, organizations often find themselves juggling multiple software stacks, resulting in friction and reduced efficiency. Third Eye aims to simplify this process by providing a unified software stack that works seamlessly with CPUs, eliminating the need for GPU-centric solutions.

The Solution: Neural DB and Learning-to-Index

Third Eye's Neural DB offers a comprehensive solution for retrieval augmented generation. By training models directly on CPUs, Neural DB eliminates the need for GPU-based embeddings and simplifies the retrieval process. With the ability to fine-tune and refine models on the fly, Neural DB provides a flexible and efficient AI ecosystem. This approach enables organizations to utilize existing CPU resources, circumvent data movement challenges, and achieve better accuracy in retrieval augmented generation tasks.

Case Studies: Third Eye's AI Ecosystem in Action

Real-world case studies demonstrate the efficacy of Third Eye's AI ecosystem and the benefits of training and fine-tuning models on CPUs. Wayfair, a prominent e-commerce company, successfully reduced latency in their retrieval systems by incorporating Third Eye's technologies. By leveraging Third Eye's CPU training capabilities, Wayfair achieved improved efficiency and cost savings in their AI workflows. Additionally, domain-specific pre-training demonstrated the superiority of focused training over more generalized approaches, further validating Third Eye's CPU training methodology.

Technical Aspects of Scaling Sparse Training on CPUs with Ray

Third Eye utilizes Ray, a distributed computing framework, to scale up sparse training on CPUs. Ray provides the necessary tools for data parallel training, enabling efficient and performant scaling. By utilizing Ray core and Ray trainer, Third Eye simplifies the developer experience, enhances fault tolerance, and facilitates broader integration capabilities. The combination of Ray and Third Eye's bold trainer enables developers to harness the full potential of CPU training with ease.

Eliminating the Need for Model Parallel Training

Traditional GPU-based training often requires model parallelism to accommodate large models. However, with CPUs, model parallel training becomes obsolete due to their scalability and capacity to handle massive models. Third Eye's CPU training approach eliminates the complexities associated with model parallel training, streamlining the overall training process.

The Benefits of Data Parallel Training

Data parallel training, where multiple CPUs handle different portions of the training data, offers several advantages. Third Eye leverages dynamic sparsity to compress gradients and reduce communication time in data parallel training. Additionally, dynamic sparsity allows for larger batch sizes, which further enhances data parallelism. By increasing batch size without sacrificing performance, Third Eye enables efficient and scalable AI training on CPUs.

Using Ray for Distributed Data Parallel Training

Ray's distributed computing capabilities complement Third Eye's CPU training methodology. By utilizing Ray's actor model, auto-scaling, and seamless integration with CPU clusters, Third Eye ensures fault-tolerant and optimized distributed data parallel training. The simplified code required for distributed training further enhances the developer experience, enabling organizations to efficiently train AI models at scale.

A Simplified Developer Experience with Bold Trainer

Third Eye's bold trainer, built on Ray, provides a user-friendly environment for developers to utilize distributed data parallel training on CPUs. The simplified code required for bold trainer implementation allows developers to focus on model training rather than intricate infrastructure management. With reduced memory usage and the option to use cost-effective CPU instances, the bold trainer offers a streamlined approach to CPU-based AI training.

Experimental Evaluation of Third Eye's CPU Training Approach

To showcase the effectiveness of Third Eye's CPU training approach, several experimental evaluations were conducted. These evaluations focused on scaling up sparse training using Bolt AI models and analyzing the impact of node size on training time.

Scaling up Sparse Training with Bolt AI Models

Third Eye's Bolt AI models were evaluated on the Critio Terabyte Benchmark, a large-scale dataset for click-through prediction. The experiments compared models of varying compressions, ranging from 50 million to 37.5 million parameters. The results demonstrated that training time increased linearly with model size and decreased linearly as the number of nodes increased. This evaluation highlighted the scalability and efficiency of Third Eye's CPU training approach when applied to real-world datasets.

Training Time Improvements with Increased Node Size

In another evaluation, a billion-parameter model was trained using Bolt AI models. The experiment aimed to determine the impact of node size on training time. The results indicated that doubling the node size led to a slight improvement in training time. However, the communication between nodes remained a bottleneck, limiting the overall reduction in training time. Despite this limitation, the experiment demonstrated the potential of Third Eye's CPU training approach to achieve faster training times with larger node configurations.

In conclusion, Third Eye's CPU training approach offers a viable alternative to GPU-based training, leveraging the power and accessibility of CPUs to train and fine-tune AI models. By utilizing dynamic sparsity, Third Eye enables efficient training on CPUs, overcoming GPU shortages and reducing costs. The introduction of Neural DB and the integration with Ray further enhance the AI ecosystem, simplifying the scalability, fault tolerance, and performance of CPU training. With real-world case studies validating the effectiveness of Third Eye's approach, it is clear that CPU training has the potential to revolutionize the AI landscape.