Geeky Gadgets

The Latest Technology News



Running a 35B Parameter AI Model on iPhone at 11 Tokens per Second

By

A diagram explaining the 35B Mixture of Experts AI architecture.

Running a 35-billion-parameter AI model directly on an iPhone may seem like an unlikely feat, but Better Stack demonstrates how this can be achieved through a combination of advanced techniques. Central to this achievement is the Mixture of Experts (MoE) architecture, which activates only 3 billion of the model’s parameters at a time, significantly reducing computational demands. By using methods like tiered quantization and efficient memory management, the model operates within the constraints of mobile hardware, balancing performance and resource efficiency. This approach highlights how even consumer devices can handle complex AI tasks with the right optimizations.

In this overview, you’ll gain insight into how the MoE model’s selective activation mechanism works, making sure efficient use of resources without compromising output quality. Explore the role of tiered quantization in compressing the model’s size while maintaining accuracy and learn how memory management strategies enable smooth operation despite limited RAM. The discussion also covers practical challenges, such as debugging issues and hardware limitations, offering a detailed look at what it takes to bring large-scale AI to mobile platforms.

Understanding the Model Architecture: Mixture of Experts

TL;DR Key Takeaways :

  • The Mixture of Experts (MoE) model enables a 35-billion-parameter AI to run efficiently on an iPhone by activating only 3 billion parameters at a time, optimizing resource usage.
  • Memory management is achieved by categorizing components into active (1.4 GB RAM) and inactive (12 GB SSD) groups, making sure smooth operation despite hardware constraints.
  • Tiered quantization reduces the model size from 19 GB to 13 GB by compressing frequently used components to 4 bits and less-used ones to 2 bits, improving performance and storage efficiency.
  • Debugging challenges, such as fixing a bug in expert size verification, were critical to making sure the model runs reliably on mobile devices.
  • The Flash Moe engine, adapted for iOS, highlights the potential for running advanced AI models on consumer hardware, paving the way for future innovations in mobile AI applications.

At the core of this implementation lies the Mixture of Experts (MoE) model, a sophisticated architecture designed to handle large-scale AI tasks with remarkable efficiency. Despite its 35 billion parameters, only 3 billion are active at any given time. This is made possible by a router mechanism that dynamically activates the most relevant “experts” for a given task, significantly reducing computational overhead.

Key structural elements of the architecture include:

  • The model comprises 40 layers, each containing 256 small experts and one shared expert.
  • During token generation, the router selectively activates the experts most relevant to the input, making sure efficient use of resources.

This design allows the model to deliver high-quality results while operating within the hardware constraints of a mobile device. By activating only the necessary components, the MoE model achieves a balance between performance and resource efficiency, making it suitable for deployment on an iPhone.

Memory Optimization: Balancing Active and Inactive Components

Running a model of this magnitude on an iPhone requires precise memory management to ensure smooth operation. The system achieves this by categorizing components into active and inactive groups, optimizing resource allocation.

  • Active components, such as embeddings, attention mechanisms, routers and shared experts, occupy approximately 1.4 GB of RAM.
  • Inactive experts, which collectively require 12 GB of storage, are offloaded to the device’s SSD and streamed into memory only when needed.

The attention mechanism, a critical component for maintaining context during token generation, remains in memory throughout the process. This approach minimizes latency and ensures seamless operation. By carefully managing the balance between active and inactive components, the system maximizes the iPhone’s limited resources without compromising the model’s performance.

Check out more relevant guides from our extensive collection on local AI that you might find useful.

Performance Optimization: Tiered Quantization

To further enhance performance, the model employs tiered quantization, a compression technique that assigns different bit depths to various components based on their usage frequency. This method reduces the model’s size while maintaining accuracy and efficiency.

  • Frequently accessed “hot” experts are quantized to 4 bits, preserving higher precision where it matters most.
  • Less frequently used “cold” experts are compressed to 2 bits, saving storage space and improving efficiency.

This optimization reduces the model’s overall size from 19 GB to 13 GB, improving caching efficiency and reducing SSD read times. The iOS page cache plays a vital role in this process, keeping frequently accessed data readily available to minimize repeated SSD reads. This combination of compression and caching ensures the model runs smoothly on the iPhone, despite its hardware limitations.

Overcoming Challenges: Debugging and Refinements

Implementing such a complex model on a mobile device was not without its challenges. One significant issue involved a bug that caused the system to loop indefinitely when handling cold experts. The problem stemmed from a function that failed to verify expert sizes correctly during activation. After identifying the root cause, the function was updated to ensure accurate size checks, resolving the issue and allowing the model to run as intended.

This debugging process underscores the importance of meticulous testing and refinement when adapting large-scale AI models for mobile platforms. Addressing such challenges is crucial to achieving reliable performance and making sure a seamless user experience.

How to Set It Up

If you’re interested in replicating this setup, the process involves several steps:

  • Clone the repository containing the model and its dependencies.
  • Apply the necessary bug fix to ensure proper functionality.
  • Use Xcode to build and install the app on your iPhone. Note: A paid Apple developer account is required for this step.
  • Load the prepackaged 13 GB model onto the app via USB.

Once installed, the app can run the model locally, generating tokens directly on your device without relying on cloud-based processing. This setup demonstrates the feasibility of deploying advanced AI models on consumer hardware, offering a glimpse into the future of mobile AI.

Performance Insights

The optimized model achieves a token generation speed of 11 tokens per second on an iPhone 17. While this performance is impressive for a mobile device, it does come with certain trade-offs. The device tends to heat up significantly during operation, which could impact long-term usability. Additionally, the model’s storage requirements may pose challenges for users with limited device capacity.

Despite these limitations, this achievement highlights the potential for running advanced AI models on consumer hardware. It demonstrates how careful optimization and resource management can unlock new possibilities for mobile AI applications.

Engine Details: Flash Moe and the iOS Port

This implementation relies on the Flash Moe engine, originally developed for MacBooks. The engine was adapted for iOS, resulting in the Flash iOS port. Its efficiency and compatibility with mobile hardware make it an ideal choice for this project, allowing high-performance AI on a compact device.

The Flash iOS port uses the unique capabilities of Apple’s hardware, including its powerful processors and advanced memory architecture. This adaptation underscores the versatility of the Flash Moe engine and its potential for allowing large-scale AI applications on a variety of platforms.

Future Implications for Mobile AI

Running a 35-billion-parameter AI model locally on an iPhone represents a significant milestone in the evolution of AI and mobile technology. By combining the Mixture of Experts model, tiered quantization and efficient memory management, this setup demonstrates the feasibility of deploying large-scale AI on consumer devices. While challenges such as heat generation and storage limitations remain, this achievement paves the way for future innovations in mobile AI, bringing powerful capabilities directly to users’ hands.

As AI continues to evolve, the ability to run complex models on mobile devices could transform industries ranging from healthcare to entertainment. This breakthrough serves as a testament to the potential of AI to enhance everyday experiences, making advanced technology more accessible than ever before.

Media Credit: Better Stack

Filed Under: AI, Guides



Disclosure: Some of our articles include affiliate links. If you buy something through one of these links, Geeky Gadgets may earn an affiliate commission. Learn about our Disclosure Policy.

Geeky Gadgets