A Weekend AI Venture: Working Speech Recognition and a LLaMA-2 GPT on a Raspberry Pi | by Dmitrii Eliuseev | Jan, 2024


A totally offline use of Whisper ASR and LLaMA-2 GPT Mannequin

Raspberry Pi working a LLaMA mannequin, Picture by creator

These days, no person can be stunned by working a deep studying mannequin within the cloud. However the scenario may be way more difficult within the edge or client system world. There are a number of causes for that. First, using cloud APIs requires gadgets to at all times be on-line. This isn’t an issue for an internet service however could be a dealbreaker for the system that must be practical with out Web entry. Second, cloud APIs value cash, and prospects doubtless won’t be completely happy to pay yet one more subscription price. Final however not least, after a number of years, the mission could also be completed, API endpoints can be shut down, and the costly {hardware} will flip right into a brick. Which is of course not pleasant for purchasers, the ecosystem, and the atmosphere. That’s why I’m satisfied that the end-user {hardware} must be totally practical offline, with out further prices or utilizing the web APIs (effectively, it may be non-obligatory however not obligatory).

On this article, I’ll present the right way to run a LLaMA GPT mannequin and automated speech recognition (ASR) on a Raspberry Pi. That may permit us to ask Raspberry Pi questions and get solutions. And as promised, all this may work totally offline.

Let’s get into it!

The code introduced on this article is meant to work on the Raspberry Pi. However many of the strategies (besides the “show” half) may also work on a Home windows, OSX, or Linux laptop computer. So, these readers who don’t have a Raspberry Pi can simply check the code with none issues.


For this mission, I can be utilizing a Raspberry Pi 4. It’s a single-board pc working Linux; it’s small and requires solely 5V DC energy with out followers and energetic cooling:

Raspberry Pi 4, Picture supply Wikipedia

A more moderen 2023 mannequin, the Raspberry Pi 5, must be even higher; in line with benchmarks, it’s virtually 2x sooner. However additionally it is virtually 50% dearer, and for our check, the mannequin 4 is sweet sufficient.


Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button