offline companion robot for my disabled husband (8GB RAM constraints) – looking for optimization advice
A partner builds a local AI robot for companionship using Mistral-7B and a repurposed wheelchair base.
In a deeply personal engineering project, a builder is developing a fully local, offline AI companion robot to provide company for her quadriplegic husband, who spends days alone in a rural home. The mobile robot is built on a repurposed power-wheelchair base and aims for autonomous conversation without an internet connection. The current prototype is a testament to resourcefulness, combining a free Lenovo ThinkPad (Intel i5, 8GB RAM) running a quantized Mistral-7B-Instruct model via llama.cpp for the LLM, a separate Jetson Nano running faster-whisper for speech recognition, and Piper TTS for voice synthesis, with output currently displayed on a TV.
The builder's primary technical challenge is maximizing the performance of the llama.cpp large language model within the severe constraint of the ThinkPad's 8GB of system RAM. She has turned to the AI community for optimization advice, specifically asking about advanced quantization techniques for smaller model sizes, Linux memory management strategies like swap or zram, and recommendations for other compact yet conversational models that could run within these limits. The project highlights the real-world application of edge AI and the push to make powerful, empathetic AI tools accessible and functional on extremely modest, repurposed hardware.
- The robot is built for a quadriplegic individual using a power-wheelchair base, Jetson Nano, and a ThinkPad, aiming for full offline operation.
- Core LLM is a quantized Mistral-7B-Instruct model running via llama.cpp on a system with only 8GB of RAM, creating a significant performance bottleneck.
- The builder is seeking expert advice on llama.cpp optimization, including quantization, Linux swap/zram, and smaller model alternatives to improve conversational quality.
Why It Matters
This project demonstrates a critical, humane use case for efficient, local AI, pushing the limits of low-resource hardware for assistive technology.