ai-phone: Running an AI-Powered SIP Phone Agent Locally on Apple Silicon

Overview of ai-phone

ai-phone is an open-source project that enables users to operate an AI-powered SIP (Session Initiation Protocol) phone agent directly on Apple Silicon hardware. By leveraging local speech processing, the project aims to facilitate automated voice communication handled by a combination of local and remote language models.

How it Works

• Local Speech Processing: The system uses MLX to perform speech recognition and synthesis on the local machine, ensuring that voice data is processed locally before being transmitted.
• Language Models: While speech remains local, the conversation logic (the language model) is processed remotely. Users are advised that their conversation transcripts and instructions are sent to the external cloud provider selected during setup.
• Hardware Requirements: The platform currently requires Apple Silicon. While the caller on the other end can use any standard phone or SIP client, the host machine running the agent must be an Apple Silicon Mac.
• SIP Integration: The tool integrates with SIP providers like SignalWire. Users are required to manage their own SIP credentials and numbers through these third-party portals.

Implementation and Usage

The software is managed via a command-line interface (CLI). Users run an installation script, followed by a setup wizard that detects system RAM to recommend suitable local models.

Core Features

• Call Handling: It supports incoming calls, outgoing calls, or both.
• Relay Mode: Advanced users can utilize a JSON-based relay mode to programmatically handle calls, allowing an external agent to exert control over each turn of the conversation.
• Operational Constraints: Currently, the system supports English language conversations and handles one call at a time. It does not install as a persistent background service, meaning users must keep the terminal active to receive incoming calls.

Verification and Performance

The developer provides documentation on call flows, network configurations (NAT), and architectural design. Performance benchmarks from late 2026 recorded a median first-audible response time of 1.50 seconds under memory pressure. Users should be aware that standard carrier charges and SIP subscription limits apply to all calls made through the platform.

For those interested in verifying the functionality, the repository includes details on testing successful call exchanges between major SIP providers. As an AGPL-3.0 licensed project, users are encouraged to review the documentation and license notes before deployment.