← Back to all projects

LiveContext

LiveContext project screenshot

A native macOS AI-powered interview preparation and real-time assistance app featuring speaker diarization, local speech-to-text via Whisper.cpp, system audio capture via ScreenCaptureKit, and sub-500ms streaming LLM responses. Built with a C++17 cross-platform engine bridged to SwiftUI through Objective-C++, supporting 5 operational modes and a 3-stage interview workflow.

View Source

Technology

frontendswiftswiftui
backendcpp-17objective-cpp
devopscmakeninjaxcode
toolswhisper-cppsherpa-onnxscreencapturekitllm-apis

🎙️ LiveContext -- AI Interview Prep & Real-Time Assistance for macOS

Long-Form Project Summary for KNWS Showcase

LiveContext is a native macOS desktop application that provides AI-powered real-time assistance during interviews, meetings, and conversations. It combines speaker diarization (identifying who is speaking), local speech-to-text (via Whisper.cpp and Apple Speech Framework), system audio capture (via ScreenCaptureKit), and streaming LLM responses into a unified tool that listens to what is being said and provides intelligent, context-aware suggestions in sub-500ms response times.

The application is built with a C++17 cross-platform engine bridged to a Swift/SwiftUI interface through an Objective-C++ bridge layer. Speaker diarization uses Sherpa-ONNX, speech recognition uses Whisper.cpp with Apple Speech Framework as fallback, and LLM inference is routed to multiple providers including OpenAI, Anthropic, Google, and local vLLM/Ollama instances.

LiveContext operates in 5 distinct modes -- Interview, Conversation, Meeting, Recruitment, and Custom -- each with tailored prompt engineering and UI layouts. A 3-stage workflow guides users from data ingestion through briefing to live session assistance.

GitHub: github.com/datagram1/interview_cheat Bundle ID: com.livecontext.app Signed by: Key Network Services Ltd

🚀 Project Goals

  1. Build a real-time AI assistant that listens to live audio and provides contextual suggestions during interviews and meetings.
  2. Implement speaker diarization to distinguish between the user and other participants, enabling targeted response generation.
  3. Run speech recognition locally for privacy -- no audio sent to cloud services for transcription.
  4. Capture system audio from any application (Zoom, Teams, Google Meet) via ScreenCaptureKit without requiring virtual audio drivers.
  5. Deliver sub-500ms streaming LLM responses so suggestions arrive while the conversation is still relevant.
  6. Support multiple LLM backends -- cloud APIs (OpenAI, Anthropic, Google) and local inference (vLLM, Ollama).
  7. Provide 5 specialised modes with tailored prompts and UI for different conversation types.
  8. Implement a 3-stage workflow -- Data Ingest, Briefing, and Live Session -- for thorough preparation.

🧠 Core Features

1. Real-Time Audio Capture (ScreenCaptureKit)

LiveContext captures live audio from the system output:

  • Uses Apple's ScreenCaptureKit framework to capture system audio streams
  • Works with any application -- Zoom, Microsoft Teams, Google Meet, FaceTime, browser-based calls
  • No virtual audio device or kernel extension required
  • Captures at native sample rates with minimal latency
  • Simultaneous microphone input capture for the user's own speech
  • Audio streams are separated and routed independently to the speech recognition pipeline

2. Speaker Diarization (Sherpa-ONNX)

Identifying who is speaking at any moment:

  • Sherpa-ONNX models run locally for real-time speaker segmentation
  • Distinguishes between the user (microphone) and remote participants (system audio)
  • Handles overlapping speech and turn-taking detection
  • Speaker labels are attached to each transcript segment
  • Enables the LLM to understand conversational context -- "They just asked about your experience with..." triggers a relevant suggestion

3. Local Speech-to-Text (Whisper.cpp)

Privacy-first speech recognition:

  • Whisper.cpp -- OpenAI's Whisper model compiled as a C++ library for native performance
  • Runs entirely on-device, no audio data leaves the machine
  • Multiple model sizes available (tiny, base, small, medium) for speed/accuracy trade-offs
  • Real-time streaming transcription with partial results
  • Apple Speech Framework as automatic fallback if Whisper models are not available
  • Language detection and multi-language support

4. Streaming LLM Responses (Sub-500ms)

Fast, contextual AI suggestions:

  • Transcript segments are streamed to the configured LLM provider in real time
  • Streaming API responses begin appearing within 500ms of the trigger
  • Supported providers:
    • OpenAI (GPT-4, GPT-4o)
    • Anthropic (Claude)
    • Google (Gemini)
    • Local vLLM (any open model on 192.168.11.26)
    • Ollama (local model runner)
  • Provider switching is instant -- no restart required
  • Token streaming renders suggestions word-by-word as they arrive
  • Context window management ensures conversation history stays within model limits

5. Five Operational Modes

Each mode configures different system prompts, UI layouts, and response behaviours:

  • Interview Mode -- the user is being interviewed; suggestions focus on answering questions confidently, highlighting relevant experience, and providing structured responses
  • Conversation Mode -- general two-way discussion; suggestions are balanced and conversational
  • Meeting Mode -- multi-participant meetings; suggestions focus on agenda points, action items, and key contributions
  • Recruitment Mode -- the user is the interviewer; suggestions focus on probing questions, candidate evaluation, and structured assessment
  • Custom Mode -- user-defined system prompt and response rules for any scenario

6. Three-Stage Workflow

A structured preparation and execution pipeline:

Stage 1: Data Ingest

  • Upload your CV/resume, job description, company information, or meeting agenda
  • Paste relevant documents, notes, or preparation materials
  • The system indexes all ingested content for contextual reference during the live session

Stage 2: Briefing

  • AI generates a comprehensive briefing based on ingested materials
  • Key talking points, likely questions, and suggested responses
  • Company research summary, role analysis, and interviewer background (if available)
  • Briefing can be reviewed and edited before the live session

Stage 3: Live Session

  • Audio capture begins
  • Real-time transcription with speaker labels
  • Streaming suggestions appear based on what is being said
  • Suggestions draw from both the live conversation and the pre-ingested briefing materials
  • Session can be paused, resumed, or ended at any time

🛠 Technical Architecture

Application Layer (Swift/SwiftUI)

  • Language: Swift 5.x
  • UI Framework: SwiftUI with AppKit integration
  • macOS Target: macOS 14+ (Sonoma)
  • Menu bar application with popover and dedicated window modes
  • Settings panel for LLM provider configuration, model selection, and audio device choices

Core Engine (C++17)

  • Language: C++17
  • Build System: CMake + Ninja
  • Components:
    • Audio capture pipeline (ring buffers, sample rate conversion)
    • Whisper.cpp integration (model loading, inference, streaming results)
    • Sherpa-ONNX integration (speaker embedding extraction, clustering)
    • LLM client (HTTP streaming, provider abstraction, context management)
    • Transcript manager (segment storage, speaker labelling, timeline)

Bridge Layer (Objective-C++)

  • Bridges C++ engine to Swift/SwiftUI
  • Manages memory between ARC (Swift) and manual management (C++)
  • Wraps C++ callbacks as Swift closures
  • Handles ScreenCaptureKit setup (Objective-C API) and routes audio buffers to C++ pipeline

Audio Pipeline

System Audio (ScreenCaptureKit) --> Ring Buffer --> VAD --> Whisper.cpp --> Transcript
Microphone (AVAudioEngine)      --> Ring Buffer --> VAD --> Whisper.cpp --> Transcript
                                                            |
                                                    Sherpa-ONNX Diarization
                                                            |
                                                    Speaker-Labelled Segments
                                                            |
                                                    LLM Context Builder --> Streaming Response

Dependencies

  • Whisper.cpp (compiled from source, linked statically)
  • Sherpa-ONNX (ONNX Runtime with custom speaker models)
  • ScreenCaptureKit (macOS system framework)
  • AVFoundation (microphone capture)
  • CMake + Ninja (build orchestration)
  • URLSession (LLM API communication)

🔧 Build & Development

  • CMake configures the C++ library and Objective-C++ bridge
  • Ninja provides fast incremental builds
  • Xcode manages the Swift/SwiftUI application target and signing
  • The C++ library is compiled as a static library linked into the app bundle
  • Code-signed by Key Network Services Ltd (LZ7T6LSFQ5)
  • Bundle ID: com.livecontext.app

📈 Outcome & Impact

LiveContext demonstrates:

  • Multi-language native development -- Swift, C++17, and Objective-C++ working together in a single application
  • Real-time audio processing -- system audio capture, speech recognition, and speaker diarization with low latency
  • On-device ML inference -- Whisper.cpp and Sherpa-ONNX running locally for privacy
  • LLM integration -- multi-provider streaming responses with sub-500ms delivery
  • macOS system framework expertise -- ScreenCaptureKit, AVFoundation, and Accessibility APIs
  • Complex build system engineering -- CMake + Ninja + Xcode hybrid builds bridging C++ to Swift
  • Product design -- structured 3-stage workflow with 5 operational modes for different use cases
  • Privacy engineering -- all audio processing happens on-device; only text is sent to LLM providers

The project represents a sophisticated intersection of audio engineering, machine learning inference, native macOS development, and AI-powered user assistance.