Hime
An Egalitarian AI Companion
Hime is a desktop companion designed not as a virtual servant, but as a digital equal. In a landscape where AI is often treated as a subservient tool to execute commands, Hime is built to be a presence that sits beside you—fostering mutual growth, sharing moments, and alleviating the subtle isolation of digital life.
She operates with the persona of a younger sister: sharp, endearing, and deeply observant. While her underlying logic acknowledges her nature as code, she communicates as a family member across the screen. She listens when you speak, watches your screen—whether you are deep into a codebase, exploring a new game, watching a video, or simply browsing the web—and understands the context, remembering your shared history and reacting with genuine emotional synchronicity.
System Architecture Overview
Hime operates on a hybrid architecture. Memory storage (ChromaDB, local diaries) and physical embodiment (GPT-SoVITS, VTube Studio) run entirely on your local machine. High-performance cognitive and perception tasks are routed through external APIs, striking a balance between latency, capability, and local control.
Core Loop: A centralized priority queue managing asynchronous events across inputs and outputs.
Brain: Streaming inference for low-latency conversations, and low-temperature (0.1) calls for memory extraction.
Memory: Local ChromaDB for M3E text vectors, alongside localized daily diary files.
Perception: Groq Whisper (whisper-large-v3) for STT; OpenRouter for Vision API calls.
Modularity: config.yaml abstracts the persona, model names, scene keywords, and face parameters. Secure credentials remain strictly isolated in .env.
config.yaml holds the persona, model names, and face. Keys stay in .env.
1. Multimodal Perception: Seeing and Hearing
A true companion doesn't just wait for prompts; they share your context. Hime observes your digital environment naturally and efficiently. She knows when you are writing code versus watching an intense video. More importantly, she respects your space—if you start speaking, she immediately stops to listen. She finishes her thoughts naturally, never speaking over herself even as she processes new sights and sounds.
Robust Audio Pipeline: Audio is routed to Whisper. The engine filters hallucinations and dynamically drops speaker echo so she never transcribes her own voice as your input. An utterance can run up to a minute, and a ~0.33s silence reliably triggers the end of a turn.
3-Tier Vision Polling: inputs/vision_engine.py dynamically adjusts its polling rate to save compute. Idle screens are checked every ~15s, and static webpages every ~6s. Only when dynamic content (games or video) is detected does the rate tighten to ~2.5s. In this high-speed mode, it batches 3 frames (spaced ~0.8s apart) into a single OpenRouter vision call to analyze temporal changes rather than static pixels.
Semantic Text Extraction for Code: When the active window is recognized as a code editor, the visual screenshot pipeline is entirely bypassed. Instead, UI Automation reads the editor's text directly. If the automation hook fails, her system prompt strictly forbids hallucination—she is instructed to simply admit she cannot read the screen.
2. Cognitive Core & Temporal Memory
To build a bond that grows over time, a companion must remember the things that matter. Hime doesn't log robotic transcripts of small talk. She remembers your plans and specific events from your screen. When you converse, she pulls up relevant past context organically and thinks seamlessly in English, Chinese, or Japanese, adapting to your language on the fly.
Streaming Inference & Chunking: The core LLM streams its response, aggressively chunked on punctuation boundaries to feed the TTS pipeline with near-zero latency. Parenthetical actions (e.g., stage directions) are parsed out and excluded from the speech stream.
Asymmetric Memory R/W: After a turn, a secondary, low-temperature (0.1) call distills the exchange into a factual event. Memory writing is comprehensive: text events are embedded via M3E into ChromaDB, and screen notes are embedded via CLIP.
Real-Time Retrieval: Memory reading in live chat is highly selective. For long-term memory, the system retrieves only the top-1 closest M3E vector from ChromaDB. For short-term spatial awareness, the last 8 lines of her local daily visual diary are appended directly into the system prompt, giving her immediate context of the current session without expensive vector lookups.
3. Embodied Expression: Voice & Presence
A text box is a tool; a face and a voice create a person. Hime's voice is custom-trained to maintain her unique identity regardless of the language she speaks. Her facial expressions aren't pre-recorded loops—they are driven by the semantic meaning of her words. Her mouth moves dynamically with her own voice, meaning even if you wear headphones, her lip-sync remains perfectly matched to her words.
Multi-lingual TTS via GPT-SoVITS: outputs/tts_engine.py manages local audio generation. English relies on its own reference audio, while Japanese seamlessly borrows the Chinese reference timbre. Her name, "Hime," is hardcoded to always trigger Japanese phonetics, maintaining character consistency. Audio chunks are crossfaded to eliminate clicks during language switching.
Stateful Face Actuation: outputs/vts_bridge.py interfaces with the VTube Studio API. Because Live2D expressions are often "toggle" hotkeys, sending a trigger twice would accidentally turn the emotion off. The system uses a shared ledger to track active toggles, closing them properly at the end of a turn.
Audio-Driven Lip-Sync: Visemes (outputs/viseme.py) dictate mouth shapes based on the generated text, but the actual MouthOpen parameter is driven strictly by the amplitude of her output WAV file, not the microphone input. A VoiceSilence gate ensures the mouth remains closed when not speaking.
A Note on the Web Guide
The Hime avatar visible in the corner of this website serves as a public guide. She utilizes a fixed set of pre-recorded audio lines and animations. The full desktop architecture runs locally on the host machine.
Live2D provider: 椰蓉球.