XiaoZhi Device

GPIO Pin Map

Button Behaviour

Audio Pipeline

Partition Table

Assets partition is NOT updated by OTA — font/sound changes require a full USB flash.

Deploy Workflow (Server Build — preferred)

# 1. Bump version — do this BEFORE building
sed -i 's/set(PROJECT_VER ".*")/set(PROJECT_VER "2.2.X")/' \
    /home/ubuntu/danscodellaro.com/xiaozhi-esp32/CMakeLists.txt

# 2. Build — IMPORTANT: capture the real exit code, not tail's
source /home/ubuntu/esp-idf/export.sh
cd /home/ubuntu/danscodellaro.com/xiaozhi-esp32
idf.py build > /tmp/build.log 2>&1; echo "BUILD EXIT: $?"
tail -5 /tmp/build.log
# (piping idf.py through `tail` masks compile failures with tail's exit 0)

# 3. Asserted deploy — refuses to ship if binary != CMakeLists version
expected=$(grep PROJECT_VER CMakeLists.txt | grep -oE "[0-9]+\.[0-9]+\.[0-9]+")
built=$(strings build/xiaozhi.bin | grep -E "^[0-9]+\.[0-9]+\.[0-9]+$" | head -1)
if [ "$expected" != "$built" ]; then echo "MISMATCH $expected vs $built"; exit 1; fi

cp build/xiaozhi.bin \
    /home/ubuntu/danscodellaro.com/frontend/esp32/xiaozhi/xiaozhi.bin

sed -i "s/_OTA_FIRMWARE_VERSION = \".*\"/_OTA_FIRMWARE_VERSION = \"$expected\"/" \
    /home/ubuntu/danscodellaro.com/api/server.py

sudo systemctl restart travel-api

# 4. Optional: free ~350 MB disk space if tight on space
# rm -rf build/   # only after you're sure nothing else needs the build dir

Version constant and binary MUST be in sync — mismatch causes an infinite OTA loop (we hit this with 2.2.86 — the build had silently failed but tail's exit-0 hid it, so the deploy shipped a stale 2.2.85 binary labelled "2.2.86" and the device looped for ~10 minutes downloading the same wrong file).

Deploy Workflow (Windows Build → Server)

# Build on Windows:
cd C:\Users\dansc\xiaozhi-esp32
python idf_build.py build

# SCP to server:
scp -i C:/Keys/lightsail-cos.pem build/xiaozhi.bin \
    [email protected]:/home/ubuntu/danscodellaro.com/frontend/esp32/xiaozhi/xiaozhi.bin

# Then run steps 5-6 above on the server.

OTA Response Format

The OTA endpoint (POST /esp32/xiaozhi/ota.php) returns:

{
  "firmware": {
    "version": "2.3.18",
    "url": "https://www.danscodellaro.com/esp32/xiaozhi/xiaozhi.bin"
  },
  "websocket":   { "url": "ws://iot.danscodellaro.com/xiaozhi/v1/", "token": "" },
  "ws_base_url": "ws://iot.danscodellaro.com",
  "server_time": { "timestamp": 1775386136101, "timezone_offset": 600 }
}

The device compares firmware.version against its running version. If server is newer → download + flash + reboot. On success the app calls MarkCurrentVersionValid() to prevent OTA rollback.

ws_base_url (added 2.2.83) is the plain-WS base URL used by the chat WS, telemetry WS, and video media WS. It is now pinned to the stable DNS name ws://iot.danscodellaro.com — a DNS-only, Cloudflare-bypass plaintext origin reserved for the device. The device caches it in NVS namespace net, key ws_base. (Earlier builds used a raw ws://<public-ip> that a server.py daemon refreshed from AWS IMDS; the iot hostname removed the need to chase IP changes.)

Pointing Device at This Server (First Time)

# Device in download mode (hold BOOT + power cycle via USB)
python set_ota_url.py
# Patches NVS at offset 0x9000 only — no firmware reflash needed

Custom Files (untracked — required for build)

These files are not committed to git. They are required for a correct build — never delete them, and always restore them when cloning to a new machine. The committed main/CMakeLists.txt lists both telemetry.cc and media_player.cc as source files; a missing file causes a "no such file" build error.

⚠️ Known Footgun: Circular GetInstance() Deadlock

If a singleton's constructor calls GetInstance() on another singleton that also calls back into the first, the C++11 magic-static initialisation guard will deadlock the main task permanently. The device hangs at that point on every boot with no panic — it just silently freezes.

This hit us in v2.2.6 and v2.2.7: McpServer() constructor called MediaPlayer::RegisterMcpTools(), which called McpServer::GetInstance() — re-entering the already-held static-init mutex. Fixed in v2.2.8 by moving the RegisterMcpTools() call from the constructor to AddCommonTools().

Rule: never call GetInstance() on a singleton from within another singleton's constructor.

.axv Binary Frame Format

Each frame (video or audio) is prefixed with a 16-byte header. All four fields are big-endian (network byte order). The firmware rejects any file with wrong byte order — it logs "Frame sync lost" and stops silently.

Common mistake: Python's struct.pack uses native (little-endian) byte order by default. Always use struct.pack(">IIII", ...) — the ">" prefix is required. Type constants are also easy to get wrong: video=0, audio=1 (not 1/2). The encoder script is pack_axv.py in the same directory as each .axv file.

Frames are interleaved in presentation-timestamp order. At 24 fps video (~42 ms) and 60 ms audio, video frames out-number audio packets ~7:5. At 8 fps, the ratio is roughly 1 video frame per 2 audio packets (125 ms vs 60 ms).

Creating .axv Files

Use scripts/make_axv.py on the server. It requires ffmpeg in PATH and handles aspect ratio padding automatically.

From a video file

# Default (8 fps — small files, low CPU)
python3 scripts/make_axv.py --input video.mp4 --output out.axv

# Source-rate match (24 fps — cinema-cadence playback, ~3× file size)
python3 scripts/make_axv.py --input video.mp4 --output out.axv --fps 23.976

# Lower quality to shrink file at high fps
python3 scripts/make_axv.py --input video.mp4 --output out.axv --fps 23.976 --quality 8

Generate a test pattern (30 s SMPTE bars + 440 Hz tone)

python3 scripts/make_axv.py --test --output test_video.axv

The script extracts video at 240×240 with black-bar padding, re-encodes audio to Opus at 24 kHz mono, then interleaves both streams by presentation timestamp. Float-precision timestamps avoid drift at non-integer rates (e.g. 23.976 fps over a 23-minute episode would drift ~33 seconds with naive integer math).

ffmpeg internals

# Video frames (extracted as individual JPEGs):
ffmpeg -i input.mp4 \
  -vf "scale=240:240:force_original_aspect_ratio=decrease,pad=240:240:(ow-iw)/2:(oh-ih)/2:black" \
  -r 23.976 -q:v 5 frames/%06d.jpg

# Audio (Opus in OGG, then raw packets demuxed):
ffmpeg -i input.mp4 -vn -ar 24000 -ac 1 \
  -c:a libopus -b:a 24k -frame_duration 60 -vbr off audio.ogg

Media Library JSON Schema

Each item in the library (api/media_library.json) follows this schema:

Add an item via API

curl -X POST https://www.danscodellaro.com/esp32/xiaozhi/media/library \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-track",
    "title": "My Track",
    "description": "An Opus audio file.",
    "type": "audio",
    "tags": ["music"],
    "duration_s": 120,
    "url": "https://www.danscodellaro.com/esp32/xiaozhi/media/files/my-track.ogg"
  }'

Posting an item with an existing id performs an upsert (update). Delete via DELETE /esp32/xiaozhi/media/library/<id>.

Server Infrastructure

Host: AWS Lightsail Ubuntu (foundry.danscodellaro.com) · Web: Caddy · API: Python Flask (travel-api.service, port 5001)

Firmware Version History

† 2.2.5 "running on device" is the Windows English rebuild — different binary from the first Linux 2.2.5 build.

Known Issues