v2 WiFi DensePose Runtime and Apps
Overview
The v2 runtime surface is a set of Rust applications and deployable libraries anchored by v2/Cargo.toml. The workspace includes the CLI (v2/crates/wifi-densepose-cli), sensing server (v2/crates/wifi-densepose-sensing-server), desktop app (v2/crates/wifi-densepose-desktop), browser WASM bindings (v2/crates/wifi-densepose-wasm), training and inference crates (v2/crates/wifi-densepose-train, v2/crates/wifi-densepose-nn, v2/crates/wifi-densepose-occworld-candle), the MediaTek bridge (v2/crates/wifi-densepose-mtk-bridge), and the default-off ruv-FANN presence head (v2/crates/ruview-fann-head). v2/crates/wifi-densepose-wasm-edge is explicitly excluded from the workspace because it targets wasm32-unknown-unknown/no_std; build it separately as noted in v2/Cargo.toml.
The CLI binary is v2/…/main.rs, which parses wifi-densepose subcommands and dispatches to module entrypoints in src/lib.rs. The operational commands are: Cognitum auth (src/auth.rs) for login/logout/whoami, Cognitum Spaces reads (src/spaces.rs), empty-room calibration over UDP (src/calibrate.rs), the calibration HTTP API (src/calibrate_api.rs), per-room enrollment/training/watch (src/room.rs), optional MAT disaster commands (src/mat.rs), and version. src/calibrate.rs binds a UDP socket, parses ESP32 CSI frames with magic 0xC511_0001, feeds wifi_densepose_signal::CalibrationRecorder, and writes a binary BaselineCalibration. src/calibrate_api.rs wraps the same recorder behind Axum routes under /api/v1/calibration/*, with one background UDP ingest task and command channels for start/stop/enroll operations. Auth commands store local Cognitum credentials and intentionally request read-only scope unless --admin or --spaces is supplied in src/auth.rs; src/spaces.rs either uses a compatibility API key or refreshes the stored OAuth session before calling Cognitum Spaces.
The sensing server executable is v2/…/main.rs, with public library modules exported from src/lib.rs. Startup parses Args, initializes telemetry, handles exit-only modes (--benchmark, --convert-model, standalone --export-rvf, --pretrain, embedding/index modes), loads runtime config and optional bootstrap baselines from --data-dir, optionally starts MQTT, constructs shared AppStateInner, spawns source tasks, and serves an Axum router. The router exposes REST and WS surfaces including /api/v1/sensing/latest, /api/v1/nodes, /api/v1/csi/*, /api/v1/vital-signs, /api/v1/model/*, /api/v1/models/*, /api/v1/recording/*, /api/v1/train/*, /ws/sensing, /ws/introspection, /api/v1/stream/pose, OAuth routes, and the additive RuField surface from src/rufield_surface.rs. Bearer/OAuth auth and host-header validation are layered after all route merges in src/main.rs, so merged /api/field and /ws/field are covered as well.
The live UDP ingest path is udp_receiver_task in v2/…/main.rs. It binds the configured UDP address, applies the ADR-296 source allowlist from src/udp_bind.rs, optionally tees admitted datagrams via src/udp_tee.rs, and then branches by wire format: JSON vendor RF events, Qualcomm CSI, MediaTek MTC1 CSI, RTL8720F radar, RTL8721Dx RAC1 CSI, ESP32 edge vitals (0xC511_0002), sync packets, edge fused vitals (0xC511_0004), WASM output (0xC511_0007), and raw ESP32 CSI (0xC511_0001). Successful sensor frames update AppStateInner fields such as latest_update, per-node NodeState, MediaTek rings, vitals, model/calibration state, and a broadcast channel consumed by /ws/sensing, MQTT, and dashboard clients. ESP32 edge-vitals and raw CSI paths build SensingUpdate objects, run room-level fusion via src/inference.rs, apply bootstrap/calibration gates, optionally attach tracked/field positions, and publish JSON to the broadcast channel.
Model packaging for the sensing server is RVF. v2/…/rvf_container.rs implements the segment format, including SEG_VEC, metadata, witness, embedding, and LoRA segments. src/rvf_pipeline.rs (loaded from src/main.rs through ProgressiveLoader) progressively loads model layers and SONA profiles for /api/v1/model/*. src/model_format.rs detects native RVF, safetensors, HuggingFace quantized blobs, and JSONL manifests; load_or_convert_model in src/main.rs converts safetensors/JSONL in memory but refuses unsupported quantized blobs with actionable errors. Standalone --export-rvf deliberately writes only a placeholder container-format demo; real trained exports are produced by training/pretraining paths, not by that shortcut.
Server-side training is split between the sensing server and the dedicated training crate. v2/…/training_api.rs adds /api/v1/train/start, /stop, /status, /pretrain, LoRA training, and /ws/train/progress. It snapshots frame_history or loads recordings, claims a single training slot with a mutex-protected status update, runs a background linear CSI-to-pose job, writes RVF output under the server data directory, and streams progress over a broadcast channel. The offline training binary is v2/…/train.rs: it loads TrainingConfig from JSON or defaults, supports --dry-run synthetic data, MM-Fi discovery, --val-dir held-out validation, --eval-only, checkpoint directories, and CUDA. With tch-backend enabled it constructs trainer::Trainer; without that feature it only verifies config/dataset plumbing. v2/…/lib.rs, src/config.rs, and src/dataset.rs define the reusable training API: bounded TrainingConfig, CsiDataset, deterministic SyntheticCsiDataset, MM-Fi layout discovery, subject-disjoint split guards, canonical metrics, model gates, WiFlow-STD, MAE, domain adaptation, and leakage protocols.
The neural inference crate v2/crates/wifi-densepose-nn is a reusable multi-backend abstraction. src/lib.rs re-exports tensors, a DensePose head, the CSI-to-visual-feature ModalityTranslator, and InferenceEngine. src/inference.rs defines the backend trait and statistics wrapper; src/onnx.rs implements an ONNX Runtime backend that loads from file/bytes, serializes ort::Session::run behind a write lock, and rejects unresolved non-positive dynamic output dimensions before reshaping. src/translator.rs can run a native weighted CSI-to-feature translator or a mock path for tests, and fails closed when native weights are absent.
v2/crates/wifi-densepose-occworld-candle is the native Candle OccWorld port. src/inference.rs exposes OccWorldCandle::load, dummy, and predict; load reads SafeTensors through src/model.rs, maps checkpoint keys, builds VQVAE components and an autoregressive transformer, and marks outputs weights_trained: true. dummy is deterministic and input-dependent but marks weights_trained: false, so consumers can benchmark shape/latency without presenting untrained trajectory priors as trained predictions.
The desktop app is a Tauri wrapper in v2/crates/wifi-densepose-desktop. src/main.rs calls wifi_densepose_desktop::run, and src/lib.rs registers Tauri commands for discovery, serial flash, OTA, WASM module management, sensing-server lifecycle, provisioning, and settings. src/…/server.rs finds sensing-server from a configured path, bundled resources, sibling executable, or PATH, then starts it as a child process and stores PID/ports in src/state.rs; if the UI does not choose a source, it explicitly passes simulated mode so demo output is marked synthetic. src/…/discovery.rs discovers ESP32 nodes through _ruview._udp.local. mDNS plus UDP broadcast on port 5006 and merges results by MAC address. src/…/wasm.rs talks to ESP32 WASM management on port 8033 (/wasm/list, multipart /wasm/upload, /wasm/{id}/{action}, /wasm/{id}, stats/info) and validates WASM magic before upload. The React UI under ui/src invokes these commands: ui/…/App.tsx routes dashboard/discovery/nodes/flash/OTA/edge modules/sensing/mesh/settings pages, and ui/…/useServer.ts polls server_status and invokes start_server/stop_server with default ports.
The browser WASM package v2/crates/wifi-densepose-wasm exposes MAT dashboard bindings. src/lib.rs initializes panic hooks/logging and re-exports src/mat.rs, which defines JavaScript-friendly disaster, triage, zone, survivor, alert, and dashboard types via wasm_bindgen. v2/…/mat-dashboard.html is a standalone dashboard example that consumes those concepts in a browser-style UI.
The edge WASM crate v2/crates/wifi-densepose-wasm-edge is a hot-loadable no-std skill library for ESP32-S3 WASM3. src/lib.rs documents the csi host imports (csi_get_phase, amplitude, variance, vitals, presence, emit/log, phase history), exports on_init, on_frame, and on_timer, defines event ID ranges, and registers many skill modules. The default wasm path is the small flagship pipeline, while src/pipeline_all.rs is a std host harness that adapts all skill modules behind EdgeSkill and runs them over a borrowed CsiFrameView. Medical modules are compiled only with medical-experimental. src/…/ghost_hunter.rs is a standalone WASM module that reads sanitized host CSI values, runs exo_ghost_hunter::GhostHunterDetector, emits events, and logs initialization.
The MediaTek bridge executable is v2/…/main.rs. It either listens for MtkCSIdump UDP datagrams (--listen, optional --csidump registration, required --listen-allow) or replays mt76 JSON dumps / .mtkcap captures (--replay). src/udp_in.rs decodes live datagrams into MtkRecords; src/dump_file.rs parses mt76 dumps; src/capture.rs records and replays decoded sessions. src/assemble.rs folds records into wifi_densepose_hardware::mediatek_csi::CsiFrame MTC1 frames, using PerRecord for live UDP (one chain per datagram) and LastChainMarker for dump/replay where chain_info can close a PPDU. src/sink.rs sends encoded MTC1 frames to the sensing server UDP port. src/provenance.rs prevents replay ambiguity: replay requires --synthetic or --captured-on <model>/<firmware>, synthetic sidecars/filenames/capture headers are sticky, and live accepted frames are physical-unvalidated unless explicitly synthetic.
v2/crates/ruview-fann-head is a default-off ruv-FANN presence-head experiment. src/lib.rs defines the incumbent linear head, prepared FannHead, context binding, digest, and abstain band. src/train.rs fits a bounded candidate or feature-matched linear control on frozen features with deterministic full-batch AdamW/MSE; src/shadow.rs scores candidates beside the incumbent without authority; src/replay.rs and binaries such as src/…/ruview-fann-replay.rs evaluate JSONL replay rows and emit reports/learning curves. It does not publish presence directly; it only produces diagnostics until the existing owner selection flow chooses a model.
The release/docs surface in v2/docs records the intended architecture. v2/…/ADR-003-neural-network-inference.md states the multi-backend inference strategy (ONNX primary, tch/Candle alternatives), while v2/docs/ddd/* capture domain language and event/model boundaries. v2/patches/ruvector-crv is a patched side package for CRV-to-ruvector experiments; its src/lib.rs maps staged CRV data into embeddings and graph/mincut-style outputs, but it is not wired into the WiFi sensing runtime paths above.
Coupling
- If the ESP32 CSI wire layout changes, update both v2/…/calibrate.rs and v2/…/main.rs: they independently parse the
0xC511_0001frame layout, includingn_subcarriers, PPDU type, flags, and IQ payload. - If firmware changes edge packet magic values, keep v2/…/main.rs and v2/…/lib.rs aligned. The server distinguishes edge vitals (
0xC511_0002), fused vitals (0xC511_0004), and WASM output (0xC511_0007); a collision routes packets to the wrong parser. - If adding routes to the sensing server, register them before the bearer-auth and host-validation layers in v2/…/main.rs. Axum layers only wrap routes already in the router, and the code comments call out
/ws/fieldpreviously bypassing auth when merged after auth layering. - If changing model file formats, update v2/…/model_format.rs, src/rvf_container.rs, and the
load_or_convert_model/run_convert_modelpaths in src/main.rs together. The server only executes RVF natively and relies on typed conversion/refusal for safetensors, JSONL, and HF quantized blobs. - If changing server training status or progress schemas, update both v2/…/training_api.rs and the desktop/UI consumers under
v2/crates/wifi-densepose-desktop/ui/src/pages/*: progress is broadcast as JSON over/ws/train/progress, while status is polled over/api/v1/train/status. - If the desktop server launch flags change, keep v2/…/server.rs and the sensing server
Argsin v2/…/main.rs in sync. The Tauri app passes--http-port,--ws-port,--udp-port,--bind, and--source; mismatched flag names break desktop-managed startup. - If ESP32 node WASM management endpoints change, update v2/…/wasm.rs and the firmware/API documentation together. The desktop command layer assumes port 8033 and paths
/wasm/list,/wasm/upload,/wasm/{id}/{action},/wasm/{id}, and stats support. - If MediaTek MTC1 frame semantics change, update v2/…/assemble.rs, src/main.rs, and the sensing server MediaTek branch in v2/…/main.rs. The bridge sets provenance/time-sync/synthetic flags and grouping mode; the server classifies snapshots and room updates based on those decoded frames.
- If a ruv-FANN candidate is promoted from shadow to authority, check v2/…/shadow.rs and the sensing-server FANN integration (v2/…/fann_shadow.rs, feature
fann-shadow). The current crate is intentionally diagnostic-only and context-bound; publishing its score requires the owner qualification path to change.
Risks
- v2/…/main.rs has load-bearing auth layer ordering. If future routes are merged after the bearer-auth middleware, authenticated deployments can silently expose those routes; the existing comments document that
/ws/fieldpreviously returned an unauthenticated WebSocket upgrade when merged in the wrong order. - v2/…/model_format.rs and src/main.rs intentionally distinguish “loadable container” from “accurate trained model.” Converting HuggingFace safetensors/JSONL to RVF makes the loader succeed, but the comments state the decoder architecture may differ; treating a converted file as validated pose accuracy can publish heuristic or mismatched-model results.
- v2/…/provenance.rs is the only guard preventing replay files from masquerading as live hardware. If
--replaystops requiring--syntheticor--captured-on, the sensing server can label fabricated or unaudited captures asmediatek:physical-unvalidated. - v2/…/training_api.rs relies on an atomic mutex claim in
claim_training_slot. Reintroducing a check-then-spawn sequence under only the outerRwLockcan spawn multiple training jobs that share status/cancel fields and overwrite each other. - v2/…/train.rs and src/dataset.rs guard against metric leakage with held-out subject checks and synthetic smoke-test labeling. If an index-level MM-Fi split is reintroduced, stride-overlapping windows can inflate validation PCK/OKS and make non-reportable metrics look like real accuracy.