Operations¶
Last update: May 28, 2026
This page covers day-to-day operations of AI-Bridge for Cisco UC — starting and stopping the server, monitoring, backup & restore, upgrades, key rotation, integrity verification, and support bundle generation.
Starting & Stopping¶
# Start the service
docker compose up -d
# Stop the service
docker compose down
# Restart the service
docker compose restart
# Check service status
docker compose ps
# Follow logs
docker compose logs -f
# Rebuild after an upgrade
docker compose up -d --build
Docker includes a built-in health check that monitors the container every 30 seconds via HTTPS GET on /health. View health status with:
Data preservation
docker compose down removes the container but preserves all volume data (secrets, clients, logs, backups). Use docker compose down -v only if you want to delete everything.
Startup Sequence¶
At every startup, the server performs the following steps automatically:
| Step | Action |
|---|---|
| 1 | Load and validate .env configuration |
| 2 | Initialize structured logging (app.log + audit.log) |
| 3 | Sync default RBAC profiles (profiles/_default/ → profiles/) |
| 4 | Run software integrity check (manifest.json + manifest.sig) |
| 5 | Verify / generate TLS certificate |
| 6 | Verify RSA key pair — generate if absent |
| 7 | Synchronize clients (create / update / purge per AUTH_CLIENTS) |
| 8 | Verify license (secrets/license.jwt) |
| 9 | Load product modules (conditional on license tier) |
| 10 | Start background watchdogs (License, Integrity, Backup) |
| 11 | Start ASGI server (HTTPS) |
Graceful shutdown
The server handles SIGTERM (docker stop) and SIGINT. All background watchdog threads are stopped cleanly via atexit handlers before the process exits.
Monitoring & Observability¶
Log Files¶
| File | Location | Content |
|---|---|---|
app.log |
LOGGING_DIR (default ./logs/) |
Application events: startup, module loading, errors, info |
audit.log |
LOGGING_DIR |
Security audit trail: all auth events, tool calls, credential operations |
Both files use structured JSON format — one JSON object per line — for easy ingestion into SIEM or log aggregation tools (Elastic, Splunk, Loki, etc.).
Log rotation: triggered when app.log or audit.log exceeds LOGGING_ROTATE_SIZE bytes (default ~5 MB). Up to LOGGING_ROTATE_FILES compressed archives are kept alongside the active file.
Console output: enabled by default (LOGGING_CONSOLE_ENABLED=1). Output is colorized by log level for interactive use.
Health & Status Endpoints¶
| Endpoint | Auth | Description |
|---|---|---|
GET /health |
None | Liveness probe — returns ok. Safe for load balancer checks. |
GET /status |
Bearer | Full server status JSON |
/status response includes:
- Server version and uptime
- License state (
VALID/GRACE/EXPIRED/INVALID), tier, expiry date, enabled products - Backup status (last backup timestamp and filename)
- Current blacklisted IP count (Fail2Ban)
- Loaded product modules
- Active client count
# Example status check
curl -sk -H "Authorization: Bearer <token>" \
https://mcp.domain.com:8443/status | python -m json.tool
Prometheus Metrics¶
GET /metrics — no authentication required. Exposes metrics in Prometheus text format.
Available metrics:
| Metric | Description |
|---|---|
| Request counters | By client, tool name, and status |
| Auth failure counters | Total and by client |
| Rate limit counters | By client, product, and service |
| Watchdog status | License, integrity, backup watchdog state |
Scrape interval
A 15-second scrape interval is recommended. The /metrics endpoint is lightweight and does not perform any Cisco UC queries.
Audit Log Event Categories¶
Every security-relevant event is written to audit.log in structured JSON. Events are organized by category:
| Category | What is logged |
|---|---|
AUTH |
Login success/failure, token issued/revoked, source IP, client identity |
ACTIONS |
Every MCP tool call — client, tool name, target host, result, execution time |
RATE_LIMITED |
Rate limit violations — per client, product, and service |
CREDENTIAL |
Store / update / delete credential operations |
TRUSTED |
SSH fingerprint trust and reject decisions |
WRITTEN / EXPORTED |
Report generation and PDF export events |
SERVER |
Startup, shutdown, license state transitions |
TRANSPORT |
TLS alerts, invalid Host/Origin headers, DNS rebinding attempts |
Audit log integrity
audit.log is an append-only structured log. It is never modified after an event is written. Treat it as an immutable audit trail and forward it to a SIEM for long-term retention.
Backup & Restore¶
Backup Encryption Model¶
| Layer | Algorithm |
|---|---|
| Archive encryption | AES-256-GCM (random session key) |
| Session key wrapping | RSA-4096 OAEP (server public key) |
| Integrity verification | SHA-512 sidecar file |
The server can create backups using the public key only. Restoring requires the RSA private key.
Triggering a Backup¶
There is no manual trigger command. Backups run automatically at the interval configured in BACKUP_INTERVAL_HOURS (default: 168 hours). To trigger sooner, either:
- Restart the server (a backup runs on each watchdog startup), or
- Temporarily reduce
BACKUP_INTERVAL_HOURSand restart
Restore — Safe Extract¶
Extract and decrypt a backup without touching the live installation. The script writes to an auto-generated folder named restore_<backup_id>/ in the project root — no --output flag is needed:
docker compose exec ai-bridge python scripts/backup_restore.py \
--file backups/backup_202605_hostname_20260508_001207.tar.gz.enc \
--key secrets/rsa_private.pem
Inspect the extracted contents in ./restore_202605_hostname_20260508_001207/ before deciding to apply them.
Restore — In-Place (--replace)¶
Replace live files with backup contents. Keep the container running — the script detects the server process, prompts for confirmation, stops it, performs the in-place extraction, then you restart the stack:
docker compose exec ai-bridge python scripts/backup_restore.py \
--file backups/backup_202605_hostname_20260508_001207.tar.gz.enc \
--key secrets/rsa_private.pem \
--replace
docker compose up -d
The script stops the server before writing
--replace mode detects the running server PID inside the container, asks for an interactive yes confirmation, then issues SIGTERM before extracting. Do not run docker compose down beforehand — the exec command would fail because the container would be stopped.
Restore After RSA Key Rotation¶
If the backup was created before a key rotation, use the archived key:
docker compose exec ai-bridge python scripts/backup_restore.py \
--file backups/old_backup.enc \
--key secrets/rsa_private.pem.archived.20260512
Archived keys are preserved in secrets/ with a .archived.YYYYMMDD suffix after rotation.
Upgrade¶
Overview¶
Upgrade packages are delivered as signed .tar.gz archives. The upgrade runner handles signature verification, file replacement, .env key merging, and pre-upgrade backup automatically.
Read the dry-run output first
Always run --dry-run before applying an upgrade. It shows all file actions, .env changes, and post-upgrade steps required.
Step-by-Step Upgrade Procedure¶
Step 1 — Verify the package integrity
Before installation, verify the archive using the DNS-published SHA-512 checksum:
# 1. Query the DNS TXT record published by the editor (uses a computer with public DNS resolution)
dig ai-bridge-for-cisco-uc-<version>.sha512.check.consulting.sourdeau.com txt +short
# 2. Compute the SHA-512 hash of the downloaded archive
sha512sum ai-bridge-for-cisco-uc-<version>.tar.gz
# 3. Compare both values — they must match exactly
# 1. Query the DNS TXT record published by the editor (uses a computer with public DNS resolution)
nslookup -q=txt ai-bridge-for-cisco-uc-<version>.sha512.check.consulting.sourdeau.com
# 2. Compute the SHA-512 hash of the downloaded archive
Get-FileHash -Algorithm SHA512 ai-bridge-for-cisco-uc-<version>.tar.gz
# 3. Compare both values — they must match exactly
Do not skip this step
If the values do not match, do not proceed. Contact contact@sourdeau.com immediately.
Step 2 — Place files in the upgrade/ directory
cp ai-bridge-for-cisco-uc-<version>.tar.gz upgrade/
tar -xzf upgrade/ai-bridge-for-cisco-uc-<version>.tar.gz \
ai-bridge-for-cisco-uc-<version>.py -C upgrade/
Step 3 — Preview (dry-run)
The dry-run report shows:
- New features and changelog
- File actions: replaced / created / unchanged
.envchanges: new keys added, always-update keys, preserved keys- Post-upgrade actions required
Step 4 — Stop the container
Step 5 — Apply the upgrade
The runner performs these steps in order:
- Verifies the RSA-PSS signature of the upgrade package
- Extracts and validates the package manifest
- Checks version (downgrades are blocked)
- Creates a pre-upgrade encrypted backup
- Applies file changes
- Merges
.envkeys
Step 6 — Rebuild and restart
.env Key Merge Behavior¶
| Key Category | Behavior |
|---|---|
| New keys | Added with default values, annotated # ADDED by upgrade vXXXXXX |
| Always-update | PROJECT_VERSION, PROJECT_NAME, PROJECT_CREATOR — always overwritten |
| Existing customer keys | Never overwritten — always preserved exactly as configured |
Your configuration is safe
The upgrade runner never overwrites customer-configured keys. Your cluster credentials, client definitions, and custom settings survive every upgrade.
Upgrade CLI Options¶
| Option | Description |
|---|---|
--dry-run |
Preview all changes without applying anything |
--no-backup |
Skip the automatic pre-upgrade backup (not recommended) |
RSA Key Rotation¶
When to Rotate¶
- Periodically as a security best practice (e.g., annually)
- Immediately after a suspected key compromise
Rotation Procedure¶
# Stop the container
docker compose down
# Run the key rotation script
docker compose run --rm ai-bridge python scripts/rotate_rsa_keys.py
# Archives current keys to: secrets/rsa_private.pem.archived.YYYYMMDD
# Generates a new RSA-4096 key pair
# The archived keys are preserved for decrypting existing backups
# Restart the container
docker compose up -d
# At startup: all client JWT tokens are automatically regenerated
# New tokens are written to clients/<name>/secrets/.token
Impact of Key Rotation¶
| Component | Impact |
|---|---|
| JWT tokens | All invalidated — new tokens auto-generated at next startup |
| OAuth client secrets | Unaffected (bcrypt-based, not RSA-dependent) |
| Backups encrypted before rotation | Decryptable using the archived key |
| New backups | Encrypted with the new public key |
Redistribute tokens after rotation
After key rotation, all AI agent users must update their configuration with the new JWT tokens from clients/<name>/secrets/.token. The old tokens are permanently invalid.
Software Integrity Verification¶
Every project file is covered by an SHA-512 manifest (manifest.json) signed with RSA-PSS (RSA-4096), producing manifest.sig.
Automatic Verification¶
- At startup: integrity check runs before any module is loaded
- Background watchdog: re-runs every 24 hours
Any tampering is logged to both app.log and audit.log with CRITICAL severity.
Manual Verification¶
docker compose exec ai-bridge python -c "from lib.integrity import verify_integrity; print(verify_integrity())"
If any file has been tampered with, the output lists each affected file and the expected vs. actual hash. The server continues to run but logs a critical integrity alert.
Show Tech (Support Bundle)¶
The show tech script assembles a diagnostic bundle for support submissions.
docker compose exec ai-bridge python scripts/show_tech.py
# Output: show_tech_<hostname>_<timestamp>.zip
Options:
| Option | Description |
|---|---|
--output <dir> |
Write bundle to a specific directory |
--no-logs |
Exclude log files from the bundle |
--no-zip |
Output as a directory instead of a .zip archive |
What Is Included¶
| Item | Included |
|---|---|
| Server version, Python version, OS info | Yes |
.env configuration (credentials and secrets redacted) |
Yes |
app.log and audit.log |
Yes (unless --no-logs) |
| Manifest verification result | Yes |
| License state (no license file content) | Yes |
| Client list (names only) | Yes |
| Python dependency versions | Yes |
What Is Never Included¶
Sensitive data is never collected
The show tech script explicitly excludes the following — they are never present in any support bundle:
- RSA private keys
- JWT tokens or OAuth client secrets
- Encrypted backup archives
- Cisco UC credentials (AXL passwords, SSH passwords)
- The
secrets/license.jwtfile
Submitting a Bundle¶
Send the bundle to support@sourdeau.com with a description of the issue.