Server Setup

Learn how to configure and run the Noid server for managing Firecracker microVMs.

Prerequisites

Before setting up the server:

  • Linux host with KVM support
  • Firecracker installed at /usr/local/bin/firecracker
  • Kernel image (vmlinux)
  • Root filesystem (rootfs.ext4)

Server Configuration

The Noid server is configured using a TOML file. The default location is ~/.noid/server.toml or a custom path specified with --config.

Basic Configuration

Create server.toml:

[server]
listen = "0.0.0.0:7654"
timeout = 30000

[paths]
kernel = "/path/to/vmlinux"
rootfs = "/path/to/rootfs.ext4"

Key settings:

  • listen: Server address and port (default: 0.0.0.0:7654)
  • timeout: Request timeout in milliseconds
  • kernel: Path to kernel image for guest VMs
  • rootfs: Path to root filesystem for guest VMs

Note: The default port is 7654, not 8080.

Full Configuration Example

[server]
listen = "0.0.0.0:7654"
timeout = 30000
workers = 4

[paths]
kernel = "/var/lib/noid/vmlinux"
rootfs = "/var/lib/noid/rootfs.ext4"
data_dir = "~/.noid"

[limits]
max_vms = 100
max_memory_per_vm = 4096

[logging]
level = "info"  # debug, info, warn, error

Data Directory Structure

The server organizes data under ~/.noid/ (or configured data_dir):

~/.noid/
├── server.toml          # Server configuration
├── config.toml          # Client configuration
├── golden/              # Golden snapshot files
│   ├── rootfs.ext4
│   ├── memory.snap
│   ├── vmstate.snap
│   └── config.json
└── users/
    └── <user-id>/       # Per-user isolation
        ├── vms/         # Virtual machines
        ├── checkpoints/ # VM checkpoints
        └── logs/        # Console logs

Golden Snapshot Directory

The ~/.noid/golden/ directory contains:

  • rootfs.ext4: Pre-installed filesystem
  • memory.snap: RAM state at snapshot time
  • vmstate.snap: CPU and device state
  • config.json: Template metadata (CPU count, memory allocation)

User Management

The server supports multi-tenant operation with user isolation.

Creating Users

Create a user with an API token:

noid-server user create <username>

This generates a 64-character hexadecimal API token for the user.

Managing Tokens

Rotate a compromised token:

noid-server user rotate-token <username>

Remove a user:

noid-server user delete <username>

Token Authentication

Tokens use SHA-256 hashing with constant-time verification. Store tokens securely - they grant full access to manage VMs.

Running the Server

Direct Execution

# With default config location
noid-server

# With custom config
noid-server --config /etc/noid/server.toml

systemd Service

Create /etc/systemd/system/noid-server.service:

[Unit]
Description=Noid Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/var/lib/noid
ExecStart=/usr/local/bin/noid-server --config /etc/noid/server.toml
Restart=always
RestartSec=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable noid-server
sudo systemctl start noid-server
sudo systemctl status noid-server

Network Configuration

Network Daemon (noid-netd)

The server requires noid-netd, a privileged daemon that manages TAP devices and NAT rules.

Installation:

See the Installation Guide for noid-netd setup.

Configuration:

The network daemon uses iptables for NAT and creates TAP devices for VM networking.

Enable IP forwarding:

echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

systemd Service for noid-netd

Create /etc/systemd/system/noid-netd.service:

[Unit]
Description=Noid Network Daemon
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/noid-netd
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable noid-netd
sudo systemctl start noid-netd

Golden Snapshot Setup

Golden snapshots dramatically speed up VM creation (from 30-60s to 5-10s).

Method 1: Automatic Provisioning

Run the provisioning script:

sudo bash scripts/provision-golden.sh

This will:

  1. Create a temporary VM named _provision
  2. Wait for complete boot
  3. Install desired applications (edit script to customize)
  4. Create a checkpoint
  5. Promote it to golden status
  6. Clean up the temporary VM

Method 2: From Existing Checkpoint

If you have a configured VM, promote its checkpoint:

sudo bash scripts/provision-golden.sh --from-checkpoint <checkpoint-id>

Customizing Golden Snapshot

Edit scripts/provision-golden.sh to pre-install tools:

# Add custom installations
apt-get install -y python3 nodejs golang
pip3 install requests numpy
npm install -g typescript

All pre-installed applications become instantly available in new VMs.

When to Rebuild Golden Snapshot

Rebuild when:

  • Kernel version changes
  • Base filesystem updates
  • Firecracker version changes
  • New tools need pre-installation
  • CPU or memory defaults change

Storage Optimization

btrfs for Zero-Copy Snapshots

On btrfs filesystems, rootfs cloning uses reflinking for instant copies with minimal disk space:

# Format partition as btrfs
mkfs.btrfs /dev/sdX

# Mount with appropriate options
mount -o compress=zstd /dev/sdX /var/lib/noid

Benefits:

  • Instant rootfs cloning (zero-copy)
  • Space-efficient until data modifications
  • Faster checkpoint operations

Note: Memory and state snapshots always require full copies regardless of filesystem.

ext4 Fallback

On ext4, the server falls back to regular file copying. This works but is slower than btrfs reflinking.

API Structure

The server exposes REST and WebSocket endpoints:

Public Endpoints

  • GET /health - Health check
  • GET /version - Server version

Authenticated Endpoints (under /v1/)

  • VM lifecycle management
  • Command execution
  • Console access
  • Checkpoint operations

All authenticated requests require a Bearer token in the Authorization header.

Logging

View server logs:

# With systemd
sudo journalctl -u noid-server -f

# Direct execution
# Logs to stdout/stderr

Configure log level in server.toml:

[logging]
level = "info"  # debug, info, warn, error

Security

Rate Limiting

The server implements rate limiting on failed authentication attempts to prevent brute force attacks.

Multi-Tenant Isolation

  • Complete database isolation per user
  • Separate filesystem directories per user
  • User can only access their own VMs

Token Security

  • 64-character hexadecimal tokens
  • SHA-256 hashing
  • Constant-time verification
  • Secure token storage

Troubleshooting

Kernel Staleness

Issue: TLS connections hang during VM boot.

Cause: CRNG (cryptographic random number generator) not initialized in guest kernel.

Solution: Rebuild kernel with proper entropy sources or use a newer kernel version (6.12.71+).

Golden Snapshot Invalidation

Issue: VMs fail to boot from golden snapshot.

Cause: Kernel or Firecracker version mismatch.

Solution: Rebuild golden snapshot after upgrading:

sudo bash scripts/provision-golden.sh

Networking Issues

Issue: VMs can't reach the internet.

Check:

# Verify IP forwarding
sysctl net.ipv4.ip_forward

# Check iptables NAT rules
sudo iptables -t nat -L

# Verify noid-netd is running
sudo systemctl status noid-netd

Database Locking

Issue: Database locked errors.

Solution: Ensure only one server instance is running. Check for stale processes:

ps aux | grep noid-server

Validation Commands

Verify server setup:

# Check Firecracker
which firecracker
firecracker --version

# Check kernel and rootfs
ls -lh /path/to/vmlinux
ls -lh /path/to/rootfs.ext4

# Check KVM
ls -l /dev/kvm

# Test server endpoint
curl http://localhost:7654/health

Next Steps