Operations

Overview

The operations module (src/ops/) provides portable utilities for managing evolver lifecycle, monitoring health, and performing maintenance. These commands are designed to work in any environment without platform dependencies.

Lifecycle Commands

Manage the evolver daemon with the lifecycle module:

node src/ops/lifecycle.js <command>

start

Start the evolver daemon in the background:

node src/ops/lifecycle.js start

What it does:

Check for Running Instance

Scans for existing evolver processes:

[Lifecycle] Already running (PIDs: 12345).

If found, exits without starting a new instance.

Spawn Detached Process

Starts node index.js --loop in detached mode:

const child = spawn('node', [script, '--loop'], {
  detached: true,
  stdio: ['ignore', logFile, logFile],
  cwd: WORKSPACE_ROOT,
  env: process.env,
});
child.unref();

Write PID File

Saves process ID to memory/evolver_loop.pid:

[Lifecycle] Started PID 12345

Output:

{
  "status": "started",
  "pid": 12345
}

Options: Start with a delay (useful after restart):

const lifecycle = require('./src/ops/lifecycle');
lifecycle.start({ delayMs: 5000 });  // Wait 5s before starting

stop

Gracefully stop all running evolver loops:

node src/ops/lifecycle.js stop

What it does:

Find Running Processes

Discovers all evolver loop PIDs via ps scan:

const pids = getRunningPids();  // Scans for 'node index.js --loop'

Send SIGTERM

Attempts graceful shutdown:

process.kill(pid, 'SIGTERM');

Waits up to 5 seconds for processes to exit.

Force Kill if Needed

If processes don’t exit, sends SIGKILL:

process.kill(pid, 'SIGKILL');

Clean Up Lock Files

Removes PID file and singleton lock:

rm memory/evolver_loop.pid
rm evolver.pid

Output:

[Lifecycle] Stopping PID 12345...
[Lifecycle] All stopped.

{
  "status": "stopped",
  "killed": [12345]
}

restart

Stop and restart the evolver daemon:

node src/ops/lifecycle.js restart

What it does:

Calls stop() to terminate running instances
Waits 2 seconds (configurable)
Calls start() to spawn a new instance

Output:

[Lifecycle] Stopping PID 12345...
[Lifecycle] All stopped.
[Lifecycle] Starting: node index.js --loop
[Lifecycle] Started PID 12346

status

Check if the evolver daemon is running:

node src/ops/lifecycle.js status

Output (running):

{
  "running": true,
  "pids": [
    {
      "pid": 12345,
      "cmd": "node /workspace/evolver/index.js --loop"
    }
  ],
  "log": "memory/evolver_loop.log"
}

Output (not running):

{
  "running": false
}

check

Perform health check and auto-restart if unhealthy:

node src/ops/lifecycle.js check

Health Criteria:

Process running: At least one evolver PID found
Log freshness: Log file modified within the last 30 minutes

Output (healthy):

{
  "healthy": true,
  "pids": [12345]
}

Output (unhealthy - not running):

{
  "healthy": false,
  "reason": "not_running"
}

[Lifecycle] Restarting...
[Lifecycle] Started PID 12346

Output (unhealthy - stagnant):

{
  "healthy": false,
  "reason": "stagnation",
  "silenceMinutes": 45
}

Run check periodically via cron to ensure the evolver stays alive:

*/10 * * * * cd /path/to/evolver && node src/ops/lifecycle.js check

log

View recent log output:

node src/ops/lifecycle.js log

This runs tail -n 20 on the log file (memory/evolver_loop.log). Options: View more lines:

const lifecycle = require('./src/ops/lifecycle');
const result = lifecycle.tailLog(100);  // Last 100 lines
console.log(result.content);

Health Monitoring

The health check module (src/ops/health_check.js) runs system diagnostics:

const { runHealthCheck } = require('./src/ops/health_check');
const health = runHealthCheck();
console.log(health);

Health Checks

Environment Variables

Checks for required secrets:

Critical: FEISHU_APP_ID, FEISHU_APP_SECRET (warning if missing)
Optional: CLAWHUB_TOKEN, OPENAI_API_KEY (info if missing)

{
  "name": "env:FEISHU_APP_ID",
  "ok": true,
  "status": "present"
}

Disk Space

Monitors root filesystem usage:

Warning: >80% used
Critical: >90% used

{
  "name": "disk_space",
  "ok": true,
  "status": "65% used"
}

Memory Usage

Checks system memory:

Critical: >95% used

{
  "name": "memory",
  "ok": true,
  "status": "72% used"
}

Process Count (Linux only)

Detects fork bombs or leaks:

Warning: >2000 processes

{
  "name": "process_count",
  "ok": true,
  "status": "342 procs"
}

Health Status

{
  "status": "ok",  // or "warning" or "error"
  "timestamp": "2026-03-09T14:30:22.123Z",
  "checks": [
    { "name": "env:FEISHU_APP_ID", "ok": true, "status": "present" },
    { "name": "disk_space", "ok": true, "status": "65% used" },
    { "name": "memory", "ok": true, "status": "72% used" },
    { "name": "process_count", "ok": true, "status": "342 procs" }
  ]
}

Cleanup Operations

The cleanup module (src/ops/cleanup.js) removes stale artifacts:

node src/ops/cleanup.js

What Gets Cleaned

Removes old GEP prompt files from memory/evolution/:

Files matching gep_prompt_*.json or gep_prompt_*.txt
Older than 24 hours
Keeps at least 10 most recent files
Enforces max 10 files total

Output:

[Cleanup] Scanning for old artifacts...
[Cleanup] Deleted 3 old GEP artifacts.

Programmatic Usage

const { run } = require('./src/ops/cleanup');
const deletedCount = run();
console.log(`Deleted ${deletedCount} files.`);

Self-Repair Mechanisms

The evolver includes built-in self-repair capabilities:

Git Rollback

On evolution failure, the evolver automatically rolls back changes:

// In solidify.js
if (validationFailed && rollbackOnFailure) {
  execSync('git checkout -- .', { cwd: repoRoot });
  execSync('git clean -fd', { cwd: repoRoot });
  console.log('[Rollback] Changes reverted.');
}

Control rollback mode:

# Hard reset (default)
EVOLVER_ROLLBACK_MODE=hard node index.js

# Stash changes (recoverable)
EVOLVER_ROLLBACK_MODE=stash node index.js

# No rollback (debugging)
EVOLVER_ROLLBACK_MODE=none node index.js

Circuit Breaker

Prevents infinite repair loops by forcing innovation:

const repairRatio = getRepairRatioInLastNCycles(8);
if (repairRatio >= strategy.repairLoopThreshold) {
  console.log('[Circuit Breaker] Forcing innovation.');
  intent = 'innovate';
}

See Strategy Presets for threshold details.

Suicide Mechanism

In loop mode, the evolver restarts itself proactively:

if (cycleCount >= maxCycles || memMb > maxRssMb) {
  console.log(`[Daemon] Restarting self (cycles=${cycleCount}, rssMb=${memMb})`);
  spawn(process.execPath, [__filename, '--loop'], { detached: true });
  process.exit(0);
}

See Loop Mode for configuration.

Process Discovery

The lifecycle module discovers running evolver processes by scanning ps output:

function getRunningPids() {
  const out = execSync('ps -e -o pid,args', { encoding: 'utf8' });
  const pids = [];
  for (const line of out.split('\n')) {
    if (line.includes('node') && line.includes('index.js') && line.includes('--loop')) {
      pids.push(parseInt(line.split(/\s+/)[0], 10));
    }
  }
  return pids.filter(isPidRunning);
}

This works across platforms and doesn’t rely on PID files (which can be stale).

Wake Triggers

The evolver supports external wake triggers via the src/ops/trigger.js module (not detailed in source, but referenced in operations index). Expected usage:

node src/ops/trigger.js wake

This likely sends a signal to wake the evolver from sleep in loop mode.

Monitoring in Production

Recommended Setup

Process Manager

Use pm2, systemd, or Docker to manage the daemon:

pm2 start "node index.js --loop" --name evolver
pm2 startup  # Enable on boot
pm2 save

Health Check Cron

Run periodic health checks:

*/10 * * * * cd /path/to/evolver && node src/ops/lifecycle.js check >> /var/log/evolver-check.log 2>&1

Log Rotation

Configure logrotate for memory/evolver_loop.log:

/path/to/evolver/memory/evolver_loop.log {
  daily
  rotate 7
  compress
  missingok
  notifempty
}

Cleanup Cron

Run artifact cleanup daily:

0 2 * * * cd /path/to/evolver && node src/ops/cleanup.js >> /var/log/evolver-cleanup.log 2>&1

Monitoring Metrics

Key metrics to track:

Metric	Command	Threshold
Process uptime	`ps -p $(cat memory/evolver_loop.pid) -o etime=`	Greater than 0
Memory usage	`ps -p $(cat memory/evolver_loop.pid) -o rss=`	Less than 500MB
Log freshness	`stat -c %Y memory/evolver_loop.log`	Less than 1800s
Cycle rate	`grep "Starting cycle" memory/evolver_loop.log \| tail -10`	1-5/min
Error rate	`grep -c "Evolution failed" memory/evolver_loop.log`	Less than 10/day

Operations Module API

For programmatic access:

const ops = require('./src/ops');

// Lifecycle
ops.lifecycle.start();
ops.lifecycle.stop();
ops.lifecycle.restart();
ops.lifecycle.status();
ops.lifecycle.checkHealth();
ops.lifecycle.tailLog(50);

// Cleanup
ops.cleanup.run();

// (Other modules: skillsMonitor, trigger, commentary, selfRepair)

Troubleshooting

Daemon Won’t Start

Problem: start command returns “already_running” but no process is visible. Solution: Remove stale lock files:

rm memory/evolver_loop.pid
rm evolver.pid
node src/ops/lifecycle.js start

Health Check Fails

Problem: check reports unhealthy due to stagnation. Solution: Inspect the log for errors:

node src/ops/lifecycle.js log

Common causes:

Evolution saturated (expected in steady-state)
Git errors blocking solidify
Validation failures

Stop Command Hangs

Problem: stop command doesn’t return. Solution: Force kill the process:

pkill -9 -f "node.*index.js.*--loop"
rm memory/evolver_loop.pid evolver.pid

Best Practices

Use lifecycle commands: Don’t manage processes manually
Run health checks periodically: Catch stagnation early
Monitor logs: Set up alerting for “Evolution failed” patterns
Enable log rotation: Prevent disk exhaustion
Run cleanup regularly: Keep artifact directories lean
Test rollback: Verify EVOLVER_ROLLBACK_MODE=stash works in your environment

Next Steps

Loop Mode

Deep dive into daemon behavior

Review Mode

Manual approval workflow

Strategy Presets

Configure evolution focus

Running Evolver

Basic execution guide

Get Started

Core Concepts

Guides

Overview

Lifecycle Commands

start

stop

restart

status

check

log

Health Monitoring

Health Checks

Health Status

Cleanup Operations

What Gets Cleaned

Programmatic Usage

Self-Repair Mechanisms

Git Rollback

Circuit Breaker

Suicide Mechanism

Process Discovery

Wake Triggers

Monitoring in Production

Recommended Setup

Monitoring Metrics

Operations Module API

Troubleshooting

Daemon Won’t Start

Health Check Fails

Stop Command Hangs

Best Practices

Next Steps

Loop Mode

Review Mode

Strategy Presets

Running Evolver

Get Started

Core Concepts

Guides

Documentation Index

​Overview

​Lifecycle Commands

​start

​stop

​restart

​status

​check

​log

​Health Monitoring

​Health Checks

​Health Status

​Cleanup Operations

​What Gets Cleaned

​Programmatic Usage

​Self-Repair Mechanisms

​Git Rollback

​Circuit Breaker

​Suicide Mechanism

​Process Discovery

​Wake Triggers

​Monitoring in Production

​Recommended Setup

​Monitoring Metrics

​Operations Module API

​Troubleshooting

​Daemon Won’t Start

​Health Check Fails

​Stop Command Hangs

​Best Practices

​Next Steps

Loop Mode

Review Mode

Strategy Presets

Running Evolver

Overview

Lifecycle Commands

start

stop

restart

status

check

log

Health Monitoring

Health Checks

Health Status

Cleanup Operations

What Gets Cleaned

Programmatic Usage

Self-Repair Mechanisms

Git Rollback

Circuit Breaker

Suicide Mechanism

Process Discovery

Wake Triggers

Monitoring in Production

Recommended Setup

Monitoring Metrics

Operations Module API

Troubleshooting

Daemon Won’t Start

Health Check Fails

Stop Command Hangs

Best Practices

Next Steps