Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Scale Agent Memory Using Multiple Nodes

  • how-to
Run several Agent Memory nodes that share a Couchbase bucket, so your deployment handles more client traffic and keeps processing memories when nodes fail.

By default, Agent Memory runs as a single node. Cluster mode lets you run several Agent Memory nodes that share 1 Couchbase bucket. Each node processes the memory blocks it ingests, and if a node stops, the surviving nodes take over any unfinished work. For more information about how nodes share work, recover from failures, and split rate limits, see Cluster Mode.

To create an Agent Memory cluster, complete the following steps:

  1. Set up the nodes.

  2. Configure the load balancer.

  3. Start the cluster.

  4. Verify the cluster deployment.

Converting or Expanding a Deployment

  • If you’re converting an existing deployment to cluster mode, keep it running while you complete Step 1 and Step 2. Changes to the .env and docker-compose.yml files you created when you deployed Agent Memory take effect only when you recreate its container in Step 3. Plan a maintenance window for Step 3 because clients must stop writing to Agent Memory until the cluster is running.

  • To add capacity to a running Agent Memory cluster, see Add Nodes to a Cluster.

Prerequisites

Before you start using Agent Memory’s cluster mode, confirm the following:

  • You have a server for each node, with Docker and Docker Compose installed. The servers share a private network with the load balancer.

  • Every node runs the same version of Agent Memory.

  • Your Couchbase cluster and bucket are prepared. For more information, see Prepare Your Couchbase Cluster.

  • You have a load balancer, such as nginx, that can terminate HTTPS, and a TLS certificate for your domain.

  • You know the requests-per-minute and tokens-per-minute limits of your model provider account.

Step 1: Set Up Nodes

Set up each node that you need for your cluster, but do not start them until Step 3.

To set up your nodes, continue from Step 2 of Deploy Agent Memory for Production, but make the following changes to the docker-compose.yml and .env files.

Update the docker-compose.yml File

Use the following docker-compose.yml file. Its image value is the pre-built Agent Memory image that you load and tag in Step 2 of Deploy Agent Memory for Production. It has no HTTPS proxy, and it publishes port 8080 only on the server’s private address. Replace <PRIVATE_IP> with that address.

name: agentmemory

services:
  agentmemory-server:
    image: agentmemory-server:latest
    container_name: agentmemory-server
    ports:
      - "<PRIVATE_IP>:8080:8080"
    env_file:
      - .env
    environment:
      - AGENTMEMORY_SERVER_HOST=0.0.0.0
      - AGENTMEMORY_SERVER_PORT=8080
    volumes:
      - agentmemory-logs:/app/logs
      - agentmemory-prometheus:/prometheus-data
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health').read()"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped

volumes:
  agentmemory-logs:
  agentmemory-prometheus:

Update the .env File

In each node’s .env file, enable cluster mode and set the rate limits for the whole cluster, not a per-node share:

AGENTMEMORY_CLUSTER_MODE=true
AGENTMEMORY_MAX_REQUESTS_PER_MINUTE=<REQUESTS_PER_MINUTE>
AGENTMEMORY_MAX_TOKENS_PER_MINUTE=<TOKENS_PER_MINUTE>

Set these limits to about 90-95% of your model provider’s limits. This leaves headroom for brief over-counting when nodes join or leave, and for other usage on the same provider account. Use the same .env values on every node for the Couchbase connection, bucket, model providers, rate limits, and cluster mode settings. For more information, see cluster mode and Rate Limiting and Throughput.

Step 2: Configure the Load Balancer

A cluster has a load balancer in front of its nodes, and every node connects to the same Couchbase bucket. Clients connect only to the load balancer, and each node runs on its own server without its own HTTPS proxy.

To use Agent Memory’s cluster mode, you must configure your load balancer to:

  • Terminate HTTPS for your domain.

  • Forward requests to port 8080 on each node’s private address.

  • Stop routing to a node that fails, and send its requests to the other nodes.

Agent Memory does not distribute ingestion or rebalance work between nodes, so distribute requests evenly, for example with round-robin routing. An uneven load balancer leaves some nodes idle while others build a queue.

The following example configures nginx. Replace <NODE_1_IP>, <NODE_2_IP>, and <NODE_3_IP> with each node’s private address, <YOUR_DOMAIN> with your domain, and <CERTIFICATE_PATH> and <PRIVATE_KEY_PATH> with the paths to your TLS certificate and private key.

upstream agentmemory {
    server <NODE_1_IP>:8080 max_fails=2 fail_timeout=10s;
    server <NODE_2_IP>:8080 max_fails=2 fail_timeout=10s;
    server <NODE_3_IP>:8080 max_fails=2 fail_timeout=10s;
}

server {
    listen 443 ssl;
    server_name <YOUR_DOMAIN>;

    ssl_certificate     <CERTIFICATE_PATH>;
    ssl_certificate_key <PRIVATE_KEY_PATH>;

    location / {
        proxy_pass http://agentmemory;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_read_timeout 300s;
    }
}

With this configuration, nginx stops sending requests to a node after 2 failed attempts within 10 seconds, and tries that node again after 10 seconds.

The proxy_next_upstream directive retries a failed request on another node. By default, nginx does not resend a request that creates data, such as a POST that adds a memory, after it sends the request to a node.

The proxy_read_timeout value allows time for synchronous memory processing, which waits for the model provider.

If your load balancer supports active health checks, point them at each node’s /health endpoint. /health only confirms that the server process responds. It does not check the Couchbase connection or the model providers.

Step 3: Start the Cluster

  • New Deployment

  • Convert Existing Deployment

Use these steps if you’re creating a new Agent Memory deployment that uses cluster mode:

  1. Start each node by running the following command in its project directory.

    docker compose up -d

  2. Point your domain at the load balancer.

  3. Verify the cluster deployment.

Do not start any new nodes until you finish draining the backlog. A node that starts without cluster mode re-queues every unfinished block, including blocks the existing node is processing. A node that starts with cluster mode does not count the existing node, so together they exceed your rate limits.

Use the following steps if you’re converting an existing Agent Memory deployment to use cluster mode:

  1. Drain the extraction backlog of your existing node.

    Blocks ingested before you enable cluster mode are not owned by any node in the cluster. If they’re still waiting for extraction, cluster mode never recovers them.

    1. Stop sending new writes to Agent Memory, for example by pausing your clients.

    2. Keep the existing node running in single-node mode until both size and processing in the queue object are 0:

      curl -s https://$YOUR_DOMAIN/health/async-batch-processor-stats | jq '.queue'

      If you have enabled authentication, add -H "Authorization: Bearer $TOKEN" with a token that carries the ams_ro role or higher. For more information about getting a token, see Configure Authentication for Self-Managed Agent Memory.

    3. Confirm that no unprocessed blocks remain in Couchbase. Run the following SQL++ query in the Capella Query Workbench, the Couchbase Server Query Workbench, or the cbq shell, with credentials that can read the bucket:

      SELECT COUNT(*) AS pending
FROM `<BUCKET_NAME>`.agentmemory.memory
WHERE status != 'ready'
  AND IFMISSINGORNULL(fail_count, 0) < 3;
      If you changed AGENTMEMORY_MAX_FAIL_COUNT from its default of 3, use your value instead.

      The query returns 0 when the backlog is drained. If the query returns a number greater than 0 while the queue is empty, restart the node with docker restart agentmemory-server to re-queue the remaining blocks. Wait until size and processing in the queue object are 0 again, then rerun the query. docker restart keeps the node’s current configuration, so the node stays reachable at your domain.

  2. Point your domain at the load balancer.

  3. Restart the existing node and start each new node using the following commands in each node’s project directory:

    docker compose down --remove-orphans && docker compose up -d

    The --remove-orphans option removes containers that are no longer in the docker-compose.yml file, such as a Traefik or nginx proxy. If you used Cloudflare Tunnel, also stop the cloudflared tunnel on the server.

  4. Verify the cluster deployment, then resume your clients.

Step 4: Verify the Cluster Deployment

To verify that a cluster is operating, wait 30 seconds after the last node starts, then run the following command on each node’s host. If your container is not named agentmemory-server, replace it with your container name.

docker exec agentmemory-server python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8080/metrics').read().decode())" | grep '^agentmemory_cluster_live_node_count'

The output shows the number of nodes you started, for example agentmemory_cluster_live_node_count 3.0 for a 3-node cluster. If this reports a lower number than you’re expecting, see Troubleshooting.

Add Nodes to a Cluster

You can add a node to a running cluster without restarting the other nodes.

  1. Set up the node, as described in Step 1: Set Up Nodes, with the same .env values as the other nodes, including AGENTMEMORY_CLUSTER_MODE=true.

  2. Start the node by running docker compose up -d in its project directory.

  3. Add the node’s private address to your load balancer, and reload the load balancer’s configuration.

  4. If you scrape metrics with an external Prometheus server, add the node to its scrape configuration.

  5. Verify the cluster. For more information, see Step 4: Verify the Cluster Deployment.

Monitor a Cluster

Each node’s embedded Prometheus instance collects metrics only from that node. To view the whole cluster in a single place, use an external Prometheus server.

  1. Add the /metrics endpoint of every node to your Prometheus server’s scrape configuration. Use each node’s private address, and run Prometheus on the same private network.

    scrape_configs:
  - job_name: agentmemory
    metrics_path: /metrics
    static_configs:
      - targets:
          - <NODE_1_IP>:8080
          - <NODE_2_IP>:8080
          - <NODE_3_IP>:8080

  2. Confirm that Prometheus reaches every node.

    curl -s "http://$PROMETHEUS_HOST:9090/api/v1/targets?state=active" | jq -r '.data.activeTargets[] | "\(.labels.instance) \(.health)"'

    Each node appears with the health up.

  3. To tell nodes apart in queries, join a metric to agentmemory_node_info on the instance label. For example, the following query shows the live node count that each node reports, labeled with its node ID:

    agentmemory_cluster_live_node_count * on(instance) group_left(node_id) agentmemory_node_info

Agent Memory exposes metrics for heartbeats, dead-node recovery, and each node’s rate limit share. Use them to build dashboards and alerts for the cluster.

For all of the cluster mode metrics that are available, expand the following section:

Show All Cluster Mode Metrics
Metric Type Description

agentmemory_cluster_live_node_count

Gauge

Number of live nodes as seen by this node.

agentmemory_cluster_per_node_rpm

Gauge

This node’s share of AGENTMEMORY_MAX_REQUESTS_PER_MINUTE.

agentmemory_node_info

Gauge

Carries this node’s ID in the node_id label. The value is always 1.

agentmemory_heartbeat_total

Counter

Heartbeat attempts, labeled by result: success, cas_mismatch, or error.

agentmemory_zombie_detected_total

Counter

Times this node found that a peer had claimed it and shut itself down.

agentmemory_recovery_sweep_total

Counter

Dead-node recovery sweeps this node has run.

agentmemory_recovery_sweep_dead_nodes_found_total

Counter

Dead nodes found across all sweeps.

agentmemory_recovery_sweep_claim_total

Counter

Attempts to claim a dead node, labeled by result: success, lost_race, no_blocks, enqueue_failed, or skipped_full_queue.

agentmemory_recovery_sweep_blocks_recovered_total

Counter

Blocks this node took over from dead nodes.

agentmemory_own_lineage_recovery_total

Counter

Recovery passes over this node’s own unfinished blocks, labeled by result: work_done, nothing, or skipped_full_queue.

agentmemory_own_lineage_blocks_recovered_total

Counter

Blocks re-queued by those recovery passes.

agentmemory_startup_own_blocks_recovered_total

Counter

Blocks this node recovered at startup.

In cluster mode, agentmemory_health_check_status{component="couchbase"} reflects heartbeat success. agentmemory_async_batch_process_outcomes_total reports result="lease_expired" when a node discards an extraction result after it loses contact with Couchbase.

Collect Diagnostics from a Cluster

To collect support diagnostics from every node in a single archive, use agentmemory-collect:

  1. On a chosen node’s host, run agentmemory-collect and pass the node URLs with --nodes.

    docker exec agentmemory-server agentmemory-collect \
  --customer $CUSTOMER_NAME \
  --ticket-id $TICKET_ID \
  --nodes http://$NODE_1_IP:8080,http://$NODE_2_IP:8080,http://$NODE_3_IP:8080

    The command uploads the archive to Couchbase Support, so you need an open support ticket. Replace the following placeholders.

    • $CUSTOMER_NAME: Your company name, using up to 50 letters, numbers, underscores, periods, or hyphens.

    • $TICKET_ID: Your Couchbase Support ticket number, up to 7 digits.

    • $NODE_1_IP, $NODE_2_IP, and $NODE_3_IP: Each node’s private address.

      If you have enabled authentication, add --auth-token with a token that each node accepts. If you have also enabled authorization, the token must carry the ams_admin role.

  2. Check the command output for errors.

    The archive leaves out any node that the command cannot reach, and the output shows Collected <URL>: error for that node.

For more information, see Collecting Support Diagnostics.

Troubleshooting

If a node reports fewer live nodes than you started

Confirm that every node registered with the cluster by running the following command in each node’s project directory.

docker compose logs agentmemory-server | grep "Node document written"

This log entry requires LOG_LEVEL set to INFO or DEBUG. INFO is the default. A node that does not show this log entry did not register. Check its AGENTMEMORY_CLUSTER_MODE value and its Couchbase connection settings.

Next Steps