Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Behavior During Node Failures

  • concept
How Cloud Native Gateway handles Couchbase Server node failures, Cloud Native Gateway instance failures, and the impact on client applications.

Couchbase Server Node Failure

When a Couchbase Server node fails, Cloud Native Gateway detects the failure and adapts its routing behavior.

KV (Data) Node Failure

  1. Connection failure detection - Cloud Native Gateway’s IO layer detects that connections to the failed node are broken (TCP reset, timeout, or connection refused).

  2. Request rerouting - Requests targeted at data nodes owned by the failed node will receive errors. Cloud Native Gateway’s retry logic may retry these operations if they’re safe to retry.

  3. Configuration update - Cloud Native Gateway receives an updated cluster configuration reflecting the failover. Once you update the configuration map, Cloud Native Gateway routes new requests to the nodes that have taken over for the failed node.

  4. Client impact - Clients may experience errors or increased latency during the failover window as Cloud Native Gateway detects the failure and receives the updated configuration. Once you update the configuration, operations resume automatically without any client action.

This behavior is dependent on how the failure is detected and handled by the Couchbase Server cluster in your deployment.

Query and Search Node Failure

You can access these services at their HTTP endpoints, and Cloud Native Gateway distributes requests across available endpoints:

  1. Cloud Native Gateway detects that the failed node’s service endpoint is unreachable.

  2. Subsequent requests are routed to other available endpoints for that service.

  3. Once the cluster configuration is updated to remove the failed node, Cloud Native Gateway stops attempting to use it.

For these services, the impact on clients is minimal as long as other nodes with the same service are available.

Cloud Native Gateway Instance Failure

Cloud Native Gateway instances can fail in different deployment scenarios. The following sections describe how failures are handled in each scenario.

Sidecar Deployment

In sidecar deployments, a Cloud Native Gateway instance failure means the Cloud Native Gateway container on a specific Couchbase pod has crashed or become unresponsive.

  1. Health check failure - The Kubernetes liveness probe detects the Cloud Native Gateway container failure.

  2. Container restart - Kubernetes automatically restarts the Cloud Native Gateway container.

  3. Service removal - While the Cloud Native Gateway container is restarting, Kubernetes removes the pod’s endpoint from the Cloud Native Gateway Service. Configured gateways (load balancers or routes) stop sending traffic to this instance.

  4. Client impact - Clients with active connections to the failed instance experience connection drops. These may manifest as errors in applications, or may be transparently retried automatically. The gRPC client in the SDK reconnects to another available Cloud Native Gateway instance (routed by a gateway or load balancer).

  5. Recovery - Once the Cloud Native Gateway container restarts and passes its health check, Kubernetes adds the endpoint back to the Service and the load balancer resumes sending traffic.

Since there are multiple Cloud Native Gateway instances (1 per Couchbase pod), the failure of a single instance does not cause an outage. Instead, traffic is redistributed to the remaining instances.

Standalone Deployment

In standalone deployments with multiple Cloud Native Gateway replicas behind a load balancer:

  1. The load balancer’s health check detects the failed instance.

  2. Traffic is redirected to healthy instances.

  3. Kubernetes (or the orchestrator) restarts the failed instance.

  4. Clients reconnect through the load balancer.

Graceful Shutdown

Cloud Native Gateway implements graceful shutdown to minimize disruption during planned maintenance. By sending a shutdown signal to the Cloud Native Gateway process, the following actions occur:

  1. Stop accepting new connections - Cloud Native Gateway stops accepting new gRPC connections and HTTP requests.

  2. Drain existing requests - In-flight requests are allowed to complete within the shutdown timeout (default 30 seconds).

  3. gRPC graceful stop - The gRPC server performs a GracefulStop, allowing active RPCs to finish.

  4. HTTP keep-alive disable - The Data API server disables keep-alive connections and waits for active requests to complete.

  5. Force shutdown - If requests do not complete within the timeout, remaining connections are forcefully closed.

In Kubernetes, this is triggered by the pod receiving a SIGTERM signal during a rolling update or scaling event. Configure the pod’s terminationGracePeriodSeconds to be at least as long as Cloud Native Gateway’s shutdown timeout to allow graceful drainage.