Upgrade a Reduced-Capacity, Online Cluster

You can upgrade a cluster from Enterprise Analytics to Operational Insights while it stays online, without a spare machine, if the cluster can serve its workload with one node fewer during the upgrade.

Understanding Reduced-Capacity Upgrade

This procedure frees one of the cluster’s own machines to act as the spare, then replaces each remaining Enterprise Analytics node with a freshly installed Operational Insights node, using swap rebalance. For the background, including how swap rebalance works and why each node is reinstalled rather than upgraded in place, see Upgrade Procedure Selection.

The procedure assumes that:

The cluster must continue to serve requests throughout the upgrade.
No spare machine is available, so the cluster runs with one node fewer until the last stage.
The cluster can serve its workload acceptably with one node fewer. If it cannot, use Upgrade a Full-Capacity, Online Cluster instead.
The cluster runs Enterprise Analytics 2.1.x (other than 2.1.0-MP3) or 2.2.x. For other versions, see Upgrade Paths.
Nodes are replaced one at a time.

The procedure has six stages.

Stage One: Prepare the Cluster

Confirm that the cluster is healthy: every node is active, and no rebalance is running or needed.
Make sure that the cluster can serve its workload with one node fewer.
Plan the upgrade for a time when brief pauses in ingestion are acceptable. Each rebalance suspends ingestion from the cluster’s links until it completes. After each rebalance, queries against the partitions that changed hands run with higher latency until the nodes that took them over have cached their data, which they do lazily, on first access. See Swap Rebalance.
Record the cluster’s configuration, so that you can compare it after the upgrade. This includes the databases, scopes, collections, and links, and the settings you have changed from their defaults.

Stage Two: Remove a Node

Rebalance one Enterprise Analytics node out of the cluster. Its machine becomes the spare for the rest of the procedure.

Run the command against a node that remains in the cluster. On an Enterprise Analytics node, couchbase-cli is in /opt/enterprise-analytics/bin:

couchbase-cli rebalance -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-remove ${OUTGOING_NODE}:8091

Wait for the rebalance to complete before you continue. For the Operational Insights Web Console and REST API equivalents, see Remove a Node and Rebalance.

From this point until Stage Six: Add Back the Last Node, the cluster serves requests with one node fewer than before.

Stage Three: Reinstall the Removed Machine

Prepare the machine whose Enterprise Analytics node has just left the cluster, by proceeding as follows:

Confirm that the rebalance that removed the node has completed successfully. The node is no longer listed on the Servers screen of the Operational Insights Web Console.

Do not continue on a machine whose node is still part of the cluster. The following steps delete the node’s local state.
Stop Enterprise Analytics on the machine:
```
systemctl stop enterprise-analytics
```
Uninstall Enterprise Analytics, including its configuration files:
- Red Hat-based distributions
- Ubuntu & Debian
dnf remove -y enterprise-analytics
If the package is held, release the hold first:
apt-mark unhold enterprise-analytics
Then remove the package:
apt-get purge -y enterprise-analytics
Delete the files that the package leaves behind. These belong to a node that’s no longer part of the cluster, and are not used again:
```
rm -rf /opt/enterprise-analytics
```
If the node’s data directory was outside /opt/enterprise-analytics, delete that directory too.

Operational Insights installs to /opt/couchbase. If that directory exists on the machine, for example because the node’s data directory was configured under it, delete it as well, so that the new installation starts empty:
```
rm -rf /opt/couchbase
```
Install Operational Insights 3.0 on the machine. Follow Install on Red Hat Enterprise or Install on Ubuntu and Debian, as appropriate for the platform.

Do not initialize the new node, or create a cluster on it. The node is configured when you add it to the cluster in the next stage.
Optionally, hold the package, so that no further upgrade occurs on the node until you choose to perform one:
- Red Hat-based distributions
- Ubuntu & Debian
Add the package name to the exclude line of /etc/dnf/dnf.conf (or /etc/yum.conf):
exclude=operational-insights
apt-mark hold operational-insights
Confirm that Operational Insights is running:
```
systemctl status operational-insights
```

Stage Four: Swap the Upgraded Node In

Add the Operational Insights node to the cluster, and remove one Enterprise Analytics node, in a single rebalance. Because the number of nodes added equals the number removed, Operational Insights performs a swap rebalance.

Every Operational Insights node runs the same fixed set of services. Do not specify services when you add the node: couchbase-cli rejects the --services option for Operational Insights and Enterprise Analytics clusters.

Run the commands against a node that remains in the cluster, rather than the node you’re removing. On an Enterprise Analytics node, couchbase-cli is in /opt/enterprise-analytics/bin. On an Operational Insights node, it’s in /opt/couchbase/bin.

Add the Operational Insights node. It’s not yet part of the cluster, and has no credentials of its own, so the values of --server-add-username and --server-add-password are placeholders:

couchbase-cli server-add -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-add https://${NEW_NODE}:18091 \
  --server-add-username Administrator \
  --server-add-password password

Rebalance, removing the Enterprise Analytics node that’s being replaced. The rebalance adds the pending node and removes the outgoing node together:
```
couchbase-cli rebalance -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-remove ${OUTGOING_NODE}:8091
```
Wait for the rebalance to complete before you continue. If it fails, resolve the cause and run the rebalance again: until it completes, the Enterprise Analytics node is still part of the cluster.

You can perform the same operation in the Web Console: on the Servers screen, click ADD SERVER to add the new node, click Remove on the outgoing node, then click Rebalance. For details of each method, see Add a Node and Rebalance and Remove a Node and Rebalance.

Stage Five: Repeat Stages Three and Four

Reinstall the machine of the node you just removed, as described in Stage Three: Reinstall the Removed Machine. Then, while any Enterprise Analytics node remains in the cluster, swap the reinstalled node in for the next one, as described in Stage Four: Swap the Upgraded Node In.

Continue until no Enterprise Analytics node remains in the cluster. One machine, the last one removed, is left outside the cluster, with Operational Insights installed.

Stage Six: Add Back the Last Node

Add the last machine back to the cluster, without removing any node, and rebalance. Because only a node is added, this is an ordinary rebalance rather than a swap rebalance:

couchbase-cli server-add -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-add https://${NEW_NODE}:18091 \
  --server-add-username Administrator \
  --server-add-password password

couchbase-cli rebalance -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD}

couchbase-cli is in /opt/couchbase/bin on every node at this point.

Confirm that the cluster has finished its upgrade. Shortly after the last Enterprise Analytics node leaves the cluster, the cluster moves to Operational Insights 3.0, and the Analytics Service restarts on every node to switch to the new version. Requests that are running at that moment fail, and can be retried.

To check, read the cluster’s compatibility version, which each node reports as 3.0.0 when the upgrade is complete:
```
curl -s -u ${USERNAME}:${PASSWORD} http://${CLUSTER_NODE}:8091/pools/default \
  | jq -r '.nodes[].prodCompatVersion'
```
Compare the cluster’s configuration with the record you made in Stage One: Prepare the Cluster.

The cluster is now fully upgraded, and back at full capacity. New features of Operational Insights 3.0 are now available. See Feature Availability During Upgrade.