Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Upgrade a Full-Capacity, Online Cluster

    You can upgrade a cluster from Enterprise Analytics to Operational Insights while it stays online and at full capacity, by using one spare machine.

    Understanding Full-Capacity Upgrade

    This procedure replaces each Enterprise Analytics node with a freshly installed Operational Insights node, using swap rebalance, so that the cluster keeps the same number of nodes throughout. For the background, including how swap rebalance works and why each node is reinstalled rather than upgraded in place, see Upgrade Procedure Selection.

    The procedure assumes that:

    • The cluster must continue to serve requests throughout the upgrade.

    • One spare machine is available, which is not part of the cluster. It meets the system requirements, and can reach the cluster’s nodes on the required ports.

    • The cluster runs Enterprise Analytics 2.1.x (other than 2.1.0-MP3) or 2.2.x. For other versions, see Upgrade Paths.

    • Nodes are replaced one at a time.

    The procedure has six stages.

    Stage One: Prepare the Spare Node

    Install Operational Insights 3.0 on the spare machine. Follow Install on Red Hat Enterprise or Install on Ubuntu and Debian, as appropriate for the platform.

    Do not initialize the new node, or create a cluster on it. The node is configured when you add it to the cluster.

    Stage Two: Prepare the Cluster

    1. Confirm that the cluster is healthy: every node is active, and no rebalance is running or needed.

    2. Plan the upgrade for a time when brief pauses in ingestion are acceptable. Each rebalance suspends ingestion from the cluster’s links until it completes. After each rebalance, queries against the partitions that changed hands run with higher latency until the nodes that took them over have cached their data, which they do lazily, on first access. See Swap Rebalance.

    3. Record the cluster’s configuration, so that you can compare it after the upgrade. This includes the databases, scopes, collections, and links, and the settings you have changed from their defaults.

    Stage Three: Swap the Spare Node In

    Add the Operational Insights node to the cluster, and remove one Enterprise Analytics node, in a single rebalance. Because the number of nodes added equals the number removed, Operational Insights performs a swap rebalance.

    Every Operational Insights node runs the same fixed set of services. Do not specify services when you add the node: couchbase-cli rejects the --services option for Operational Insights and Enterprise Analytics clusters.

    Run the commands against a node that remains in the cluster, rather than the node you’re removing. On an Enterprise Analytics node, couchbase-cli is in /opt/enterprise-analytics/bin. On an Operational Insights node, it’s in /opt/couchbase/bin.

    1. Add the Operational Insights node. It’s not yet part of the cluster, and has no credentials of its own, so the values of --server-add-username and --server-add-password are placeholders:

      couchbase-cli server-add -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-add https://${NEW_NODE}:18091 \
  --server-add-username Administrator \
  --server-add-password password

    2. Rebalance, removing the Enterprise Analytics node that’s being replaced. The rebalance adds the pending node and removes the outgoing node together:

      couchbase-cli rebalance -c ${CLUSTER_NODE}:8091 \
  -u ${USERNAME} -p ${PASSWORD} \
  --server-remove ${OUTGOING_NODE}:8091

    3. Wait for the rebalance to complete before you continue. If it fails, resolve the cause and run the rebalance again: until it completes, the Enterprise Analytics node is still part of the cluster.

    You can perform the same operation in the Web Console: on the Servers screen, click ADD SERVER to add the new node, click Remove on the outgoing node, then click Rebalance. For details of each method, see Add a Node and Rebalance and Remove a Node and Rebalance.

    The cluster now has the same number of nodes as before, one of which runs Operational Insights.

    Stage Four: Reinstall the Removed Machine

    Prepare the machine whose Enterprise Analytics node has just left the cluster, by proceeding as follows:

    1. Confirm that the rebalance that removed the node has completed successfully. The node is no longer listed on the Servers screen of the Operational Insights Web Console.

      Do not continue on a machine whose node is still part of the cluster. The following steps delete the node’s local state.

    2. Stop Enterprise Analytics on the machine:

      systemctl stop enterprise-analytics

    3. Uninstall Enterprise Analytics, including its configuration files:

      • Red Hat-based distributions

      • Ubuntu & Debian

      dnf remove -y enterprise-analytics

      If the package is held, release the hold first:

      apt-mark unhold enterprise-analytics

      Then remove the package:

      apt-get purge -y enterprise-analytics

    4. Delete the files that the package leaves behind. These belong to a node that’s no longer part of the cluster, and are not used again:

      rm -rf /opt/enterprise-analytics

      If the node’s data directory was outside /opt/enterprise-analytics, delete that directory too.

      Operational Insights installs to /opt/couchbase. If that directory exists on the machine, for example because the node’s data directory was configured under it, delete it as well, so that the new installation starts empty:

      rm -rf /opt/couchbase

    5. Install Operational Insights 3.0 on the machine. Follow Install on Red Hat Enterprise or Install on Ubuntu and Debian, as appropriate for the platform.

      Do not initialize the new node, or create a cluster on it. The node is configured when you add it to the cluster in the next stage.

    6. Optionally, hold the package, so that no further upgrade occurs on the node until you choose to perform one:

      • Red Hat-based distributions

      • Ubuntu & Debian

      Add the package name to the exclude line of /etc/dnf/dnf.conf (or /etc/yum.conf):

      exclude=operational-insights
      apt-mark hold operational-insights

    7. Confirm that Operational Insights is running:

      systemctl status operational-insights

    Stage Five: Repeat Stages Three and Four

    Swap the newly installed Operational Insights node in for the next Enterprise Analytics node, as described in Stage Three: Swap the Spare Node In. Then reinstall the machine that node ran on, as described in Stage Four: Reinstall the Removed Machine.

    Continue until no Enterprise Analytics node remains in the cluster. One machine, the last one removed, is left outside the cluster, with Operational Insights installed.

    Stage Six: Finish the Upgrade

    1. Confirm that the cluster has finished its upgrade. Shortly after the last Enterprise Analytics node leaves the cluster, the cluster moves to Operational Insights 3.0, and the Analytics Service restarts on every node to switch to the new version. Requests that are running at that moment fail, and can be retried.

      To check, read the cluster’s compatibility version, which each node reports as 3.0.0 when the upgrade is complete:

      curl -s -u ${USERNAME}:${PASSWORD} http://${CLUSTER_NODE}:8091/pools/default \
  | jq -r '.nodes[].prodCompatVersion'

    2. Compare the cluster’s configuration with the record you made in Stage Two: Prepare the Cluster.

    3. Repurpose the machine that remains outside the cluster, or uninstall Operational Insights from it. See Uninstall.

    The upgrade is complete. New features of Operational Insights 3.0 are now available. See Feature Availability During Upgrade.