Viewing cluster status

The Central Management Dashboard provides instant visibility into IVX cluster status. You can drill down to other pages in the Central Management Web UI to view details. You can also use Central Management and Virtual Execution CLI commands to view status information.

Prerequisites

Monitor, Operator, or Admin access

Viewing the cluster status using the Central Management dashboard

Use the Central Management Dashboard to check the health of the cluster and its components.

A healthy cluster has the following characteristics:

Virtual ExecutionIVX appliances (nodes) are connected to the Central Management appliance.
The cluster status in Ready status.
The cluster utilization percentage is in the normal range.
The nodes are in Ready status.
There is at least one broker.
Sensors that are managed by the Central Management appliance that manages the cluster are connected and healthy.
Hybrid appliances that are connected to the Central Management appliance that manages the cluster are healthy.

To view cluster status:

Log in to the Central Management Web UI.
Click the Dashboard tab, if it is not already displayed.
Select the cluster from the group menu.
Review the information in the Summary pane.
Click the cluster name, and review the high-level information about the cluster.
Click the cluster name again to open the Details dialog box. View more detailed information on the Summary tab.
Click the Nodes tab to view information about the nodes.
Click the node name to view more detailed information about the node.
Click the Enrolled Sensors tab to view information about the sensors and hybrid appliances.

Viewing cluster status using the Central Management Web UI

Use the Appliances > Clusters page in the Central Management Web UI to view a list of the clusters this Central Management appliance manages and their overall status. You can then click a cluster name for more detail.

The following table describes the cluster status fields.

Field	Description
Name	The name of the cluster.
Connection	The state of the connection between the nodes and the cluster. OK or Connected—At least one node is connected to the cluster. Disconnected—No node is connected to the cluster.
Health	OK or Healthy—The cluster and the individual nodes are healthy. Warning—Configuration settings are not the same on all nodes. Critical—The cluster does not have a broker node.

Field	Description
Utilization (hourly)	The average cluster utilization level over the last hour: Healthy, Warning, Critical. See Viewing cluster utilization for information about the three levels.
OS	The Intelligent Virtual Execution - Server system image version that is running on the nodes.
Guest Image	The guest images version installed on the nodes.

Field

Description

Utilization (hourly)

The average cluster utilization level over the last hour: Healthy, Warning, Critical.

See Viewing cluster utilization for information about the three levels.

The Intelligent Virtual Execution - Server system image version that is running on the nodes.

Guest Image

The guest images version installed on the nodes.

To view cluster status:

Log in to the Central Management Web UI.
Select Appliances > Clusters.
Click the cluster name in the Name column to view additional details about the cluster, the nodes, and the enrolled sensors.

Viewing cluster status using the Central Management CLI

Use the commands in this section to monitor the status of the cluster and its components.

To view cluster status:

Log in to the Central Management CLI.
Enable the CLI enable mode:
```
cm-hostname > enable
```
To view summary information:
```
cm-hostname # show cmc mvx cluster
```

To view brief information:

cm-hostname # show cmc mvx cluster brief

To view detailed information about a specific cluster:
```
cm-hostname # show cmc mvx cluster <cluster name>
```
To view information about the nodes in a specific cluster:
```
cm-hostname # show cmc mvx cluster <cluster name> nodes
```

Examples

The following example shows summary information about the Cluster-Acme cluster (the only cluster being managed by this Central Management appliance).

cm-1 # show cmc mvx cluster

MVX Cluster                          : Cluster-Acme

    Health OK                        : yes
    Health severity                  : OK
    Master broker                    : vx-1
    Member node count                : 2
    All connected                    : yes
    Description:

The following example shows brief information about the Cluster-Acme cluster (the only
cluster being managed by this Central Management appliance).

cm-1 # show cmc mvx cluster brief

CLUSTER NAME     HEALTHY CONNECTED NODES MASTER
------------     ------- --------- ----- ------
Cluster-Acme     yes     yes        2    vx-1

The following example shows detailed information about the Cluster-Acme cluster.

cm-1 (config) # show cmc mvx cluster detail

MVX cluster                        : Cluster-Acme

    Health OK                      : yes
    Health severity                : OK
    Master broker                  : vx-1
    Member node count              : 2
    All connected                  : yes
    Description:

Health Status:
    Nodes connected all            : yes
    System configuration in sync   : yes
    System software version match  : yes
    Security content version match : yes
    Guest-images version match     : yes
    Master Node Selected           : yes
    Broker selected                : yes

Update Status:
    Latest OS version installed    : yes
    GI update available            : no

Member Status (Total 2 Nodes):
    Brokers:
        vx-1 (master) 10.11.121.12 ok

Compute Nodes:
        vx-2 10.11.121.18 ok

The following information shows information about the nodes in the Cluster-Acme cluster.

cm-1 # show cmc mvx cluster Cluster-Acme nodes

NODES: Cluster-Acme CONNECTED     HEALTHY     ADDRESS
--------            ---------     -------     -------
vx-1                yes           yes         10.11.121.12
vx-2                yes           yes         10.11.121.18
Brokers (active)
vx-1                yes           yes         10.11.121.12
Brokers (ready)
vx-2                yes           yes         10.11.121.18

Output fields

Field	Description
Health OK	Whether the cluster is healthy.
Health severity	Severity of the cluster's health condition: OK—The cluster is healthy. NOT READY—The cluster has no master configuration. WARNING—The configuration settings on the nodes do not match and need to be synchronized with the master configuration, the security content versions on the nodes do not match, or the node is not healthy. DEGRADED—One or more nodes in the cluster are not connected to the Central Management appliance, or the nodes are not running the same system image. CRITICAL—The cluster has no broker.
Master broker	Hostname of the master broker.
Member node count	Number of nodes in the cluster (including both brokers and compute nodes).
All connected	Whether all members of the cluster are connected.
Description	Description of the cluster (if one was provided).
Health Status
Nodes connected all	Whether all nodes are connected.
System configuration in sync	Whether relevant detection-related configuration settings match on all nodes.
System software version match	Whether the same Virtual Execution software version is running on all nodes.
Security content version match	Whether the same security content version is running on all nodes.
Guest- images version match	Whether the same versions of guest images are installed on all nodes.
Master Node Selected	Whether the master configuration is defined.
Broker selected	Whether a broker is defined.
Update Status
Latest OS version installed	Whether the latest Virtual Execution software version is running on all nodes.
GI update available	Whether the latest version of guest images is running on all nodes.
Member Status	The hostname, IP address, and health status of each broker node. The health status can be ok, WARN, or CRIT.
Brokers	The hostname, IP address, and health status of each broker node. The health status can be ok, WARN, or CRIT.
Compute Nodes	The hostname, IP address, and health status of each compute node. The health status can be ok, WARN, or CRIT.

Viewing cluster status using the Virtual Execution CLI

Use the commands in this section to monitor the status of a cluster and its member nodes.

To view cluster status:

Log in to the Virtual Execution CLI.
Enable the CLI enable mode:
```
vx-hostname > enable
```
To view cluster status:
```
vx-hostname # show mvx cluster status
```
To view detailed information about each node in the cluster:
```
vx-hostname # show mvx cluster status detail
```

Examples

The following example shows the status of the Cluster-Acme cluster from the vx-1 member node.

vx-1 # show mvx cluster status

MVX Cluster Status

    Cluster Name : Cluster-Acme
    Cluster Health : ok
    Cluster Size : 3
    Cluster Utilization : 48 %

The following example shows the status of the Cluster-Acme cluster and its nodes from the
vx-2 member node.

vx-2 # show mvx cluster status detail

MVX Cluster Status

    Cluster Name : Cluster-Acme
    Cluster Health : ok
    Cluster Size : 3
    Cluster Utilization : 57 %

MVX Cluster Nodes

    Node Name : vx-1
    Node Health : ok
    Node Address : 10.1.1.1
    Node Role : broker
    Node Utilization : 65 %

    Node Name : vx-2
    Node Health : ok
    Node Address : 10.1.1.2
    Node Role : node
    Node Utilization : 96 %

    Node Name : vx-3
    Node Health : ok
    Node Address : 10.1.1.3
    Node Role : node
    Node Utilization : 83 %

The following example shows the status from the vx-4 node, which is not yet a member of
a cluster.

vx-4 # show mvx cluster status

MVX Cluster nodes status is not yet available.

Output fields

Field	Description
IVX Cluster status
Cluster Name	The cluster name.
Cluster Health	The cluster health: ok—The cluster is healthy. degraded—One or more nodes are unhealthy or offline. critical—All nodes are unhealthy or no broker is available.
Cluster Size	The number of nodes in the cluster.
Cluster Utilization	The current cluster utilization, as a percentage of the capacity of the whole cluster. This calculation is based on the cluster's submission queue size and the member nodes' VM load. NOTE: Any node in a critical health state is not included in this calculation.
IVX Cluster nodes
Node Name	The node name.
Node Health	The node health: ok—The node is healthy. critical—Some major functionality is not working (for example, there is no broker node) or the node is offline (for example, it is reloading after an upgrade).
Node Address	The IP address of the Virtual Execution appliance (node).
Node Role	The role of the node in the cluster: broker—broker node node—compute node
Node Utilization	The current node utilization, as a percentage of capacity. This calculation is based on the node's VM load.

Viewing cluster database statistics using the Virtual Execution CLI

Use the commands in this section to view cluster database statistics for each broker in an IVX cluster.

To view cluster database statistics:

Log in to the CLI of any broker (Virtual Execution appliance) in the cluster.
Enable the CLI enable mode:
```
vx-hostname > enable
```

View the statistics:

vx-hostname # show mvx node cdbmgr status

Examples

The following example from the vx-1 broker shows the statistics for it and the other broker in the cluster.

vx-1 # show mvx node cdbmgr status

    CdbMgr Node Stats:
        Ip Address                                 : 10.11.121.12
        CDB Node ID                                : 7xxxxxxx-bxxx-4xxx-gxxx-2xxxxx
        Status                                     : Up and running
        Load (Data size)                           : 94.29 KB
        Owns (% of cluster data stored on node)    : 100.0%

        Ip Address                                 : 10.11.121.18
        CDB Node ID                                : bxxxxxxx-cxxx-5xxx-axxx-2xxxxx
        Status                                     : Up and running
        Load (Data size)                           : 70.33 KB
        Owns (% of cluster data stored on node)    : 100.0%

The following example shows the output of the command on a compute node that is not a
broker.

vx-2 # show mvx node cdbmgr status
% Broker functionality is disabled

Output fields

Field	Description
Ip Address	The IP address of the broker.
CDB Node ID	The unique identifier for the database node.
Status	The current status of the broker: Up and running—The broker is running normally. Up and leaving cluster—The broker is running and is leaving the cluster. Up and joining cluster—The broker is running and is joining the cluster. Node is down—The broker is not running. Down and joining cluster—The broker is not running and is joining the cluster. Down and leaving cluster—The broker is not running and is leaving the cluster.
Load (Data size)	The total amount of data this broker is handling. The data includes submission data from sensors and analysis results from compute nodes.
Owns (% of cluster data stored on node)	The percentage of the overall cluster data stored on this broker. For example, 100.0% means all submission and analysis results from all member brokers have been replicated to storage on this broker.