High availability Receivers are used in primary and secondary mode so that the secondary Receiver can take over functions when the primary Receiver fails, providing better data collection than provided by a single Receiver.
Important
If you are required to comply with FIPS regulations, do not use this feature. High Availability Receivers are not FIPS-compliant.
Setting up high availability Receiver requires two Receivers: one acting as the primary or preferred primary and the other as secondary to monitor the primary continuously. When the secondary determines that the primary has failed, it stops the primary and takes over its function.
The IPMI cards eliminate the possibility of both DS NICs using the shared IP address and MAC at the same time by shutting down the failed receiver. The IPMI cards are connected with a cross-over or straight-through cable to the other Receiver. The Receivers are connected with a cross-over or straight-through cable on the heartbeat NIC. There is a management NIC for communication with the Trellix ESM, and a data source NIC for collecting data.
Note
The Heartbeat connection requires a network latency of 75 ms or better to prevent High Availability Receivers fail over and any issues. Any network latency issue requires Customer Support to fix the high availability receiver and improve the network.
When the primary Receiver is running properly and the secondary Receiver is in secondary mode, this is happening:
The Receivers communicate constantly over the dedicated heartbeat NIC and the management NIC.
Any certificates that are received, such as OPSEC or Estreamer, are passed to the other Receiver in the pair.
All data sources use the data source NIC.
Each Receiver monitors and reports its own health. This includes internal health items like disk errors, database crashes, and lost links on NICs.
Trellix ESM communicates with the receivers periodically to determine their status and health.
Any new configuration information is sent to both the primary and secondary receiver.
Trellix ESM sends policy to both the primary and secondary receiver.
Stop/Reboot/Call Home apply to each receiver independently.
Primary Receiver failure
Determination of primary Receiver failure is the responsibility of the secondary receiver. It must determine that failure quickly and accurately to minimize data loss. On failover, all data since the primary last sent data to Trellix ESM and Trellix Enterprise Security Manager - Enterprise Log Manager is lost. The amount of data lost depends on the throughput of the Receiver and the rate at which Trellix ESM pulls data from the Receiver. These competing processes must be carefully balanced to optimize data availability.
When the primary Receiver fails completely (power loss, CPU failure) there is no heartbeat communication with the primary Receiver. Corosync recognizes the loss of communication and marks the primary Receiver as failed. Pacemaker on the secondary Receiver requests that the IPMI card on the primary Receiver shut down the primary Receiver. The secondary Receiver then assumes the shared IP address and MAC address, and starts all collectors.
Secondary Receiver failure
The secondary failure process occurs when the secondary Receiver is no longer responding to the heartbeat communication. This means the system has been unable to communicate with the secondary Receiver after trying to do so for a time using the management and heartbeat interfaces.
If the primary is unable to get heartbeat and integrity signals, corosync marks the secondary as failed and pacemaker uses the secondary's IPMI card to shut it down.
Primary health problem
There are two reasons for a fail-over:
Loss of heartbeat, caused by bad NIC or cable, or dead or unresponsive or rebooting Trellix Enterprise Security Manager - Event Receiver
Database is no longer running or has no PID
Secondary health problem
When the health of the secondary Receiver is severely compromised, this occurs:
The secondary Receiver reports health problems to Trellix ESM when queried and kills the corosync and pacemaker processes.
If the secondary Receiver is still part of the cluster, it removes itself from the cluster and is unavailable if there is a primary Receiver failure.
The health problem is analyzed and a repair tried.
If the health problem is resolved, the Receiver is returned to normal operation using the Return to service procedure.
If the health problem is not resolved, the Replace a failed Receiver process is initiated.
Returning to service
When a Receiver is returned to service after a failure (for example, restart after a power failure, hardware repair, or network repair), the following occurs:
Receivers in high availability mode do not start collecting data on startup. They are in secondary mode until they are set as primary.
The preferred primary device assumes the role of primary and starts using the shared data source IP address to collect data. If there is no preferred primary device, the device that is primary starts using the shared data source and collects data.
Upgrading Receiver-HA
The Receiver-HA upgrade process upgrades both receivers sequentially, starting with the secondary receiver. It occurs like this:
The upgrade tarball file is uploaded to Trellix ESM and applied to the secondary Receiver.
You switch the role of the primary and secondary Receiver, using the Switch Receiver-HA roles process, so the Receiver that was upgraded is now the primary Receiver and the one that has not yet been upgraded is secondary.
The upgrade tarball is applied to the new secondary receiver.
You once again switch the role of the primary and secondary Receiver, using the Switch Receiver-HA roles process, so the original Receiver roles are assumed once again.
When upgrading, it is best not to have a preferred primary Receiver.
If your Receiver-HA is set up with a preferred primary, it is best to change the setting before upgrading. On the HA Receiver tab, select None in the Preferred primary device field. This allows you to use the Fail-over option, which is not available with a preferred primary setting. After both Receivers are upgraded, you can apply the preferred primary setting again.