EDRF On-prem FAQs and troubleshoot

How to collect all relevant logs from ePO and Windows endpoints?

You can use the Minimum Escalation Requirement (MER) tool, which is a part of the Self-Service Supportability Orchestrator (SSSO) tool, to collect any relevant log and system information and create an archive with all the information, which can be submitted to Trellix. Download the MER tool. Alternatively, you can download from the ePO by navigating to Menu → Software Catalog → Utilities & Connectors → MER for ePO.

The MER tool collects any relevant log and system information and create an archive with all the information, which can be submitted to Trellix.

How to collect all relevant logs on the EDR Telemetry Store appliance?

If you encounter issues with the EDR Telemetry Store, you can collect diagnostic data to assist with debugging. The diagnostic command gathers logs, configuration files, and key system statistics required for analysis.

To collect EDR Telemetry Store diagnostics:

Before you retrieve logs, change the default password of the config user:
1. From your local machine, connect to the appliance as the config user.
```
ssh config@<EDR Telemetry Store_IP_address>
```
2. When prompted, enter the default password: config.
3. You are required to immediately change your password. Re-enter your current password (config ) and then create a new password.
Connect to the appliance as the admin user and run the enable command to enter enable mode.
Run the following command:
```
collect opensearch diagnostics
```
The system collects all relevant logs and archives them into a .tar.gz file in the /opt/trellix/uploads/diagnostics/opt/trellix/uploads/diagnostics/ directory.
To access the .tar.gz file, you must log in as the config user. You cannot access the file as an admin user.
Use the scp command to copy all the files in the diagnostics folder from the EDR Telemetry Store appliance to your local machine:
```
scp -r config@<EDR Telemetry Store_IP_address>:/opt/trellix/uploads/diagnostics/ \path_to_your_local_directory\
```

How to analyze EDR Telemetry Store crashes?

Follow steps 1 - 4 in the How to collect all relevant logs on the EDR Telemetry Store appliance section.
Review journalctl logs to determine the time of the crash.
Open the following log file to identify related errors:
/var/log/opensearch/Trellix-trace-store.log

How to troubleshoot over-sharding?

Follow steps 1 - 4 in the How to collect all relevant logs on the EDR Telemetry Store appliance section.

Over-sharding issue occurs if the total number of shards in the cluster is too high. You can identify the over-sharding issues when All shards failed error appears in logs when executing queries.

Additionally, to identify the over-sharding status:

Check the heap value for the data nodes.

curl -X GET --cert /etc/opensearch/certs/opensearch-admin.pem --key /etc/opensearch/certs/opensearch-admin-key.pem --cacert /etc/opensearch/certs/ca.pem "https://localhost:9200/_cat/nodes?v=true&h=name,heap.max"

Check the total number of shards.

curl -X GET --cert /etc/opensearch/certs/opensearch-admin.pem --key /etc/opensearch/certs/opensearch-admin-key.pem --cacert /etc/opensearch/certs/ca.pem "https://localhost:9200/_cluster/stats?filter_path=indices.shards.total"

Compare the values. If the total number of shards exceeds the heap value, the cluster is over-sharded.

To resolve over-sharding issues:
- Delete older indices.
```
curl -X DELETE --cert /etc/opensearch/certs/opensearch-admin.pem --key /etc/opensearch/certs/
opensearch-admin-key.pem --cacert /etc/opensearch/certs/ca.pem
```
- Reduce the number of shards for future indices using one of the following options.
  - Update the index template or ISM
    Place the eop-va-rpm package in the uploads folder for the config user.
    In configure mode, as an admin user, enter this command:
    upgrade package eop-va-rpm
    Verify that changes to the template or ISM are reflected in the cluster.
  - Reconfigure the cluster
    Run the setup command:
    opensearch cluster setup
    Select option 4 for ISM and index template settings.
    Enter y to modify the settings.
    Modify the values for number_of_shards in the index template and min_doc_count in ISM.
    On the summary page, press 5 to regenerate the configuration.
    When the new setup file is generated, run the deployment command:
    opensearch cluster deploy
Note
Make sure that you maintain fewer than 20–25 shards per GB of JVM heap on each node

How to troubleshoot index rollover failure?

Follow steps 1 - 4 in the How to collect all relevant logs on the EDR Telemetry Store appliance section.
Index rollover failure occurs if the cluster has reached the maximum number of shards allowed per node, and you see the Expand source error.
To troubleshoot:

How to manually delete events from the EDR Telemetry Store based on time and type?

You can use the following curl command to delete events older than a specific timeframe (for example, 2 days) across all indices. This command also includes an eventType filter to target specific data.

Note
This command must be executed from the EDR Telemetry Store appliance CLI.

curl -X POST "https://localhost:9200/edrf-traces-*/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "time": {
              "lte": "now-2d" 
            }
          }
        },
        {
          "term": {
            "eventType": "Network Accessed" 
          }
        }
      ]
    }
  }
}
' --cert /etc/opensearch/certs/opensearch-admin.pem --key /etc/opensearch/certs/opensearch-admin-key.pem --cacert /etc/opensearch/certs/ca.pem

How to troubleshoot command permission issues?

Caution
The root user has complete control over the operating system. Incorrect modifications to system-critical files can cause system failure.

Ensure commands are executed as the root user.
To log in as the root user:
1. Exit the admin user session.
2. Run the following command:
```
shell
```
If a console timeout occurs, exit from all levels and re-login to the console.

How to troubleshoot when nodes are not joining the cluster?

On the cluster manager node, run the setup command only once to generate the artifact:
```
opensearch cluster setup
```
Note
Do not run this command on any other node.
Verify that the artifact resides in the following directory on the cluster manager node:
```
/opt/trellix/uploads/config
```
On all nodes (including the cluster manager node), run the deployment command:
```
opensearch cluster deploy
```
Select the Download from remote server (SSH/SCP) option.
Enter the following connection details when prompted:
- Remote server: The IP address of the cluster manager node.
- User: config

How to enable debug logging for DXL Broker (Windows and Linux)

To enable debug logging for Data Exchange Layer Broker and its components on the Windows and Linux endpoints, see the Trellix Knowledge Base article, How to enable debug logging for Data Exchange Layer Broker (Windows and Linux) - KB95314.

How to create a local log file for EDRF?

Create a log file on Windows:

To review EDRF logs, run the following commands in the command prompt as an administrator and extract the log files:

"c:\Program Files\Trellix\XClient\xclient.exe" -g c:\xclient_g_<YYYMMDD>.log

"c:\Program Files\Trellix\XClient\xclient.exe" -x c:\xclient_x_<YYYMMDD>.log

Create a log file on Linux:

Run the following commands in the Linux terminal as an administrator and extract the log files:

sudo /opt/Trellix/XClient/bin/xclient -g /tmp/xclient_<YYYMMDD>.log

sudo /opt/Trellix/XClient/bin/xclient -x /tmp/xclient_config_<YYYMMDD>.log

Create a log file on macOS:

Run the following commands in the macOS terminal as an administrator and extract the log files:

sudo /Library/Trellix/xclient/xclient.app/Contents/MacOS/xclient -g /tmp/xclient_g_20251012.log

sudo /Library/Trellix/xclient/xclient.app/Contents/MacOS/xclient -x /tmp/xclient_x_20251012.log

How do I update the OpenSearch truststore for custom ePO certificates?

If your ePO console uses a custom certificate, the mTLS connection fails because the OpenSearch truststore does not recognize the custom certificate. To resolve this issue, manually add the ePO Orion Root CA to the OpenSearch server.

Retrieve the Orion Root CA certificate using one of the following methods:

From the browser:

https://<ePO_IP>:8443/remote/epo.command.createAgentHandlerCertificateAuthority

Note
Enter your ePO administrator credentials when prompted.

From the CLI:

curl https://<ePO_IP>:8443/remote/epo.command.createAgentHandlerCertificateAuthority -u <admin_username> -k

Copy the entire certificate output, including the ----BEGIN CERTIFICATE---- and ----END CERTIFICATE---- tags.
Log on to the EDR Telemetry Store appliance using SSH and switch to the root user.
```
enable
[sudo] password for admin:
su
```
Open the OpenSearch CA certificate file:
```
nano /etc/opensearch/certs/ca.pem
```
Append the copied certificate to the end of the file, then save the changes.
Restart the OpenSearch service:
```
systemctl restart opensearch
```