EDR Telemetry Store provides a scalable architecture for large-scale data workloads. You can expand the cluster configuration over time as your storage needs increase. As a result, you do not need to provision the full target capacity during the initial deployment.
EDR Telemetry Store scaling options
Scaling option | Description | EDR Telemetry Store implementation | Benefits and limitations |
|---|---|---|---|
Horizontal scaling | Increases capacity by adding nodes to the cluster (for example, expanding from 3 to 6 data nodes). | (Recommended) EDR Telemetry Store automatically redistributes data (shards) and workload across the new nodes. | Provides high scalability and fault tolerance. |
Vertical scaling | Increases capacity by upgrading the resources of existing nodes (for example, changing an m6i.4xlarge instance to m6i.8xlarge). | Use only when specific existing nodes are resource-constrained (such as high CPU usage). | Scalability is limited by the physical constraints of the hardware. |
Shard rebalancing
When you add a new data node to the cluster, the cluster manager automatically detects the node and initiates shard rebalancing.
During this process, the cluster manager migrates a portion of the existing data shards (both primary and replicas) from the existing nodes to the new node. This process occurs dynamically without service interruption, immediately distributing the indexing, searching, and storage load across the expanded hardware pool.
Resilience and high availability
Horizontal scaling improves resilience by increasing the pool of nodes available for replica shard distribution.
Larger clusters can tolerate simultaneous node failures without data loss. For example, a 15-node cluster provides greater resilience against single-node failures than a 3-node cluster.
Note
To ensure high availability, you must configure the replication factor correctly and distribute nodes across availability zones.