Scaling Guide

In an ObjectRocket for Elasticsearch instance, all of the data is stored in dedicated data nodes which can be scaled up or out. Whether you scale out (add data nodes), or scale up (change plan size) will depend on what the current state of the cluster is and what your needs are.

Adding data nodes is the easiest way to grow your cluster. If you’re using the 8GB plan size, for example, that means each of your data nodes has 8GB of storage. Whenever you need more capacity, you can simply add a new data node, which will add an another 8GB of capacity. On our platform, data nodes can be automatically added with our RocketScale technology, or manually via the UI.

However, if you have large shards, have a large number of data nodes already, or have an application where fewer/larger nodes provides a performance advanatge, a plan resize may be the right path. A plan resize takes every data node in your cluster and moves it to a different plan size (e.g. 8GB to 16GB) and then adds/removes any extra data nodes that may or may not be needed. Plan resizes will require a ticket to support, but are generally completed without downtime.

Adding a data node

A new instance comes with two data nodes and more can be easily added after the initial instance creation. To add more nodes go to the Elasticsearch instance in the Control Panel and simply click the Data Nodes heading, then the Add Data Node button and follow the prompts. Additional charges apply, so take a look at our pricing page for details.

_images/add_datanode.png

Removing a data node

To ensure that data nodes are always removed safely, requests to remove a data node must be made via a support ticket. From there, we will remove the requested number of data nodes for you as quickly as possible.

RocketScale™

RocketScale™ is an ObjectRocket technology that will automatically add data nodes as you need them. Our service monitors the space utilization of each data node and when it sees one hovering above a pre-set threshold for a set amount of time, it will automtically add a data node to ensure your cluster does not run out of space.

By default, all clusters have this feature enabled and will add a new node when storage usage on any node stays above 85% for a period of 10 minutes (with some exceptions discussed below).

You can disable and/or modify the thresholds at any time, but if you disable it, it may impact our ability to guarantee that your cluster never goes red due to a lack of capacity.

How to configure RocketScale

In the ObjectRocket UI, click on the RocketScale Button for the instance in question.

_images/esrocketscalebutton.png

You will have the option to:

  • Enable or disable RocketScale on the instance
    • Disabling it will stop the ObjectRocket service from automatically adding data nodes when you need them
  • Set the RocketScale Disk Usage Threshold for the instance
    • This threshold determines the level of storage utilization that must be exceeded on a data node before RocketScale kicks in
    • This setting can be set higher or lower based on your needs. Keep in mind that setting this threshold above 90% could put your cluster at risk of going red as it fills up.
    • This setting is limited to a range of 70-95%
  • Set Advanced Rocketscale settings
    • Rocketscale has additional built in settings that you can tune to control the behavior of Rocketscale for your instance
    • Number of Minutes in Violation (default = 10 minutes, range = 1 - 1000). This is the window of time that the utlization must be above the threshold before a new data node is added.
    • Max Node Limit (default = 12, range = 3 - 20) - The maximum number of nodes that your cluster can have before Rocketscale will stop adding more. Even if you cross the Disk Usage Threshold, RocketScale will not add any more data nodes once your cluster has reached this size.
    • Cooldown time (default = 600 seconds, range = 300 - 10000) - This is the amount of time in seconds that our service will wait between checks of node storage utilization.
_images/esrocketscalewadvanced.png

Conditions that Prevent RocketScale from Adding Nodes

RocketScale is designed to stop adding nodes if it determines that adding data nodes will not help reduce utilization, when a cluster is in a bad state, or when a plan resize may be a better option.

Currently the conditions that will stop RocketScale from adding a node are:

  • RocketScale is disabled If RocketScale is disabled, it will not attempt to add nodes. It is worth noting though that our support team may still add a node if your cluster fills up and is in danger of failing.
  • Cluster has more data nodes than shards Elasticsearch can’t split a shard, so any data nodes beyond the number of total shards will not help and we will not add any more nodes.
  • Cluster has more nodes than the max node limit setting In most scenarios, we’ve found that clusters with more than 12 data nodes will function more efficiently with a larger plan size and fewer data nodes. However, this setting can be modified for larger clusters where more data nodes are desired.
  • Cluster has nodes with low utilization If your cluster has nodes with less than 40% utilization, but you also have a node with high utilization, it may be a sign that something is wrong with the cluster that requires the attention of our support team. If RocketScale detects this scenario it will turn scaling off for the instance and alert our support team to look at your cluster.