ObjectRocket

Automated online compaction

ObjectRocket Marketing
ObjectRocket skyline

We are excited to announce Automated Online Compaction on the ObjectRocket platform for MongoDB.

Automated Online Compaction allows MongoDB instances to be compacted online and in the background on the ObjectRocket platform. The application will only experience a replica set election in order to start using the newly compacted slave. Without this feature, applications experience extended downtime when a collection is compacted or when a database is repaired.

Compactions can be scheduled, and windows defined for when the final stepDown() takes place. Users can turn on the feature and not have to worry about MongoDB fragmenting over time. The instance is kept in a nice tidy form and it’s all automated.

All databases fragment over time, some worse than others depending on the underlaying design. In a generic sense; fragmentation occurs when deletes create spaces that new data or updated data can’t reuse. MongoDB fragments just like most popular databases. Even when using Powerof2Sizes we found that we spent a large amount of our DBA time working to keep customers databases compact. We felt that if we charge base on disk space footprint, it’s only right to help customers keep that footprint tidy. But the stock commands didn’t work for us because of they require service interruptions. Anything we built had to be automated, online, work in parallel at scale, and present a minimal impact to customers.

To this end, we have been working over the last few months to build this feature, and had to release a couple other components in order to make this possible. First, we built a component that allows the user to specify a window when they would like a stepDown() to be performed. Then we needed to build out a complete state machine for MongoDB replica sets. In order for this feature to work properly, our code needed to understand the state of all replica sets in a cluster at any given point in time, understand failures, and understand how to recover. We also needed a scheduler component to allow the scheduled stepDown and compactions. We needed to ensure we took into account backups being run, balancer activity, and overall availability impact to the cluster.

With that work done, we then could build a component that performed complete compactions of a replica set in the background, and almost 100% transparent to the user and calling application. The new feature is called Automated Online Compaction for MongoDB.

Here is how it works:

    1. User requests a compaction manually or they have set a compaction schedule.
    2. Per shard, a SECONDARY is selected for compaction, and compaction starts.

 

 

It should be reiterated, while the compaction is done in the background on a SECONDARY, in order to rotate it to PRIMARY an election must take place. Users must be aware of this fact, it’s best practice anyway.

Under the covers, here is what we are doing:

We track the Step Down window (if defined) for the instance:

{
  "stepdown_window" : {
    "scheduled" : true,
    "end" : ISODate("2014-04-24T14:16:00Z"),
    "ran_in_window" : false,
    "enabled" : true,
    "start" : ISODate("2014-04-23T14:16:00Z"),
    "weekly" : true
  }
}

We keep track of each replica set member and it’s state. We update the metadata so we know the state of every slave. State is “syncing” then once completed is “compressed”.

{
  "compression" : {
    "state" : "compressing",
    "balancer_stopped_by_check" : false,
    "updated" : "2014-04-23 20:57:49.449538",
    "shards" : [
      {
        "state" : "compressing",
        "updated" : "2014-04-23 20:57:02.126299",
        "shardstr" : "abcdef/myserver1:27017,myserver2:27017,myserver3:27017",
        "members" : [
          {
            "state" : "syncing",
            "updated" : "2014-04-23 20:57:49.449521",
            "name" : "myserver1:27017"
          }
        ],
        "updated_ts" : 1398286622
      }
    ]
  }
}

Once all the SECONDARY slaves are {state: “compressed”} we wait for the scheduled Step Down window:

{
  "state" : "awaiting_stepdown",
  ...
  "members" : [
    {
      "state" : "compressed",
      "updated" : "2014-04-23 21:07:03.627440",
      "name" : "myserver1:27017"
    },
    {
      "state" : "compressed",
      "updated" : "2014-04-23 21:27:01.384174",
      "name" : "myserver2:27017"
    }
  ]
}

And lastly, we compact the remaining SECONDARY (the previous PRIMARY):

{
  "members" : [
    {
      "state" : "compressed",
      "updated" : "2014-04-23 21:12:01.467046",
      "name" : "myserver1:27017"
    },
    {
      "state" : "compressed",
      "updated" : "2014-04-23 21:32:01.334504",
      "name" : "myserver2:27017"
    },
    {
      "state" : "compressed",
      "updated" : "2014-04-23 22:07:01.419483",
      "name" : "myserver3:27017"
    }
  ]
}

In order to get started with Automated Online Compaction, navigate to your instances view and select the compaction button, then schedule a stepdown window on the settings page. You can optionally choose to run the compaction weekly as well.

Exit mobile version