Heroku’s announcement in July regarding the shut down of the Compose MongoDB add-on (effective on October 15, 2019) presented ObjectRocket an opportunity to support Heroku users of MongoDB. We provide a simple, low-friction migration tool to help existing Compose add-on users move over to the ObjectRocket MongoDB add-on. This blog reveals what is going on behind the curtain with our migration wizard, and explains what happens during the migration process and how we implemented the technology.
Migrating data between MongoDB instances is nothing new. It does require some MongoDB knowledge (MongoDB version, storage size, configuration, etc.) and planning to ensure that all of the data is migrated successfully. At ObjectRocket, we have been doing migrations for customers since the beginning of our fully-managed MongoDB product back in 2012. MongoDB instances can be migrated via several different methods and we always try to tailor the method to the customer’s requirements.
No Downtime
If a user is willing to accept some down time, the standard and well documented mongodump and mongorestore utilities are straightforward and easy to execute. However, if a user requires no downtime to their application or MongoDB instance during the migration, it becomes a more challenging endeavor.
Given how Heroku platform users typically leverage various technologies within an add-on model, we felt that it was important to prioritize application up-time and simplicity in any migration tool we created to support them. The migration tool that we built allows for customers with replica sets in a standard type configuration to easily move data from one MongoDB instance to another, without needing to stop ongoing writes/deletes to the original MongoDB instance. In order to not lose the new data coming in from the application, the application must keep writing to the source MongoDB instance during the migration. The only change that the customer needs to do in the application is to update their connection strings or configuration variables after the new destination MongoDB instance is ready.
Migration Wizard
In the spirit of simplicity and low friction, we wanted to create a migration wizard that can walk users through a flow that would provide the migration tool the appropriate data, give users updates as the migration progressed, and provide a user-driven method of finalization.
To get started, we just need a couple of things from the Compose customer:
- A user with oplog access on the Compose MongoDB source instance
- Basic inputs from the Compose MongoDB instance including:
- MongoDB version
- MongoDB storage engine
- Plan/Instance size
- URI
See ObjectRocket migration tool documentation for further details.
To continue down the path of low friction, we had the ability to provision the destination ObjectRocket MongoDB instance on behalf of the user within the migration wizard. Once the user gives the tool authorization, we can ensure that the destination ObjectRocket MongoDB instance is provisioned and configured appropriately for the migration.
The ObjectRocket MongoDB migration tool, along with the documentation, allows Compose customers to completely self-serve their MongoDB migration on their own timeline. If at any time there are questions or issues with the migration, customers can reach out to the ObjectRocket Fanatical Support® Team.
What’s Really Happening
There are several methods for migrating data from one MongoDB instance to another. With the driving priorities of NO DOWNTIME and ease-of-use, we opted to use the mongo-connector open source project because we already had experience with it and it is a fairly mature project with all of the pieces that are needed to meet our requirements.
Per the project’s Readme, “mongo-connector creates a pipeline from a MongoDB cluster to one or more target systems, such as Solr, Elasticsearch, or another MongoDB cluster. It synchronizes data in MongoDB to the target then tails the MongoDB oplog, keeping up with operations in MongoDB in real-time.” There were even some additional scripts that were created by other contributors that made it easier for us.
The migration flow runs in a container in a Kubernetes environment. This makes it easy to quickly scale resources to match migration demand.
With the basic inputs plugged in, the tool does the rest! Just press Start Migration on the migration tool UI and the tool automatically performs the following workflow:
- Runs mongo-connector Decode “timestamp” Utility Script that captures the most recent oplog entry on the source MongoDB instance and translates it to the format that the mongo-connector is going to want – this gives us a starting point to track. The MongoDB URI captured in the wizard is used to authenticate to the Compose instance.
- Syncs users between the source and destination MongoDB instances.
- Dumps the data from the source MongoDB instance using the mongodump command.
- Runs the mongostore command to put the copied data into the destination MongoDB instance. This gives us the majority of the data that we need for the migration but we still need to capture any new data added by applications since the migration started.
- Starts the mongo-connector. The connector will leverage the file created from the Decode “timestamp” utility script to go back to the original snapshot and recreate the activity that occurred (and is continuing to occur) on the source MongoDB instance. The connector updates the data on the destination MongoDB instance based on the captured activity, and displays a table of the synced data.
- After the source MongoDB instance and destination MongoDB instance are synced, we present the user with the ability to finalize the migration when they are ready. To finalize the migration, the user needs to 1) change the connection strings for their application to point to the destination MongoDB instance, and then 2) click Finalize Migration on the UI. We will continue to tail the source MongoDB instance, up until either the user clicks Finalize Migration OR the time period of 24 hours from the beginning of the migration expires.
Next Steps
Creating a tool for NO DOWNTIME MongoDB migrations has proven valuable beyond our original Heroku Compose MongoDB use case. Our own ObjectRocket Fanatical Support® Team is already finding value via efficiencies in their day-to-day migrations. We plan to leverage the tool as a feature for standard ObjectRocket MongoDB customers and are exploring opportunities to continue with the spirit of the mongo-connector open source project to connect with other technologies.
Our database experts here at ObjectRocket are always available to help you with the migration and implementation at no charge. Email us at support@objectrocket.com to get started!
