[FIXED] How do deploy to Heroku with long release tasks

Issue

We have a rails v6 app running in Heroku. We have a Postgres DB and we use elastic search. When we push new releases, there are usually two things that need to happen in each release phase:

1. Run database migrations
2. Update the elastic search index Currently, both of these operations happen in a bash script invoked by the Procfile (details below).

The problem is that even though the new code isn't "released" until after the release tasks finish, the database migrations take effect immediately. The DB migrations often introduce breaking changes that cause errors until the corresponding code is released. Normally this wouldn't be a major issue, but Elasticsearch reindexing takes almost two hours to complete (and it needs to happen after migrations). So during that reindexing time, the database has been updated but the code hasn't been released yet. And that's too long for the site to be broken or in maintenance mode.

Database migrations and reindexing are pretty common operations and Heroku is very popular. So my question is what is the correct way to orchestrate this kind of release without multiple hours of downtime?

Uncessful ideas

My first idea was to perform the migrations after the reindexing. But often the migrations modify DB fields that get used during re-indexing. So that won't work.

I also thought about trying to perform the re-indexing on a new/different Elasticsearch index, and then point the app at the newer one when the process completes. I'm not sure how to do this, but it is also problematic because the newly released code often needs the updated index to work properly. So we would still potentially break the site while the reindexing is happening.

Details

Procfile

release:      bash ./release-tasks.sh
rails:        bundle exec bin/rails s -p 3000

Here's a simplified version of our release-tasks.sh script:

echo "Running migrations on an existing DB...";
rake db:migrate;
# This effects the production DB immediately 

echo "Reindexing..."
rake searchkick:reindex:all
# This takes two hours to finish before the code goes live

Solution

DB migration should not introduce breaking changes. Your migrations should be "safe", i.e. your pre-deployment code and post-deployment code should both work if the migration ran.

For instance, you should not remove a column from your database unless the pre-deployment code has the column in question in the self.ignored_columns.

Check strong_migrations for more info. The gem page lists the Potentially dangerous operations and provides the safe alternative to run them.

Answered By - AbM

Answer Checked By - Mildred Charles (PHPFixing Admin)