Add side_by_side mode to update_materialized_view

README.md

@@ -91,7 +91,8 @@ hierarchies of dependent views.
 
 Scenic offers a `replace_view` schema statement, resulting in a `CREATE OR
 REPLACE VIEW` SQL query which will update the supplied view in place, retaining
-all dependencies. Materialized views cannot be replaced in this fashion.
+all dependencies. Materialized views cannot be replaced in this fashion, though
+the `side_by_side` update strategy may yield similar results (see below).
 
 You can generate a migration that uses the `replace_view` schema statement by
 passing the `--replace` option to the `scenic:view` generator:
@@ -137,7 +138,7 @@ end
 ```
 
 Scenic even provides a `scenic:model` generator that is a superset of
-`scenic:view`.  It will act identically to the Rails `model` generator except
+`scenic:view`. It will act identically to the Rails `model` generator except
 that it will create a Scenic view migration rather than a table migration.
 
 There is no special base class or mixin needed. If desired, any code the model
@@ -185,6 +186,44 @@ you would need to refresh view B first, then right after refresh view A. If you
 would like this cascading refresh of materialized views, set `cascade: true`
 when you refresh your materialized view.
 
+## Can I update the definition of a materialized view without dropping it?
+
+No, but Scenic can help you approximate this behavior with its `side_by_side`
+update strategy.
+
+Generally, changing the definition of a materialized view requires dropping it
+and recreating it, either without data or with a non-concurrent refresh. The
+materialized view will be locked for selects during the refresh process, which
+can cause problems in your application if the refresh is not fast.
+
+The `side_by_side` update strategy prepares the new version of the view under a
+temporary name. This includes copying the indexes from the original view and
+refreshing the data. Once prepared, the original view is dropped and the new
+view is renamed to the original view's name. This process minimizes the time the
+view is locked for selects at the cost of additional disk space.
+
+You can generate a migration that uses the `side_by_side` strategy by passing
+the `--side-by-side` option to the `scenic:view` generator:
+
+```sh
+$ rails generate scenic:view search_results --materialized --side-by-side
+      create  db/views/search_results_v02.sql
+      create  db/migrate/[TIMESTAMP]_update_search_results_to_version_2.rb
+```
+
+The migration will look something like this:
+
+```ruby
+class UpdateSearchResultsToVersion2 < ActiveRecord::Migration
+  def change
+    update_view :search_results,
+      version: 2,
+      revert_to_version: 1,
+      materialized: { side_by_side: true }
+  end
+end
+```
+
 ## I don't need this view anymore. Make it go away.
 
 Scenic gives you `drop_view` too:
@@ -234,18 +273,18 @@ It's our experience that maintaining a library effectively requires regular use
 of its features. We're not in a good position to support MySQL, SQLite or other
 database users.
 
-Scenic *does* support configuring different database adapters and should be
+Scenic _does_ support configuring different database adapters and should be
 extendable with adapter libraries. If you implement such an adapter, we're happy
 to review and link to it. We're also happy to make changes that would better
 accommodate adapter gems.
 
 We are aware of the following existing adapter libraries for Scenic which may
 meet your needs:
 
-* [`scenic_sqlite_adapter`](<https://github.com/pdebelak/scenic_sqlite_adapter>)
-* [`scenic-mysql_adapter`](<https://github.com/EmpaticoOrg/scenic-mysql_adapter>)
-* [`scenic-sqlserver-adapter`](<https://github.com/ClickMechanic/scenic_sqlserver_adapter>)
-* [`scenic-oracle_adapter`](<https://github.com/cdinger/scenic-oracle_adapter>)
+- [`scenic_sqlite_adapter`](https://github.com/pdebelak/scenic_sqlite_adapter)
+- [`scenic-mysql_adapter`](https://github.com/EmpaticoOrg/scenic-mysql_adapter)
+- [`scenic-sqlserver-adapter`](https://github.com/ClickMechanic/scenic_sqlserver_adapter)
+- [`scenic-oracle_adapter`](https://github.com/cdinger/scenic-oracle_adapter)
 
 Please note that the maintainers of Scenic make no assertions about the
 quality or security of the above adapters.
@@ -255,26 +294,24 @@ quality or security of the above adapters.
 ### Used By
 
 Scenic is used by some popular open source Rails apps:
-[Mastodon](<https://github.com/mastodon/mastodon/>),
-[Code.org](<https://github.com/code-dot-org/code-dot-org>), and
-[Lobste.rs](<https://github.com/lobsters/lobsters/>).
+[Mastodon](https://github.com/mastodon/mastodon/),
+[Code.org](https://github.com/code-dot-org/code-dot-org), and
+[Lobste.rs](https://github.com/lobsters/lobsters/).
 
 ### Related projects
 
-- [`fx`](<https://github.com/teoljungberg/fx>) Versioned database functions and
+- [`fx`](https://github.com/teoljungberg/fx) Versioned database functions and
   triggers for Rails
 
-
 ### Media
 
 Here are a few posts we've seen discussing Scenic:
 
-- [Announcing Scenic - Versioned Database Views for Rails](<https://thoughtbot.com/blog/announcing-scenic--versioned-database-views-for-rails>) by Derek Prior for thoughtbot
-- [Effectively Using Materialized Views in Ruby on Rails](<https://pganalyze.com/blog/materialized-views-ruby-rails>) by Leigh Halliday for pganalyze
-- [Optimizing String Concatenation in Ruby on Rails](<https://dev.to/pimp_my_ruby/from-slow-to-lightning-fast-optimizing-string-concatenation-in-ruby-on-rails-28nk>)
-- [Materialized Views In Ruby On Rails With Scenic](<https://www.ideamotive.co/blog/materialized-views-ruby-rails-scenic>) by Dawid Karczewski for Ideamotive
-- [Using Scenic and SQL views to aggregate data](<https://dev.to/weareredlight/using-scenic-and-sql-views-to-aggregate-data-226k>) by André Perdigão for Redlight Software
-
+- [Announcing Scenic - Versioned Database Views for Rails](https://thoughtbot.com/blog/announcing-scenic--versioned-database-views-for-rails) by Derek Prior for thoughtbot
+- [Effectively Using Materialized Views in Ruby on Rails](https://pganalyze.com/blog/materialized-views-ruby-rails) by Leigh Halliday for pganalyze
+- [Optimizing String Concatenation in Ruby on Rails](https://dev.to/pimp_my_ruby/from-slow-to-lightning-fast-optimizing-string-concatenation-in-ruby-on-rails-28nk)
+- [Materialized Views In Ruby On Rails With Scenic](https://www.ideamotive.co/blog/materialized-views-ruby-rails-scenic) by Dawid Karczewski for Ideamotive
+- [Using Scenic and SQL views to aggregate data](https://dev.to/weareredlight/using-scenic-and-sql-views-to-aggregate-data-226k) by André Perdigão for Redlight Software
 
 ### Maintainers
 

lib/generators/scenic/materializable.rb

@@ -14,7 +14,14 @@ module Materializable
           type: :boolean,
           required: false,
           desc: "Adds WITH NO DATA when materialized view creates/updates",
-          default: false
+          default: false,
+          aliases: ["--no-data"]
+        class_option :side_by_side,
+          type: :boolean,
+          required: false,
+          desc: "Uses side-by-side strategy to update materialized view",
+          default: false,
+          aliases: ["--side-by-side"]
         class_option :replace,
           type: :boolean,
           required: false,
@@ -35,6 +42,25 @@ def replace_view?
       def no_data?
         options[:no_data]
       end
+
+      def side_by_side?
+        options[:side_by_side]
+      end
+
+      def materialized_view_update_options
+        set_options = {no_data: no_data?, side_by_side: side_by_side?}
+          .select { |_, v| v }
+
+        if set_options.empty?
+          "true"
+        else
+          string_options = set_options.reduce("") do |memo, (key, value)|
+            memo + "#{key}: #{value}, "
+          end
+
+          "{ #{string_options.chomp(", ")} }"
+        end
+      end
     end
   end
 end

lib/generators/scenic/view/templates/db/migrate/update_view.erb

@@ -5,7 +5,7 @@ class <%= migration_class_name %> < <%= activerecord_migration_class %>
     <%= method_name %> <%= formatted_plural_name %>,
       version: <%= version %>,
       revert_to_version: <%= previous_version %>,
-      materialized: <%= no_data? ? "{ no_data: true }" : true %>
+      materialized: <%= materialized_view_update_options %>
   <%- else -%>
     <%= method_name %> <%= formatted_plural_name %>, version: <%= version %>, revert_to_version: <%= previous_version %>
   <%- end -%>

lib/scenic/adapters/postgres.rb

@@ -4,6 +4,10 @@
 require_relative "postgres/indexes"
 require_relative "postgres/views"
 require_relative "postgres/refresh_dependencies"
+require_relative "postgres/side_by_side"
+require_relative "postgres/index_creation"
+require_relative "postgres/index_migration"
+require_relative "postgres/temporary_name"
 
 module Scenic
   # Scenic database adapters.
@@ -155,17 +159,26 @@ def create_materialized_view(name, sql_definition, no_data: false)
       # @param no_data [Boolean] Default: false. Set to true to create
       #   materialized view without running the associated query. You will need
       #   to perform a refresh to populate with data.
+      # @param side_by_side [Boolean] Default: false. Set to true to create the
+      #   new version under a different name and atomically swap them, limiting
+      #   the time that a view is inaccessible at the cost of doubling disk usage
       #
       # @raise [MaterializedViewsNotSupportedError] if the version of Postgres
       #   in use does not support materialized views.
       #
       # @return [void]
-      def update_materialized_view(name, sql_definition, no_data: false)
+      def update_materialized_view(name, sql_definition, no_data: false, side_by_side: false)
         raise_unless_materialized_views_supported
 
-        IndexReapplication.new(connection: connection).on(name) do
-          drop_materialized_view(name)
-          create_materialized_view(name, sql_definition, no_data: no_data)
+        if side_by_side
+          SideBySide
+            .new(adapter: self, name: name, definition: sql_definition)
+            .update
+        else
+          IndexReapplication.new(connection: connection).on(name) do
+            drop_materialized_view(name)
+            create_materialized_view(name, sql_definition, no_data: no_data)
+          end
         end
       end
 
@@ -253,15 +266,19 @@ def populated?(name)
         end
       end
 
+      # A decorated ActiveRecord connection object with some Scenic-specific
+      # methods. Not intended for direct use outside of the Postgres adapter.
+      #
+      # @api private
+      def connection
+        Connection.new(connectable.connection)
+      end
+
       private
 
       attr_reader :connectable
       delegate :execute, :quote_table_name, to: :connection
 
-      def connection
-        Connection.new(connectable.connection)
-      end
-
       def raise_unless_materialized_views_supported
         unless connection.supports_materialized_views?
           raise MaterializedViewsNotSupportedError

lib/scenic/adapters/postgres/index_creation.rb

@@ -0,0 +1,68 @@
+module Scenic
+  module Adapters
+    class Postgres
+      # Used to resiliently create indexes on a materialized view. If the index
+      # cannot be applied to the view (e.g. the columns don't exist any longer),
+      # we log that information and continue rather than raising an error. It is
+      # left to the user to judge whether the index is necessary and recreate
+      # it.
+      #
+      # Used when updating a materialized view to ensure the new version has all
+      # apprioriate indexes.
+      #
+      # @api private
+      class IndexCreation
+        # Creates the index creation object.
+        #
+        # @param connection [Connection] The connection to execute SQL against.
+        # @param speaker [#say] (ActiveRecord::Migration) The object used for
+        #   logging the results of creating indexes.
+        def initialize(connection:, speaker: ActiveRecord::Migration.new)
+          @connection = connection
+          @speaker = speaker
+        end
+
+        # Creates the provided indexes. If an index cannot be created, it is
+        # logged and the process continues.
+        #
+        # @param indexes [Array<Scenic::Index>] The indexes to create.
+        #
+        # @return [void]
+        def try_create(indexes)
+          Array(indexes).each(&method(:try_index_create))
+        end
+
+        private
+
+        attr_reader :connection, :speaker
+
+        def try_index_create(index)
+          success = with_savepoint(index.index_name) do
+            connection.execute(index.definition)
+          end
+
+          if success
+            say "index '#{index.index_name}' on '#{index.object_name}' has been created"
+          else
+            say "index '#{index.index_name}' on '#{index.object_name}' is no longer valid and has been dropped."
+          end
+        end
+
+        def with_savepoint(name)
+          connection.execute("SAVEPOINT #{name}")
+          yield
+          connection.execute("RELEASE SAVEPOINT #{name}")
+          true
+        rescue
+          connection.execute("ROLLBACK TO SAVEPOINT #{name}")
+          false
+        end
+
+        def say(message)
+          subitem = true
+          speaker.say(message, subitem)
+        end
+      end
+    end
+  end
+end

lib/scenic/adapters/postgres/index_migration.rb

@@ -0,0 +1,70 @@
+module Scenic
+  module Adapters
+    class Postgres
+      # Used during side-by-side materialized view updates to migrate indexes
+      # from the original view to the new view.
+      #
+      # @api private
+      class IndexMigration
+        # Creates the index migration object.
+        #
+        # @param connection [Connection] The connection to execute SQL against.
+        # @param speaker [#say] (ActiveRecord::Migration) The object used for
+        #   logging the results of migrating indexes.
+        def initialize(connection:, speaker: ActiveRecord::Migration.new)
+          @connection = connection
+          @speaker = speaker
+        end
+
+        # Retreives the indexes on the original view, renames them to avoid
+        # collisions, retargets the indexes to the destination view, and then
+        # creates the retargeted indexes.
+        #
+        # @param from [String] The name of the original view.
+        # @param to [String] The name of the destination view.
+        #
+        # @return [void]
+        def migrate(from:, to:)
+          source_indexes = Indexes.new(connection: connection).on(from)
+          retargeted_indexes = source_indexes.map { |i| retarget(i, to: to) }
+          source_indexes.each(&method(:rename))
+
+          if source_indexes.any?
+            say "indexes on '#{from}' have been renamed to avoid collisions"
+          end
+
+          IndexCreation
+            .new(connection: connection, speaker: speaker)
+            .try_create(retargeted_indexes)
+        end
+
+        private
+
+        attr_reader :connection, :speaker
+
+        def retarget(index, to:)
+          new_definition = index.definition.sub(
+            /ON (.*)\.#{index.object_name}/,
+            'ON \1.' + to + " "
+          )
+
+          Scenic::Index.new(
+            object_name: to,
+            index_name: index.index_name,
+            definition: new_definition
+          )
+        end
+
+        def rename(index)
+          temporary_name = TemporaryName.new(index.index_name).to_s
+          connection.rename_index(index.object_name, index.index_name, temporary_name)
+        end
+
+        def say(message)
+          subitem = true
+          speaker.say(message, subitem)
+        end
+      end
+    end
+  end
+end