GH-510: Add support for asynchronously reading data from disk using multiple threads (#518)

* update accent4j version

* add configuration to enable_async_data_reads

* update documentation

* add async data reading

* update changelog

* fix bug in how ToggleQueue checks for whether it is read optimized or not

* fix copyright year

* upgrade to ccl 3.2.0 for contains/not_contains support

* set accent4j dependency to release version of 1.14.0

Author: jtnelson

CHANGELOG.md

@@ -41,7 +41,8 @@ We made several changes to improve the safety, scalability and operational effic
 
 ##### New Functionality and Enhancements
 * Reduced the amount of heap space required for essential storage metadata.
-* Added the `enable_efficient_metadata` configuration option to further reduce the amount of heap space required for essential storage metadata. When this option is set to `true`, metadata will occupy approximately one-third less heap space and likely improve overall system performance due to a decrease in garbage collection pauses (although per-operation performance may be slightly affected by additional overhead).
+* **Efficient Metadata:** Added the `enable_efficient_metadata` configuration option to further reduce the amount of heap space required for essential storage metadata. When this option is set to `true`, metadata will occupy approximately one-third less heap space and likely improve overall system performance due to a decrease in garbage collection pauses (although per-operation performance may be slightly affected by additional overhead).
+* **Asynchronous Data Reads:** Added the `enable_async_data_reads` configuration option to allow Concourse Server to *potentially* use multiple threads to read data from disk. When data records are either no longer cached or not eligible to ever be cached (due to space limitations), Concourse Server streams the relevant information from disk on-demand. By default, this is a synchronous process and the performance is linear based on the number of Segment files in the database. With this new configuration option, Concourse Server can now stream the data using multiple threads. Even under high contention, the read performance should be no worse than the default synchronous performance, but there may be additional overhead that reduces peak performance on a per-operation basis.
 * Improved write performance of the `set` method in large transactions by creating normalized views of existing data, which are consulted during the method’s implicit `select` read operation.
 * Improved the performance of the `verifyOrSet` method by removing redundant internal verification that occurred while finalizing the write.
 

build.gradle

@@ -121,7 +121,7 @@ subprojects {
         compile 'joda-time:joda-time:2.2'
         compile 'org.apache.thrift:libthrift:0.20.0'
         compile 'commons-configuration:commons-configuration:1.9'
-        compile group: 'com.cinchapi', name: 'accent4j', version: '1.13.1', changing:true
+        compile group: 'com.cinchapi', name: 'accent4j', version: '1.14.0', changing:true
         compile 'com.cinchapi:lib-config:1.5.1'
         compile group: 'com.cinchapi', name: 'lib-cli', version: '1.1.1', changing:true
 

concourse-server/conf/concourse.yaml

@@ -4,7 +4,7 @@
 
 # The path to the file where access credentials for Concourse Server are
 # stored. For optimal security, this file should be placed in a separate
-# directory from Concourse Server with more restrictive operating system 
+# directory from Concourse Server with more restrictive operating system
 # permissions
 #
 # DEFAULT {$concourse.home}/.access
@@ -26,8 +26,8 @@ buffer_directory:
 buffer_page_size:
 
 # The listener port (1-65535) for client connections. Choose a port between
-# 49152 and 65535 to minimize the possibility of conflicts with other services
-# on this host.
+# 49152 and 65535 to minimize the possibility of conflicts with other
+# services on this host.
 #
 # DEFAULT: 1717
 client_port:
@@ -40,9 +40,9 @@ client_port:
 # DEFAULT: {$user.home}/concourse/db
 database_directory:
 
-# The default environment that is automatically loaded when the server
-# starts and is used whenever a client does not specify an environment
-# for the connection.
+# The default environment that is automatically loaded when the server starts
+# and is used whenever a client does not specify an environment for the
+# connection.
 #
 # DEFAULT: default
 default_environment:
@@ -53,17 +53,17 @@ default_environment:
 # DEFAULT: false
 enable_console_logging:
 
-# The amount of memory that is allocated to the Concourse Server JVM.
-# Concourse requires a minimum heap size of 256MB to start, but much
-# more is recommended to ensure that read and write operations avoid
-# expensive disk seeks where possible. Concourse generally sets both
-# the initial and maximum heap sizes to the specified value, so there
-# must be enough system memory available for Concourse Server to start.
-#
-# Be careful and avoid setting the heap size too large because this may
-# cause longer garbage collection (gc) pauses or interfere with the ability
-# of Concourse Server to memory map (mmap) certain data files. We
-# recommend the following sizing guidelines:
+# The amount of memory that is allocated to the Concourse Server JVM. Concourse
+# requires a minimum heap size of 256MB to start, but much more is recommended
+# to ensure that read and write operations avoid expensive disk seeks where
+# possible. Concourse generally sets both the initial and maximum heap sizes to
+# the specified value, so there must be enough system memory available for
+# Concourse Server to start.
+#
+# Be careful and avoid setting the heap size too large because this may cause
+# longer garbage collection (gc) pauses or interfere with the ability of
+# Concourse Server to memory map (mmap) certain data files. We recommend the
+# following sizing guidelines:
 #
 # SYSTEM MEMORY    | Recommended heap_size
 # -----------------------------------------------------------
@@ -89,46 +89,46 @@ http_port:
 http_enable_cors:
 
 # A comma separated list of default URIs that are permitted to access HTTP
-# endpoints. By default (if enabled), the value of this preference is set
-# to the wildcard character '*' which means that any origin is allowed access.
+# endpoints. By default (if enabled), the value of this preference is set to
+# the wildcard character '*' which means that any origin is allowed access.
 # Changing this value to a discrete list will set the default origins that are
 # permitted, but individual endpoints may override this value.
 #
 # DEFAULT: (allow any origin)
 http_cors_default_allow_origin:
 
-# A comma separated list of default headers that are sent in response to a
-# CORS preflight request to indicate which HTTP headers can be used when
-# making the actual request. By default (if enabled), the value of this
-# preference is set to the wildcard character '*' which means that any headers
-# specified in the preflight request are allowed. Changing this value to a
-# discrete list will set the default headers that are permitted, but individual
-# endpoints may override this value.
+# A comma separated list of default headers that are sent in response to a CORS
+# preflight request to indicate which HTTP headers can be used when making the
+# actual request. By default (if enabled), the value of this preference is set
+# to the wildcard character '*' which means that any headers specified in the
+# preflight request are allowed. Changing this value to a discrete list will
+# set the default headers that are permitted, but individual endpoints may
+# override this value.
 #
 # DEFAULT: (allow any headers)
 http_cors_default_allow_headers:
 
 # A comma separated list of default methods that are sent in response to a
-# CORS preflight request to indicate which HTTP methods can be used when making
-# the actual request. By default (if enabled), the value of this preference is
-# set to the wildcard character '*' which means that any method specified in the
-# preflight request is allowed. Changing this value to a discrete list will set
-# the default methods that are permitted, but individual endpoints may override
-# this value.
+# CORS preflight request to indicate which HTTP methods can be used when
+# making the actual request. By default (if enabled), the value of this
+# preference is set to the wildcard character '*' which means that any method
+# specified in the preflight request is allowed. Changing this value to a
+# discrete list will set the default methods that are permitted, but individual
+# endpoints may override this value.
 #
 # DEFAULT: (allow any method)
 http_cors_default_allow_methods:
 
-# The initial root password for Concourse Server. This password is used to set 
-# up the initial administrator account when the server is first run. It is strongly
-# recommended to change this password immediately after the initial setup to maintain 
-# security.
+# The initial root password for Concourse Server. This password is used to set
+# up the initial administrator account when the server is first run. It is
+# strongly recommended to change this password immediately after the initial
+# setup to maintain security.
 #
 # DEFAULT: "admin"
 init_root_password:
 
-# The initial root username for Concourse Server. This username is associated with the
-# initial administrator account. It is strongly
+# The initial root username for Concourse Server. This username is associated
+# with the initial administrator account.
 #
 # DEFAULT: "admin"
 init_root_username:
@@ -148,8 +148,8 @@ jmx_port:
 #
 # ERROR: critical information when the system reaches a potentially fatal
 #    state and may not operate normally.
-# WARN:  useful information when the system reaches a less than ideal state but
-#    can continue to operate normally.
+# WARN:  useful information when the system reaches a less than ideal state
+#    but can continue to operate normally.
 # INFO:  status information about the system that can be used for sanity
 #     checking.
 # DEBUG: detailed information about the system that can be used to diagnose
@@ -164,16 +164,16 @@ log_level:
 
 # The length of the longest substring that will be indexed for fulltext search.
 #
-# This value does not mean that longer words will not be indexed. It simply means 
-# that, for any indexable value (e.g. a String), any substring that is longer than
-# the value of this preference will not be added to the search index. The effect is
-# that search strings containing any words with a length greater than the value of 
-# this preference will return 0 results. 
-
-# For best performance, this value should be set to the longest word length of any 
-# possible search string. To be safe, we recommend setting this value to be the 
-# length of the longest possible word in the search language. For example, the longest 
-# possible word in English is about 40 characters long.
+# This value does not mean that longer words will not be indexed. It simply
+# means that, for any indexable value (e.g. a String), any substring that is
+# longer than the value of this preference will not be added to the search
+# index. The effect is that search strings containing any words with a length
+# greater than the value of this preference will return 0 results.
+#
+# For best performance, this value should be set to the longest word length of
+# any possible search string. To be safe, we recommend setting this value to be
+# the length of the longest possible word in the search language. For example,
+# the longest possible word in English is about 40 characters long.
 #
 # DEFAULT: 40
 max_search_substring_length:
@@ -187,10 +187,10 @@ max_search_substring_length:
 # DEFAULT: automatically chosen based on the client_port
 shutdown_port:
 
-# The listener port (1-65535) for remote debugger connections. Choose a port between
-# 49152 and 65535 to minimize the possibility of conflicts with other services
-# on this host. If the value of this preference is set to 0, then remote debugging for
-# Concourse Server is disabled.
+# The listener port (1-65535) for remote debugger connections. Choose a port
+# between 49152 and 65535 to minimize the possibility of conflicts with other
+# services on this host. If the value of this preference is set to 0, then
+# remote debugging for Concourse Server is disabled.
 #
 # DEFAULT: 0
 remote_debugger_port:
@@ -199,12 +199,31 @@ remote_debugger_port:
 ###  EXPERIMENTAL CONFIGURATION FOR CONCOURSE SERVER  ###
 #########################################################
 
-# Automatically use a combination of defragmentation, garbage collection and load 
-# balancing within the data files to optimize storage for read performance.
+# Potentially use multiple threads to asynchronously read data from disk.
+#
+# When enabled, reads will typically be faster when accessing data that is too
+# large to ever fit in memory or no longer cached due to memory constraints.
+#
+# This setting is particularly useful for search data since those indexes are
+# not cached by default (unless ENABLE_SEARCH_CACHE is enabled). Even if search
+# records are cached, this setting may still provide a performance boost if the
+# size of some search metadata exceeds the limits of what is cacheable in
+# memory.
+#
+# NOTE: There might be some overhead that could make some reads slower if all
+# their relevant segment metadata is cached and there is high contention.
+#
+# DEFAULT: false
+enable_async_data_reads:
+
+# Automatically use a combination of defragmentation, garbage collection and
+# load balancing within the data files to optimize storage for read
+# performance.
 #
-# The compaction process may run continuously in the background without disrupting 
-# reads or writes. The storage engine uses a specific strategy to determine how 
-# data files should be reorganized to improve the performance of read operations. 
+# The compaction process may run continuously in the background without
+# disrupting reads or writes. The storage engine uses a specific strategy to
+# determine how data files should be reorganized to improve the performance of
+# read operations.
 #
 # DEFAULT: false
 enable_compaction:
@@ -215,29 +234,29 @@ enable_compaction:
 # for essential metadata by a third. As a result, overall system performance may
 # improve due to a reduction in garbage collection pauses.
 #
-# However, this setting may increase CPU usage and slightly reduce peak performance
-# on a per-operation basis due to weaker reference locality.
+# However, this setting may increase CPU usage and slightly reduce peak
+# performance on a per-operation basis due to weaker reference locality.
 #
 # DEFAULT: false
 enable_efficient_metadata:
 
-# Maintain and in-memory cache of the data indexes used to respond to search commands.
-# Search indexes tend to be much larger than those used for primary and secondary 
-# lookups, so enabling the search cache may cause memory issues (and overall 
-# performance degradation) if search is heavily used. Furthermore, indexing and 
-# write performance may suffer if cached search indexes must be incrementally kept 
-# current.
+# Maintain and in-memory cache of the data indexes used to respond to search
+# commands. Search indexes tend to be much larger than those used for primary
+# and secondary lookups, so enabling the search cache may cause memory issues
+# (and overall performance degradation) if search is heavily used. Furthermore,
+# indexing and write performance may suffer if cached search indexes must be
+# incrementally kept current.
 #
 # DEFAULT: false
 enable_search_cache:
 
 # Attempt to optimize verify commands by using special lookup records.
 #
-# A lookup record only contains data for a single field. The database does not cache
-# lookup records, so, while generating one is theoretically faster than generating a
-# full or partial record, repeated attempts to verify data in the same field (e.g. a 
-# counter whose value is stored against a single locator/key) or record may be slower 
-# due to lack of caching.
+# A lookup record only contains data for a single field. The database does not
+# cache lookup records, so, while generating one is theoretically faster than
+# generating a full or partial record, repeated attempts to verify data in the
+# same field (e.g. a counter whose value is stored against a single
+# locator/key) or record may be slower due to lack of caching.
 #
 # DEFAULT: false
 enable_verify_by_lookup:
@@ -247,22 +266,22 @@ enable_verify_by_lookup:
 #############################################
 init:
 
-  # Configuration for the root user. If provided, will override values for flat config
-  # options that are prefixed with "init_"
+  # Configuration for the root user. If provided, will override values for flat
+  # config options that are prefixed with "init_"
   root:
 
-    # The initial root password for Concourse Server. This password is used to set 
-    # up the initial administrator account when the server is first run. It is 
-    # strongly recommended to change this password immediately after the initial setup 
-    # to maintain security.
+    # The initial root password for Concourse Server. This password is used to
+    # set up the initial administrator account when the server is first run. It
+    # is strongly recommended to change this password immediately after the
+    # initial setup to maintain security.
     #
     # DEFAULT: the value of the init_root_password option, if available.
     # Otherwise "admin"
     password:
 
-    # The initial root username for Concourse Server. This username is associated 
-    # with the initial administrator account. It is strongly
+    # The initial root username for Concourse Server. This username is
+    # associated with the initial administrator account.
     #
-    # DEFAULT: the value of the init_root_username option, if available. 
+    # DEFAULT: the value of the init_root_username option, if available.
     # Otherwise "admin"
-    username:
+    username:

concourse-server/src/main/java/com/cinchapi/concourse/server/GlobalState.java

@@ -290,6 +290,28 @@ public final class GlobalState extends Constants {
      */
     public static String INIT_ROOT_USERNAME = "admin";
 
+    /**
+     * Potentially use multiple threads to asynchronously read data from disk.
+     * <p>
+     * When enabled, reads will typically be faster when accessing data too
+     * large to fit in memory or no longer cached due to memory constraints.
+     * </p>
+     * <p>
+     * This setting is particularly useful for search data since those indexes
+     * are not cached by default (unless {@link #ENABLE_SEARCH_CACHE} is
+     * enabled). Even if search records are cached, this setting may still
+     * provide a performance boost if the size of some search metadata exceeds
+     * the limits of what is cacheable in memory.
+     * </p>
+     * <p>
+     * <strong>NOTE:</strong> There might be some overhead that could make some
+     * reads slower if all their relevant segment metadata is cached and there
+     * is high contention.
+     * </p>
+     */
+    @Experimental
+    public static boolean ENABLE_ASYNC_DATA_READS = false;
+
     /**
      * Automatically use a combination of defragmentation, garbage collection
      * and load balancing within the data files to optimize storage for read
@@ -421,6 +443,9 @@ public final class GlobalState extends Constants {
         MAX_SEARCH_SUBSTRING_LENGTH = config.getOrDefault(
                 "max_search_substring_length", MAX_SEARCH_SUBSTRING_LENGTH);
 
+        ENABLE_ASYNC_DATA_READS = config.getOrDefault("enable_async_data_reads",
+                ENABLE_ASYNC_DATA_READS);
+
         ENABLE_COMPACTION = config.getOrDefault("enable_compaction",
                 ENABLE_COMPACTION);
 

concourse-server/src/main/java/com/cinchapi/concourse/server/storage/db/Database.java

@@ -55,6 +55,7 @@
 import com.cinchapi.common.base.CheckedExceptions;
 import com.cinchapi.common.base.Verify;
 import com.cinchapi.common.collect.concurrent.ThreadFactories;
+import com.cinchapi.common.concurrent.JoinableExecutorService;
 import com.cinchapi.concourse.annotate.Restricted;
 import com.cinchapi.concourse.server.GlobalState;
 import com.cinchapi.concourse.server.concurrent.AwaitableExecutorService;
@@ -181,6 +182,12 @@ private static Segment findSegment(Collection<Segment> segments,
      */
     private static final Cache<Composite, CorpusRecord> DISABLED_CORPUS_CACHE = new NoOpCache<>();
 
+    /**
+     * Global flag to indicate if async data reads are enabled
+     */
+    // Copied here as a final variable for (hopeful) performance gains.
+    private static final boolean ENABLE_ASYNC_DATA_READS = GlobalState.ENABLE_ASYNC_DATA_READS;
+
     /**
      * Global flag that indicates if compaction is enabled.
      */
@@ -299,6 +306,12 @@ private static Segment findSegment(Collection<Segment> segments,
      */
     private transient boolean running = false;
 
+    /**
+     * A {@link JoinableExecutorService} that facilitates
+     * {@link #ENABLE_ASYNC_DATA_READS async data reading} if enabled.
+     */
+    private transient JoinableExecutorService reader;
+
     /**
      * We hold direct references to the current Segment. This pointer changes
      * whenever the database triggers a sync operation.
@@ -373,6 +386,11 @@ private static Segment findSegment(Collection<Segment> segments,
      */
     private transient AwaitableExecutorService writer;
 
+    /**
+     * Dynamic configuration.
+     */
+    private transient Options options = new Options();
+
     /**
      * Construct a Database that is backed by the default location which is in
      * {@link GlobalState#DATABASE_DIRECTORY}.
@@ -919,6 +937,11 @@ public void start() {
             // @formatter:on
             Logger.info("Database is running with compaction {}.",
                     ENABLE_COMPACTION ? "ON" : "OFF");
+
+            reader = ENABLE_ASYNC_DATA_READS ? new JoinableExecutorService(
+                    Math.max(3, Runtime.getRuntime().availableProcessors()),
+                    ThreadFactories.namingThreadFactory("DatabaseReader"))
+                    : null;
         }
 
     }
@@ -944,6 +967,9 @@ public void stop() {
             }
             fullCompaction.shutdownNow();
             incrementalCompaction.shutdownNow();
+            if(reader != null) {
+                reader.shutdown();
+            }
         }
     }
 
@@ -1000,6 +1026,7 @@ public boolean verify(Write write, long timestamp) {
      * @param toks {@code query} split by whitespace
      * @return the CorpusRecord
      */
+    @SuppressWarnings("unchecked")
     private CorpusRecord getCorpusRecord(Text key, Text infix) {
         masterLock.readLock().lock();
         try {
@@ -1009,8 +1036,25 @@ private CorpusRecord getCorpusRecord(Text key, Text infix) {
                     : DISABLED_CORPUS_CACHE;
             return cache.get(composite, () -> {
                 CorpusRecord $ = CorpusRecord.createPartial(key, infix);
-                for (Segment segment : segments) {
-                    segment.corpus().seek(composite, $);
+                if(options.enableAsyncCorpusDataReads()) {
+                    int i = 0;
+                    Fragment<Text, Text, Position>[] fragments = new Fragment[segments
+                            .size()];
+                    Runnable[] tasks = new Runnable[segments.size()];
+                    for (Segment segment : segments) {
+                        Fragment<Text, Text, Position> fragment = new Fragment<>(
+                                key, infix);
+                        fragments[i] = fragment;
+                        tasks[i++] = () -> segment.corpus().seek(composite,
+                                fragment);
+                    }
+                    reader.join(tasks);
+                    $.append(fragments);
+                }
+                else {
+                    for (Segment segment : segments) {
+                        segment.corpus().seek(composite, $);
+                    }
                 }
                 return $;
             });
@@ -1029,14 +1073,32 @@ private CorpusRecord getCorpusRecord(Text key, Text infix) {
      * @param key
      * @return the IndexRecord
      */
+    @SuppressWarnings("unchecked")
     private IndexRecord getIndexRecord(Text key) {
         masterLock.readLock().lock();
         try {
             Composite composite = Composite.create(key);
             return indexCache.get(composite, () -> {
                 IndexRecord $ = IndexRecord.create(key);
-                for (Segment segment : segments) {
-                    segment.index().seek(composite, $);
+                if(options.enableAsyncIndexDataReads()) {
+                    int i = 0;
+                    Fragment<Text, Value, Identifier>[] fragments = new Fragment[segments
+                            .size()];
+                    Runnable[] tasks = new Runnable[segments.size()];
+                    for (Segment segment : segments) {
+                        Fragment<Text, Value, Identifier> fragment = new Fragment<>(
+                                key, null);
+                        fragments[i] = fragment;
+                        tasks[i++] = () -> segment.index().seek(composite,
+                                fragment);
+                    }
+                    reader.join(tasks);
+                    $.append(fragments);
+                }
+                else {
+                    for (Segment segment : segments) {
+                        segment.index().seek(composite, $);
+                    }
                 }
                 return $;
             });
@@ -1112,14 +1174,32 @@ private Record<Identifier, Text, Value> getLookupRecord(Identifier record,
      * @param identifier
      * @return the TableRecord
      */
+    @SuppressWarnings("unchecked")
     private TableRecord getTableRecord(Identifier identifier) {
         masterLock.readLock().lock();
         try {
             Composite composite = Composite.create(identifier);
             return tableCache.get(composite, () -> {
                 TableRecord $ = TableRecord.create(identifier);
-                for (Segment segment : segments) {
-                    segment.table().seek(composite, $);
+                if(options.enableAsyncTableDataReads()) {
+                    int i = 0;
+                    Fragment<Identifier, Text, Value>[] fragments = new Fragment[segments
+                            .size()];
+                    Runnable[] tasks = new Runnable[segments.size()];
+                    for (Segment segment : segments) {
+                        Fragment<Identifier, Text, Value> fragment = new Fragment<>(
+                                identifier, null);
+                        fragments[i] = fragment;
+                        tasks[i++] = () -> segment.table().seek(composite,
+                                fragment);
+                    }
+                    reader.join(tasks);
+                    $.append(fragments);
+                }
+                else {
+                    for (Segment segment : segments) {
+                        segment.table().seek(composite, $);
+                    }
                 }
                 return $;
             });
@@ -1146,6 +1226,7 @@ private TableRecord getTableRecord(Identifier identifier) {
      * @param key
      * @return the TableRecord
      */
+    @SuppressWarnings("unchecked")
     private TableRecord getTableRecord(Identifier identifier, Text key) {
         masterLock.readLock().lock();
         try {
@@ -1157,8 +1238,25 @@ private TableRecord getTableRecord(Identifier identifier, Text key) {
                 Composite composite = Composite.create(identifier, key);
                 table = tablePartialCache.get(composite, () -> {
                     TableRecord $ = TableRecord.createPartial(identifier, key);
-                    for (Segment segment : segments) {
-                        segment.table().seek(composite, $);
+                    if(options.enableAsyncTableDataReads()) {
+                        int i = 0;
+                        Fragment<Identifier, Text, Value>[] fragments = new Fragment[segments
+                                .size()];
+                        Runnable[] tasks = new Runnable[segments.size()];
+                        for (Segment segment : segments) {
+                            Fragment<Identifier, Text, Value> fragment = new Fragment<>(
+                                    identifier, key);
+                            fragments[i] = fragment;
+                            tasks[i++] = () -> segment.table().seek(composite,
+                                    fragment);
+                        }
+                        reader.join(tasks);
+                        $.append(fragments);
+                    }
+                    else {
+                        for (Segment segment : segments) {
+                            segment.table().seek(composite, $);
+                        }
                     }
                     return $;
                 });
@@ -1312,6 +1410,44 @@ public boolean contains(String key, long record) {
 
     }
 
+    /**
+     * Dynamic runtime configuration options.
+     *
+     * @author Jeff Nelson
+     */
+    private final class Options {
+
+        /**
+         * Return {@code true} if {@link CorpusRecord corpus records} should be
+         * read from disk asynchronously.
+         * 
+         * @return the optional configuration value.
+         */
+        boolean enableAsyncCorpusDataReads() {
+            return running && ENABLE_ASYNC_DATA_READS;
+        }
+
+        /**
+         * Return {@code true} if {@link IndexRecord index records} should be
+         * read from disk asynchronously.
+         * 
+         * @return the optional configuration value.
+         */
+        boolean enableAsyncIndexDataReads() {
+            return running && ENABLE_ASYNC_DATA_READS;
+        }
+
+        /**
+         * Return {@code true} if {@link TableRecord table records} should be
+         * read from disk asynchronously.
+         * 
+         * @return the optional configuration value.
+         */
+        boolean enableAsyncTableDataReads() {
+            return running && ENABLE_ASYNC_DATA_READS;
+        }
+    }
+
     /**
      * {@link Cache} wrapper that is aware of whether the {@link Database} is
      * running and behaves accordingly.

concourse-server/src/main/java/com/cinchapi/concourse/server/storage/db/Fragment.java

@@ -0,0 +1,127 @@
+/*
+ * Copyright (c) 2013-2025 Cinchapi Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.cinchapi.concourse.server.storage.db;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import com.cinchapi.concourse.server.io.Byteable;
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * A {@link Fragment} represents a subset of another {@link Record}.
+ * <p>
+ * {@link Fragment Fragments} are used to asynchronously read {@link Revision
+ * revisions} into memory to later be {@link Record#append(Fragment...) merged}
+ * into a complete {@link Record}.
+ * </p>
+ * <p>
+ * Regardless of their locator or key type, {@link Fragment Fragments} are
+ * optimized for O(1) accumulation of {@link Revision revisions} in a
+ * <strong>single thread</strong> and do not support any query operations.
+ * </p>
+ *
+ * @author Jeff Nelson
+ */
+@NotThreadSafe
+public class Fragment<L extends Byteable & Comparable<L>, K extends Byteable & Comparable<K>, V extends Byteable & Comparable<V>>
+        extends Record<L, K, V> {
+
+    /**
+     * The accumulated {@link Revision revisions}.
+     */
+    private final List<Revision<L, K, V>> revisions = new ArrayList<>();
+
+    /**
+     * Construct a new instance.
+     * 
+     * @param locator
+     * @param key
+     */
+    protected Fragment(L locator, K key) {
+        super(locator, key);
+    }
+
+    @Override
+    public void append(Revision<L, K, V> revision) {
+        revisions.add(revision);
+    }
+
+    @Override
+    public int cardinality() {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean contains(K key, V value) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean contains(K key, V value, long timestamp) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public Set<V> get(K key) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public Map<K, Set<V>> getAll() {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public Map<K, Set<V>> getAll(long timestamp) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean isEmpty() {
+        return revisions.isEmpty();
+    }
+
+    @Override
+    public Set<K> keys() {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public Set<K> keys(long timestamp) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    protected Map<K, Set<V>> $createDataMap() {
+        return ImmutableMap.of();
+    }
+
+    /**
+     * Return the accumulated {@link Revision revisions} in this
+     * {@link Fragment}.
+     * 
+     * @return the {@link Revision revisions}
+     */
+    List<Revision<L, K, V>> revisions() {
+        return revisions;
+    }
+
+}

concourse-server/src/main/java/com/cinchapi/concourse/server/storage/db/Record.java

@@ -127,6 +127,18 @@ protected Record(L locator, @Nullable K key) {
         this.partial = key != null;
     }
 
+    /**
+     * {@link #append(Revision) Append} every {@link Fragment#revisions()
+     * revision} from each of the {@code fragments}.
+     * 
+     * @param fragments
+     */
+    public void append(Fragment<L, K, V>... fragments) {
+        for (Fragment<L, K, V> fragment : fragments) {
+            fragment.revisions().forEach(this::append);
+        }
+    }
+
     /**
      * Append {@code revision} to the record by updating the in-memory indices.
      * The {@code revision} must have:

concourse-server/src/main/java/com/cinchapi/concourse/server/storage/db/TableRecord.java

@@ -41,8 +41,8 @@
  * A logical grouping of data for a single entity.
  * <p>
  * This is the primary view of stored data within Concourse, similar to a Row in
- * a traditional database. PrimaryRecords are designed to efficiently handle
- * direct/non-query reads.
+ * a traditional database. A {@link TableRecord} is designed to efficiently
+ * handle direct/non-query reads.
  * </p>
  * 
  * @author Jeff Nelson