how to compact an Oracle NoSQL database

[dario-vega]

Let me start with a brief description extracted from the documentation about how Oracle NoSQL uses the disk to store data.

Storage Nodes use their available disk space for two purposes:

• To store your data.
• To save reserved files.

Reserved files consist of data that has already been replicated to active replica nodes. The purpose of storing a copy of this data is to use it for Replica Nodes that lose contact with the Master Node. Losing contact typically occurs because Replica nodes are shut down, a network partition event occurs, or another transient problem occurs.

The Storage Node is primarily designed to consume the amount of disk space you assign it and to use the remaining disk space to save the reserved files. Each Storage Node manages its available disk space, leaving 5 GB free for recovery purposes. Your intervention is typically not required in this disk management process unless a storage node exceeds its available disk capacity.

Note: If a Storage Node (SN) consumes more than what is assigned as storagedirsize, including leaving 5 GB of space free, the SN automatically attempts to free up disk space by deleting reserved files (not your data files), until more than 5 GB of space is available. If the Storage Node is unable to free up enough space, it suspends write operations to the node. Read operations continue as normal. Write operations resume automatically once the node obtains sufficient free disk space.

The most common reason a node's storage directory fills up is because of reserved files. If the Storage Node exceeds its disk threshold, it continues to delete the reserved files until the threshold is no longer exceeded. With a very low impact on the performances.

Be sure to monitor disk usage regularly using the statistics provided, as described in Monitoring Disk Usage.

Snapshot

When you create a snapshot, it is stored in a subdirectory of the SN. But these snapshots don't become persistent backups unless they are copied to separate storage. It is your responsibility to copy each of the snapshots to another location, preferably on a different machine, for data safety.

It is also your responsibility to manage the space consumed by snapshots.

Impact of Erasure with snapshots

Snapshot-based backups create hard-links to original files. Until these backups are copied to their target location (complete off-store copy work) and the corresponding hard-links are removed (performing a snapshot remove command), erasure doesn't process obsolete data in those files. Erasure ignores files with hard-links to them.

Avoiding Disk Usage Violation

The storage engine does not consider the data consumed by snapshots when it collects information about disk space usage. Initially, the files in the snapshot are considered to be part of the live data of the store. Over time, though, as older files are cleaned and deleted, their presence in the snapshot causes the files to be retained and use the disk space that is not taken into account by the storage engine. It could cause a disk usage violation, in which case further writes to the store are disabled. To avoid this problem, users should delete snapshot files at regular intervals.

The DbSpace tool

The DbSpace displays the disk space utilization for an environment. Is it another way to obtain information about the space consumed to store your data and the space used by the reserved files. You can execute the following command for each disk in your store. Behind the scenes, Oracle NoSQL uses Berkeley JE.

[opc@node1-nosql ~]$ java -cp $KVHOME/lib/kvstore.jar com.sleepycat.je.util.DbSpace -h /home/opc/nosql/data/disk1/rg1-rn1/env
                      % Utilized
  File    Size (kB)  Avg  Min  Max
--------  ---------  ---- ---  ---
00000006     499109    0    0    0
00000007       2828   85   85   85
 TOTALS      501938    1    1    1

1,048,575 kB are used by additional reserved files:
0x5
[opc@node1-nosql ~]$ java -cp $KVHOME/lib/kvstore.jar com.sleepycat.je.util.DbSpace -h /home/opc/nosql/data/disk1/rg1-rn1/env.old
                      % Utilized
  File    Size (kB)  Avg  Min  Max
--------  ---------  ---- ---  ---
00000005    1048575   90   90   90
00000006     499109   90   90   90
00000007        591   98   98   98
 TOTALS     1548277   90   90   90

4,194,301 kB are used by additional reserved files:
0x1-0x4

Compact an Oracle NoSQL database by restoring from a snapshot

The snapshots do not include the reserved files. After Restoring Directly from a Snapshot, You have 2 directories env and env.old - without having the same files. You can delete the directory env.old to use less space. It is a way to delete the reserved files and compact your store.

/home/opc/nosql/data/disk1/rg1-rn1/env.old:
total 5.5G
-rw-rw-r--. 1 opc opc    0 Jan 10 07:12 je.lck
-rw-rw-r--. 1 opc opc 1.0G Jan 10 08:56 00000001.jdb
-rw-rw-r--. 1 opc opc 1.0G Jan 10 09:00 00000002.jdb
-rw-rw-r--. 1 opc opc 1.0G Jan 10 09:03 00000003.jdb
-rw-rw-r--. 1 opc opc 1.0G Jan 10 09:14 00000004.jdb
-rw-rw-r--. 2 opc opc 1.0G Jan 10 09:20 00000005.jdb
-rw-rw-r--. 2 opc opc 488M Jan 10 10:17 00000006.jdb
-rw-rw-r--. 1 opc opc  208 Jan 10 10:17 election.persisted
-rw-rw-r--. 1 opc opc 592K Jan 10 10:17 00000007.jdb

/home/opc/nosql/data/disk1/rg1-rn1/recovery:
total 0

/home/opc/nosql/data/disk1/rg1-rn1/env:
total 1.5G
-rw-rw-r--. 2 opc opc 1.0G Jan 10 09:20 00000005.jdb
-rw-rw-r--. 2 opc opc 488M Jan 10 10:17 00000006.jdb
-rw-rw-r--. 1 opc opc    0 Jan 10 10:19 je.lck
-rw-rw-r--. 1 opc opc  208 Jan 10 10:23 election.persisted
-rw-rw-r--. 1 opc opc 2.8M Jan 10 10:26 00000007.jdb

/home/opc/nosql/data/disk1/rg1-rn1/snapshots:
total 0

if a customer wants to compact a store (e.g UAT cluster), Can they use this procedure? yes, you can

• delete all current snapshots
• create a snapshot
• restore from the snapshot
• delete the snapshot
• delete the env.old directory
• create a new snapshot for backup

ATTENTION Due to the distributed nature and scale of the Oracle NoSQL Database, when executing this procedure be sure that it is applied to all SN and all disks used by each SN.