MagickCache is a free, open-source, high-performance repository for securely storing images, image sequences, video, audio, metadata, and arbitrary binary content. Resources are memory-mapped for fast access, and image retrieval can be limited to a specific region for additional efficiency.
Resources can be stored permanently or assigned a time-to-live (TTL), after which they expire and can be removed. A single MagickCache repository can scale to billions of resources, making it well suited to digital media archiving, asset management, and content delivery workloads.
MagickCache works in concert with ImageMagick. Download and install MagickCache, then create and populate a repository with your media and metadata. MagickCache is available as both a command-line utility (magick-cache) and a C API.
- Key Concepts
- Create a Digital Media Repository
- Put Content in the Repository
- Get Content from the Repository
- Identify Repository Content
- Delete and Expire Content
- MagickCache Is Not Just for Images
- Delete a Repository
- Security
- Portability
- MagickCache API
- ImageMagick Access
| Term | Meaning |
|---|---|
| Repository | A directory on your filesystem that holds resources (for example, /opt/dmr). |
| Resource | A single stored item: an image, a blob, or a piece of metadata. |
| IRI | The identifier for a resource, in the form project/type/resource-path. |
| Passkey | A secret that identifies you as the owner of a resource (or of the repository). |
| Passphrase | An optional secret used to scramble image content before it is stored. |
| TTL | An optional time-to-live after which a resource is considered expired. |
Resource types are image, blob, and meta.
You need a location to store and retrieve your content. Create a repository on your local filesystem:
magick-cache -passkey ~/.passkey create /opt/dmrHere, ~/.passkey is a file containing your repository passkey. The passkey can contain any binary content, including a password, phrase, image, or random data. It is sensitive to every character, including control characters and trailing newlines.
One way to create a passkey without stray control characters is:
echo -n "myPasskey" > ~/.passkey
chmod 600 ~/.passkeyFor best results, use a passkey of at least eight characters, and restrict access to the file as shown above. Keep it safe: without the passkey, you cannot identify, delete, or expire resources in the repository.
For the lowest latency and best performance, store your repository on an SSD.
You only need to create a repository once. A single repository can hold billions of images, videos, audio files, blobs, and metadata resources, but you may also create multiple repositories if you prefer to keep content separate.
Add a cast photo to the repository:
magick-cache -passkey ~/.passkey put /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson 20240508-rebecca-ferguson.jpgThe resource identifier is an IRI composed of project/type/resource-path. In this example, the project is movies, the type is image, and the resource path is mission-impossible/cast/rebecca-ferguson.
Each resource path should uniquely identify a resource. If you need multiple versions of the same image, give each version a distinct identifier.
To have a resource expire after two days, add the -ttl option:
magick-cache -passkey ~/.passkey -ttl "2 days" put /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson 20240508-rebecca-ferguson.jpgOnce the TTL has elapsed, the resource is marked as expired and can be removed with the expire command (see Delete and Expire Content).
To prevent the repository owner from viewing image content, scramble the image with a passphrase:
magick-cache -passkey ~/.passkey -passphrase ~/.passphrase -ttl "2 days" put /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson 20240508-rebecca-ferguson.jpgThe same passphrase is required when retrieving the image.
Only image resources are scrambled. Blob and metadata resources are stored as-is. If you need additional privacy for those, encrypt or obfuscate them before storing them.
To retrieve a resource:
magick-cache -passkey ~/.passkey get /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.pngThe original image was stored as a JPEG; it is converted to PNG on retrieval because of the output file extension.
To retrieve only a portion of an image, provide a geometry with an offset:
magick-cache -passkey ~/.passkey -extract 100x100+0+0 get /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.pngTo resize the image instead, omit the offset:
magick-cache -passkey ~/.passkey -extract 100x100 get /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.pngIf the image was scrambled, supply the same passphrase that was used to store it:
magick-cache -passkey ~/.passkey -passphrase ~/.passphrase get /opt/dmr movies/image/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.pngList the resources under a path:
magick-cache -passkey ~/.passkey identify /opt/dmr movies/image/mission-impossible/castEach entry includes the IRI, image dimensions (for images), content size, TTL, expiration status, and creation date. Expired resources are marked with an asterisk (*).
MagickCache also supports wildcard resource types. Quote the IRI so your shell does not expand the asterisk:
magick-cache -passkey ~/.passkey identify /opt/dmr 'movies/*/mission-impossible/cast'Other users may store content alongside yours in the same repository. However, you cannot get, identify, delete, or expire resources that were created with a different passkey.
The repository owner can access all resources:
magick-cache -passkey ~/.passkey identify /opt/dmr /Delete a specific resource:
magick-cache -passkey ~/.passkey delete /opt/dmr movies/image/mission-impossible/cast/rebecca-fergusonDelete all expired resources under a path:
magick-cache -passkey ~/.passkey expire /opt/dmr movies/image/mission-impossible/castExpiration is not automatic: a resource whose TTL has elapsed remains in the repository, flagged as expired, until expire is run. Consider running it periodically, for example from a scheduled job.
You can store video, audio, metadata, or arbitrary files using the blob and meta resource types:
magick-cache -passkey ~/.passkey put /opt/dmr movies/blob/mission-impossible/cast/rebecca-ferguson 20240508-rebecca-ferguson.mp4magick-cache -passkey ~/.passkey put /opt/dmr movies/meta/mission-impossible/cast/rebecca-ferguson 20240508-rebecca-ferguson.txt| Type | Accepts |
|---|---|
image |
Any format supported by ImageMagick. |
meta |
Text metadata. |
blob |
Any text or binary content (video, audio, documents, and so on). |
The repository owner can delete all content:
magick-cache -passkey ~/.passkey delete /opt/dmr /Use caution. This operation permanently removes all repository content and cannot be undone.
MagickCache is not intended to provide cryptographic security. Instead, it generates resource identifiers from secure-quality hashes so that resource locations are difficult to predict.
- A resource can be accessed by the repository owner or by a user who holds the passkey it was stored with.
- Anyone with sufficient privileges to read the repository files directly on disk may also be able to access stored resources.
- For additional privacy, image content can be scrambled with a passphrase before storage. The same passphrase is required to retrieve the image.
- Blob and metadata resources are never scrambled. Encrypt them yourself if they are sensitive.
- Protect your passkey and passphrase files with restrictive file permissions, and keep backups: a lost passkey cannot be recovered.
A MagickCache repository is self-contained and portable. You can move or copy it to another location or host and continue using it, provided you supply the same repository passkey.
Every command-line operation (create, put, get, identify, delete, and expire) is also available through the MagickCache C API.
ImageMagick can read and write MagickCache resources directly using the dmr: prefix. The examples below use the ImageMagick 7 magick command; with ImageMagick 6, substitute convert.
Retrieve a resource:
magick -define dmr:path=/opt/dmr -define dmr:passkey="$HOME/.passkey" \
dmr:movies/image/mission-impossible/cast/rebecca-ferguson \
rebecca-ferguson.pngStore or replace a resource:
magick rebecca-ferguson.png \
-define dmr:path=/opt/dmr -define dmr:passkey="$HOME/.passkey" \
dmr:movies/image/mission-impossible/cast/rebecca-fergusonStore metadata by setting the dmr:meta define:
magick -define dmr:path=/opt/dmr -define dmr:passkey="$HOME/.passkey" \
-define dmr:meta="Ilsa Faust" xc: \
dmr:movies/meta/mission-impossible/cast/rebecca-ferguson