Skip to content

Commit

Permalink
Merge pull request #49 from kontent-ai/up_down_migrations
Browse files Browse the repository at this point in the history
Up down migrations
  • Loading branch information
winklertomas committed Jun 7, 2023
2 parents 0288dcc + 3cd3ae7 commit 7fe9d35
Show file tree
Hide file tree
Showing 13 changed files with 381 additions and 52 deletions.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ The supported commands are divided into groups according to their target, at thi
* `migration add --name <migration name>` – Generates a script file (in JavaScript or TypeScript) for running a migration on a [Kontent.ai](https://kontent.ai/) project.
* The file is stored in the `Migrations` directory within the root of your repository.
* Add your migration script in the body of the `run` function using the [Kontent.ai Management SDK](https://github.com/kontent-ai/management-sdk-js) that was injected via the `apiClient` parameter.
* Add your rollback script in the body of the `rollback` function using the [Kontent.ai Management SDK](https://github.com/kontent-ai/management-sdk-js) that was injected via the `apiClient` parameter.
* To choose between JavaScript and TypeScript when generating the script file, use the `--template-type` option, such as `--template-type "javascript"`.
* The migration template contains an `order` property that is used to run a batch of migrations (range or all) in the specified order. Order can be one of the two types - `number` or `date`.
* Ordering by `number` has a higher priority. The `order` must be a unique positive integer or zero. There may be gaps between migrations, for example, the following sequence is perfectly fine 0,3,4,5,10
Expand All @@ -120,7 +121,9 @@ The supported commands are divided into groups according to their target, at thi
* By adding `--range` you need to add value in form of `number:number` in case of number ordering or in the format of `Tyyyy-mm-dd-hh-mm-ss:yyyy-mm-dd-hh-mm-ss` in case of date order.
> When using the range with dates, only the year value is mandatory and all other values are optional. It is fine to have a range specified like `T2023-01:2023-02`. It will take all migrations created in January of 2023. Notice the T at the beginning of the Date range. It helps to separate date ordering from number order.
* You can execute a migration against a specific project (options `--project <YOUR_PROJECT_ID> --api-key <YOUR_MANAGEMENT_API_KEY>`) or environment stored in the local configuration file (option `--environment <YOUR_ENVIRONMENT_NAME>`).
* To execute your `rollback` scripts instead of `run` scripts, use the switch option `--rollback` or shortly `-b`.
* After each run of a migration script, the CLI logs the execution into a status file. This file holds data for the next run to prevent running the same migration script more than once. You can choose to override this behavior, for example for debugging purposes, by using the `--force` parameter.
> Note: For every migration there is only one record in the status file. Calling run/rollback continously overrides the that record with new data.
* You can choose whether you want to keep executing the migration scripts even if one migration script fails (option `--continue-on-error`) or whether you want to get additional information logged by HttpService into the console (option `--log-http-service-errors-to-console`).

* `backup --action [backup|restore|clean]` - This command enables you to use [Kontent.ai backup manager](https://github.com/kontent-ai/backup-manager-js)
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@kontent-ai/cli",
"version": "0.7.1",
"version": "0.7.3",
"description": "Command line interface tool that can be used for generating and runningKontent.ai migration scripts",
"main": "./lib/index.js",
"types": "./lib/types/index.d.ts",
Expand Down
35 changes: 27 additions & 8 deletions src/cmds/migration/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { createManagementClient } from '../../managementClientFactory';
import { getPluginsFilePath, loadMigrationsExecutionStatus } from '../../utils/statusManager';
import { IMigration } from '../../models/migration';
import { IRange } from '../../models/range';
import { IStatus, Operation } from '../../models/status';
import { loadStatusPlugin } from '../../utils/status/statusPlugin';

const runMigrationCommand: yargs.CommandModule = {
Expand Down Expand Up @@ -63,6 +64,12 @@ const runMigrationCommand: yargs.CommandModule = {
default: false,
type: 'boolean',
},
rollback: {
alias: 'b',
describe: 'Call rollback function from the migration',
default: false,
type: 'boolean',
},
})
.conflicts('all', 'name')
.conflicts('range', 'name')
Expand Down Expand Up @@ -115,9 +122,12 @@ const runMigrationCommand: yargs.CommandModule = {
const runRange = argv.range && (exports.getRange(argv.range) || getRangeDate(argv.range));
const logHttpServiceErrorsToConsole = argv.logHttpServiceErrorsToConsole;
const continueOnError = argv.continueOnError;
const rollback = argv.rollback;
let migrationsResults: number = 0;
const runForce = argv.force;

const operation: Operation = rollback ? 'rollback' : 'run';

if (argv.environment) {
const environments = getEnvironmentsConfig();

Expand All @@ -133,7 +143,14 @@ const runMigrationCommand: yargs.CommandModule = {
logHttpServiceErrorsToConsole,
});

await loadMigrationsExecutionStatus(plugin?.readStatus ?? null);
const migrationOptions = {
client: apiClient,
projectId: projectId,
operation: operation,
saveStatusFromPlugin: plugin?.saveStatus ?? null,
};

const migrationsStatus = await loadMigrationsExecutionStatus(plugin?.readStatus ?? null);

if (runAll || runRange) {
let migrationsToRun = await loadMigrationFiles();
Expand All @@ -148,17 +165,17 @@ const runMigrationCommand: yargs.CommandModule = {
if (runForce) {
console.log('Skipping to check already executed migrations');
} else {
migrationsToRun = skipExecutedMigrations(migrationsToRun, projectId);
migrationsToRun = skipExecutedMigrations(migrationsStatus, migrationsToRun, projectId, operation);
}

if (migrationsToRun.length === 0) {
console.log('No migrations to run.');
}

const sortedMigrationsToRun = migrationsToRun.sort(orderComparator);
const sortedMigrationsToRun = migrationsToRun.sort(orderComparator(rollback));
let executedMigrationsCount = 0;
for (const migration of sortedMigrationsToRun) {
const migrationResult = await runMigration(migration, apiClient, projectId, plugin?.saveStatus ?? null);
const migrationResult = await runMigration(migrationsStatus, migration, migrationOptions);

if (migrationResult > 0) {
if (!continueOnError) {
Expand All @@ -179,7 +196,7 @@ const runMigrationCommand: yargs.CommandModule = {
module: migrationModule,
};

migrationsResults = await runMigration(migration, apiClient, projectId, plugin?.saveStatus ?? null);
migrationsResults = await runMigration(migrationsStatus, migration, migrationOptions);
}

process.exit(migrationsResults);
Expand Down Expand Up @@ -268,8 +285,8 @@ const checkForInvalidOrder = (migrationsToRun: IMigration[]): void => {
}
};

const skipExecutedMigrations = (migrations: IMigration[], projectId: string): IMigration[] => {
const executedMigrations = getSuccessfullyExecutedMigrations(migrations, projectId);
const skipExecutedMigrations = (migrationStatus: IStatus, migrations: IMigration[], projectId: string, operation: Operation): IMigration[] => {
const executedMigrations = getSuccessfullyExecutedMigrations(migrationStatus, migrations, projectId, operation);
const result: IMigration[] = [];

for (const migration of migrations) {
Expand All @@ -283,7 +300,7 @@ const skipExecutedMigrations = (migrations: IMigration[], projectId: string): IM
return result;
};

const orderComparator = (migrationPrev: IMigration, migrationNext: IMigration) => {
const comparator = (migrationPrev: IMigration, migrationNext: IMigration) => {
if (typeof migrationPrev.module.order === 'number' && typeof migrationNext.module.order === 'number') {
return migrationPrev.module.order - migrationNext.module.order;
}
Expand All @@ -295,6 +312,8 @@ const orderComparator = (migrationPrev: IMigration, migrationNext: IMigration) =
return typeof migrationPrev.module.order === 'number' ? -1 : 1;
};

const orderComparator = (rollback: boolean) => (migrationPrev: IMigration, migrationNext: IMigration) => rollback ? -comparator(migrationPrev, migrationNext) : comparator(migrationPrev, migrationNext);

const formatDate = (date: string, time: string) => {
if (time === '') {
time = '00:00';
Expand Down
3 changes: 3 additions & 0 deletions src/models/status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ export interface IMigrationStatus {
readonly success: boolean;
readonly order: number | Date;
readonly time: Date;
readonly lastOperation?: Operation;
}

export interface IStatus {
[projectId: string]: IMigrationStatus[];
}

export type Operation = 'run' | 'rollback';
4 changes: 4 additions & 0 deletions src/tests/__snapshots__/createMigration.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ const migration = {
order: 1,
run: async (apiClient) => {
},
rollback: async(apiClient) => {
},
};
module.exports = migration;
Expand All @@ -19,6 +21,8 @@ const migration: MigrationModule = {
order: 1,
run: async (apiClient) => {
},
rollback: async(apiClient) => {
},
};
export default migration;
Expand Down
1 change: 1 addition & 0 deletions src/tests/__snapshots__/statusManager.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

exports[`Status manager Project has success status in status manager file 1`] = `
Object {
"lastOperation": "run",
"name": "migration1",
"order": 1,
"success": true,
Expand Down
2 changes: 1 addition & 1 deletion src/tests/cmds/run/runMigration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ const migrations: IMigration[] = [
];

jest.spyOn(statusManager, 'markAsCompleted').mockImplementation(async () => {});
jest.spyOn(statusManager, 'loadMigrationsExecutionStatus').mockImplementation(async () => {});
jest.spyOn(statusManager, 'loadMigrationsExecutionStatus').mockImplementation(async () => ({}));
jest.spyOn(migrationUtils, 'loadMigrationFiles').mockReturnValue(
new Promise((resolve) => {
resolve(migrations);
Expand Down
202 changes: 202 additions & 0 deletions src/tests/cmds/run/runMigrationWithRollback.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
import yargs from 'yargs';
import { IMigration } from '../../../models/migration';
import * as migrationUtils from '../../../utils/migrationUtils';
import * as statusManager from '../../../utils/statusManager';
import { IStatus } from '../../../models/status';

const { handler } = require('../../../cmds/migration/run');

const migrations: IMigration[] = [
{
name: 'test1',
module: {
order: 1,
run: async () => {
console.log('test1');
},
rollback: async () => {
console.log('rollback 1');
},
},
},
{
name: 'test2',
module: {
order: 2,
run: async () => {
console.log('test2');
},
rollback: async () => {
console.log('rollback 2');
},
},
},
{
name: 'test3',
module: {
order: 3,
run: async () => {
console.log('test3');
},
rollback: async () => {
console.log('rollback 3');
},
},
},
];

const migrationStatus: IStatus = {
'fcb801c6-fe1d-41cf-af91-ec13802a1ed2': [
{
name: 'test1',
success: true,
order: 1,
time: new Date(Date.now()),
lastOperation: 'run',
},
{
name: 'test2',
success: true,
order: 2,
time: new Date(Date.now()),
lastOperation: 'rollback',
},
{
name: 'test3',
success: true,
order: 3,
time: new Date(Date.now()),
lastOperation: 'run',
},
],
};

const migrationStatusAllRollbacks: IStatus = {
'fcb801c6-fe1d-41cf-af91-ec13802a1ed2': [
{
name: 'test1',
success: true,
order: 1,
time: new Date(Date.now()),
lastOperation: 'rollback',
},
{
name: 'test2',
success: true,
order: 2,
time: new Date(Date.now()),
lastOperation: 'rollback',
},
{
name: 'test3',
success: true,
order: 3,
time: new Date(Date.now()),
lastOperation: 'rollback',
},
],
};

describe('run migration command tests', () => {
let mockExit: jest.SpyInstance<never, [code?: number | undefined]>;
beforeEach(() => {
jest.restoreAllMocks();
jest.clearAllMocks();

jest.spyOn(statusManager, 'markAsCompleted').mockImplementation(async () => {});

jest.spyOn(migrationUtils, 'loadMigrationFiles').mockReturnValue(
new Promise((resolve) => {
resolve(migrations);
})
);

mockExit = jest.spyOn(process, 'exit').mockImplementation(() => {
return undefined as never;
});
});

it('with date range all, all rollbacks should be called', async () => {
jest.spyOn(statusManager, 'loadMigrationsExecutionStatus').mockImplementation(async () => ({}));

const args: any = yargs.parse([], {
apiKey: '',
projectId: 'fcb801c6-fe1d-41cf-af91-ec13802a1ed2',
range: '',
all: true,
rollback: 'true',
});

const migration1 = jest.spyOn(migrations[0].module, 'rollback');
const migration2 = jest.spyOn(migrations[1].module, 'rollback');
const migration3 = jest.spyOn(migrations[2].module, 'rollback');

await handler(args);

const migration3Order = migration3.mock.invocationCallOrder[0];
const migration2Order = migration2.mock.invocationCallOrder[0];
const migration1Order = migration1.mock.invocationCallOrder[0];

expect(migration1).toBeCalledTimes(1);
expect(migration2).toBeCalledTimes(1);
expect(migration3).toBeCalledTimes(1);

expect(migration3Order).toBeLessThan(migration2Order);
expect(migration2Order).toBeLessThan(migration1Order);

expect(mockExit).toHaveBeenCalledWith(0);
});

it('with date range all, only rollback from first and last migrations should should be called', async () => {
jest.spyOn(statusManager, 'loadMigrationsExecutionStatus').mockImplementation(async () => migrationStatus);

const args: any = yargs.parse([], {
apiKey: '',
projectId: 'fcb801c6-fe1d-41cf-af91-ec13802a1ed2',
range: '',
all: true,
rollback: 'true',
});

const migration1 = jest.spyOn(migrations[0].module, 'rollback');
const migration2 = jest.spyOn(migrations[1].module, 'rollback');
const migration3 = jest.spyOn(migrations[2].module, 'rollback');

await handler(args);

const migration3Order = migration3.mock.invocationCallOrder[0];
const migration1Order = migration1.mock.invocationCallOrder[0];

expect(migration3).toBeCalledTimes(1);
expect(migration2).not.toBeCalled();

expect(migration3Order).toBeLessThan(migration1Order);
expect(migration1).toBeCalledTimes(1);

expect(mockExit).toHaveBeenCalledWith(0);
});

it('with date range all, only rollback from first and last migrations should should be called', async () => {
jest.spyOn(statusManager, 'loadMigrationsExecutionStatus').mockImplementation(async () => migrationStatusAllRollbacks);

const args: any = yargs.parse([], {
apiKey: '',
projectId: 'fcb801c6-fe1d-41cf-af91-ec13802a1ed2',
range: '',
all: true,
rollback: 'true',
});

const migration1 = jest.spyOn(migrations[0].module, 'rollback');
const migration2 = jest.spyOn(migrations[1].module, 'rollback');
const migration3 = jest.spyOn(migrations[2].module, 'rollback');

await handler(args);

expect(migration3).not.toBeCalled();
expect(migration2).not.toBeCalled();
expect(migration1).not.toBeCalled();

expect(mockExit).toHaveBeenCalledWith(0);
});
});
Loading

0 comments on commit 7fe9d35

Please sign in to comment.