[RND-623] initial matchup to MetaEd api-schema plugin #300

bradbanister · 2023-09-22T17:58:34Z

No description provided.

bradbanister · 2023-11-03T16:00:32Z

Meadowlark-js/backends/meadowlark-elasticsearch-backend/test/QueryElasticsearch.test.ts

@@ -12,16 +12,19 @@ import {
  AuthorizationStrategy,
  TraceId,
  ResourceInfo,
+  MetaEdProjectName,
+  MetaEdResourceName,
+  SemVer,


More branded strings to avoid getting them confused

bradbanister · 2023-11-03T16:01:49Z

Meadowlark-js/backends/meadowlark-mongodb-backend/test/integration/Config.ts

@@ -15,7 +15,12 @@ export const setupConfigForIntegration = async () => {
    return;
  }

-  const path = join(process.cwd(), '.env');
+  // First look for .env file paired with this Config.ts file
+  let path = join(__dirname, '.env');


allow for different .envs in each backend rather than force one big .env at root

I really like this

bradbanister · 2023-11-03T16:02:23Z

Meadowlark-js/backends/meadowlark-mongodb-backend/test/integration/Delete.test.ts

  };

  const referencedDocumentInfo: DocumentInfo = {
    ...newDocumentInfo(),
-    documentIdentity: { natural: 'delete5' },
+    documentIdentity: [{ natural: 'delete5' }],


Document identity structure changed to enforce ordering

bradbanister · 2023-11-03T16:03:57Z

Meadowlark-js/backends/meadowlark-mongodb-backend/test/integration/Upsert.test.ts

    expect(result.outboundRefs).toMatchInlineSnapshot(`
      [
-        "QtykK4uDYZK7VOChNxRsMDtOcAu6a0oe9ozl2Q",
+        "QtykK4uDYZK7VOChzZNTfqxhrlznYfnql5kIHg",


MeadowlarkIds changed due to DocumentIdentity changing to enforce ordering

bradbanister · 2023-11-03T16:04:51Z

Meadowlark-js/backends/meadowlark-postgresql-backend/src/model/MeadowlarkDocument.ts

@@ -90,12 +97,12 @@ export interface MeadowlarkDocument extends MeadowlarkDocumentId {
 */
 export function newMeadowlarkDocument(): MeadowlarkDocument {
  return {
-    meadowlark_id: undefined as unknown as MeadowlarkId,
-    document_uuid: undefined as unknown as DocumentUuid,


This was just wrong

bradbanister · 2023-11-03T16:06:30Z

Meadowlark-js/backends/meadowlark-postgresql-backend/test/setup/Config.ts

@@ -9,9 +9,17 @@ import { CachedEnvironmentConfigProvider, Config } from '@edfi/meadowlark-utilit
 import { join } from 'path';

 export const setupConfigForIntegration = async () => {
-  const path = join(process.cwd(), '.env');
+  // First look for .env file paired with this Config.ts file
+  let path = join(__dirname, '.env');


allow for different .envs in each backend rather than force one big .env at root

bradbanister · 2023-11-03T16:07:37Z

Meadowlark-js/backends/meadowlark-postgresql-backend/test/setup/Config.ts

-    throw new Error('Cannot run integration tests because there is no .env file in the repository root directory.');
+    // eslint-disable-next-line no-console
+    console.error(`Cannot run integration tests because there is no .env file in '${path}'`);
+    process.exit(-1);


Consistency with mongodb's setup

bradbanister · 2023-11-03T16:08:37Z

Meadowlark-js/package.json

    "jest": "^29.7.0",
    "jest-environment-node": "^29.7.0",
    "kafkajs": "^2.2.4",
    "lerna": "^6.6.2",
    "mongodb": "^5.9.0",
    "pg": "^8.11.3",
    "pg-format": "^1.0.4",
-    "prettier": "3.0.3",
+    "prettier": "2.8.8",


Prettier 3.x not compatible with latest Jest's inline snapshot writing

bradbanister · 2023-11-03T16:09:29Z

Meadowlark-js/packages/meadowlark-core/src/Constants.ts

-  static readonly uriVersion33b: string = 'v3.3b';
-
-  static readonly projectVersion33b: string = '3.3.1-b';
-


simplifying this

bradbanister · 2023-11-03T16:09:51Z

Meadowlark-js/packages/meadowlark-core/src/Logger.ts

  method: string,
  statusCode?: number,
  message?: string,
  extra?: any,
 ): void {
  if (isDebugEnabled()) {
-    Logger.debug(`${moduleName}.${method} ${statusCode ?? ''} ${message ?? ''}`.trimEnd(), request.traceId, extra);
+    Logger.debug(`${moduleName}.${method} ${statusCode ?? ''} ${message ?? ''}`.trimEnd(), traceId, extra);


simplifying

bradbanister · 2023-11-03T16:10:18Z

Meadowlark-js/packages/meadowlark-core/src/api-schema/ApiSchemaLoader.ts

+
+const moduleName = 'ApiSchemaFinder';
+
+const dataStandard50pre1Path: string = path.resolve(__dirname, '../ds-schemas/DataStandard-5.0.0-pre.1.json');


Hardcoded to DS 5.0-pre.1 right now

bradbanister · 2023-11-03T16:13:02Z

Meadowlark-js/packages/meadowlark-core/src/extraction/BuildDocumentInfo.ts

-): Promise<DocumentInfo> {
-  const documentIdentity: DocumentIdentity = extractDocumentIdentity(matchingMetaEdModel, body);
+): DocumentInfo {
+  const documentIdentity: DocumentIdentity = extractDocumentIdentity(resourceSchema, body);


All the extractors are rewritten, probably easier to look at them directly rather than try to follow the diff

bradbanister · 2023-11-03T16:15:04Z

Meadowlark-js/packages/meadowlark-core/src/handler/MetadataHandler.ts

- */
-export async function metaed(_frontendRequest: FrontendRequest): Promise<FrontendResponse> {
-  const modelNpmPackage = modelPackageFor(Constants.uriVersion33b);
-  const { metaEd, metaEdConfiguration } = await loadMetaEdState(modelNpmPackage);


MetaEd is removed

bradbanister · 2023-11-03T16:17:09Z

Meadowlark-js/packages/meadowlark-core/src/handler/UriBuilder.ts

+
+// TODO: See RND-382 on what to do with this
+function versionAbbreviationFor(_resourceVersion: SemVer): ProjectShortVersion {
+  return '3.3b' as ProjectShortVersion;


Keeping this for now, RND-382 will probably change to '5.0.0-pre.1' for existing hardcode.

Meadowlark-js/packages/meadowlark-core/src/index.ts

Meadowlark-js/packages/meadowlark-core/test/validation/DocumentValidator.test.ts.ignore

andonyns · 2023-11-06T16:36:46Z

Meadowlark-js/packages/meadowlark-core/test/integration/metaed/MetaEdValidation.test.ts.ignore

Same comment for this file. Should it be renamed? or removed?

bradbanister force-pushed the RND-623 branch from 0f73541 to 607eb6a Compare September 25, 2023 19:04

bradbanister force-pushed the RND-623 branch from 411c3c6 to b2a0ac0 Compare October 13, 2023 20:40

bradbanister force-pushed the RND-623 branch 2 times, most recently from 7b37aaf to 705309c Compare October 28, 2023 19:24

bradbanister force-pushed the RND-623 branch from d72e882 to c0f903b Compare November 1, 2023 22:36

bradbanister marked this pull request as ready for review November 3, 2023 15:59

bradbanister commented Nov 3, 2023

View reviewed changes

bradbanister requested review from andonyns, stephenfuqua, DavidJGapCR and jleiva-gap November 3, 2023 16:21

andonyns reviewed Nov 6, 2023

View reviewed changes

Meadowlark-js/packages/meadowlark-core/src/index.ts Outdated Show resolved Hide resolved

andonyns reviewed Nov 6, 2023

View reviewed changes

Meadowlark-js/packages/meadowlark-core/test/validation/DocumentValidator.test.ts.ignore Outdated Show resolved Hide resolved

andonyns reviewed Nov 6, 2023

View reviewed changes

bradbanister added 3 commits November 6, 2023 11:50

[RND-623] initial matchup to MetaEd api-schema plugin

8563ce6

[RND-623] implement new identity and reference extraction

154d3d8

[RND-623] ready for test updates

5ae9a2f

bradbanister added 22 commits November 6, 2023 11:50

[RND-623] test fixup

969cecb

[RND-623] updated apischema

9372f40

sync

6bb6b4a

[RND-623] fix test

ef52527

more test fixup

3f3c2c5

unit tests passing

dee7df7

[RND-623] mongo test updates

73c049b

[RND-623] fix postgres integration

5b0d654

update ds

176e9e2

fastify debug config

010d3e2

[RND-623] ignore optional descriptor references that aren't there

8dd5b00

simplify identity structure

7f77375

[RND-623] update api schema

f91751f

[RND-623] fix superclass info extractor bug

a621571

[RND-623] use abstract resource info

46e8d36

[RND-623] Update DS

fe2104b

setup for merge with RND-651

b024216

[RND-623] with abstract resource info, with overposting disabled

19018f5

update api schema

04884e1

test fixup

ce4f21c

remove one e2e, skip other

32785eb

remove test credentials

ca07400

bradbanister force-pushed the RND-623 branch from 3a2fd99 to ca07400 Compare November 6, 2023 17:51

lint fixup

4a3cd53

bradbanister force-pushed the RND-623 branch from ebe5ab7 to 4a3cd53 Compare November 6, 2023 18:24

fixup commented out tests

b942c70

bradbanister force-pushed the RND-623 branch from 51e0e55 to b942c70 Compare November 6, 2023 20:36

andonyns approved these changes Nov 6, 2023

View reviewed changes

stephenfuqua merged commit 9be8824 into main Nov 6, 2023
13 checks passed

stephenfuqua deleted the RND-623 branch November 6, 2023 21:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RND-623] initial matchup to MetaEd api-schema plugin #300

[RND-623] initial matchup to MetaEd api-schema plugin #300

bradbanister commented Sep 22, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

andonyns Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

bradbanister Nov 3, 2023

andonyns Nov 6, 2023

		static readonly uriVersion33b: string = 'v3.3b';

		static readonly projectVersion33b: string = '3.3.1-b';


		const moduleName = 'ApiSchemaFinder';

		const dataStandard50pre1Path: string = path.resolve(__dirname, '../ds-schemas/DataStandard-5.0.0-pre.1.json');

[RND-623] initial matchup to MetaEd api-schema plugin #300

[RND-623] initial matchup to MetaEd api-schema plugin #300

Conversation

bradbanister commented Sep 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment