Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rush] add support for sharding phases #4652

Merged
merged 65 commits into from
May 25, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
b49a165
add test project
aramissennyeydd Apr 16, 2024
6f93177
add support for sharded phases
aramissennyeydd Apr 16, 2024
ed7abc0
move to .shards folder and rename properties
aramissennyeydd Apr 16, 2024
02e154a
remove dev logging
aramissennyeydd Apr 16, 2024
06f1a4c
add cobuild test repo for shards
aramissennyeydd Apr 16, 2024
c4a88e3
reduce the timeout period
aramissennyeydd Apr 16, 2024
fc625c1
fix bugs with shards in cobuilds
aramissennyeydd Apr 16, 2024
db8ad00
minor script updates
aramissennyeydd Apr 17, 2024
eb80e93
pr fixes
aramissennyeydd Apr 25, 2024
403734c
move to a new plugin
aramissennyeydd Apr 25, 2024
1bf1465
adjust to sharding config
aramissennyeydd Apr 25, 2024
7d34e74
isolate sharding work to ShardedPhaseOperationPlugin
aramissennyeydd Apr 26, 2024
6c8cf5c
move operationSettings inside of IOperationRunner
aramissennyeydd Apr 26, 2024
19b0dbd
revert state
aramissennyeydd Apr 26, 2024
044bcbb
run rush update
aramissennyeydd Apr 26, 2024
540453e
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd Apr 26, 2024
31dd787
add changesets
aramissennyeydd Apr 26, 2024
d4159e9
add support for sharding to the jest plugin
aramissennyeydd Apr 26, 2024
7821702
remove .shards remnants
aramissennyeydd Apr 26, 2024
ab0b208
support coverage output in jest
aramissennyeydd Apr 26, 2024
e6b1eb4
rename test cases
aramissennyeydd Apr 26, 2024
efa189f
revert metadata changes
aramissennyeydd Apr 26, 2024
7f24299
addressing PR feedback
aramissennyeydd Apr 30, 2024
73d3d77
arrays..
aramissennyeydd Apr 30, 2024
226fd81
don't need to format collator command
aramissennyeydd Apr 30, 2024
3fd51e3
remove from command-line.json
aramissennyeydd Apr 30, 2024
a3eb413
adjust how the logs get written
aramissennyeydd Apr 30, 2024
67598bf
support for configuring collator configuration
aramissennyeydd Apr 30, 2024
058eee7
move to shard being additional command
aramissennyeydd Apr 30, 2024
52f53ca
Apply suggestions from code review
aramissennyeydd May 1, 2024
e176bab
address pr comments
aramissennyeydd May 1, 2024
ba0924f
organize imports
aramissennyeydd May 1, 2024
9016dfc
move parent folder definition to static and use operationmetadatamana…
aramissennyeydd May 1, 2024
845a9b6
move operation settings to operation and standardize normalize log fi…
aramissennyeydd May 1, 2024
ee88bd5
replace all
aramissennyeydd May 1, 2024
4683793
remove heft specific updates
aramissennyeydd May 1, 2024
735c159
remove unnecessary change logs
aramissennyeydd May 1, 2024
817dc44
remove test project from the rest of rush
aramissennyeydd May 1, 2024
aea2ee7
revert typo fix
aramissennyeydd May 1, 2024
aea9238
fix lock file
aramissennyeydd May 2, 2024
354a872
fix test cases
aramissennyeydd May 2, 2024
8c49e12
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd May 10, 2024
f21108c
remove unnecessary phase
aramissennyeydd May 23, 2024
83b53ad
Apply suggestions from code review
aramissennyeydd May 23, 2024
42f63c2
address more PR feedback - create an initial pre-shard stage
aramissennyeydd May 23, 2024
64befae
adjust local dev setup and fix bug in operation creation
aramissennyeydd May 23, 2024
df66a93
remove reference to env vars
aramissennyeydd May 23, 2024
14a00eb
fix linting issues
aramissennyeydd May 23, 2024
791097f
add shards to readme
aramissennyeydd May 23, 2024
fd76a79
test with weights enabled as well
aramissennyeydd May 23, 2024
42d98fc
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd May 23, 2024
bb7a494
fix merge issue
aramissennyeydd May 23, 2024
a18b56b
remove console.log
aramissennyeydd May 23, 2024
11ffb2e
fixing output folder argument format
aramissennyeydd May 25, 2024
b2ddb4f
fix api report
aramissennyeydd May 25, 2024
291349b
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd May 25, 2024
f8f66be
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd May 25, 2024
48365b1
fix merge issue
aramissennyeydd May 25, 2024
3ff8688
fix test case
aramissennyeydd May 25, 2024
1cf78ff
Use a const.
iclanton May 25, 2024
a3f623f
fixes after local testing
aramissennyeydd May 25, 2024
37ddc9b
fix how we get the output folder
aramissennyeydd May 25, 2024
c1484a3
remove quote handling
aramissennyeydd May 25, 2024
d73fd00
Merge remote-tracking branch 'upstream/main' into jest-sharding
aramissennyeydd May 25, 2024
90caded
fix unit test
aramissennyeydd May 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ common/autoinstallers/*/.npmrc
# Heft temporary files
.cache
.heft

.shards

# Common toolchain intermediate files
temp
Expand Down
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// This is a workaround for https://github.com/eslint/eslint/issues/3458
require('local-eslint-config/patch/modern-module-resolution');
// This is a workaround for https://github.com/microsoft/rushstack/issues/3021
require('local-eslint-config/patch/custom-config-package-names');

module.exports = {
extends: ['local-eslint-config/profile/node'],
parserOptions: { tsconfigRootDir: __dirname }
};
8 changes: 8 additions & 0 deletions build-tests/heft-jest-shards-test/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# heft-jest-reporter-test

This project illustrates configuring Jest reporters in a minimal [Heft](https://www.npmjs.com/package/@rushstack/heft) project


Please see the [Jest configuration](./config/jest.config.json),
the [Getting started with Heft](https://rushstack.io/pages/heft_tutorials/getting_started/),
and ["jest" task](https://rushstack.io/pages/heft_tasks/jest/) articles for more information.
35 changes: 35 additions & 0 deletions build-tests/heft-jest-shards-test/config/heft.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"$schema": "../../../apps/heft/src/schemas/heft.schema.json",

// TODO: Add comments
"phasesByName": {
"build": {
"cleanFiles": [{ "includeGlobs": ["dist", "lib", "lib-commonjs"] }],

"tasksByName": {
"typescript": {
"taskPlugin": {
"pluginPackage": "@rushstack/heft-typescript-plugin"
}
},
"lint": {
"taskDependencies": ["typescript"],
"taskPlugin": {
"pluginPackage": "@rushstack/heft-lint-plugin"
}
}
}
},

"test": {
"phaseDependencies": ["build"],
"tasksByName": {
"jest": {
"taskPlugin": {
"pluginPackage": "@rushstack/heft-jest-plugin"
}
}
}
}
}
}
15 changes: 15 additions & 0 deletions build-tests/heft-jest-shards-test/config/jest.config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"extends": "@rushstack/heft-jest-plugin/includes/jest-shared.config.json",
"coverageDirectory": "<rootDir>/coverage",
"collectCoverage": true,
"testMatch": ["<rootDir>/lib/**/*.test.cjs"],
"collectCoverageFrom": [
"lib/**/*.cjs",
"!lib/**/*.d.ts",
"!lib/**/*.test.cjs",
"!lib/**/test/**",
"!lib/**/__tests__/**",
"!lib/**/__fixtures__/**",
"!lib/**/__mocks__/**"
]
}
17 changes: 17 additions & 0 deletions build-tests/heft-jest-shards-test/config/rush-project.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"$schema": "../../../libraries/rush-lib/src/schemas/rush-project.schema.json",

"operationSettings": [
{
"operationName": "_phase:build",
"outputFolderNames": ["lib", "dist"]
},
{
"operationName": "_phase:test",
"outputFolderNames": ["coverage", "temp/coverage"],
"sharding": {
"count": 6
}
}
]
}
59 changes: 59 additions & 0 deletions build-tests/heft-jest-shards-test/config/typescript.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/**
* Configures the TypeScript plugin for Heft. This plugin also manages linting.
*/
{
"$schema": "https://developer.microsoft.com/json-schemas/heft/v0/typescript.schema.json",

/**
* If provided, emit these module kinds in addition to the modules specified in the tsconfig.
* Note that this option only applies to the main tsconfig.json configuration.
*/
"additionalModuleKindsToEmit": [
// {
// /**
// * (Required) Must be one of "commonjs", "amd", "umd", "system", "es2015", "esnext"
// */
// "moduleKind": "amd",
//
// /**
// * (Required) The name of the folder where the output will be written.
// */
// "outFolderName": "lib-amd"
// }
],

/**
* If true, emit CommonJS module output to the folder specified in the tsconfig "outDir" compiler option with the .cjs extension alongside (or instead of, if TSConfig specifies ESNext) the default compilation output.
*/
"emitCjsExtensionForCommonJS": true,

/**
* If true, emit ESNext module output to the folder specified in the tsconfig "outDir" compiler option with the .mjs extension alongside (or instead of, if TSConfig specifies ESNext) the default compilation output.
*/
// "emitMjsExtensionForESModule": true,

/**
* Describes the way files should be statically coped from src to TS output folders
*/
"staticAssetsToCopy": {
/**
* File extensions that should be copied from the src folder to the destination folder(s).
*/
"fileExtensions": [".css", ".png"]

/**
* Glob patterns that should be explicitly included.
*/
// "includeGlobs": [
// "some/path/*.js"
// ],

/**
* Glob patterns that should be explicitly excluded. This takes precedence over globs listed
* in "includeGlobs" and files that match the file extensions provided in "fileExtensions".
*/
// "excludeGlobs": [
// "some/path/*.css"
// ]
}
}
136 changes: 136 additions & 0 deletions build-tests/heft-jest-shards-test/coverage.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
{
"/Users/aramis.sennyey/Projects/rushstack/build-tests/heft-jest-shards-test/src/index.ts": {
"path": "/Users/aramis.sennyey/Projects/rushstack/build-tests/heft-jest-shards-test/src/index.ts",
"all": true,
"statementMap": {
"0": {
"start": {
"line": 1,
"column": 0
},
"end": {
"line": 1,
"column": 92
}
},
"1": {
"start": {
"line": 2,
"column": 0
},
"end": {
"line": 2,
"column": 59
}
},
"2": {
"start": {
"line": 3,
"column": 0
},
"end": {
"line": 3,
"column": 0
}
},
"3": {
"start": {
"line": 4,
"column": 0
},
"end": {
"line": 4,
"column": 51
}
},
"4": {
"start": {
"line": 5,
"column": 0
},
"end": {
"line": 5,
"column": 15
}
},
"5": {
"start": {
"line": 6,
"column": 0
},
"end": {
"line": 6,
"column": 1
}
}
},
"s": {
"0": 0,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"branchMap": {
"0": {
"type": "branch",
"line": 1,
"loc": {
"start": {
"line": 1,
"column": 223
},
"end": {
"line": 6,
"column": 1
}
},
"locations": [
{
"start": {
"line": 1,
"column": 223
},
"end": {
"line": 6,
"column": 1
}
}
]
}
},
"b": {
"0": [0]
},
"fnMap": {
"0": {
"name": "(empty-report)",
"decl": {
"start": {
"line": 1,
"column": 223
},
"end": {
"line": 6,
"column": 1
}
},
"loc": {
"start": {
"line": 1,
"column": 223
},
"end": {
"line": 6,
"column": 1
}
},
"line": 1
}
},
"f": {
"0": 0
}
}
}
24 changes: 24 additions & 0 deletions build-tests/heft-jest-shards-test/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"name": "heft-jest-shards-test",
"description": "This project illustrates configuring Jest sharding in a minimal Heft project",
"version": "1.0.0",
"private": true,
"scripts": {
"build": "heft build --clean",
"start": "heft build-watch",
"_phase:build": "heft run --only build -- --clean",
"_phase:test": "heft run --only test -- --clean --max-workers=1"
},
"devDependencies": {
"@jest/types": "29.5.0",
"local-eslint-config": "workspace:*",
"@rushstack/heft": "workspace:*",
"@rushstack/heft-jest-plugin": "workspace:*",
"@rushstack/heft-lint-plugin": "workspace:*",
"@rushstack/heft-typescript-plugin": "workspace:*",
"@types/heft-jest": "1.0.1",
"eslint": "~8.7.0",
"typescript": "~5.3.3",
"@types/node": "18.17.15"
}
}
6 changes: 6 additions & 0 deletions build-tests/heft-jest-shards-test/src/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

export function add(a: number, b: number): number {
return a + b;
}
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/index.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('othertests', () => {
it.each(new Array(100).fill(0))('should succeed', async (i) => {
await new Promise((resolve) => setTimeout(resolve, 10));
expect(1 + 1).toEqual(2);
});
});
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/othertests.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('index.test', () => {
it.each(new Array(100).fill(0))('should succeed', async () => {
await new Promise((resolve) => setTimeout(resolve, 10));
expect(1 + 1).toEqual(2);
});
});
10 changes: 10 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/test.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('tests', () => {
it.each(new Array(100).fill(0))('should succeed', async (i) => {
await new Promise((resolve) => setTimeout(resolve, 10));
// comment
expect(1 + 1).toEqual(2);
});
});
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/test2.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('othertests', () => {
it.each(new Array(100).fill(0))('should succeed', async (i) => {
await new Promise((resolve) => setTimeout(resolve, 10));
expect(1 + 1).toEqual(2);
});
});
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/test3.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('othertests', () => {
it.each(new Array(100).fill(0))('should succeed', async (i) => {
await new Promise((resolve) => setTimeout(resolve, 10));
expect(1 + 1).toEqual(2);
});
});
9 changes: 9 additions & 0 deletions build-tests/heft-jest-shards-test/src/test/test4.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
// See LICENSE in the project root for license information.

describe('othertests', () => {
it.each(new Array(100).fill(0))('should succeed', async (i) => {
await new Promise((resolve) => setTimeout(resolve, 10));
expect(1 + 1).toEqual(2);
});
});
Loading