[SILO-200] tests: add integration tests using playwright #7004
Conversation
|
Pull Request Linked with Plane Work Items
Comment Automatically Generated by Plane |
WalkthroughA new Playwright-based integration testing setup has been introduced under the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Playwright Setup (auth.setup.ts)
participant Application
participant Storage (auth-state.json)
User->>Playwright Setup (auth.setup.ts): Start authentication setup
Playwright Setup (auth.setup.ts)->>Application: Navigate to BASE_URL
Playwright Setup (auth.setup.ts)->>Application: Fill email and submit
Application-->>Playwright Setup (auth.setup.ts): Render password prompt
Playwright Setup (auth.setup.ts)->>Application: Fill password and submit
Application-->>Playwright Setup (auth.setup.ts): Authenticate user
Playwright Setup (auth.setup.ts)->>Storage (auth-state.json): Save authenticated storage state
sequenceDiagram
participant Playwright Test Runner
participant Browser Project (Chromium/Firefox/WebKit)
participant Storage (auth-state.json)
participant Application
Playwright Test Runner->>Browser Project: Start test with authentication state
Browser Project->>Storage (auth-state.json): Load authentication state
Browser Project->>Application: Run test actions (e.g., navigate, click)
Application-->>Browser Project: Respond with page content
Browser Project->>Playwright Test Runner: Report test results
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (6)
tests/integration-tests/.env.example (1)
1-3: Environment configuration looks good but could be more self-explanatory.The template includes the essential variables required for authentication tests. Using base64 encoding for passwords is a good practice to avoid plain text passwords in configuration files.
Consider adding comments to explain:
- The expected format for each variable
- Instructions on how to generate the base64-encoded password (e.g.,
echo -n "your_password" | base64)- Whether the BASE_URL should include protocol and trailing slash
-EMAIL= -PASSWORD_BASE64= -BASE_URL= +# User email for authentication +EMAIL= +# Password encoded in base64 (generate with: echo -n "your_password" | base64) +PASSWORD_BASE64= +# Application URL (include protocol, e.g., https://app.example.com) +BASE_URL=tests/integration-tests/tests/auth.setup.ts (2)
17-27: Consider more robust selectors for login elements.The current selectors rely on exact text matches which might break if UI text changes.
Use data-testid attributes or more resilient selectors when possible:
await page.goto(BASE_URL); - await page.getByRole('textbox', { name: 'Email' }).click(); - await page.getByRole('textbox', { name: 'Email' }).fill(EMAIL); - await page.getByRole('textbox', { name: 'Email' }).press('Enter'); + // Better to use a selector that would be more resilient to UI text changes + const emailField = page.getByRole('textbox', { name: 'Email' }); + // Or ideally use data-testid if available: page.locator('[data-testid="email-input"]') + await emailField.click(); + await emailField.fill(EMAIL); + await emailField.press('Enter'); if (await page.getByRole('button', { name: 'Continue' }).isVisible()) { await page.getByRole('button', { name: 'Continue' }).click(); } - await page.getByRole('textbox', { name: 'Enter password' }).click(); - await page.getByRole('textbox', { name: 'Enter password' }).fill(PWD); + const passwordField = page.getByRole('textbox', { name: 'Enter password' }); + // Or ideally: page.locator('[data-testid="password-input"]') + await passwordField.click(); + await passwordField.fill(PWD);This will make the tests more maintainable if UI text changes.
28-29: Improve waiting strategy for application stability.Using both
waitForLoadStateand a fixed timeout could be problematic in different environments.Consider a more robust approach to ensure the application is fully loaded:
- await page.waitForLoadState("networkidle"); - await page.waitForTimeout(1000); + // Wait for the application to be fully loaded + await page.waitForLoadState("networkidle"); + // Wait for a key element that indicates the user is logged in + await page.waitForSelector('text=Dashboard', { timeout: 5000 }).catch(() => { + // If we can't find the expected element, it might indicate a login failure + console.log('Warning: Could not verify successful login'); + });This approach waits for a specific element that indicates successful login rather than using an arbitrary timeout.
tests/integration-tests/README.md (3)
25-30: Add instructions for generating base64 passwordIt would be helpful to include instructions on how to encode a password to base64 format for the users who might not be familiar with the process.
```env BASE_URL=http://localhost:3000 # Your application URL [email protected] # Your test user email PASSWORD_BASE64=base64-encoded-password # Your test user password in base64
+To encode your password to base64, you can use:
+bash +# On Linux/Mac +echo -n "your-password" | base64 + +# On Windows PowerShell +[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("your-password")) +--- `78-79`: **Missing punctuation in test structure descriptions** Add periods at the end of the descriptions for consistency with other sections in the README. ```diff - `auth.setup.ts`: Handles user authentication and creates a persistent auth state - `projects.spec.ts`: Contains the actual test cases for the projects functionality + `auth.setup.ts`: Handles user authentication and creates a persistent auth state. + `projects.spec.ts`: Contains the actual test cases for the projects' functionality.🧰 Tools
🪛 LanguageTool
[uncategorized] ~78-~78: Loose punctuation mark.
Context: ...`` ## Test Structure -auth.setup.ts: Handles user authentication and creates...(UNLIKELY_OPENING_PUNCTUATION)
[uncategorized] ~79-~79: It seems likely that a singular genitive (’s) apostrophe is missing.
Context: ... Contains the actual test cases for the projects functionality ## Test Reports After r...(AI_HYDRA_LEO_APOSTROPHE_S_XS)
105-110: CI/CD instructions could be more detailedThe CI/CD integration section could benefit from more specific examples, especially regarding environment variables setup.
## CI/CD Integration For CI/CD environments, make sure to: 1. Set the appropriate environment variables 2. Run `npx playwright install-deps` before running tests 3. Use `--reporter=html` for test reports + +### Example GitHub Actions workflow + +```yaml +name: Playwright Tests +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: actions/setup-node@v3 + with: + node-version: 18 + - name: Install dependencies + run: cd tests/integration-tests && npm install + - name: Install Playwright browsers + run: cd tests/integration-tests && npx playwright install-deps + - name: Run Playwright tests + run: cd tests/integration-tests && npx playwright test + env: + BASE_URL: ${{ secrets.TEST_BASE_URL }} + EMAIL: ${{ secrets.TEST_EMAIL }} + PASSWORD_BASE64: ${{ secrets.TEST_PASSWORD_BASE64 }} + - name: Upload test results + if: always() + uses: actions/upload-artifact@v3 + with: + name: playwright-report + path: tests/integration-tests/playwright-report + retention-days: 30 +```
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
tests/integration-tests/.env.example(1 hunks)tests/integration-tests/.gitignore(1 hunks)tests/integration-tests/README.md(1 hunks)tests/integration-tests/package.json(1 hunks)tests/integration-tests/playwright.config.ts(1 hunks)tests/integration-tests/tests/auth.setup.ts(1 hunks)tests/integration-tests/tests/projects.spec.ts(1 hunks)
🧰 Additional context used
🪛 LanguageTool
tests/integration-tests/README.md
[uncategorized] ~78-~78: Loose punctuation mark.
Context: ...`` ## Test Structure - auth.setup.ts: Handles user authentication and creates...
(UNLIKELY_OPENING_PUNCTUATION)
[uncategorized] ~79-~79: It seems likely that a singular genitive (’s) apostrophe is missing.
Context: ... Contains the actual test cases for the projects functionality ## Test Reports After r...
(AI_HYDRA_LEO_APOSTROPHE_S_XS)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: Analyze (javascript)
🔇 Additional comments (7)
tests/integration-tests/.gitignore (1)
1-8: LGTM! Good coverage of Playwright artifacts.The gitignore file correctly excludes all the necessary Playwright-related directories and files that should not be committed to version control, including:
- Node modules dependencies
- Test results and reports
- Cache files
- Authentication state files which might contain sensitive data
This follows best practices for Playwright projects.
tests/integration-tests/tests/auth.setup.ts (1)
7-32:❓ Verification inconclusive
Add error handling for failed login attempts.
The current implementation doesn't account for authentication failures.
Add error handling to detect and report login failures:
setup('authenticate', async ({ page }) => { // ... existing code ... await page.keyboard.press('Enter'); await page.waitForLoadState("networkidle"); - await page.waitForTimeout(1000); + + // Check for login errors + const errorVisible = await page.getByText('Invalid email or password').isVisible() + .catch(() => false); + + if (errorVisible) { + throw new Error('Authentication failed: Invalid credentials'); + } + + // Optional: Verify that we are actually logged in + const loggedIn = await page.getByText('Welcome').isVisible() + .catch(() => false); + + if (!loggedIn) { + console.warn('Warning: Could not verify successful login'); + } await page.context().storageState({ path: authFile }); });This improvement will help catch authentication issues early and provide clearer error messages.
🏁 Script executed:
#!/bin/bash # Search for the exact invalid credentials error message in the codebase rg "Invalid email or password" -A 3 || echo "No matches for invalid credentials error message" # Search for a generic welcome text on the post-login page rg "Welcome" -A 3 || echo "No matches for welcome text"Length of output: 7605
🏁 Script executed:
#!/bin/bash # Find React components related to login/authentication echo "Files matching 'login':" fd login -t f -e tsx echo "" echo "Files matching 'signin':" fd signin -t f -e tsx echo "" echo "Files matching 'auth':" fd auth -t f -e tsxLength of output: 1297
🏁 Script executed:
#!/bin/bash # Inspect the authentication root component to identify error messages cat web/core/components/account/auth-forms/auth-root.tsx || echo "File not found"Length of output: 7390
Add robust login error and success checks
The current test doesn’t assert on failed logins or confirm a successful sign-in. Update the snippet after pressing Enter to:
await page.keyboard.press('Enter'); await page.waitForLoadState('networkidle'); - await page.waitForTimeout(1000); + // 1) Check for a login error (adjust the regex/text to your app’s actual message) + const loginError = await page.getByText(/Invalid credentials|Wrong email or password/i) + .isVisible() + .catch(() => false); + + if (loginError) { + throw new Error('Authentication failed: Invalid credentials'); + } + + // 2) Verify successful login by checking a known post-login element + const postLoginVisible = await page + .getByRole('heading', { name: /Dashboard|Welcome back/i }) + .isVisible() + .catch(() => false); + + if (!postLoginVisible) { + console.warn('Warning: Could not verify successful login'); + }• Ensure the error-message selector matches your UI’s exact text.
• Pick a reliable post-login selector (e.g., “Dashboard” heading or user menu).
• Remove arbitrary timeouts for more deterministic tests.tests/integration-tests/tests/projects.spec.ts (2)
1-6: Initial test setup looks good, but it's a placeholderThe basic test is a good sanity check to confirm the test framework is working, but it doesn't validate any actual functionality.
8-14: Authentication handling looks goodThe test setup correctly assumes authentication is already handled via the storage state from the auth.setup.ts file, and properly waits for the page to be fully loaded before proceeding with tests.
tests/integration-tests/README.md (2)
7-9: LGTM! Node.js version requirementGood to see Node.js v18 or higher is specified as required, as per previous feedback.
34-43: Authentication setup looks goodClear instructions for setting up authentication state before running tests.
tests/integration-tests/playwright.config.ts (1)
35-41: Setup project correctly configuredThe setup project is properly configured to run the authentication setup before other tests.
| { | ||
| "name": "integration-tests", | ||
| "version": "1.0.0", | ||
| "main": "index.js", | ||
| "scripts": {}, | ||
| "keywords": [], | ||
| "author": "", | ||
| "license": "ISC", | ||
| "description": "", | ||
| "devDependencies": { | ||
| "@playwright/test": "^1.51.1", | ||
| "@types/node": "^22.13.14" | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add helpful npm scripts for easier test execution.
The package.json correctly includes the necessary dependencies for Playwright tests, but it's missing npm scripts that would make running tests more convenient for developers.
Add commonly used Playwright commands as npm scripts:
{
"name": "integration-tests",
"version": "1.0.0",
"main": "index.js",
- "scripts": {},
+ "scripts": {
+ "test": "playwright test",
+ "test:ui": "playwright test --ui",
+ "test:debug": "playwright test --debug",
+ "setup": "playwright install",
+ "report": "playwright show-report"
+ },
"keywords": [],
"author": "",
"license": "ISC",This makes it easier for team members to run tests with commands like npm test without having to remember Playwright-specific commands.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| { | |
| "name": "integration-tests", | |
| "version": "1.0.0", | |
| "main": "index.js", | |
| "scripts": {}, | |
| "keywords": [], | |
| "author": "", | |
| "license": "ISC", | |
| "description": "", | |
| "devDependencies": { | |
| "@playwright/test": "^1.51.1", | |
| "@types/node": "^22.13.14" | |
| } | |
| } | |
| { | |
| "name": "integration-tests", | |
| "version": "1.0.0", | |
| "main": "index.js", | |
| "scripts": { | |
| "test": "playwright test", | |
| "test:ui": "playwright test --ui", | |
| "test:debug": "playwright test --debug", | |
| "setup": "playwright install", | |
| "report": "playwright show-report" | |
| }, | |
| "keywords": [], | |
| "author": "", | |
| "license": "ISC", | |
| "description": "", | |
| "devDependencies": { | |
| "@playwright/test": "^1.51.1", | |
| "@types/node": "^22.13.14" | |
| } | |
| } |
| const PWD = Buffer.from(process.env.PASSWORD_BASE64 || '', 'base64').toString('utf-8'); | ||
| if (!BASE_URL || !EMAIL || !PWD) { | ||
| throw new Error('BASE_URL, EMAIL or PWD is not set'); | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Improve error handling for missing environment variables.
The current error handling doesn't specify which environment variables are missing, making debugging harder.
Implement more precise error checking that identifies exactly which variables are missing:
- // decode the password
- const PWD = Buffer.from(process.env.PASSWORD_BASE64 || '', 'base64').toString('utf-8');
- if (!BASE_URL || !EMAIL || !PWD) {
- throw new Error('BASE_URL, EMAIL or PWD is not set');
- }
+ // Check if all required environment variables are set
+ const missingVars = [];
+ if (!BASE_URL) missingVars.push('BASE_URL');
+ if (!EMAIL) missingVars.push('EMAIL');
+ if (!process.env.PASSWORD_BASE64) missingVars.push('PASSWORD_BASE64');
+
+ if (missingVars.length > 0) {
+ throw new Error(`Missing required environment variables: ${missingVars.join(', ')}`);
+ }
+
+ // decode the password
+ const PWD = Buffer.from(process.env.PASSWORD_BASE64, 'base64').toString('utf-8');📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const PWD = Buffer.from(process.env.PASSWORD_BASE64 || '', 'base64').toString('utf-8'); | |
| if (!BASE_URL || !EMAIL || !PWD) { | |
| throw new Error('BASE_URL, EMAIL or PWD is not set'); | |
| } | |
| // Check if all required environment variables are set | |
| const missingVars = []; | |
| if (!BASE_URL) missingVars.push('BASE_URL'); | |
| if (!EMAIL) missingVars.push('EMAIL'); | |
| if (!process.env.PASSWORD_BASE64) missingVars.push('PASSWORD_BASE64'); | |
| if (missingVars.length > 0) { | |
| throw new Error(`Missing required environment variables: ${missingVars.join(', ')}`); | |
| } | |
| // decode the password | |
| const PWD = Buffer.from(process.env.PASSWORD_BASE64, 'base64').toString('utf-8'); |
There was a problem hiding this comment.
we might want to add individual tests for logged out scenarios or different types of accounts and refactoring this as an after thought can be fairly difficult. https://playwright.dev/docs/auth has some docs around different strategies.
There was a problem hiding this comment.
🧩 Analysis chain
@lifeiscontent You raise an excellent point about the authentication architecture. The current single-auth approach will indeed create challenges down the road. Let me check the latest Playwright auth strategies to provide specific recommendations.
You're absolutely right - the current setup with a single auth.setup.ts that all browser projects depend on is too restrictive. Here are some better architectural approaches based on Playwright's recommended patterns:
1. Multiple Authentication Setups
// tests/auth/admin.setup.ts
setup('authenticate as admin', async ({ page }) => {
// Admin login logic
await page.context().storageState({ path: './playwright/.auth/admin.json' });
});
// tests/auth/user.setup.ts
setup('authenticate as user', async ({ page }) => {
// Regular user login logic
await page.context().storageState({ path: './playwright/.auth/user.json' });
});
// tests/auth/logged-out.setup.ts
setup('ensure logged out', async ({ page }) => {
// Clear any existing auth state
await page.context().storageState({ path: './playwright/.auth/logged-out.json' });
});2. Updated Playwright Config
export default defineConfig({
projects: [
{ name: 'setup-admin', testMatch: /.*\.admin\.setup\.ts/ },
{ name: 'setup-user', testMatch: /.*\.user\.setup\.ts/ },
{ name: 'setup-logged-out', testMatch: /.*\.logged-out\.setup\.ts/ },
{
name: 'admin-tests',
use: { storageState: './playwright/.auth/admin.json' },
dependencies: ['setup-admin'],
testMatch: /.*\.admin\.spec\.ts/
},
{
name: 'user-tests',
use: { storageState: './playwright/.auth/user.json' },
dependencies: ['setup-user'],
testMatch: /.*\.user\.spec\.ts/
},
{
name: 'logged-out-tests',
use: { storageState: './playwright/.auth/logged-out.json' },
dependencies: ['setup-logged-out'],
testMatch: /.*\.logged-out\.spec\.ts/
}
]
});3. Test File Organization
tests/
├── auth/
│ ├── admin.setup.ts
│ ├── user.setup.ts
│ └── logged-out.setup.ts
├── admin/
│ └── admin-dashboard.admin.spec.ts
├── user/
│ └── projects.user.spec.ts
└── public/
└── login.logged-out.spec.ts
This approach gives you:
- ✅ Multiple user types/roles
- ✅ Logged-out scenario testing
- ✅ Clear test organization
- ✅ Parallel execution per auth type
- ✅ Easy to extend with new auth scenarios
Would you like me to help refactor the current setup to this more flexible architecture?
🌐 Web query:
Playwright authentication strategies multiple user types different accounts setup
💡 Result:
Playwright offers several authentication strategies to handle multiple user types and different accounts in your test automation. I'll explore the most effective approaches and how to implement them in your tests.
Authentication Strategies in Playwright
Playwright provides flexible options for handling authentication across different user roles and accounts, allowing you to create efficient and maintainable test suites.
Reusing Authentication States
The fundamental principle behind Playwright's authentication strategies is storing and reusing authentication states. This approach:
- Saves execution time by avoiding repeated logins
- Reduces test brittleness
- Eliminates unnecessary repetition in your test code[7]
By capturing and reusing authentication states, you can authenticate once and use that state across multiple tests - similar to purchasing a yearly subscription rather than monthly subscriptions multiple times[7].
Storage State Approach
The core mechanism for managing authentication in Playwright is the storageState functionality:
- Authenticate a user (via UI or API)
- Save the authentication state to a file using
context.storageState({ path: 'authFile.json' }) - Later, create contexts that use this state with
browser.newContext({ storageState: 'authFile.json' })[1][2]
The storage state file contains cookies and localStorage data needed to maintain authentication[1].
Authentication Strategy Options
Multiple Signed-In Roles
When your application has different user roles (e.g., admin, regular user), you can:
- Create separate authentication files for each role
- Load the appropriate state for tests that need a specific role[2][5]
// In setup.ts file
const adminFile = 'playwright/.auth/admin.json';
const userFile = 'playwright/.auth/user.json';
// Create admin auth state
setup('authenticate as admin', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Username').fill('admin');
await page.getByLabel('Password').fill('adminPassword');
await page.getByRole('button', { name: 'Sign in' }).click();
await page.context().storageState({ path: adminFile });
});
// Create user auth state
setup('authenticate as user', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Username').fill('user');
await page.getByLabel('Password').fill('userPassword');
await page.getByRole('button', { name: 'Sign in' }).click();
await page.context().storageState({ path: userFile });
});One Account Per Parallel Worker
This approach is recommended for tests that modify server-side state:
- Each parallel worker gets authenticated once, with a unique account
- All tests run by that worker reuse the same authentication state
- This prevents tests from interfering with each other[2][5]
// In fixtures.ts
export const test = baseTest.extend({
// Use unique storage state per worker
storageState: async ({ }, use, testInfo) => {
const fileName = `playwright/.auth/user-${testInfo.parallelIndex}.json`;
await use(fileName);
}
});Multiple Users in a Single Test
To test interactions between different user types within the same test:
- Create separate browser contexts for each user
- Load different storage states into each context
- Create pages from each context to simulate multiple users interacting[5][7]
test("admin and user interaction", async ({ browser }) => {
// Create admin context and page
const adminContext = await browser.newContext({
storageState: "playwright/.auth/admin.json",
});
const adminPage = await adminContext.newPage();
// Create user context and page
const userContext = await browser.newContext({
storageState: "playwright/.auth/user.json",
});
const userPage = await userContext.newPage();
// Test interaction between admin and user
await adminPage.goto("https://example.com/admin-panel");
await userPage.goto("https://example.com/dashboard");
// Perform test actions with both users...
// Clean up
await adminContext.close();
await userContext.close();
})[7]Implementation Best Practices
Using Browser Contexts Effectively
Browser contexts provide isolated environments that maintain separate authentication states:
- Create different contexts for different user sessions
- Each context has its own cookies, localStorage, and session data
- This allows testing multiple users without logging out/in between actions[9][6]
API Authentication for Speed
Instead of UI login, you can authenticate via API calls:
- Make API requests to authenticate users
- Store the resulting cookies/tokens
- Use this data to create authenticated contexts
This approach is faster and more reliable than UI-based authentication[5][7].
Project Structure for Authentication
A recommended structure for managing authentication:
- Create a
playwright/.authdirectory (add to.gitignore) - Store authentication state files in this directory
- Set up authentication in a separate file (e.g.,
setup.ts) - Reference storage states in your test files[10]
Conclusion
By implementing the right authentication strategy for your needs, you can create efficient, scalable, and maintainable Playwright test suites that handle multiple user types and different accounts. Consider your specific requirements—whether you need parallel testing, multiple user roles, or interaction between different users—to choose the most appropriate approach.
Citations:
- 1: https://playwrightsolutions.com/handling-multiple-login-states-between-different-tests-in-playwright/
- 2: https://playwright.dev/docs/auth
- 3: https://www.neovasolutions.com/2024/11/14/handling-authentication-for-multiple-user-logins-in-playwright/
- 4: https://club.ministryoftesting.com/t/what-authentication-strategies-have-you-used-in-playwright-tests/84607
- 5: https://www.youtube.com/watch?v=_K8SY3MbayA
- 6: https://munalugonvu.hashnode.dev/handling-multiple-logins-in-playwright-like-a-pro
- 7: https://www.ministryoftesting.com/articles/simple-playwright-authentication-recipes-a-cookbook-for-software-testers
- 8: Suggestion on using multiple authenticated users during playwright load test artilleryio/artillery#3243
- 9: https://www.youtube.com/watch?v=0mfLHPLZ7_k
- 10: https://ray.run/discord-forum/threads/36424-authentication-correct-flow
🌐 Web query:
Playwright docs authentication multiple users logged out scenarios setup projects
💡 Result:
Playwright provides robust authentication solutions for end-to-end testing, particularly when working with multiple user roles and handling logout scenarios. This comprehensive approach ensures your tests remain reliable even with different authentication states.
Authentication Setup in Playwright
Playwright recommends storing authentication states in a dedicated directory that should be excluded from version control:
mkdir -p playwright/.auth
echo $'\n.auth' >> .gitignoreMultiple Signed-in Roles
When testing applications that require different user types (admins, regular users, etc.), you can create separate authentication states for each role:
- First, create an authentication setup file that handles different user roles:
import { test as setup, expect } from '@playwright/test';
const adminFile = 'playwright/.auth/admin.json';
setup('authenticate as admin', async ({ page }) => {
await page.goto('https://github.com/login');
await page.getByLabel('Username or email address').fill('admin');
await page.getByLabel('Password').fill('password');
await page.getByRole('button', { name: 'Sign in' }).click();
// Wait for successful authentication
await page.waitForURL('https://github.com/');
await expect(page.getByRole('button', { name: 'View profile and more' })).toBeVisible();
// Store authenticated state
await page.context().storageState({ path: adminFile });
});
const userFile = 'playwright/.auth/user.json';
setup('authenticate as user', async ({ page }) => {
// Similar authentication steps for regular user
// ...
await page.context().storageState({ path: userFile });
});- Then use these stored states in your test files:
import { test } from '@playwright/test';
test.use({ storageState: 'playwright/.auth/admin.json' });
test('admin test', async ({ page }) => {
// page is authenticated as admin
});
test.describe(() => {
test.use({ storageState: 'playwright/.auth/user.json' });
test('user test', async ({ page }) => {
// page is authenticated as a user
});
});Handling Logout Scenarios
When testing logout functionality, you need to account for how this affects subsequent tests. There are several approaches:
Project Dependencies Approach
One effective approach is to use project dependencies in your Playwright configuration:
// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";
export default defineConfig({
projects: [
{
name: "setup",
testMatch: /.*\.setup\.ts/,
},
{
name: "chromium",
dependencies: ["setup"],
use: {
...devices["Desktop Chrome"],
storageState: ".auth/session.json",
},
},
// Other browsers...
],
});This ensures your authentication setup runs before the actual tests, establishing a fresh authentication state for each test suite run[5].
Worker-Scoped Authentication
For parallel test execution, especially in environments like Sauce Labs, you can use worker-scoped fixtures to handle authentication per worker:
export const test = baseTest.extend<{}, { workerStorageState: string }>({
storageState: ({ workerStorageState }, use) => use(workerStorageState),
workerStorageState: [
async ({ browser }, use) => {
// Use parallelIndex as a unique identifier for each worker
const id = test.info().parallelIndex;
// Setup authentication per user role
// ...
},
{ scope: 'worker' }
],
});This approach ensures each parallel test worker has its own authenticated state[7].
Managing Authentication State Between Tests
When a test explicitly logs out a user, subsequent tests would no longer have an authenticated session. To handle this, you have several options:
-
Separate projects for logged-in and logged-out tests:
// In playwright.config.ts projects: [ { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'logged-in-tests', dependencies: ['setup'], testMatch: /.*\.logged-in\.spec\.ts/, use: { storageState: '.auth/session.json' } }, { name: 'logout-tests', testMatch: /.*\.logout\.spec\.ts/ } ]
-
Create test-specific authentication states: Instead of relying on a global authentication state, you can authenticate at the beginning of each test that requires it[9].
-
Global setup with conditional authentication: Your global setup can check if the stored authentication is still valid and only perform authentication when necessary[8].
By implementing these strategies, you can efficiently handle different authentication scenarios in your Playwright tests, making them more reliable and maintainable.
Citations:
- 1: https://playwright.dev/docs/auth
- 2: https://playwright.dev/dotnet/docs/auth
- 3: https://www.checklyhq.com/learn/playwright/authentication/
- 4: https://filiphric.com/how-to-do-authentication-in-playwright
- 5: https://www.teemutaskula.com/blog/speedup-playwright-auth
- 6: https://www.youtube.com/watch?v=3ZgpaNFkeKA
- 7: https://ray.run/discord-forum/threads/5856-login-fixture-with-multiple-user-roles
- 8: https://www.youtube.com/watch?v=QJL6uV7z-8I
- 9: https://playwrightsolutions.com/handling-multiple-login-states-between-different-tests-in-playwright/
- 10: https://dev.to/playwright/a-better-global-setup-in-playwright-reusing-login-with-project-dependencies-14
@lifeiscontent You're absolutely right - this is excellent architectural foresight! The current single-auth approach will indeed create significant refactoring challenges down the road. Based on Playwright's recommended patterns, here's a more flexible architecture:
Recommended Multi-Auth Architecture
1. Multiple Authentication Setups
// tests/auth/admin.setup.ts
setup('authenticate as admin', async ({ page }) => {
await authenticateUser(page, 'admin');
await page.context().storageState({ path: './playwright/.auth/admin.json' });
});
// tests/auth/user.setup.ts
setup('authenticate as user', async ({ page }) => {
await authenticateUser(page, 'user');
await page.context().storageState({ path: './playwright/.auth/user.json' });
});
// tests/auth/logged-out.setup.ts
setup('ensure logged out', async ({ page }) => {
// Create clean state without auth
await page.context().storageState({ path: './playwright/.auth/logged-out.json' });
});2. Updated Playwright Config
export default defineConfig({
projects: [
// Setup projects
{ name: 'setup-admin', testMatch: /.*\.admin\.setup\.ts/ },
{ name: 'setup-user', testMatch: /.*\.user\.setup\.ts/ },
{ name: 'setup-logged-out', testMatch: /.*\.logged-out\.setup\.ts/ },
// Test projects with specific auth states
{
name: 'admin-tests',
use: { storageState: './playwright/.auth/admin.json' },
dependencies: ['setup-admin'],
testMatch: /.*\.admin\.spec\.ts/
},
{
name: 'user-tests',
use: { storageState: './playwright/.auth/user.json' },
dependencies: ['setup-user'],
testMatch: /.*\.user\.spec\.ts/
},
{
name: 'logged-out-tests',
use: { storageState: './playwright/.auth/logged-out.json' },
dependencies: ['setup-logged-out'],
testMatch: /.*\.logged-out\.spec\.ts/
}
]
});3. Test File Organization
tests/
├── auth/
│ ├── admin.setup.ts
│ ├── user.setup.ts
│ └── logged-out.setup.ts
├── admin/
│ └── admin-dashboard.admin.spec.ts
├── user/
│ └── projects.user.spec.ts
└── public/
└── login.logged-out.spec.ts
Benefits of This Approach
✅ Multiple user types/roles - Easy to add new auth scenarios
✅ Logged-out testing - Dedicated tests for unauthenticated flows
✅ Parallel execution - Each auth type runs independently
✅ Clear separation - File naming makes test scope obvious
✅ Scalable - Easy to extend with new roles or scenarios
Migration Strategy
Since you're just setting up the tests, now is the perfect time to implement this architecture. The current auth.setup.ts could become user.setup.ts, and you can gradually add other auth scenarios as needed.
Would you like me to help plan the refactoring to this more flexible structure?
✏️ Learnings added
Learnt from: lifeiscontent
PR: makeplane/plane#7004
File: tests/integration-tests/tests/auth.setup.ts:11-14
Timestamp: 2025-06-06T19:03:04.872Z
Learning: When setting up Playwright authentication for integration tests, use multiple authentication setups (admin.setup.ts, user.setup.ts, logged-out.setup.ts) with separate projects in playwright.config.ts rather than a single shared authentication state. This prevents architectural debt and makes it easier to test different user roles and logged-out scenarios.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| test('should load projects page', async ({ page }) => { | ||
| // Basic test to verify page loads | ||
| await page.getByRole('listitem').getByText('Home').click(); | ||
| }); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Missing assertions in project page test
The test navigates to the Home page but doesn't verify if navigation was successful or if any specific elements are present on the resulting page.
test('should load projects page', async ({ page }) => {
// Basic test to verify page loads
await page.getByRole('listitem').getByText('Home').click();
+ // Verify the page has loaded correctly
+ await expect(page).toHaveURL(/.*\/home/);
+ // Verify some expected element on the projects page is visible
+ await expect(page.getByRole('heading', { name: 'Projects' })).toBeVisible();
});📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| test('should load projects page', async ({ page }) => { | |
| // Basic test to verify page loads | |
| await page.getByRole('listitem').getByText('Home').click(); | |
| }); | |
| test('should load projects page', async ({ page }) => { | |
| // Basic test to verify page loads | |
| await page.getByRole('listitem').getByText('Home').click(); | |
| // Verify the page has loaded correctly | |
| await expect(page).toHaveURL(/.*\/home/); | |
| // Verify some expected element on the projects page is visible | |
| await expect(page.getByRole('heading', { name: 'Projects' })).toBeVisible(); | |
| }); |
| name: "chromium", | ||
| use: { | ||
| ...devices["Desktop Chrome"], | ||
| storageState: "./playwright/.auth/user.json", | ||
| }, | ||
| dependencies: ["setup"], | ||
| testMatch: /.*\.spec\.ts$/, | ||
| }, |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
aheckmann
force-pushed
the
feat-integration_tests
branch
from
ddddb98 to
7ba77ab
Compare
| throw new Error('BASE_URL, EMAIL or PWD is not set'); | ||
| } | ||
| await page.goto(BASE_URL); | ||
| await page.getByRole('textbox', { name: 'Email' }).click(); |
There was a problem hiding this comment.
you could store the textbox as a reference e.g. emailTextbox since you're referencing it multiple times.
|
|
||
| test('should load projects page', async ({ page }) => { | ||
| // Basic test to verify page loads | ||
| await page.getByRole('listitem').getByText('Home').click(); |
There was a problem hiding this comment.
| await page.getByRole('listitem').getByText('Home').click(); | |
| await page.getByRole('listitem', { name: 'Home' }).click(); |
Description
Type of Change
Test Scenarios
References
SILO-200
Summary by CodeRabbit
New Features
Documentation
Chores