Images not being included in mochawesome report

[Pablo-Quiroga-Globant]

Describe the bug

Diff images are not being included into mochawesome reports. The directory to place the images is being created, though the path doesn't look right.

To Reproduce
Steps to reproduce the behavior:

1. Create a Cypress project and add mochawesome as a dependency.

    "cypress": "^10.6.0",
    "cypress-mochawesome-reporter": "^3.2.2",

2. Use the following Cypress config:

const { defineConfig } = require('cypress');
const visualRegression = require('@frsource/cypress-plugin-visual-regression-diff/dist/plugins');

module.exports = defineConfig({
  reporter: 'cypress-mochawesome-reporter',
  e2e: {
    baseUrl: 'https://your.website.test/',
    setupNodeEvents(on, config) {
      require('cypress-mochawesome-reporter/plugin')(on);
      visualRegression.initPlugin(on, config);
    },
  },
});

3. Create a test that includes cy.matchImage().
4. After test end, go to cypress/reports/html and open the index.html file containing test results.

Expected behavior
I'm expecting to see on the report is the original screenshot in case there's no diff. If there is any diff, I expect to see the original screenshot, the diff and the actual. For actual behavior, see attached screenshot.

Screenshots

The url of the file is pointing to file:///{root_path}/cypress/reports/html/screenshots{root_path}/cypress/integration/__image_snapshots__/Error%20page%20should%20validate%20there%20are%20no%20visual%20changes%20#0.actual.png

Note: {root_path} is the full path to the project folder.

Please complete the following information:

• OS and version: Ubuntu 20.04
• Browser and version: Electron (the one included with Cypress 10.6.0)
• Cypress version: 10.6.0

Additional context

I'm using cypress-mochawesome-reports plugin. I can see the diff on the Cypress UI, but when I run the tests on headless mode (the default on the CI) the images are not being added to the report. It creates a set of empty folders (where the image should be).

I'm using cypress-mochawesome-reports default config, and visual regression plugin defaults. The issue becomes more evident when I turn on embeddedScreenshots on for the reporter:

reporter: 'cypress-mochawesome-reporter',
reporterOptions: {
   embeddedScreenshots: true,
},

I'm getting the following error:

Start generate report process
Read and merge jsons from "{root_path}/cypress/reports/html/.jsons"
Enhance report
An error was thrown in your plugins file while executing the handler for the after:run event.

The error we received was:

Error: ENOENT: no such file or directory, open '{root_path}/cypress/screenshots{root_path}/cypress/integration/__image_snapshots__/Error page should validate there are no visual changes #0.actual.png'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)
    at convertImageToBase64 ({root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:111:52)
    at {root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:104:11
    at Array.map (<anonymous>)
    at createScreenshotsContextList ({root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:100:22)
    at attachScreenshotsToTestContext ({root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:76:9)
    at {root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:34:24
    at Array.forEach (<anonymous>)
    at {root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:32:19
    at Array.forEach (<anonymous>)
    at attachScreenshotsToSuiteTestsContext ({root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:31:12)
    at {root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:16:7
    at Array.forEach (<anonymous>)
    at enhanceReport ({root_path}/node_modules/cypress-mochawesome-reporter/lib/enhanceReport.js:6:18)
    at mergeAndCreate ({root_path}/node_modules/cypress-mochawesome-reporter/lib/generateReport.js:19:3)
    at async Promise.all (index 0)
    at async generateReport ({root_path}/node_modules/cypress-mochawesome-reporter/lib/generateReport.js:62:22)
    at async afterRunHook ({root_path}/node_modules/cypress-mochawesome-reporter/lib/index.js:35:3)
    at async Object.handler ({root_path}/node_modules/cypress-mochawesome-reporter/plugin.js:9:5)

I believe that the issue is related to the path of the image being sent to the test context.

Thank you!

[Pablo-Quiroga-Globant]

@FRSgit Hello! Anything else I can do to help?

[FRSgit]

@Pablo-Quiroga-Globant Hey, I think there is at least one issue with cypress-mochawesome-reporter that needs to be resolved before your issue can be handled. Have a look here.
To sum up - currently any screenshot with any special characters in it's name won't appear in HTML report generated by cypress-mochawesome and we need a PR with a fix there. I unfortunately won't do it this week - so if you're in rush, please have a look there 🙏

[luqy]

I also encounter the same problem, whether it is necessary to use special characters

[FRSgit]

Hey @luqy,
Thanks for the suggestion, but it would be a breaking change for our users - all of the existing screenshots would need to be removed & regenerated once again. Since it's an obvious error in mochawesome package I'd say it's much better to address the issue there

[FRSgit]

Just an update: cypress-mochawesome-reporter has been just fixed LironEr/cypress-mochawesome-reporter#106.

Now I can finish up work resolving this issue, stay tuned! 😁

[mikstime]

We are facing a similar issue. Apparently, the compared images do not exist in the file system.

Dependencies:

{
  "cypress": "10.8.0",
  "cypress-mochawesome-reporter": "3.2.3",
  "@frsource/cypress-plugin-visual-regression-diff": "3.1.2",
}

Cypress Error

cy.task('cp-visual-regression-diff-compareImages') failed with the following error:

> ENOENT: no such file or directory, open 'report-desktop/all-screenshots/cp-visual-regression-diff_snapshots/cypress/e2e/desktop/screenshot/image_snapshots/screen check1 #0.actual.png.png

Stack trace

From Node.js Internals:
  Error: ENOENT: no such file or directory, open 'report-desktop/all-screenshots/__cp-visual-regression-diff_snapshots__/cypress/e2e/desktop/screenshot/__image_snapshots__/screen check1 #0.actual.png.png'
      at Object.openSync (node:fs:585:3)
      at Object.readFileSync (node:fs:453:35)
      at scaleImageAndWrite (<hidden path>/node_modules/@frsource/cypress-plugin-visual-regression-diff/src/image.utils.ts:64:24)
      at compareImagesTask (<hidden path>/node_modules/@frsource/cypress-plugin-visual-regression-diff/src/task.hook.ts:74:33)
      at invoke (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:234:16)
      at <unknown> (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:59:14)
      at tryCatcher (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/node_modules/bluebird/js/release/util.js:16:23)
      at Function.Promise.attempt.Promise.try (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/node_modules/bluebird/js/release/method.js:39:29)
      at Object.wrapChildPromise (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:58:23)
      at RunPlugins.taskExecute (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:240:10)
      at RunPlugins.execute (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:160:21)
      at EventEmitter.<anonymous> (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:257:12)
      at EventEmitter.emit (node:events:390:28)
      at EventEmitter.emit (node:domain:475:12)
      at process.<anonymous> (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:33:22)
      at process.emit (node:events:390:28)
      at process.emit (node:domain:475:12)
      at process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/mikstime/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
      at emit (node:internal/child_process:917:12)
      at processTicksAndRejections (node:internal/process/task_queues:84:21)

[mikstime]

Cypress Error

cy.task('cp-visual-regression-diff-compareImages') failed with the following error:

> ENOENT: no such file or directory, open 'report-desktop/all-screenshots/cp-visual-regression-diff_snapshots/cypress/e2e/desktop/screenshot/image_snapshots/screen check1 #0.actual.png.png

After taking a screenshot, it is moved to another folder, so that it could be compared with the original one later. How it's implemented:

1. on('after:screenshot') handler asynchronously moves the file and returns {path: newPath}.
2. cy.screenshot(oldPath, {onAfterScreenshot: (el, {path}) => {…}}) is called to retrieve newPath.

So onAfterScreenshot is expected to have newPath instead of oldPath. But in practice this does not happen as of cypress 10.10.0.

@FRSgit I wonder if you have faced similar issues during development process?

[FRSgit]

Sorry @mikstime for lack of responses - I'm knee-deep into issue #173 at the moment as it seems more critical :<

I don't know if I understood your comment correctly. Can you confirm that the "path replacement" that was implemented in a return method of on('after:screenshot') handler have worked previously but is not anymore since [email protected]? If so, we need to file an issue request on cypress repo.

[mikstime]

Can you confirm that the "path replacement" that was implemented in a return method of on('after:screenshot') handler have worked previously but is not anymore since [email protected]?

I’ll reproduce this issue on a blank project and let you know soon

[mikstime]

Turns out there was a problem with custom defined on event binder which had no support for return values.

Cypress has a limitation of adding a single event handler per event. Since your plugin adds custom 'before:browser:launch' handler, you should consider exporting it.

[mikstime]

Apparently, mochawesome reporter can't proceed screenshots outside the report folder. And this plugin currently moves screenshots outside screenshots folder.

It seems like option to disable "path replacement" functionality and slight modification of the comparison logic could do the thing. @FRSgit WDYT?

[mikstime]

I found a way to create an adaptor for mochawesome report. But in order to effortlessly maintain it the ability to modify imgPath is needed. Would you mind consider accepting this PR.

[mikstime]

Are there any updates on this issue?

[AnnVb]

Any update on this issue?

[BlaneyXYZ]

Any word on a solution?

[AlexRusbridge]

Hey @FRSgit, just wondering if you have an update on this.

We're currently using uktrade/cypress-image-diff to perform VR testing for our web app, but the inability to update baseline images all at once is slowing down our development times.

Looking at the other plugins available, this one seems to support the functionality we need. However, as we're a large development team with a range of OS's and browsers, we run VR testing in the pipeline to ensure consistency. This means that producing a diff report is a key feature for us.

Either a native report talked about in discussion #96, or the ability to pull in screenshots through a third-party plugin like mochawesome.

Do you have a rough ETA of when either of these would be implemented, or a workaround we can use in the meantime?

Thanks,
Alex

[FRSgit]

Hey everyone!

The newest status is that mochawesome most probably contains a bug, but it hasn't been pursued.

Priorities (from my perspective):

1. I'd really like somebody to try to step forward and fix this issue in mochawesome-reporter repo. It important that this gets fixed directly in the reporter code. Unfortunately, I won't be able to tackle it - currently have too many things on my plate right now.
2. There is a PR that might provide a bit hackish (but working!) workaround to this problem. I'll try to merge it to the main and release it this week.