Implement Step.RunHidden to not show a step in the report

[botondberes]

Hi Anton,

good news! We purchased our license today!
That being said, I would like to request 2 features to be prioritized please.
One being this issue regarding a Step.RunHidden method that does exactly what Run does but it simply does not publish its metrics so they won't show up in the report.
I took the liberty and went ahead and tried to implement this functionality with my limited F# knowledge and the attached patch is the result. Unfortunately I couldn't figure out how to create overloads in F# (some sources say it's not possible) so I created a new method instead but I'm sure you'll find a better way of implementing it.
RunHidden.patch
As a little background on why we need this: We're load testing a MS SQL server and for that I'm preparing SQLCommands in a Step prior to the actual execution, then I cleanup using a Step after. This leads to the report showing 3 steps (Prepare, Execute, Cleanup) but the Preparation and Cleanup are not something our clients are interested in, the only important value is the time it took to execute the stored procedure. I need to prepare the command prior to execution in the ScenarioContext because the context will determine whether or not we use individual fake values, constants or real data.

As for the second feature, we would love to be able to use this #643. I'm thinking something along the lines of:

NBomberRunner
	.RegisterScenarios([..scenarios])
	.WithTestName(name)
        .Sequential(true)
	.Run();

If you have any questions, please don't hesitate to ask.
I look forward to your response.

Regards,
Botond

[AntyaDev]

Hi @botondberes ,
Thank you for choosing NBomber) It's fantastic news.

We can prioritize the features you asked for. Next week, we will release version 5.6, and afterward, we will commence work on version 5.7, which may incorporate those features. Let me double-check the backlog for version 5.7, but I believe it's realistic.

Once again, thank you for choosing NBomber! Feel free to reach out if you have any further questions or need assistance.

[botondberes]

Thank you! We look forward to the next versions!

[AntyaDev]

Hi @botondberes
You mentioned that "the Preparation and Cleanup are not something our clients are interested in"
question: why do you need them modeled as Step? Did you consider implementing them as regular C# methods?

{
    Prepare(); // it's a regular C# method invocation
    var step = Step.Run("Execute")
    Cleanup();
}

Does it make sense?

[botondberes]

Hi Anton,

We do want the Prepare and Cleanup steps to be measured and potentially be tracked by a ReportingSink etc. Actually, having the results for all steps in the .txt log file would also be helpful, we just don't want those steps to show on the HTML report which we share with our clients.

Thanks in advance!

[AntyaDev]

Ah I got it now, thanks

[AntyaDev]

Hi @botondberes
Will it be ok if the user specifies hidden = true and the output reports (HTML, TXT, MD) do not contain this step?

Step.run(name, ctx, run, hidden = true)

I noticed that you mentioned only HTML reports, but I think this can be confusing. As a user, I would think that this would apply to all types of reports.

[AntyaDev]

@botondberes I just started thinking about this functionality differently :)

Since you want to filter out some of the steps, would it be better if we provide some callback that will be invoked before reports are created?

NBomberRunner
    .RegisterScenarios(scenario)
    .WithReportFinalizer(data =>
    {
        var stepStats = data.ScenarioStats[0].StepStats;

        data.ScenarioStats[0].StepStats = stepStats.Where(x => x.StepName != "hidden").ToArray();

        return data;
    })
    .Run();    

[botondberes]

I think we would prefer having a boolean attached to the Step instead of having to filter by name but I could make both versions work for us.

[AntyaDev]

@botondberes Please try latest beta: https://www.nuget.org/packages/NBomber/5.7.0-beta.1
It contains this feature: #690

[AntyaDev]

@botondberes

I think we would prefer having a boolean attached to the Step instead of having to filter by name but I could make both versions work for us.

• The approach with boolean Step.Run(name, hidden: true) is less ergonomic than ReportFinalizer. I think that ReportFinalizer can solve a class of problems like Hidden Steps or Hidden Scenarios. With ReportFinilizer, we could even add options to inject custom HTML, custom report descriptions, etc.
• Also, ReportFinalizer is a more elegant addition since it logically sits in the processing pipeline as a finalized step before returning the final result. Conversely, Step.RunHidden can't be added without changing several components because it sits in the core engine and requires an extension.
  Step.RunHidden can be treated differently. For example, I would consider that when hidden: true, we should filter stats for the ReportingSinks.

Do you think it makes sense?

[botondberes]

I see, yes that makes sense. I won't be able to test it today but should have some time tomorrow.

[AntyaDev]

Thanks. It is still not the final solution. We will wait for your feedback.

[botondberes]

I've tested it and it works fine. This is what I've used:

.WithReportFinalizer(data =>
{
	var stepStats = data.ScenarioStats[0].StepStats;
	data.ScenarioStats[0].StepStats = stepStats.Where(x => !x.StepName.Contains(Constants.SKIP_REPORT)).ToArray();
	return data;
})

Thanks for the quick implementation!

[AntyaDev]

@botondberes
You are welcome!

Just in case, you can use:

data.GetScenarioStats("scenario name").StepStats

I will include this in the docs when published.