Implement capability to run arbitrary scripts as part of testing

[dhrogers]

We need to expand the testing capability beyond just image comparison.

Running an arbitrary script would be a good next step.

[pnav]

Hi David,

Isn't this what the unit testing harness accomplishes? Now, those need to be more fully fleshed out, but the point of those are to verify functional units apart from the gold image tests.

See tests/src/framework/Application.cpp for an example.

If that's not what you have in mind, can you elaborate more on what you mean by "running an arbitrary script"?

• Paul

[GregAbram]

We’re thinking about tests that use apps other than gxywriter. For example, we’re working on sampling…particle trace…trace-to-tube… render. The first and (soon) the last steps will use state files for setup, but it uses a separate app (sampletrace) to wrap the four steps. So I was thinking about having shell files in the tests directory that are searched for and executed (like it searches for .state files and gxywriter’s them). The shell files might refer to state files that have a different extension than .state so they don’t get caught up in the current approach. The assumption would be that the scripts would produce images for testing just like the current approach, but perhaps using several distinct steps and maybe using alternative apps to create the rendering. On Aug 1, 2019, at 3:36 PM, Paul Navrátil <[email protected]<mailto:[email protected]>> wrote: Hi David, Isn't this what the unit testing harness accomplishes? Now, those need to be more fully fleshed out, but the point of those are to verify functional units apart from the gold image tests. See tests/src/framework/Application.cpp for an example. If that's not what you have in mind, can you elaborate more on what you mean by "running an arbitrary script"? * Paul — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAHZWRSHZVRO53CHE5T6RPDQCNJNVA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3L6XZA#issuecomment-517467108>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAHZWRXAD5TQOR6VSDZCJCTQCNJNVANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401>

[pnav]

got it. yep we should certainly test apps beyond gxywriter. ideally each should have its own test. On Aug 1, 2019, at 17:17, Greg Abram <[email protected]<mailto:[email protected]>> wrote: We’re thinking about tests that use apps other than gxywriter. For example, we’re working on sampling…particle trace…trace-to-tube… render. The first and (soon) the last steps will use state files for setup, but it uses a separate app (sampletrace) to wrap the four steps. So I was thinking about having shell files in the tests directory that are searched for and executed (like it searches for .state files and gxywriter’s them). The shell files might refer to state files that have a different extension than .state so they don’t get caught up in the current approach. The assumption would be that the scripts would produce images for testing just like the current approach, but perhaps using several distinct steps and maybe using alternative apps to create the rendering. On Aug 1, 2019, at 3:36 PM, Paul Navrátil <[email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: Hi David, Isn't this what the unit testing harness accomplishes? Now, those need to be more fully fleshed out, but the point of those are to verify functional units apart from the gold image tests. See tests/src/framework/Application.cpp for an example. If that's not what you have in mind, can you elaborate more on what you mean by "running an arbitrary script"? * Paul — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAHZWRSHZVRO53CHE5T6RPDQCNJNVA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3L6XZA#issuecomment-517467108>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAHZWRXAD5TQOR6VSDZCJCTQCNJNVANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAI74G62HCA6BU6HLNEIRELQCNOGPA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3MBNEA#issuecomment-517478032>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAI74G2YMN43IIV7HRZ674DQCNOGPANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401>

[GregAbram]

Would allow us to test the multiserver, too. On Aug 1, 2019, at 4:19 PM, Paul Navrátil <[email protected]<mailto:[email protected]>> wrote: got it. yep we should certainly test apps beyond gxywriter. ideally each should have its own test. On Aug 1, 2019, at 17:17, Greg Abram <[email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: We’re thinking about tests that use apps other than gxywriter. For example, we’re working on sampling…particle trace…trace-to-tube… render. The first and (soon) the last steps will use state files for setup, but it uses a separate app (sampletrace) to wrap the four steps. So I was thinking about having shell files in the tests directory that are searched for and executed (like it searches for .state files and gxywriter’s them). The shell files might refer to state files that have a different extension than .state so they don’t get caught up in the current approach. The assumption would be that the scripts would produce images for testing just like the current approach, but perhaps using several distinct steps and maybe using alternative apps to create the rendering. On Aug 1, 2019, at 3:36 PM, Paul Navrátil <[email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> wrote: Hi David, Isn't this what the unit testing harness accomplishes? Now, those need to be more fully fleshed out, but the point of those are to verify functional units apart from the gold image tests. See tests/src/framework/Application.cpp for an example. If that's not what you have in mind, can you elaborate more on what you mean by "running an arbitrary script"? * Paul — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAHZWRSHZVRO53CHE5T6RPDQCNJNVA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3L6XZA#issuecomment-517467108>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAHZWRXAD5TQOR6VSDZCJCTQCNJNVANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAI74G62HCA6BU6HLNEIRELQCNOGPA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3MBNEA#issuecomment-517478032>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAI74G2YMN43IIV7HRZ674DQCNOGPANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAHZWRUJHX3BI3WS7DB55CTQCNOO7A5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3MBRBY#issuecomment-517478535>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAHZWRRVH6CPTSRQWDZNQJTQCNOO7ANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401>

[dhrogers]

I have a new application 'raysample' which can be used to test the new ray-based sampling capability. It creates images which could be compared to some 'gold' image, the way the current testing works.

[pnav]

Excellent! I'll find it and integrate it into the testing structure.

…

________________________________ From: David H. Rogers <[email protected]> Sent: Friday, August 2, 2019 3:45 PM To: TACC/Galaxy <[email protected]> Cc: Paul Navratil <[email protected]>; Assign <[email protected]> Subject: Re: [TACC/Galaxy] Implement capability to run arbitrary scripts as part of testing (#10) I have a new application 'raysample' which can be used to test the new ray-based sampling capability. It creates images which could be compared to some 'gold' image, the way the current testing works. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAI74G6POYXSZNEHWCCMAULQCSMG7A5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OZRUA#issuecomment-517839056>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAI74GYF42W2YATCJT6L6MLQCSMG7ANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401>

[GregAbram]

I’ve enhanced sample tracer so it takes 2 state files - one specifying the sampling, and one specifying the rendering. Here’s one of 4 renderings from it - will look way cooler when Intel fixes the sprinkles. [cid:[email protected]] Greg Abram [email protected]<mailto:[email protected]> On Aug 2, 2019, at 3:01 PM, Paul Navrátil <[email protected]<mailto:[email protected]>> wrote: Excellent! I'll find it and integrate it into the testing structure.

…

________________________________ From: David H. Rogers <[email protected]<mailto:[email protected]>> Sent: Friday, August 2, 2019 3:45 PM To: TACC/Galaxy <[email protected]<mailto:[email protected]>> Cc: Paul Navratil <[email protected]<mailto:[email protected]>>; Assign <[email protected]<mailto:[email protected]>> Subject: Re: [TACC/Galaxy] Implement capability to run arbitrary scripts as part of testing (#10) I have a new application 'raysample' which can be used to test the new ray-based sampling capability. It creates images which could be compared to some 'gold' image, the way the current testing works. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAI74G6POYXSZNEHWCCMAULQCSMG7A5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OZRUA#issuecomment-517839056>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAI74GYF42W2YATCJT6L6MLQCSMG7ANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#10?email_source=notifications&email_token=AAHZWRXBL3HIZ4D7NEXJIUDQCSOCTA5CNFSM4IIWE3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3O2Q6A#issuecomment-517843064>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAHZWRTOV3XRFLCNTV7NMCLQCSOCTANCNFSM4IIWE3LA>. This message is from an external sender. Learn more about why this matters.<https://ut.service-now.com/sp?id=kb_article&number=KB0011401>