Searchkit Save to file rather than browser download.

[rbaugh]

Overview

There isn't a way to save SK reports to the file system for historical/snapshot recording. This will allow for saving files to the system and you can also schedule jobs with the new action to have your reports saved automatically. A CiviRule could likely send an email out when the file is created, but that is not for this PR.

Before

Currently the only option is to download the report from the browser. We have a need to be able to keep historical/snapshot reports but there isn't a way unless you manually download and save yourself.

After

With a new saveFile API4 action, this allows you to save a report output to the file system.

Technical Details

This adds a new bool to the Download action which prevents the headers from being written when not array format and also prevents the exit once the file contents are available. Secondly, this adds a new action, saveFile, for SearchDisplay which then leverages the new bool property of the Download to capture the file data and it then saves it to the file system. It is also recording the file to the File and EntityFile entities so that you can pull reports back out. This doesn't include any reports, just the ability to save the report to a file on the system and log it in the File entities.

In the new action, you can specify a file name and also a new folder that the file can be saved to. It is using the AbstractRunAction in order to capture the additional run fields that are just passed on to the Download action. With the bool, we can then capture the results of the download and save to a file.

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

Quick links for reviewers...

• ➡️ PR Demo site (can be rebuilt with the command test this please).
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[demeritcowboy]

There was a previous attempt that was similar (in spirit) to this which didn't get merged but I think the use-case was mostly solved by the DB Entity display option (add a display of type DB Entity). At least it seems like it would solve your use-case.

[rbaugh]

From the looks of it, the DB entity just refills the tables doesn’t it? The purpose of this is to be able to go back and compare say memberships from previous months or years.

[rbaugh]

@demeritcowboy I looked through the SK code and the Entity overwrites the data so we don't have a snapshot in time which this PR would allow you to capture. I would be willing to remove the SaveFile API action from this PR if we could keep the bool flag on the Download action. This way I can leverage the Download and store the file on my own and not have the process killed as if the download was via the browser.

[rbaugh]

Also, I think with just the bool attribute to the download, you could then make a call to the API to get the file and use the file to email the report. I know that the email option was on the roadmap so this could potentially allow for completing the option for emailing as well.

[demeritcowboy]

I don't really have a stake in it anymore. If someone wants to try this out and ok it and if @colemanw is ok with the architecture then I'm not going to stand in the way.

But just also offering that it sounds like just the following would satisfy your use-case if put it in a file and cron schedule it. This is also how some people are doing mailings, just with adding one more line to do the mailing.

#!/bin/sh
THE_DATE=`date +%Y%m%d%H%M%S`
cv api4 --user=admin_cms_user SearchDisplay.download savedSearch=saved_search_name display=search_display_name format=csv > report_$THE_DATE.csv

[rbaugh]

Looking back over the getEntityTypes function, the use of the EntityFile entry in my SaveFile Action wouldn't work anyways so I can pull that portion of this PR. @colemanw if we could determine if it is feasible to just have the bool added to the Download action, I'll clean up the PR so it could be merge ready.

[rbaugh]

So it looks like if I add the civicrm_search_display to the getEntityTypes function, then it allows for querying the files back out in APIv4 and can now be used in SK as well. I can use feedback on what impacts adding the bool to the Download action could cause. Otherwise it seems like a simple addition that can open up possibilities of creating snapshots that you can save out or you could even use for emailing the reports or both.

I saw the comment in the getEntityTables function in regards to ACL performance, so hopefully adding the search display doesn't cause any undue performance impact.