Improve CSV and JSON output

[silverhack]

It is a common request to have a compliance report separated in CSV or JSON output, rather than exported in single RAW files, for easier consumption by other tools. Actually, CSV, JSON and CLIXML output are a bit redundant and probably not useful, so instead saving metadata into RAW files, it would be nice to be able to export pass/fails compliance results into a well formatted CSV and JSON files.

On the other hand, the output should be consistent across all formats. As an improvement, CSV and JSON outputs should contain the same fields and same field name in both cases. That consolidated output could then be used for further processing and for easier consumption by other tools.

Finally, the Excel output was deprecated in Monkey365 and will be removed two releases later (0.91.4).

[silverhack]

Hi all,

The proposed CSV will have the following headers:

TIMESTAMP
TENANTID
TENANTNAME
SUBSCRIPTIONID
SUBSCRIPTIONNAME
UNIQUEID
PROVIDER
FINDINGID
FINDINGTITLE
FINDINGTYPE
FINDINGTAGS
SERVICENAME
SEVERITYID
SEVERITY
FINDINGDESCRIPTION
FINDINGRATIONALE
FINDINGREMEDIATION
FINDINGREFERENCEURL
RESOURCELOCATION
STATUS
RESOURCETYPE
RESOURCEID
RESOURCENAME
RESOURCEGROUP
RESOURCETAGS
COMPLIANCE
NOTES
MONKEY365VERSION

It is strongly based on the Open CyberSecurity Schema Framework schema for cloud findings.

For JSON data, property names will be converted to camelCase.

Please, do let me know if you have ideas about how to improve CSV and JSON data output.

Thanks,

[olivierdumon]

Hi @silverhack
I just wanted to know if there was an update on this request.
I'm very interested by the global CSV export option.
Thanks for this great tool.
Good day
Olivier

[silverhack]

Hey @olivierdumon sorry for the late reply and thank you! I'm glad you like the tool :D

Regarding improving output, yes, I'm still working on it. Please note that actually monkey365 is dealing with multiple services (Azure, Entra ID, Exchange, SharePoint, etc..) and every single service has its own schema. Unstructured data is hard to normalise into a common schema that can be consumed by others channels, such as CSV or JSON.

I'm very close to update the tool with these and other improvements. The roadmap will be as follows:

1. A minor version of the tool will be released on this/next week
2. Once the minor version is published, I'll start upgrading some internal modules in order to give support for other formats
3. Test new changes with multiple subscriptions/tenants
4. Update docs
5. Release the new version

Thanks!

[olivierdumon]

Hi @silverhack

Thank you for your feedback

I'll be following your project closely :)
Thank you for your work

[silverhack]

Hey all,

I'm currently working in JSON and CSV data outputs. As previously mentioned, the JSON output is based on the Open CyberSecurity Schema Framework schema for cloud findings.

The following is an example of output:

{
  "metadata": {
    "eventCode": "aad_sbd_enabled",
    "product": {
      "name": "Monkey365",
      "vendorName": "Monkey365",
      "version": "0.98"
    },
    "version": "1.1.0"
  },
  "severityId": 0,
  "severity": "Unknown",
  "status": "New",
  "statusCode": "pass",
  "statusDetail": null,
  "statusId": 1,
  "unmapped": {
    "provider": "EntraID",
    "pluginId": "aad0024",
    "apiType": "EntraIDPortal",
    "resource": "EntraIDPortal"
  },
  "activityName": "Create",
  "activityId": 1,
  "findingInfo": {
    "createdTime": "2024-08-21T11:47:48Z",
    "description": "Security defaults in Microsoft Entra ID (Azure Active Directory) make it easier to be secure and help protect your organization. Security defaults
 contain preconfigured security settings for common attacks.Microsoft is making security defaults available to everyone. The goal is to ensure that all organizations 
have a basic level of security-enabled at no extra cost. The use of security defaults however will prohibit custom settings which are being set with more advanced set
tings.",
    "productId": "Monkey365",
    "title": "Ensure Security Defaults is disabled on Microsoft Entra ID",
    "id": "Monkey365-aad-sbd-enabled-a4807c0361194a9a9da91e02458bd3ff-zxuQ2OfB3Ag"
  },
  "resources": {
    "cloudPartition": "6",
    "region": null,
    "data": null,
    "group": {
      "name": "General"
    },
    "labels": null,
    "name": null,
    "type": null,
    "id": null
  },
  "categoryName": "Findings",
  "categoryId": 2,
  "className": "Detection",
  "classId": 2004,
  "cloud": {
    "account": {
      "name": "Contoso",
      "type": "AzureADAccount",
      "typeId": "6",
      "id": "a4807c03-6119-4a9a-9da9-1e02458bd3ff"
    },
    "organization": {
      "name": "Contoso",
      "id": "a4807c03-6119-4a9a-9da9-1e02458bd3ff"
    },
    "provider": "Microsoft365",
    "region": "global"
  },
  "time": "2024-08-21T11:47:48Z",
  "remediation": {
    "description": "From Azure Console1. Sign in to the Azure portal as a security administrator, Conditional Access administrator, or global administrator.2. Bro
wse to Microsoft Entra ID  Properties.3. Select Manage security defaults.4. Set the Enable security defaults toggle to No.5. Select Save.",
    "references": [
      "https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/users-default-permissions",
      "http://www.rebeladmin.com/2019/04/step-step-guide-restrict-azure-ad-administration-portal/",
      "https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults",
      "https://techcommunity.microsoft.com/t5/azure-active-directory-identity/introducing-security-defaults/ba-p/1061414"
    ]
  },
  "typeId": 200401,
  "typeName": "Create"
}

Please, do let me know in comments below if you have ideas about how to improve CSV and JSON data output.

Thanks in advance,

[olivierdumon]

Hi @silverhack
This example seems enough exhaustive for me :)
Thanks

[silverhack]

Thanks @olivierdumon!

Still working on CSV and CLIXML options, but I will try to upload some CSV,JSON and CLIXML examples soon.

Cheers,

[silverhack]

Hi all,

Finally, the CSV will have the following properties for Microsoft 365:

TIMESTAMP
TENANTID
TENANTNAME
UNIQUEID
PROVIDER
FINDINGID
FINDINGTITLE
FINDINGTYPE
FINDINGTAGS
SERVICENAME
SEVERITYID
SEVERITY
FINDINGDESCRIPTION
FINDINGRATIONALE
FINDINGREMEDIATION
FINDINGREFERENCEURL
RESOURCELOCATION
STATUS
RESOURCETYPE
RESOURCEID
RESOURCENAME
RESOURCEGROUP
RESOURCETAGS
COMPLIANCE
NOTES
MONKEY365VERSION

The following properties are for Azure subscriptions:

TIMESTAMP
TENANTID
TENANTNAME
SUBSCRIPTIONID
SUBSCRIPTIONNAME
UNIQUEID
PROVIDER
FINDINGID
FINDINGTITLE
FINDINGTYPE
FINDINGTAGS
SERVICENAME
SEVERITYID
SEVERITY
FINDINGDESCRIPTION
FINDINGRATIONALE
FINDINGREMEDIATION
FINDINGREFERENCEURL
RESOURCELOCATION
STATUS
RESOURCETYPE
RESOURCEID
RESOURCENAME
RESOURCEGROUP
RESOURCETAGS
COMPLIANCE
NOTES
MONKEY365VERSION

Cheers,

[silverhack]

Hi all,

I'm testing the new modules with some subscriptions and tenants, and hopefully it will be merged into main branch soon. In the meantime, I've attached some examples in different formats:

• monkey3654b94cd8c950c40aebd9135f0aeb0244d27081825.csv

• monkey3654b94cd8c950c40aebd9135f0aeb0244d20240827142648.json

• monkey3654b94cd8c950c40aebd9135f0aeb0244d20240827142648.zip

The above files are some examples and you can import it using the following methods:

JSON

Just use the following command

$json = Get-Content -Raw .\monkey3654b94cd8c950c40aebd9135f0aeb0244d20240827142648.json | ConvertFrom-Json

CSV

For CSV files, the following can be used:

$csv = Get-Content -Raw .\monkey3654b94cd8c950c40aebd9135f0aeb0244d27081825.csv | ConvertFrom-Csv

CLIXML

GitHub not allowed to upload XML files, so this was uploaded into a zip file. The following command can be used:

Expand-Archive .\monkey3654b94cd8c950c40aebd9135f0aeb0244d20240827142648.zip monkey
$cliXml = Import-Clixml .\monkey\monkey3654b94cd8c950c40aebd9135f0aeb0244d20240827142648.clixml

Cheers,

[silverhack]

Hi all,

Already implemented in main branch. Please, raise an issue if you find a problem.

Go, go go!