Merge pull request #66 from microsoft/FUAM

Author: itsnotaboutthecell

monitoring/README.md

@@ -1,4 +1,3 @@
 # Fabric Monitoring
 
-Assets in this folder are meant to enhance your monitoring experience with Fabric
-
+Assets in this folder are meant to enhance your monitoring experience with Fabric

monitoring/fabric-unified-admin-monitoring/README.md

@@ -0,0 +1,66 @@
+# Fabric Unified Admin Monitoring (FUAM)
+
+Solution accelerator for Fabric platform administrator monitoring.
+
+![image](./media/general/fuam_cover.png)
+
+## Introduction
+
+Fabric Unfied Admin Monitoring (short: FUAM) is a solution to enable a holistic monitoring on top of Microsoft Fabric. Today monitoring for Fabric can be done through different reports, apps and tools. 
+
+
+FUAM has the goal to provide a more holistic view on top of the various information, which can be extracted from Fabric, allowing it's users to analyze at a very high level, but also to deep dive into specific artifacts for a more fine granular data analysis.
+
+
+![image](./media/general/fuam_monitoring_map_cover.png)
+
+
+FUAM is completely build with Fabric capabilities with Pipelines and Notebooks as the main tool to extract and transform data. All of the data is stored in it's raw format, but also in Delta Parquet, enabling the user to directly use it through Power BI Direct Lake or the Lakehouse SQL Endpoint via SQL. FUAM comes with a set of standard report enabling a quick overview on the data, but it's intended to give the users all tool to customize or build the own reports on top of the data model. Through the openness of Fabric it's of course also possible to combine it with your own data enabling you to do the analysis you wish.
+
+
+## FUAM Content
+
+
+FUAM extracts the following data from the tenant:
+
+- Tenant Settings
+- Delegated Tenant Settings
+- Activities
+- Workspaces
+- Capacities
+- Capacity Metrics
+- Tenant meta data (Scanner API)
+- Capacity Refreshables
+- Git Connections
+- Engine level insights (coming soon in optimization module)
+
+### Architecture
+The architecture of FUAM is built on Fabric items like Pipelines, Notebooks, Lakehouses, Semantic Models and Power BI reports. We have built the component in a modular structure, which helps you to extend FUAM with your own modules. This architecture design helps to maintain the solution also with ease.
+
+The data ingestion logic is orchastrated and parametizable, which allows to use the main orchestration pipeline for initial and incremental data loads. FUAM Lakehouse is one of the core component in the architecture. All the data is transformed and persisted in a way, which open amazing capabilities analyzing the collected data in a semantic model with DirectLake mode.
+
+![image](./media/general/fuam_architecture.png)
+
+### Benefits
+- Modular building blocks
+- One-source-of-truth approach
+- End-to-end report experience
+- Intelligent optimization module (coming soon)
+- Long term CU utilization analysis
+- Bring your own data & connect FUAM
+
+## Setup
+
+Are you ready to try FUAM out? We have prepared two step-by-step documentations, which support you to deploy FUAM in your workspace on your tenant:
+
+#### [Click here to **deploy/upgrade** FUAM](/monitoring/fabric-unified-admin-monitoring/how-to/How_to_deploy_FUAM.md)
+
+
+## Support
+The FUAM solution accelerator template **is not official Microsoft service**.
+
+**Ideas/Suggestions:** Submit your ideas and suggestions as an issue in this repository.
+
+**Bug Reports:** We maintain a backlog in the project issues page. Let us know if you run into any problems or share your suggestions by adding an entry into the issues section.
+
+**Important:** Please, do not open a support ticket in case of an issue regarding the templates. In case of any questions, issues regarding the templates, please create an issue in this repository.

monitoring/fabric-unified-admin-monitoring/config/deployment_order.json

@@ -0,0 +1,186 @@
+[
+    {
+        "name": "FUAM Master DEV",
+        "fuam_id": "eb764c8b-cf3b-55be-adf4-348fe9233657"
+    },
+    {
+        "name": "fuam pbi-service-api admin",
+        "fuam_id": "30c5df6c-5353-562f-9d7a-01a9eec0250b"
+    },
+    {
+        "name": "fuam fabric-service-api admin",
+        "fuam_id": "ea79859d-a1fb-5a17-a815-da0602762a8a"
+    },
+    {
+        "name": "FUAM_Lakehouse.Lakehouse",
+        "fuam_id": "729eb8a2-8070-5ed8-ad43-dccbc00b32af"
+    },
+    {
+        "name": "FUAM_Config_Lakehouse.Lakehouse",
+        "fuam_id": "81e56522-cafc-5ce8-8070-002382f38d4a"
+    },
+    {
+        "name": "FUAM_Staging_Lakehouse.Lakehouse",
+        "fuam_id": "63716a7d-def7-5cfb-b07e-16e54f516cc9"
+    },
+    {
+        "name": "Generate_Calendar_Table.Notebook",
+        "fuam_id": "fa9500ac-2738-5288-a823-239c83e997dd"
+    },
+    {
+        "name": "Init_FUAM_Lakehouse_Tables.Notebook",
+        "fuam_id": "f7fbb76e-da62-5fe8-a598-f2843775a00f"
+    },
+    {
+        "name": "01_Transfer_Active_Items_Unit.Notebook",
+        "fuam_id": "d587f38d-ef29-54e5-ab85-2703576c8db9"
+    },
+    {
+        "name": "Load_Items_E2E.DataPipeline",
+        "fuam_id": "038f8eb2-532f-5ac0-9c2e-734b5fce4837"
+    },
+    {
+        "name": "02_Transfer_Activities_Unit.Notebook",
+        "fuam_id": "79eae50d-9aa9-50f9-99e2-0436ba9ab649"
+    },
+    {
+        "name": "03_Aggregate_Activities_Unit.Notebook",
+        "fuam_id": "b87e115f-b8d0-5505-b264-8cd986ededc3"
+    },
+    {
+        "name": "Load_Activities_E2E.DataPipeline",
+        "fuam_id": "2c7e10f5-7d44-5acf-a471-0b0d1fa6f949"
+    },
+    {
+        "name": "01_Transfer_Capacities_Unit.Notebook",
+        "fuam_id": "164524ea-e82b-5f52-a752-1ee9ac0d3808"
+    },
+    {
+        "name": "Load_Capacities_E2E.DataPipeline",
+        "fuam_id": "13f115cc-e9e2-591e-afb5-5ec0b820973a"
+    },
+    {
+        "name": "01_Transfer_Capacity_Refreshables_Unit.Notebook",
+        "fuam_id": "6910d6b4-546d-57a9-84db-7a004813cbb8"
+    },
+    {
+        "name": "Load_Capacity_Refreshables_E2E.DataPipeline",
+        "fuam_id": "8c28e8a4-0ede-5600-b754-1b440d7e7b74"
+    },
+    {
+        "name": "02_Transfer_Workspaces_Unit.Notebook",
+        "fuam_id": "7f851b21-59b2-5ca7-9ab0-9869d9b32ba5"
+    },
+    {
+        "name": "Load_PBI_Workspaces_E2E.DataPipeline",
+        "fuam_id": "ca8aed99-4fff-58d4-8c52-e559b4af2d2b"
+    },
+    {
+        "name": "01_Transfer_Incremental_Inventory_Unit.Notebook",
+        "fuam_id": "b2971e0b-f5c8-5e37-849f-fc245dca21fd"
+    },
+    {
+        "name": "Load_Inventory_E2E.DataPipeline",
+        "fuam_id": "1ebc7e31-c85c-5669-80ac-4e60c2312cb6"
+    },
+    {
+        "name": "01_Transfer_Delegated_Tenant_Settings_Overrides_Unit.Notebook",
+        "fuam_id": "1aa06487-46b4-5b0d-86c8-d0a2ffd3607a"
+    },
+    {
+        "name": "01_Transfer_Tenant_Admin_Settings_Unit.Notebook",
+        "fuam_id": "4d9bbbb0-0d45-5737-928d-ea910ca156e7"
+    },
+    {
+        "name": "Load_Tenant_Settings_E2E.DataPipeline",
+        "fuam_id": "5b3737e4-4ffe-56f6-bb98-ecf4496aa813"
+    },
+    {
+        "name": "Load_Delegated_Tenant_Settings_Overrides_E2E.DataPipeline",
+        "fuam_id": "6f620dc5-1bbf-5284-9143-4c0db4f10452"
+    },
+    {
+        "name": "01_Transfer_Git_Connections_Unit.Notebook",
+        "fuam_id": "2d451082-7969-50fb-a564-f9f8448d778e"
+    },
+    {
+        "name": "Load_Git_Connections_E2E.DataPipeline",
+        "fuam_id": "6be51f60-f72d-51f4-ba74-afb5561752f4"
+    },
+    {
+        "name": "FUAM_Backup_Lakehouse.Lakehouse",
+        "fuam_id": "23845d54-5fa7-5568-88ba-a4847e25878e"
+    },
+    {
+        "name": "01_FUAM_Lakehouse_Backup.Notebook",
+        "fuam_id": "356043fc-da88-5bab-9eaf-586787c6da6d"
+    },
+    {
+        "name": "02_FUAM_Lakehouse_Optimization.Notebook",
+        "fuam_id": "6a60b8f4-2512-5832-8c57-aa0bc6463d8d"
+    },
+    {
+        "name": "Maintenance_for_FUAM.DataPipeline",
+        "fuam_id": "51caae68-d672-5081-ab83-2d33a79ad380"
+    },
+    {
+        "name": "01_Transfer_CapacityMetricData_Timepoints_Unit.Notebook",
+        "fuam_id": "2a4d82cd-7581-5548-a184-301407eadbc1"
+    },
+    {
+        "name": "02_Transfer_CapacityMetricData_ItemKind_Unit.Notebook",
+        "fuam_id": "fc8c385f-18f8-556f-986b-a360cc855827"
+    },
+    {
+        "name": "03_Transfer_CapacityMetricData_ItemOperation_Unit.Notebook",
+        "fuam_id": "38e5dae7-b770-535e-85a3-512d20d8c556"
+    },
+    {
+        "name": "Load_Capacity_Metrics_E2E.DataPipeline",
+        "fuam_id": "20918174-b1be-5c2c-9d6a-c0813660137a"
+    },
+    {
+        "name": "Load_FUAM_Data_E2E.DataPipeline",
+        "fuam_id": "7e182286-079c-5370-bdb1-c34567fff85f"
+    },
+    {
+        "name": "FUAM_Item_SM.SemanticModel",
+        "fuam_id": "896543f5-c24f-5e3e-89a7-b3bd3fcf498e"
+    },
+    {
+        "name": "FUAM_Item_Analyzer_Report.Report",
+        "fuam_id": "84b6a5b7-4150-5a30-b2d3-07ffdca8f818"
+    },
+    {
+        "name": "FUAM_Core_SM.SemanticModel",
+        "fuam_id": "2d2f2490-9f07-5eb0-acbb-6eea72a03527"
+    },
+    {
+        "name": "FUAM_Core_Report.Report",
+        "fuam_id": "0c1d8145-fa2c-55d1-91a3-a30aa115d1a7"
+    },
+    {
+        "name": "FUAM_Semantic_Model_Meta_Data_Analyzer_SM.SemanticModel",
+        "fuam_id": "f7d9f33b-9d76-584d-b880-afc497bbcff0"
+    },
+    {
+        "name": "FUAM_Semantic_Model_Meta_Data_Analyzer_Report.Report",
+        "fuam_id": "7b9a41c8-7cf2-502a-9bb3-4c538ee4477f"
+    },
+    {
+        "name": "FUAM_SQL_Endpoint_Analyzer_SM.SemanticModel",
+        "fuam_id": "d3c9522f-3bbb-563f-8405-f45bd4f9b7c0"
+    },
+    {
+        "name": "FUAM_SQL_Endpoint_Analyzer_Report.Report",
+        "fuam_id": "2b4d3fb4-315c-5f0b-b399-aea33f6e104e"
+    },
+    {
+        "name": "FUAM_Gateway_Monitoring_From_Files_SM.SemanticModel",
+        "fuam_id": "dff06bbb-5fd2-554b-863d-2cb12e6e94ff"
+    },
+    {
+        "name": "FUAM_Gateway_Monitoring_From_Files_Report.Report",
+        "fuam_id": "0f3125cc-6324-5520-bd53-6fa9f9d5487c"
+    }
+]

monitoring/fabric-unified-admin-monitoring/config/item_config.yaml

@@ -0,0 +1,77 @@
+workspace: 
+  src: 'FUAM Master DEV'
+  trg: 'FUAM Master TEST'
+connections:
+  pbi_connection :
+    src: fuam pbi-service-api admin
+    trg: fuam pbi-service-api admin
+  fabric_connection: 
+    src: fuam fabric-service-api admin
+    trg: fuam fabric-service-api admin
+
+items:
+  - FUAM_Lakehouse.Lakehouse
+  - FUAM_Config_Lakehouse.Lakehouse
+  - FUAM_Staging_Lakehouse.Lakehouse
+
+  - Generate_Calendar_Table.Notebook
+  - Init_FUAM_Lakehouse_Tables.Notebook
+
+  - 01_Transfer_Active_Items_Unit.Notebook
+  - Load_Items_E2E.DataPipeline
+
+  - 02_Transfer_Activities_Unit.Notebook
+  - 03_Aggregate_Activities_Unit.Notebook
+  - Load_Activities_E2E.DataPipeline
+
+  - 01_Transfer_Capacities_Unit.Notebook
+  - Load_Capacities_E2E.DataPipeline
+
+  - 01_Transfer_Capacity_Refreshables_Unit.Notebook
+  - Load_Capacity_Refreshables_E2E.DataPipeline
+
+  - 02_Transfer_Workspaces_Unit.Notebook
+  - Load_PBI_Workspaces_E2E.DataPipeline
+
+  - 01_Transfer_Incremental_Inventory_Unit.Notebook
+  - Load_Inventory_E2E.DataPipeline
+
+  - 01_Transfer_Delegated_Tenant_Settings_Overrides_Unit.Notebook
+  - 01_Transfer_Tenant_Admin_Settings_Unit.Notebook
+  - Load_Tenant_Settings_E2E.DataPipeline
+  - Load_Delegated_Tenant_Settings_Overrides_E2E.DataPipeline
+
+  - 01_Transfer_Git_Connections_Unit.Notebook
+  - Load_Git_Connections_E2E.DataPipeline
+
+  - FUAM_Backup_Lakehouse.Lakehouse
+  - 01_FUAM_Lakehouse_Backup.Notebook
+  - 02_FUAM_Lakehouse_Optimization.Notebook
+  - Maintenance_for_FUAM.DataPipeline
+
+
+  - 01_Transfer_CapacityMetricData_Timepoints_Unit.Notebook
+  - 02_Transfer_CapacityMetricData_ItemKind_Unit.Notebook
+  - 03_Transfer_CapacityMetricData_ItemOperation_Unit.Notebook
+  - Load_Capacity_Metrics_E2E.DataPipeline
+
+  - Load_FUAM_Data_E2E.DataPipeline
+
+  - FUAM_Item_SM.SemanticModel
+  - FUAM_Item_Analyzer_Report.Report
+
+  - FUAM_Core_SM.SemanticModel
+  - FUAM_Core_Report.Report
+
+  - FUAM_Semantic_Model_Meta_Data_Analyzer_SM.SemanticModel
+  - FUAM_Semantic_Model_Meta_Data_Analyzer_Report.Report
+
+  - FUAM_SQL_Endpoint_Analyzer_SM.SemanticModel
+  - FUAM_SQL_Endpoint_Analyzer_Report.Report
+
+  - FUAM_Gateway_Monitoring_From_Files_SM.SemanticModel
+  - FUAM_Gateway_Monitoring_From_Files_Report.Report
+
+fuam_lakehouse_semantic_models:
+  - FUAM_Core_SM.SemanticModel
+  - FUAM_Item_SM.SemanticModel