Support for Ecosystem Node Type and Define a flow section within core.json

[LeighFinegold]

Feature Request

The current implementation of the architecture-as-code framework expects all nodes to be defined within a single document. However, in enterprise systems, ecosystems are composed of multiple interconnected systems, and business flows often span these systems. To better represent real-world scenarios and facilitate enterprise-wide architectural documentation, I propose two enhancements:

Enhancement 1: Introduce a Node Type for Ecosystem

A new node type, eco-system, should be added to represent a collection of related systems within a larger ecosystem. This will help organize and abstract complex architecture into manageable layers, aligning better with enterprise structures.

At a larger scale, Traderx could be considered as an ecosystem consisting of a Trading System, Ref Data System, Trade System, and Position System.
Example -

Each can be defined as a system node and grouped under an overarching ecosystem node.

Enhancement 2: Define Flows Within Core.json:

As part of #426, The ability to define business flows was added. There was an expectation that eventually something would bundle flow definitions with node architectures so that node references could be derived. The proposal here is actually to define the flows section in core.json such that there is a direct link to everything that spans the ecosystem.

Benefits

1. Enhanced Documentation: The ecosystem and flow enhancements would enable more intuitive and comprehensive documentation for enterprise systems.
2. Improved Traversal with CALM-Hub: With the planned rollout of CALM-Hub, these features will enable automatic traversal to bundle related CALM documents for building documentation websites.
3. Scalability: Allows for modular and distributed/FrontToBack documentation, reflecting the distributed nature of modern enterprise systems.

Alternative approaches

The flow.json spec is enhanced such that at the top of the flow document every architecture spec (core.json) that contains all referenced nodes is specified.

[rocketstack-matt]

Enhancement 1: Introduce a Node Type for Ecosystem
tl;dr I agree with the proposal to add an ecosystem node-type,

However, I don't agree that current implementation expects all nodes to be defined in a single document.

You could already have a document that shows

{
  "$schema": "https://calm.finos.org/draft/2024-10/meta/calm.json",
  "nodes": [
    {
      "unique-id": "traderx-trading-system",
      "node-type": "system",
      "name": "Trading System",
      "description": "Simple Trading System",
      "detailed-architecture": "https://traderx.finos.org/trading-system"
    },
    {
      "unique-id": "traderx-refdata-system",
      "node-type": "system",
      "name": "Ref Data System",
      "description": "Account & Product Reference Data System",
      "detailed-architecture": "https://traderx.finos.org/ref-data-system"
    },
    {
      "unique-id": "traderx-trade-system",
      "node-type": "system",
      "name": "Trade System",
      "description": "Trade Booking System",
      "detailed-architecture": "https://traderx.finos.org/trade-system"
    },
    {
      "unique-id": "traderx-position-system",
      "node-type": "system",
      "name": "Position System",
      "description": "POsition Keeper",
      "detailed-architecture": "https://traderx.finos.org/position-system"
    }
  ]
}

This is in effect the ecosystem you're talking about, the individual services would then be represented in the additional documents. All that being said, I think for enterprises being able to have a node show all the different ecosystems is in itself useful.

Enhancement 2: Define Flows Within Core.json:
I assume you don't mean move the flows definition into core, rather add a new optional collection to node which would reference the definition from flows.json - the same way we do it with interfaces and controls?

[LeighFinegold]

Thanks Matt. Yes that example is what I was thinking. Maybe with an additional node-type we would have that with relationship.

{
  "$schema": "https://calm.finos.org/draft/2024-10/meta/calm.json",
  "nodes": [
   {
      "unique-id": "traderx-ecosystem",
      "node-type": "ecosystem",
      "name": "Trading EcoSystem",
      "description": "TraderX Ecosystem"
    }
    {
      "unique-id": "traderx-trade-system",
      "node-type": "system",
      "name": "Trade System",
      "description": "Simple Trade System",
      "detailed-architecture": "https://traderx.finos.org/trading-system"
    },
    {
      "unique-id": "traderx-refdata-system",
      "node-type": "system",
      "name": "Ref Data System",
      "description": "Account & Product Reference Data System",
      "detailed-architecture": "https://traderx.finos.org/ref-data-system"
    },
    {
      "unique-id": "traderx-trade-system",
      "node-type": "system",
      "name": "Trade System",
      "description": "Trade Booking System",
      "detailed-architecture": "https://traderx.finos.org/trade-system"
    },
    {
      "unique-id": "traderx-position-system",
      "node-type": "system",
      "name": "Position System",
      "description": "Position Keeper",
      "detailed-architecture": "https://traderx.finos.org/position-system"
    }
  ],
  "relationships": [
  {
      "unique-id": "traderx-ecosystem-is-composed-of",
      "relationship-type": {
        "composed-of": {
          "container": "traderx-ecosystem",
          "nodes": [
            "traderx-position-system",
            "traderx-trade-system",
            "traderx-refdata-system",
            "traderx-trading-system"           
          ]
        }
      }
    },
  
  ]
}

In relation to enhancement 2, I did mean flows referenced in core at same level as nodes, controls etc.

Thinking about this some more. Flows actually reference relationships from core atm. The relationships are the ones that reference the nodes. Do we leverage detailed-architecture on relationships today?

What happens if I need to reference a relationship between services of nodes accross systems. Where is that relationship defined?

If the trading system is this:-

{
  "$schema": "https://calm.finos.org/draft/2024-10/meta/calm.json",
  "nodes": [
    {
      "unique-id": "traderx-trading-system",
      "node-type": "system",
      "name": "Trade System",
      "description": "Simple Trade System"
    },
   {
      "unique-id": "web-gui",
      "node-type": "service",
      "name": "Web GUI",
      "description": "Allows employees to manage accounts and book trades",
      "data-classification": "Confidential",
      "run-as": "systemId"
    }
  ],
  "relationships": [
  {
      "unique-id": "traderx-trading-system-is-composed-of",
      "relationship-type": {
        "composed-of": {
          "container": "traderx-trading-system",
          "nodes": [
            "web-gui" 
          ]
        }
      }
    }  
  ]
}

And this for reference data system

{
  "$schema": "https://calm.finos.org/draft/2024-10/meta/calm.json",
  "nodes": [
    {
      "unique-id": "traderx-refdata-system",
      "node-type": "system",
      "name": "Refdata System",
      "description": "Simple Refdata System",
    },
   {
      "unique-id": "accounts-service",
      "node-type": "service",
      "name": "Accounts Service",
      "description": "Allows for retrieval of account information",
      "data-classification": "Confidential",
      "run-as": "systemId"
    }
  ],
  "relationships": [
  {
      "unique-id": "traderx-refdata-system-is-composed-of",
      "relationship-type": {
        "composed-of": {
          "container": "traderx-refdata-system",
          "nodes": [
            "accounts-service" 
          ]
        }
      }
    }  
  ]
}

Do we define it at the ecosystem level only? Or on each system level calm document, do we specify the connecting node and then reference it via detailed architecture.

{
  "$schema": "https://calm.finos.org/draft/2024-10/meta/calm.json",
  "nodes": [
    {
      "unique-id": "traderx-refdata-system",
      "node-type": "system",
      "name": "Refdata System",
      "description": "Simple Refdata System",
    },
   {
      "unique-id": "accounts-service",
      "node-type": "service",
      "name": "Accounts Service",
      "description": "Allows for retrieval of account information",
      "data-classification": "Confidential",
      "run-as": "systemId"
    }
   {
      "unique-id": "web-gui",
      "node-type": "service",
      "detailed-architecture": "https://traderx.finos.org/trading-system/nodes/web-gui"

    }
  ],
  "relationships": [
  {
      "unique-id": "traderx-refdata-system-is-composed-of",
      "relationship-type": {
        "composed-of": {
          "container": "traderx-refdata-system",
          "nodes": [
            "accounts-service" 
          ]
        }
      }
    },
  {
      "unique-id": "web-gui-uses-account-service",
      "description": "Looks up account service",
      "relationship-type": {
        "connects": {
          "source": {
            "node": "web-gui"
          },
          "destination": {
            "node": "account-service"
          }
        }
      },
      "protocol": "HTTPS"
    },
  ]
}

Or should we just reference it on relationship

{
      "unique-id": "web-gui-uses-account-service",
      "description": "Looks up account service",
      "relationship-type": {
        "connects": {
          "source": {
            "node": "web-gui"
          "detailed-architecture": "https://traderx.finos.org/trading-system/nodes/web-gui"
          },
          "destination": {
            "node": "account-service"
          }
        }
      },
      "protocol": "HTTPS"
}

[rocketstack-matt]

I think this can work neatly if we put the flows on nodes, even though flows are composed of relationships . . . .

Let's assume the following hierarchy of node types:

ecosystem
-- system
---- service
---- db
---- etc.

Now higher level nodes have a more detailed architecture, either explicitly, where they have an actual URL reference to another doc, or implicitly, where we should assume that everything they consist of is in the same document, so in the case of your example above the traders-ecosystem has implicit detailed architecture (it's all in the same doc) whilst the systems it is composed of have explicit detailed architecture.

Now, if the ecosystem node contained a collection of flows, those flows should be able to reference any relationships defined between the system nodes, whilst the system nodes, in their detailed architecture docs would be able to reference relationships they declare between their services, dbs, etc.