[FEATURE] Configure default child/nested stack execution order per parent

[ltshb]

Is your feature request related to a problem? Please describe.
Currently child/nested stack are always executed after the parent stack (unless the --reverse option is given). This is fine for most use case, but there is some use case where the opposite would be better. Here below an good example for this reverse order use case

CDN/
   cloudfront_a/
          stack.tm.hcl
          service_aa/
               stack.tm.hcl
               k8s_namespace.tf
          service_bb/
               stack.tm.hcl
               bucket.tf
   cloudfront_b/
          stack.tm.hcl
          service_ba/
              stack.tm.hcl
          service_bb/
              stack.tm.hcl
k8s/
   stack.tm.hcl
   load_balancer.tf

In the example above you have a folder containing Content Delivery Networks like for example AWS Cloudfronts. In each Cloudfront stack you have the services that are routed by CF cache behavior. Some of the services are redirected to a k8s cluster load balancer that leaves outside of the CDN folder, but some services are a simple S3 bucket. In this case Cloudfront needs to know the bucket
domain name and is therefor depending on the service which means that cloudfront nested/child stack should be executed before and not after.

Describe the solution you'd like
It would be good to be able to change the default execution order of nested stack not only by CLI option but by parent stack or even better by a global config on a parent folder. So we could have in the CDN folder above (which is not a stack) a config file containing something like this

terramate {
  orchestration = {
    reverse = true
  }
}

This config would have only effect on the current folder and its sub folders

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[i4ki]

Hi @ltshb

The parent/child nested ordering is an implicit ordering added at runtime. You can think of it as if they add hidden before clause in the parent stack to all of its children stacks.

This implicit filesystem ordering works great in many setups but not in some advanced cases, as reported by some users, so we are planning to disable this in the next release with a configuration flag.

Example:

terramate {
  config {
    order_of_execution {
      nested = false
    }
  }
}

If disabled this way, then you will be free to use after and before clauses to and from child stacks, given that no cycles are introduced, it will work.

We are not sure yet if allowing to reverse the nested ordering just in some branches of the filesystem is a good idea because it can make it very hard to reason about the overall ordering of execution of the whole project, and also it can become hard to change, move or clone stacks to other places, etc.

The config you proposed suggests that this would reverse the entire graph (similarly to --reverse flag), not only the filesystem implicit order, but then it does not make much sense because then all your stack.after and stack.before would have the opposite behavior, which is just confusing.

So, talking specifically about your problem, do you think the disabling of the nested behavior of the execution order would work for you? In this case, you can add a stack.before = [".."] to make a child stack execute before its parent stack.

[soerenmartius]

Hi @ltshb

The parent/child nested ordering is an implicit ordering added at runtime. You can think of it as if they add hidden before clause in the parent stack to all of its children stacks.

This implicit filesystem ordering works great in many setups but not in some advanced cases, as reported by some users, so we are planning to disable this in the next release with a configuration flag.

Example:

terramate {
  config {
    order_of_execution {
      nested = false
    }
  }
}

If disabled this way, then you will be free to use after and before clauses to and from child stacks, given that no cycles are introduced, it will work.

We are not sure yet if allowing to reverse the nested ordering just in some branches of the filesystem is a good idea because it can make it very hard to reason about the overall ordering of execution of the whole project, and also it can become hard to change, move or clone stacks to other places, etc.

The config you proposed suggests that this would reverse the entire graph (similarly to --reverse flag), not only the filesystem implicit order, but then it does not make much sense because then all your stack.after and stack.before would have the opposite behavior, which is just confusing.

So, talking specifically about your problem, do you think the disabling of the nested behavior of the execution order would work for you? In this case, you can add a stack.before = [".."] to make a child stack execute before its parent stack.

do we plan to make this configurable for specific directories and their nested structure or for entire projects only?

[i4ki]

@soerenmartius I think this is yet to be defined.

[i4ki]

@soerenmartius only globally for now, until we get more experience with the feature and hear user's feedbacks. It's easy to extend later on to subtrees only.