Add a page to discuss potential consequences (risks?) of interactions

[lukasvermeer]

Consequences of interactions
Interactions between two experiments can have one of two detrimental effects. In the realm of product development, we (usually) care much more about inference and decision errors than we case about bias.

Interactions can introduce bias in the measurement
Because interactions can affect the estimate of the effects of both experiments, they might cause us to overestimate or underestimate these effects. We might think an experiment has a 5% lift, when in fact it only has a 4% lift which is biased upwards by an interaction effect with another experiment.

Interactions can cause inference and decision errors
In the case that interaction effects are severe, they might cause us to conclude that the effect of one experiment is negative when in fact it would be positive were it not for a drastically negative interaction effect. These sorts of directional errors would not only cause us to overestimate or underestimate the size of the effect, but also cause us to make wrong inference (e.g. we conclude the treatment is bad when it is good) and wrong decisions (e.g. we stop the test when we should have rolled it out).

These types of errors are much worse from a product development point of view. Luckily they are more rare, and more easy to detect, because much more drastic interaction effects are required for such a decision reversal.