Optimized HC-set finalization - changing expected results (#712)

* Initial implementation of building and minimizing fw-rules directly from connectivity properties. Signed-off-by: Tanya <[email protected]> * Fixed lint errors. Signed-off-by: Tanya <[email protected]> * Updating (some of) expected results for explainability queries, according to more condensed optimized output. Signed-off-by: Tanya <[email protected]> * Fixed converting fw-rules to connectivity properties, while taking into account TCP/non-TCP protocol restriction. Signed-off-by: Tanya <[email protected]> * Optimized handling IpBlocks in optimized fw-rules minimization Signed-off-by: Tanya <[email protected]> * Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping. Signed-off-by: Tanya <[email protected]> * Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping. Signed-off-by: Tanya <[email protected]> * More optimization in calculation partial ns grouping. Signed-off-by: Tanya <[email protected]> * Fixed lint error Signed-off-by: Tanya <[email protected]> * Refining basic namespace grouping by finding more opportunities to use properties in containing connections. Signed-off-by: Tanya <[email protected]> * One more refinemenet of basic namespace grouping Signed-off-by: Tanya <[email protected]> * One more refinemenet of basic namespace grouping Signed-off-by: Tanya <[email protected]> * More refinemenets of peer grouping from properties Signed-off-by: Tanya <[email protected]> * More refinemenets of peer grouping from properties Signed-off-by: Tanya <[email protected]> * More refinements of peer grouping from properties Signed-off-by: Tanya <[email protected]> * Added outputEndpoints option handling to PeerSetElement. Refined ns-set pairs grouping computation -trying starting from src_peers and from dst_peers and choosing a more compact grouping. Added grouping by full IpBlock. Signed-off-by: Tanya <[email protected]> * Fixing lint errors. Signed-off-by: Tanya <[email protected]> * Fixing handling txt-no_fw_rules format in the optimized solution Signed-off-by: Tanya <[email protected]> * Fixing lint error Signed-off-by: Tanya <[email protected]> * Fix: taking into account connectivity restriction (TCP/non-TCP) in generation of dot output in optimized solution Signed-off-by: Tanya <[email protected]> * Small fixes in txt_no_fw_rules_format Signed-off-by: Tanya <[email protected]> * Small fixes in txt_no_fw_rules_format Signed-off-by: Tanya <[email protected]> * Added grouping by dns entries to the optimized algorithm. Signed-off-by: Tanya <[email protected]> * Changed expected results of connectivity map query tests according to optimized runs Signed-off-by: Tanya <[email protected]> * Changed optimized semantic diff query implementation according to the optimized fw-rules minimization algorithm. Fixed get_connection_set_and_peers_from_cube. Changed some of the expected results of semantic diff tests. Signed-off-by: Tanya <[email protected]> * Keeping every dns entry separate in minimization of fw rules. Updated more semantic diff expected results. Signed-off-by: Tanya <[email protected]> * Updated containment, permits, forbids expected results. Signed-off-by: Tanya <[email protected]> * Cleaning up unused code and refactoring accordingly. Signed-off-by: Tanya <[email protected]> * Fixed lint error. Signed-off-by: Tanya <[email protected]> * Changed default to be the optimized run. Signed-off-by: Tanya <[email protected]> * In opt='debug' the result explanation should ne according to the optimized run. Signed-off-by: Tanya <[email protected]> * Restoring resource in scheme, changed by mistake. Signed-off-by: Tanya <[email protected]> * Updating more expected results. Signed-off-by: Tanya <[email protected]> * Small optimizations. Signed-off-by: Tanya <[email protected]> * Initial implementation of building and minimizing fw-rules directly from connectivity properties. Signed-off-by: Tanya <[email protected]> * Fixed lint errors. Signed-off-by: Tanya <[email protected]> * Updating (some of) expected results for explainability queries, according to more condensed optimized output. Signed-off-by: Tanya <[email protected]> * Fixed converting fw-rules to connectivity properties, while taking into account TCP/non-TCP protocol restriction. Signed-off-by: Tanya <[email protected]> * Optimized handling IpBlocks in optimized fw-rules minimization Signed-off-by: Tanya <[email protected]> * Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping. Signed-off-by: Tanya <[email protected]> * Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping. Signed-off-by: Tanya <[email protected]> * More optimization in calculation partial ns grouping. Signed-off-by: Tanya <[email protected]> * Fixed lint error Signed-off-by: Tanya <[email protected]> * Refining basic namespace grouping by finding more opportunities to use properties in containing connections. Signed-off-by: Tanya <[email protected]> * One more refinemenet of basic namespace grouping Signed-off-by: Tanya <[email protected]> * One more refinemenet of basic namespace grouping Signed-off-by: Tanya <[email protected]> * More refinemenets of peer grouping from properties Signed-off-by: Tanya <[email protected]> * More refinemenets of peer grouping from properties Signed-off-by: Tanya <[email protected]> * More refinements of peer grouping from properties Signed-off-by: Tanya <[email protected]> * Added outputEndpoints option handling to PeerSetElement. Refined ns-set pairs grouping computation -trying starting from src_peers and from dst_peers and choosing a more compact grouping. Added grouping by full IpBlock. Signed-off-by: Tanya <[email protected]> * Fixing lint errors. Signed-off-by: Tanya <[email protected]> * Fixing handling txt-no_fw_rules format in the optimized solution Signed-off-by: Tanya <[email protected]> * Fixing lint error Signed-off-by: Tanya <[email protected]> * Fix: taking into account connectivity restriction (TCP/non-TCP) in generation of dot output in optimized solution Signed-off-by: Tanya <[email protected]> * Small fixes in txt_no_fw_rules_format Signed-off-by: Tanya <[email protected]> * Small fixes in txt_no_fw_rules_format Signed-off-by: Tanya <[email protected]> * Added grouping by dns entries to the optimized algorithm. Signed-off-by: Tanya <[email protected]> * Changed expected results of connectivity map query tests according to optimized runs Signed-off-by: Tanya <[email protected]> * Changed optimized semantic diff query implementation according to the optimized fw-rules minimization algorithm. Fixed get_connection_set_and_peers_from_cube. Changed some of the expected results of semantic diff tests. Signed-off-by: Tanya <[email protected]> * Keeping every dns entry separate in minimization of fw rules. Updated more semantic diff expected results. Signed-off-by: Tanya <[email protected]> * Updated containment, permits, forbids expected results. Signed-off-by: Tanya <[email protected]> * Cleaning up unused code and refactoring accordingly. Signed-off-by: Tanya <[email protected]> * Fixed lint error. Signed-off-by: Tanya <[email protected]> * Changed default to be the optimized run. Signed-off-by: Tanya <[email protected]> * In opt='debug' the result explanation should ne according to the optimized run. Signed-off-by: Tanya <[email protected]> * Restoring resource in scheme, changed by mistake. Signed-off-by: Tanya <[email protected]> * Updating more expected results. Signed-off-by: Tanya <[email protected]> * Small optimizations. Signed-off-by: Tanya <[email protected]> * Small optimizations. Signed-off-by: Tanya <[email protected]> --------- Signed-off-by: Tanya <[email protected]>
IBM · May 5, 2024 · bc23aca · bc23aca
1 parent 5408921
commit bc23aca
Show file tree

Hide file tree

Showing 153 changed files with 1,074 additions and 5,372 deletions.
diff --git a/nca/CoreDS/CanonicalHyperCubeSet.py b/nca/CoreDS/CanonicalHyperCubeSet.py
@@ -41,7 +41,6 @@ class CanonicalHyperCubeSet:
  def __init__(self, dimensions, allow_all=False):
  self.layers = dict() # layers are w.r.t active dimensions
  self.all_dimensions_list = dimensions # ordered list of all dimensions
- self.all_dim_types = [DimensionsManager().get_dimension_type_by_name(dim_name) for dim_name in dimensions]
  # init ordered list of active dimensions:
  if allow_all:
  self.active_dimensions = [] # names (for non-active dimensions everything is allowed)
@@ -172,8 +171,9 @@ def _get_entire_space_cube(self, dimensions_list_restriction=None):
  dimensions_list_restriction = self.all_dimensions_list
  dimensions_list_ordered = self._get_dimensions_subset_by_order(dimensions_list_restriction)
  cube_res = []
+ dimensions_manager = DimensionsManager()
  for dim_name in dimensions_list_ordered:
- cube_res.append(DimensionsManager().get_dimension_domain_by_name(dim_name, True))
+ cube_res.append(dimensions_manager.get_dimension_domain_by_name(dim_name, True))
  return cube_res
 
  def __len__(self):
@@ -228,7 +228,10 @@ def __contains__(self, item):
  """
  if len(item) < len(self.all_dimensions_list):
  raise Exception("input item len mismatch")
- for index, dim_type in enumerate(self.all_dim_types):
+ dimensions_manager = DimensionsManager()
+ all_dim_types = [dimensions_manager.get_dimension_type_by_name(dim_name)
+ for dim_name in self.all_dimensions_list]
+ for index, dim_type in enumerate(all_dim_types):
  if dim_type == DimensionsManager.DimensionType.DFA:
  assert (isinstance(item[index], str))
  else:
@@ -604,9 +607,10 @@ def get_cube_str(self, cube):
  :return: str representation for cube's values
  """
  res = ""
+ dimensions_manager = DimensionsManager()
  for dim_index, dim_values in enumerate(cube):
  dim_name = self.active_dimensions[dim_index]
- res += DimensionsManager().get_dim_values_str(dim_values, dim_name) + ", "
+ res += dimensions_manager.get_dim_values_str(dim_values, dim_name) + ", "
  return f"({res})"
 
  def _is_last_dimension(self):
@@ -666,11 +670,12 @@ def _get_aligned_cube_by_new_active_dimensions(cube, current_active_dimensions,
  for index, dim_name in enumerate(current_active_dimensions):
  current_active_dimensions_dict[dim_name] = index
  aligned_cube_values = []
+ dimensions_manager = DimensionsManager()
  for active_dim_name in new_active_dimensions:
  if active_dim_name in current_active_dimensions_dict:
  aligned_cube_values.append(cube[current_active_dimensions_dict[active_dim_name]])
  else:
- aligned_cube_values.append(DimensionsManager().get_dimension_domain_by_name(active_dim_name, True))
+ aligned_cube_values.append(dimensions_manager.get_dimension_domain_by_name(active_dim_name, True))
  return aligned_cube_values
 
  def _set_active_dimensions(self, dim_names_set):
@@ -828,8 +833,9 @@ def reduce_active_dimensions(self):
  # reduce by searching for active dimensions on which entire domain is allowed for all the cubes
  dimensions_to_reduce = []
  values_per_dimension = self._get_values_sets_per_active_dimension()
+ dimensions_manager = DimensionsManager()
  for dim_name, values_set in values_per_dimension.items():
- dim_domain = DimensionsManager().get_dimension_domain_by_name(dim_name)
+ dim_domain = dimensions_manager.get_dimension_domain_by_name(dim_name)
  if {dim_domain} == values_set:
  dimensions_to_reduce.append(dim_name)
  dimensions_to_reduce = self._get_dimensions_subset_by_order(dimensions_to_reduce)

diff --git a/nca/CoreDS/ConnectivityCube.py b/nca/CoreDS/ConnectivityCube.py
@@ -29,8 +29,9 @@ def __init__(self, dimensions_list=None):
  self.dimensions_list = dimensions_list if dimensions_list else self.all_dimensions_list
  self.named_ports = set() # used only in the original solution
  self.excluded_named_ports = set() # used only in the original solution
+ dimensions_manager = DimensionsManager()
  for dim in self.dimensions_list:
- dim_value = DimensionsManager().get_dimension_domain_by_name(dim, True)
+ dim_value = dimensions_manager.get_dimension_domain_by_name(dim, True)
  self.set_dim_directly(dim, dim_value)
 
  def copy(self):
@@ -46,17 +47,6 @@ def copy(self):
  res.set_dim_directly(dim_name, dim_value.copy())
  return res
 
- def is_empty_dim(self, dim_name):
- """
- Returns True iff a given dimension is empty
- :param str dim_name: the given dimension name
- """
- if self.get_dim_directly(dim_name) != DimensionsManager().get_empty_dimension_by_name(dim_name):
- return False
-
- # for "dst_ports" can have named ports in original solution
- return not self.named_ports and not self.excluded_named_ports if dim_name == "dst_ports" else True
-
  def is_full_dim(self, dim_name):
  """
  Returns True iff a given dimension is full
@@ -171,18 +161,22 @@ def has_active_dim(self):
  """
  Returns True iff the cube has at least one active dimension. Otherwise, returns False.
  """
+ dimensions_manager = DimensionsManager()
  for dim in self.dimensions_list:
- if self.get_dim_directly(dim) != DimensionsManager().get_dimension_domain_by_name(dim):
+ if self.get_dim_directly(dim) != dimensions_manager.get_dimension_domain_by_name(dim):
  return True
  return False
 
  def is_empty(self):
  """
  Returns True iff the cube has at least one empty dimension. Otherwise, returns False.
  """
+ dimensions_manger = DimensionsManager()
  for dim in self.dimensions_list:
- if self.is_empty_dim(dim):
- return True
+ if self.get_dim_directly(dim) == dimensions_manger.get_empty_dimension_by_name(dim):
+ # for "dst_ports" can have named ports in original solution
+ if dim != "dst_ports" or (not self.named_ports and not self.excluded_named_ports):
+ return True
  return False
 
  def get_ordered_cube_and_active_dims(self):
@@ -192,10 +186,11 @@ def get_ordered_cube_and_active_dims(self):
  """
  cube = []
  active_dims = []
+ dimensions_manager = DimensionsManager()
  # add values to cube by required order of dimensions
  for dim in self.dimensions_list:
  dim_value = self.get_dim_directly(dim)
- if dim_value != DimensionsManager().get_dimension_domain_by_name(dim):
+ if dim_value != dimensions_manager.get_dimension_domain_by_name(dim):
  if isinstance(dim_value, MinDFA):
  cube.append(dim_value)
  else:

diff --git a/nca/CoreDS/ConnectivityProperties.py b/nca/CoreDS/ConnectivityProperties.py
@@ -149,10 +149,11 @@ def get_cube_dict(self, cube, is_txt=False):
  :rtype: dict
  """
  cube_dict = {}
+ dimensions_manager = DimensionsManager()
  for i, dim in enumerate(self.active_dimensions):
  dim_values = cube[i]
- dim_type = DimensionsManager().get_dimension_type_by_name(dim)
- dim_domain = DimensionsManager().get_dimension_domain_by_name(dim)
+ dim_type = dimensions_manager.get_dimension_type_by_name(dim)
+ dim_domain = dimensions_manager.get_dimension_domain_by_name(dim)
  if dim_domain == dim_values:
  continue # skip dimensions with all values allowed in a cube
  if dim in ['protocols', 'methods']:
@@ -167,7 +168,7 @@ def get_cube_dict(self, cube, is_txt=False):
  values_list = ','.join(str(interval) for interval in values_list)
  else:
  # TODO: should be a list of words for a finite len DFA?
- values_list = DimensionsManager().get_dim_values_str(dim_values, dim)
+ values_list = dimensions_manager.get_dim_values_str(dim_values, dim)
  cube_dict[dim] = values_list
  return cube_dict
 
@@ -452,9 +453,9 @@ def get_all_conns_props_per_domain_peers():
  This is a compact way to represent all peers connections, but it is an over-approximation also containing
  IpBlock->IpBlock connections. Those redundant connections will be eventually filtered out.
  """
- src_peers = BasePeerSet().get_peer_set_by_indices(DimensionsManager().get_dimension_domain_by_name("src_peers"))
- dst_peers = BasePeerSet().get_peer_set_by_indices(DimensionsManager().get_dimension_domain_by_name("dst_peers"))
- return ConnectivityProperties.make_conn_props_from_dict({"src_peers": src_peers, "dst_peers": dst_peers})
+ # optimization: src_peers and dst_peers have the same domain
+ peers = BasePeerSet().get_peer_set_by_indices(DimensionsManager().get_dimension_domain_by_name("src_peers"))
+ return ConnectivityProperties.make_conn_props_from_dict({"src_peers": peers, "dst_peers": peers})
 
  @staticmethod
  def make_empty_props():

diff --git a/nca/FWRules/MinimizeBasic.py b/nca/FWRules/MinimizeBasic.py
@@ -108,25 +108,30 @@ def _get_pods_grouping_by_labels(self, pods_set, ns, extra_pods_set):
  @staticmethod
  def get_connection_set_and_peers_from_cube(the_cube, peer_container,
  relevant_protocols=ProtocolSet(True)):
+ all_peers = peer_container.get_all_peers_group(True)
  conn_cube = the_cube.copy()
- src_peers = conn_cube["src_peers"] or peer_container.get_all_peers_group(True)
+ src_peers = conn_cube["src_peers"] or all_peers
  conn_cube.unset_dim("src_peers")
- dst_peers = conn_cube["dst_peers"] or peer_container.get_all_peers_group(True)
+ dst_peers = conn_cube["dst_peers"] or all_peers
  conn_cube.unset_dim("dst_peers")
  protocols = conn_cube["protocols"]
  conn_cube.unset_dim("protocols")
- if not conn_cube.has_active_dim() and (protocols.is_whole_range() or protocols == relevant_protocols):
+ has_active_dim = conn_cube.has_active_dim()
+ if not has_active_dim and (protocols == relevant_protocols or protocols.is_whole_range()):
  conns = ConnectionSet(True)
  else:
  conns = ConnectionSet()
  protocol_names = ProtocolSet.get_protocol_names_from_interval_set(protocols)
+ if has_active_dim:
+ props = ConnectivityProperties.make_conn_props(conn_cube)
+ else:
+ props = ConnectivityProperties.make_all_props()
  for protocol in protocol_names:
- if conn_cube.has_active_dim():
- conns.add_connections(protocol, ConnectivityProperties.make_conn_props(conn_cube))
+ if has_active_dim:
+ conns.add_connections(protocol, props)
  else:
  if ConnectionSet.protocol_supports_ports(protocol) or ConnectionSet.protocol_is_icmp(protocol):
- conns.add_connections(protocol,
- ConnectivityProperties.get_all_conns_props_per_config_peers(peer_container))
+ conns.add_connections(protocol, props)
  else:
  conns.add_connections(protocol, True)
  return conns, src_peers, dst_peers

diff --git a/nca/FWRules/MinimizeCsFWRulesOpt.py b/nca/FWRules/MinimizeCsFWRulesOpt.py
@@ -41,7 +41,7 @@ def compute_minimized_fw_rules_per_connection(self, connections, peer_props,
  """
  The main function for creating the minimized set of fw-rules for a given connection set
 
- :param connections: the allowed connections for the given peer pairs, of type ConnectionSet
+ :param ConnectionSet connections: the allowed connections for the given peer pairs, of type ConnectionSet
  :param ConnectivityProperties peer_props: peers (src,dst) for which communication is allowed over the given connections
  :param ConnectivityProperties peer_props_in_containing_connections: peers in connections that contain the current
  connection set
@@ -193,14 +193,13 @@ def _compute_partial_ns_grouping(self, ns_set, is_src_ns):
  dim_peers = conn_cube[dim_name]
  other_dim_peers = conn_cube[other_dim_name].canonical_form()
  curr_ns_set = set()
- curr_ns_peers = PeerSet()
  for ns in ns_set:
  ns_peers = PeerSet(self.cluster_info.ns_dict[ns])
- curr_covered = ConnectivityProperties.make_conn_props_from_dict({dim_name: ns_peers,
-  other_dim_name: other_dim_peers})
- if ns_peers.issubset(dim_peers) and (curr_covered & self.peer_props_without_ns_expr):
- curr_ns_set.add(ns)
- curr_ns_peers |= ns_peers
+ if ns_peers.issubset(dim_peers):
+ curr_covered = ConnectivityProperties.make_conn_props_from_dict({dim_name: ns_peers,
+  other_dim_name: other_dim_peers})
+ if curr_covered & self.peer_props_without_ns_expr:
+  curr_ns_set.add(ns)
  if curr_ns_set:
  ns_set_to_peer_set[frozenset(curr_ns_set)] |= other_dim_peers
  for curr_ns_set, other_dim_peers in ns_set_to_peer_set.items():
@@ -216,12 +215,14 @@ def _compute_partial_ns_grouping(self, ns_set, is_src_ns):
  # ensure that the found pairs (with and without IpBlocks) are at least partially included
  # in the current connections' properties (rather than being wholly contained
  # in containing connections' properties)
- if self.peer_props_without_ns_expr & curr_covered_without_ip_block:
- self.peer_props_without_ns_expr -= curr_covered_without_ip_block
+ peer_props_without_ns_expr_updated = self.peer_props_without_ns_expr - curr_covered_without_ip_block
+ if self.peer_props_without_ns_expr != peer_props_without_ns_expr_updated:
+ self.peer_props_without_ns_expr = peer_props_without_ns_expr_updated
  self.base_elem_pairs.add((curr_ns_set, other_dim_peers_without_ip_block) if is_src_ns
  else (other_dim_peers_without_ip_block, curr_ns_set))
- if self.peer_props_without_ns_expr & curr_covered_ip_block:
- self.peer_props_without_ns_expr -= curr_covered_ip_block
+ peer_props_without_ns_expr_updated = self.peer_props_without_ns_expr - curr_covered_ip_block
+ if self.peer_props_without_ns_expr != peer_props_without_ns_expr_updated:
+ self.peer_props_without_ns_expr = peer_props_without_ns_expr_updated
  self.base_elem_pairs.add((curr_ns_set, other_dim_peers_ip_block) if is_src_ns
  else (other_dim_peers_ip_block, curr_ns_set))
 
@@ -249,8 +250,8 @@ def _compute_full_ipblock_and_dns_grouping(self, is_src_ns):
  self._add_to_map_if_covered(dim_name, ipblock.get_peer_set(), other_dim_name, other_dim_peers,
  ipblock_dnsentry_to_peer_set)
  dns_entries = dim_peers.get_dns_entries()
- if dns_entries:
- self._add_to_map_if_covered(dim_name, dns_entries, other_dim_name, other_dim_peers,
+ for dns_entry in dns_entries:
+ self._add_to_map_if_covered(dim_name, PeerSet({dns_entry}), other_dim_name, other_dim_peers,
  ipblock_dnsentry_to_peer_set)
  for curr_peers, other_dim_peers in ipblock_dnsentry_to_peer_set.items():
  curr_peers = PeerSet(set(curr_peers)) # peel off the frozenset