Merge pull request #33 from opennetworkinglab/rework-l2-briding-exercise

Rework l2 bridging exercise

EXERCISE-1.md

@@ -1,10 +1,10 @@
-# Exercise 1 - P4Runtime basics
+# Exercise 1: P4Runtime basics
 
 This exercise provides a hands-on introduction to the P4Runtime API. You will be
 asked to:
 
 1. Take a look at the starter P4 code
-2. Compile it for the BMv2 software switch and understand the output (the P4Info 
+2. Compile it for the BMv2 software switch and understand the output (the P4Info
    and BMv2 JSON files)
 3. Start Mininet with a 2x2 topology of `stratum_bmv2` switches
 4. Use the P4Runtime Shell to manually insert table entries in one of the
@@ -137,29 +137,29 @@ You can ignore the ONOS one for now, we will use that in exercise 3 and 4.
 To make sure the container is started without errors, you can use the `make
 mn-log` command to show the Mininet log. Verify that you see the following
 output (press Ctrl-C to exit):
- 
+
 ```
-$ make mn-log 
+$ make mn-log
 docker-compose logs -f mininet
 Attaching to mininet
 mininet    | *** Error setting resource limits. Mininet's performance may be affected.
 mininet    | *** Creating network
 mininet    | *** Adding hosts:
-mininet    | h1a h1b h1c h2 h3 h4 
+mininet    | h1a h1b h1c h2 h3 h4
 mininet    | *** Adding switches:
-mininet    | leaf1 leaf2 spine1 spine2 
+mininet    | leaf1 leaf2 spine1 spine2
 mininet    | *** Adding links:
-mininet    | (h1a, leaf1) (h1b, leaf1) (h1c, leaf1) (h2, leaf1) (h3, leaf2) (h4, leaf2) (spine1, leaf1) (spine1, leaf2) (spine2, leaf1) (spine2, leaf2) 
+mininet    | (h1a, leaf1) (h1b, leaf1) (h1c, leaf1) (h2, leaf1) (h3, leaf2) (h4, leaf2) (spine1, leaf1) (spine1, leaf2) (spine2, leaf1) (spine2, leaf2)
 mininet    | *** Configuring hosts
-mininet    | h1a h1b h1c h2 h3 h4 
+mininet    | h1a h1b h1c h2 h3 h4
 mininet    | *** Starting controller
-mininet    | 
+mininet    |
 mininet    | *** Starting 4 switches
 mininet    | leaf1 stratum_bmv2 @ 50001
 mininet    | leaf2 stratum_bmv2 @ 50002
 mininet    | spine1 stratum_bmv2 @ 50003
 mininet    | spine2 stratum_bmv2 @ 50004
-mininet    | 
+mininet    |
 mininet    | *** Starting CLI:
 ```
 
@@ -235,7 +235,7 @@ If the shell started successfully, you should see the following output:
 ```
 *** Connecting to P4Runtime server at host.docker.internal:50001 ...
 *** Welcome to the IPython shell for P4Runtime ***
-P4Runtime sh >>>    
+P4Runtime sh >>>
 ```
 
 #### Available commands
@@ -276,7 +276,7 @@ You should see the following output:
 ```
 *** Attaching to Mininet CLI...
 *** To detach press Ctrl-D (Mininet will keep running)
-mininet> 
+mininet>
 ```
 
 ### Insert static NDP entries

EXERCISE-2.md

@@ -1,4 +1,4 @@
-# Exercise 2 - Yang, OpenConfig, and gNMI basics
+# Exercise 2: Yang, OpenConfig, and gNMI basics
 
 This set of exercises is designed to give you more exposure to YANG, OpenConfig,
 and gNMI.

EXERCISE-3.md

@@ -1,4 +1,4 @@
-# Exercise 3 - Using ONOS as the control plane
+# Exercise 3: Using ONOS as the control plane
 
 This exercise provides a hands-on introduction to ONOS where you will learn how
 to:
@@ -38,7 +38,7 @@ Requesting ONOS to pre-load the following built-in apps:
 * `gui`: ONOS web user interface (available at <http://localhost:8181/onos/ui>)
 * `drivers.bmv2`: BMv2/Stratum drivers based on P4Runtime, gNMI, and gNOI
 * `lldpprovider`: LLDP-based link discovery application (used in Exercise 4)
-* `hostprovider`: Host discovery application (used in Exercise 5)
+* `hostprovider`: Host discovery application (used in Exercise 4)
 
 
 Once ONOS has started, you can check its log using the `make onos-log` command.

EXERCISE-4.md

@@ -1,6 +1,8 @@
-# Exercise 4: Enabling link discovery via P4Runtime packet I/O
+# Exercise 4: Enabling ONOS built-in services
 
-In this exercise, you will be asked to integrate the ONOS built-in link
+In this exercise, you will use P4Runtime packet I/O to implement discovery
+of dynamic network components like links between switches or the
+presence of hosts. For this you have to integrate the ONOS built-in link
 discovery service with your P4 program. ONOS performs link discovery by using
 controller packet-in/out. To make this work, you will need to apply simple
 changes to the starter P4 code, validate the P4 changes using PTF-based data
@@ -232,7 +234,7 @@ bar. If you are using the tutorial VM, open up a browser (e.g. Firefox) to
 
 On the same page where the ONOS topology view is shown:
 * Press `L` to show device labels;
-* Press `A` multiple times until you see link stats, in either 
+* Press `A` multiple times until you see link stats, in either
   packets/seconds (pps) or bits/seconds.
 
 Link stats are derived by ONOS by periodically obtaining the port counters for
@@ -242,6 +244,177 @@ counters.
 In this case, you should see approx 1 packet/s, as that's the rate of
 packet-outs generated by the `lldpprovider` app.
 
+## Host discovery & L2 bridging
+
+By fixing Packet I/O in the pipeline interpreter we did not only get link
+discovery, but also enabled the built-in `hostprovider` app to perform *host*
+discovery. The controller snoops incoming ARP/NDP packets on the switch and
+deduces where a host is connected to from the packets (meta) data. With the
+knowledge about the topology it can then insert L2 unicast rules to enable
+connectivity.
+
+### Overview
+
+The ONOS app assumes that hosts of a given subnet are all connected to the same
+leaf, and two interfaces of two different leaves cannot be configured with the
+same IPv6 subnet. In other words, L2 bridging is allowed only for hosts
+connected to the same leaf.
+
+The Mininet script [topo.py](mininet/topo.py) used in this tutorial defines 4
+subnets:
+
+* `2001:1:1::/64` with 3 hosts connected to `leaf1` (`h1a`, `h1b`, and `h1c`)
+* `2001:1:2::/64` with 1 hosts connected to `leaf1` (`h2`)
+* `2001:2:3::/64` with 1 hosts connected to `leaf2` (`h3`)
+* `2001:2:4::/64` with 1 hosts connected to `leaf2` (`h4`)
+
+The same IPv6 prefixes are defined in the [netcfg.json](netcfg.json) file and
+are used to provide interface configuration to ONOS.
+
+### Our P4 implementation of L2 bridging
+
+The P4 code defines tables to forward packets based on the Ethernet address,
+precisely, two distinct tables, to handle two different types of L2 entries:
+
+1. Unicast entries: which will be filled in by the control plane when the
+   location (port) of new hosts is learned.
+2. Broadcast/multicast entries: used replicate NDP Neighbor Solicitation
+   (NS) messages to all host-facing ports;
+
+For (2), unlike ARP messages in IPv4, which are broadcasted to Ethernet
+destination address FF:FF:FF:FF:FF:FF, NDP messages are sent to special Ethernet
+addresses specified by RFC2464. These addresses are prefixed with 33:33 and the
+last four octets are the last four octets of the IPv6 destination multicast
+address. The most straightforward way of matching on such IPv6
+broadcast/multicast packets, without digging in the details of RFC2464, is to
+use a ternary match on `33:33:**:**:**:**`, where `*` means "don't care".
+
+For this reason, our solution defines two tables. One that matches in an exact
+fashion `l2_exact_table` (easier to scale on switch ASIC memory) and one that
+uses ternary matching `l2_ternary_table` (which requires more expensive TCAM
+memories, usually much smaller).
+
+These tables are applied to packets in an order defined in the `apply` block
+of the ingress pipeline (`IngressPipeImpl`):
+
+```
+if (!l2_exact_table.apply().hit) {
+    l2_ternary_table.apply();
+}
+```
+
+The ternary table has lower priority and it's applied only if a matching entry
+is not found in the exact one.
+
+**Note**: To keep things simple, we won't be using VLANs to segment our L2
+domains. As such, when matching packets in the `l2_ternary_table`, these will be
+broadcasted to ALL host-facing ports.
+
+### ONOS L2BridgingComponent
+
+We already provide an ONOS app that controls the L2 bridging parts of the P4
+program. The source code of it is located here: `app/src/main/java/org/p4/p4d2/tutorial/L2BridgingComponent.java`
+
+This app component defines two event listener located at the bottom of the
+`L2BridgingComponent` class, `InternalDeviceListener` for device events (e.g.
+connection of a new switch) and `InternalHostListener` for host events (e.g. new
+host discovered). These listeners in turn call methods like:
+
+* `setUpDevice()`: responsible for creating a multicast group for all host-facing
+  ports and inserting flow rules to broadcast/multicast packets such as ARP and
+  NDP messages;
+
+* `learnHost()`: responsible for inserting unicast L2 entries based on the
+  discovered host location.
+
+To support reloading the app implementation, these methods are also called at
+component activation for all devices and hosts known by ONOS at the time of
+activation (look for methods `activate()` and `setUpAllDevices()`).
+
+To keep things simple, our broadcast domain will be restricted to a single
+device, i.e. we allow packet replication only for ports of the same leaf switch.
+As such, we can exclude ports going to the spines from the multicast group. To
+determine whether a port is expected to be facing hosts or not, we look at the
+interface configuration in [netcfg.json](netcfg.json) file (look for the `ports`
+section of the JSON file).
+
+### Examine flow rules and groups
+
+Check the ONOS flow rules, you should now see flow rules installed by our app.
+For example, to show all flow rules installed so far on device `leaf1`:
+
+```
+onos> flows -s any device:leaf1
+deviceId=device:leaf1, flowRuleCount=...
+    ADDED, bytes=0, packets=0, table=IngressPipeImpl.l2_exact_table, priority=10, selector=[hdr.ethernet.dst_addr=0xbb00000001], treatment=[immediate=[IngressPipeImpl.set_egress_port(port_num=0x1)]]
+    ...
+    ADDED, bytes=0, packets=0, table=IngressPipeImpl.l2_ternary_table, priority=10, selector=[hdr.ethernet.dst_addr=0x333300000000&&&0xffff00000000], treatment=[immediate=[IngressPipeImpl.set_multicast_group(gid=0xff)]]
+    ADDED, bytes=3596, packets=29, table=IngressPipeImpl.l2_ternary_table, priority=10, selector=[hdr.ethernet.dst_addr=0xffffffffffff&&&0xffffffffffff], treatment=[immediate=[IngressPipeImpl.set_multicast_group(gid=0xff)]]
+    ...
+```
+
+To show also the groups installed so far, you can use the `groups` command. For
+example to show groups on `leaf1`:
+```
+onos> groups any device:leaf1
+deviceId=device:leaf1, groupCount=2
+   id=0x63, state=ADDED, type=CLONE, bytes=0, packets=0, appId=org.onosproject.core, referenceCount=0
+       id=0x63, bucket=1, bytes=0, packets=0, weight=-1, actions=[OUTPUT:CONTROLLER]
+   id=0xff, state=ADDED, type=ALL, bytes=0, packets=0, appId=org.onosproject.ngsdn-tutorial, referenceCount=0
+       id=0xff, bucket=1, bytes=0, packets=0, weight=-1, actions=[OUTPUT:3]
+       id=0xff, bucket=2, bytes=0, packets=0, weight=-1, actions=[OUTPUT:4]
+       id=0xff, bucket=3, bytes=0, packets=0, weight=-1, actions=[OUTPUT:5]
+       id=0xff, bucket=4, bytes=0, packets=0, weight=-1, actions=[OUTPUT:6]
+```
+
+The `CLONE` group is the same introduced in Exercise 3, which maps to P4Runtime
+`CloneSessionEntry` and it's used to clone packets to the controller via
+packet-in.
+
+The `ALL` group is a new one, created by our app (`appId=org.onosproject.ngsdn-tutorial`).
+Groups of type `ALL` in ONOS map to P4Runtime `MulticastGroupEntry`, in this
+case used to broadcast NDP NS packets to all host-facing ports. This group is
+installed by `L2BridgingComponent.java`, and is used by an entry in the P4
+`l2_ternary_table` (look for flow rule with
+`treatment=[immediate=[IngressPipeImpl.set_multicast_group(gid=0xff)]`)
+
+### Test L2 bridging on Mininet
+
+To verify that L2 bridging works as intended, send a ping between hosts in the
+same subnet:
+
+```
+mininet> h1a ping h1b
+PING 2001:1:1::b(2001:1:1::b) 56 data bytes
+64 bytes from 2001:1:1::b: icmp_seq=2 ttl=64 time=0.580 ms
+64 bytes from 2001:1:1::b: icmp_seq=3 ttl=64 time=0.483 ms
+64 bytes from 2001:1:1::b: icmp_seq=4 ttl=64 time=0.484 ms
+...
+```
+
+Differently from exercise 1, here we have NOT set any NDP static entry.
+Instead, NDP NS and NA packets are handled by the data plane thanks to the ALL
+group and `l2_ternary_table`'s flow rule described above. Moreover, given the
+ACL flow rules to clone NDP packets to the controller, hosts can be discovered
+by ONOS. Host discovery events are used by `L2BridgingComponent.java` to insert
+entries in the P4 `l2_exact_table` to enable forwarding between hosts in the
+same subnet. Check the ONOS log, you should see messages related to the
+discovery of host `h1b` who is now receiving NDP NS messages from `h1a` and
+replying with NDP NA ones to them:
+
+```
+INFO  [L2BridgingComponent] HOST_ADDED event! host=00:00:00:00:00:1B/None, deviceId=device:leaf1, port=4
+INFO  [L2BridgingComponent] Adding L2 unicast rule on device:leaf1 for host 00:00:00:00:00:1B/None (port 4)...
+```
+
+### Visualize hosts on the ONOS web UI
+
+Using the ONF Cloud Tutorial Portal, click on the "ONOS UI" button in the top
+bar. If you are using the tutorial VM, open up a browser (e.g. Firefox) to
+<http://127.0.0.1:8181/onos/ui>.
+
+To toggle showing hosts on the topology view, press `H` on your keyboard.
+
 ## Congratulations!
 
 You have completed the fourth exercise!