rfnoc_radio_loopback with rx and tx from same block gives error: read property `scaling@OUTPUT_EDGE:0'

[TSlivede]

Issue Description

rfnoc_radio_loopback --rx-blockid '0/Radio#1' --tx-blockid '0/Radio#1' --duration 2

doesn't work and prints error:

[ERROR] [RFNOC::GRAPH::DETAIL] Adding edge 0/DDC#1:0 -> 0/DUC#1:0 without disabling property_propagation_active will lead to unresolvable graph!
Error: RfnocError: Adding edge without disabling property_propagation_active will lead to unresolvable graph!

If rx-blockid and tx-blockid are different, it works as expected (My real usecase is not the example. I am using all 4 rx-channels. The example just seems like the easiest way to reproduce the problem. Edit: After finding the workaround it seems that this problem is independent of my problem.).

As the error message mentioned that property_propagation could be disabled, I tried modifying the example. I made my own copy of rfnoc_radio_loopback.cpp, copied connect_through_blocks directly into it and changed the last graph->connect call to disable property propagation (I set skip_property_propagation to true).

This solves the unresolvable graph error, but leads to another error:

Error: RuntimeError: AccessError: Attempting to read property `scaling@OUTPUT_EDGE:0' before it was initialized!

This error appears to occur at the graph->commit call.

Setup Details

• USRP:
  • N310
  • UHD 4.0.0.0-0-g90ce6062 (installed via mender, downloaded with uhd_images_downloader from ettus)
  • FPGA-image: uhd_image_loader --args "type=n3xx,fpga=HG"
• host:
  • Ubuntu 20
  • UHD 4.0.0.0-154-gb061af4f

Expected Behavior

I expected to be able to use rx and tx from the same radio control block in loopback configuration (receive and directly send out again), at least when I disable property propagation.

Actual Behaviour

I got the errors mentioned above.

Steps to reproduce the problem

Modified rfnoc_radio_loopback code:

//
// Copyright 2016 Ettus Research LLC
// Copyright 2018 Ettus Research, a National Instruments Company
//
// SPDX-License-Identifier: GPL-3.0-or-later
//

// Example UHD/RFNoC application: Connect an rx radio to a tx radio and
// run a loopback.

#include <uhd/rfnoc/block_id.hpp>
#include <uhd/rfnoc/mb_controller.hpp>
#include <uhd/rfnoc/radio_control.hpp>
#include <uhd/rfnoc_graph.hpp>
#include <uhd/types/tune_request.hpp>
#include <uhd/utils/graph_utils.hpp>
#include <uhd/utils/math.hpp>
#include <uhd/utils/safe_main.hpp>
#include <boost/format.hpp>
#include <boost/program_options.hpp>
#include <chrono>
#include <csignal>
#include <iostream>
#include <thread>
#include <numeric>

namespace po = boost::program_options;
using uhd::rfnoc::radio_control;
using namespace std::chrono_literals;

using uhd::rfnoc::rfnoc_graph;
using uhd::rfnoc::block_id_t;
using uhd::rfnoc::graph_edge_t;

void my_connect_through_blocks(rfnoc_graph::sptr graph,
    const block_id_t src_blk,
    const size_t src_port,
    const block_id_t dst_blk,
    const size_t dst_port)
{
    // First, create a chain from the source block to a stream endpoint
    auto block_chain = get_block_chain(graph, src_blk, src_port, true);
    UHD_LOG_TRACE("GRAPH_UTILS", "Found source chain for " + src_blk.to_string());
    // See if dst_blk is in our block_chain already
    const bool dst_found = std::accumulate(block_chain.begin(),
        block_chain.end(),
        false,
        [dst_blk, dst_port](bool dst_found, const graph_edge_t edge) {
            // This is our "accumulator" function that checks if the current_blk's ID and
            // input port match what we're looking for
            return dst_found
                   || (dst_blk.to_string() == edge.dst_blockid
                       && dst_port == edge.dst_port);
        });
    // If our dst_blk is in the chain already, make sure its the last element and continue
    if (dst_found) {
        UHD_LOG_TRACE(
            "GRAPH_UTILS", "Found dst_blk (" + dst_blk.to_string() + ") in source chain");
        while (dst_blk.to_string() == block_chain.back().dst_blockid
               && dst_port == block_chain.back().dst_port) {
            UHD_LOG_TRACE("GRAPH_UTILS",
                boost::format(
                    "Last block (%s:%d) doesn't match dst_blk (%s:%d); removing.")
                    % block_chain.back().dst_blockid % block_chain.back().dst_port
                    % dst_blk.to_string() % dst_port);
            block_chain.pop_back();
        }
    } else {
        // If we hadn't found dst_blk, find it now, then merge the two chain
        auto dest_chain = get_block_chain(graph, dst_blk, dst_port, false);
        block_chain.insert(block_chain.end(), dest_chain.begin(), dest_chain.end());
        UHD_LOG_TRACE(
            "GRAPH_UTILS", "Found destination chain for " + dst_blk.to_string());
    }

    // Finally, make all of the connections in our chain.
    // If we have SEPs in the chain, find them and directly
    // call connect on the src and dst blocks since calling
    // connect on SEPs is invalid
    std::string src_to_sep_id;
    size_t src_to_sep_port         = 0;
    bool has_src_to_sep_connection = false;
    std::string sep_to_dst_id;
    size_t sep_to_dst_port         = 0;
    bool has_sep_to_dst_connection = false;

    for (auto edge : block_chain) {
        if (uhd::rfnoc::block_id_t(edge.dst_blockid).match(uhd::rfnoc::NODE_ID_SEP)) {
            has_src_to_sep_connection = true;
            src_to_sep_id             = edge.src_blockid;
            src_to_sep_port           = edge.src_port;
        } else if (uhd::rfnoc::block_id_t(edge.src_blockid)
                       .match(uhd::rfnoc::NODE_ID_SEP)) {
            has_sep_to_dst_connection = true;
            sep_to_dst_id             = edge.dst_blockid;
            sep_to_dst_port           = edge.dst_port;
        } else {
            graph->connect(
                edge.src_blockid, edge.src_port, edge.dst_blockid, edge.dst_port);
        }
    }
    if (has_src_to_sep_connection && has_sep_to_dst_connection) {
        graph->connect(src_to_sep_id, src_to_sep_port, sep_to_dst_id, sep_to_dst_port, true);
    } else if (has_src_to_sep_connection != has_sep_to_dst_connection) {
        throw uhd::runtime_error(
            "[graph_utils] Incomplete path. Only one SEP edge found.");
    }
}

/****************************************************************************
 * SIGINT handling
 ***************************************************************************/
static bool stop_signal_called = false;
void sig_int_handler(int)
{
    stop_signal_called = true;
}

/****************************************************************************
 * main
 ***************************************************************************/
int UHD_SAFE_MAIN(int argc, char* argv[])
{
    // variables to be set by po
    std::string args, rx_ant, tx_ant, rx_blockid, tx_blockid, ref, pps;
    size_t total_num_samps, spp, rx_chan, tx_chan;
    double rate, rx_freq, tx_freq, rx_gain, tx_gain, rx_bw, tx_bw, total_time, setup_time;
    bool rx_timestamps;

    // setup the program options
    po::options_description desc("Allowed options");
    // clang-format off
    desc.add_options()
        ("help", "help message")
        ("args", po::value<std::string>(&args)->default_value(""), "UHD device address args")
        ("spp", po::value<size_t>(&spp)->default_value(0), "Samples per packet (reduce for lower latency)")
        ("rx-freq", po::value<double>(&rx_freq)->default_value(0.0), "Rx RF center frequency in Hz")
        ("tx-freq", po::value<double>(&tx_freq)->default_value(0.0), "Tx RF center frequency in Hz")
        ("rx-gain", po::value<double>(&rx_gain)->default_value(0.0), "Rx RF center gain in Hz")
        ("tx-gain", po::value<double>(&tx_gain)->default_value(0.0), "Tx RF center gain in Hz")
        ("rx-ant", po::value<std::string>(&rx_ant), "Receive antenna selection")
        ("tx-ant", po::value<std::string>(&tx_ant), "Transmit antenna selection")
        ("rx-blockid", po::value<std::string>(&rx_blockid)->default_value("0/Radio#0"), "Receive radio block ID")
        ("tx-blockid", po::value<std::string>(&tx_blockid)->default_value("0/Radio#1"), "Transmit radio block ID")
        ("rx-chan", po::value<size_t>(&rx_chan)->default_value(0), "Channel index on receive radio")
        ("tx-chan", po::value<size_t>(&tx_chan)->default_value(0), "Channel index on transmit radio")
        ("rx-bw", po::value<double>(&rx_bw), "RX analog frontend filter bandwidth in Hz")
        ("tx-bw", po::value<double>(&tx_bw), "TX analog frontend filter bandwidth in Hz")
        ("rx-timestamps", po::value<bool>(&rx_timestamps)->default_value(false), "Set timestamps on RX")
        ("setup", po::value<double>(&setup_time)->default_value(0.1), "seconds of setup time")
        ("nsamps", po::value<size_t>(&total_num_samps)->default_value(0), "total number of samples to receive")
        ("rate", po::value<double>(&rate)->default_value(0.0), "Sampling rate")
        ("duration", po::value<double>(&total_time)->default_value(0), "total number of seconds to receive")
        ("int-n", "Tune USRP with integer-N tuning")
        ("ref", po::value<std::string>(&ref)->default_value("internal"), "clock reference (internal, external, mimo, gpsdo)")
        ("pps", po::value<std::string>(&pps)->default_value("internal"), "PPS source (internal, external, mimo, gpsdo)")
    ;
    // clang-format on
    po::variables_map vm;
    po::store(po::parse_command_line(argc, argv, desc), vm);
    po::notify(vm);

    // print the help message
    if (vm.count("help")) {
        std::cout << boost::format("RFNoC: Radio loopback test %s") % desc << std::endl;
        std::cout
            << std::endl
            << "This application streams data from one radio to another using RFNoC.\n"
            << std::endl;
        return ~0;
    }

    /************************************************************************
     * Create device and block controls
     ***********************************************************************/
    std::cout << std::endl;
    std::cout << boost::format("Creating the RFNoC graph with args: %s...") % args
              << std::endl;
    uhd::rfnoc::rfnoc_graph::sptr graph = uhd::rfnoc::rfnoc_graph::make(args);

    // Create handles for radio objects
    uhd::rfnoc::block_id_t rx_radio_ctrl_id(rx_blockid);
    uhd::rfnoc::block_id_t tx_radio_ctrl_id(tx_blockid);
    // This next line will fail if the radio is not actually available
    uhd::rfnoc::radio_control::sptr rx_radio_ctrl =
        graph->get_block<uhd::rfnoc::radio_control>(rx_radio_ctrl_id);
    uhd::rfnoc::radio_control::sptr tx_radio_ctrl =
        graph->get_block<uhd::rfnoc::radio_control>(tx_radio_ctrl_id);
    std::cout << "Using RX radio " << rx_radio_ctrl_id << ", channel " << rx_chan
              << std::endl;
    std::cout << "Using TX radio " << tx_radio_ctrl_id << ", channel " << tx_chan
              << std::endl;
    size_t rx_mb_idx = rx_radio_ctrl_id.get_device_no();

    /************************************************************************
     * Set up radio
     ***********************************************************************/
    // Connect the RX radio to the TX radio
    my_connect_through_blocks(
        graph, rx_radio_ctrl_id, rx_chan, tx_radio_ctrl_id, tx_chan);
    graph->commit();

    rx_radio_ctrl->enable_rx_timestamps(rx_timestamps, rx_chan);

    // Set time and clock reference
    if (vm.count("ref")) {
        // Lock mboard clocks
        for (size_t i = 0; i < graph->get_num_mboards(); ++i) {
            graph->get_mb_controller(i)->set_clock_source(ref);
        }
    }
    if (vm.count("pps")) {
        // Lock mboard clocks
        for (size_t i = 0; i < graph->get_num_mboards(); ++i) {
            graph->get_mb_controller(i)->set_time_source(pps);
        }
    }

    // set the sample rate
    if (rate <= 0.0) {
        rate = rx_radio_ctrl->get_rate();
    } else {
        std::cout << boost::format("Setting RX Rate: %f Msps...") % (rate / 1e6)
                  << std::endl;
        rate = rx_radio_ctrl->set_rate(rate);
        std::cout << boost::format("Actual RX Rate: %f Msps...") % (rate / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the center frequency
    if (vm.count("rx-freq")) {
        std::cout << boost::format("Setting RX Freq: %f MHz...") % (rx_freq / 1e6)
                  << std::endl;
        uhd::tune_request_t tune_request(rx_freq);
        if (vm.count("int-n")) {
            tune_request.args = uhd::device_addr_t("mode_n=integer");
        }
        rx_radio_ctrl->set_rx_frequency(rx_freq, rx_chan);
        std::cout << boost::format("Actual RX Freq: %f MHz...")
                         % (rx_radio_ctrl->get_rx_frequency(rx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-freq")) {
        std::cout << boost::format("Setting TX Freq: %f MHz...") % (tx_freq / 1e6)
                  << std::endl;
        uhd::tune_request_t tune_request(tx_freq);
        if (vm.count("int-n")) {
            tune_request.args = uhd::device_addr_t("mode_n=integer");
        }
        tx_radio_ctrl->set_tx_frequency(tx_freq, tx_chan);
        std::cout << boost::format("Actual TX Freq: %f MHz...")
                         % (tx_radio_ctrl->get_tx_frequency(tx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the rf gain
    if (vm.count("rx-gain")) {
        std::cout << boost::format("Setting RX Gain: %f dB...") % rx_gain << std::endl;
        rx_radio_ctrl->set_rx_gain(rx_gain, rx_chan);
        std::cout << boost::format("Actual RX Gain: %f dB...")
                         % rx_radio_ctrl->get_rx_gain(rx_chan)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-gain")) {
        std::cout << boost::format("Setting TX Gain: %f dB...") % tx_gain << std::endl;
        tx_radio_ctrl->set_tx_gain(tx_gain, tx_chan);
        std::cout << boost::format("Actual TX Gain: %f dB...")
                         % tx_radio_ctrl->get_tx_gain(tx_chan)
                  << std::endl
                  << std::endl;
    }

    // set the IF filter bandwidth
    if (vm.count("rx-bw")) {
        std::cout << boost::format("Setting RX Bandwidth: %f MHz...") % (rx_bw / 1e6)
                  << std::endl;
        rx_radio_ctrl->set_rx_bandwidth(rx_bw, rx_chan);
        std::cout << boost::format("Actual RX Bandwidth: %f MHz...")
                         % (rx_radio_ctrl->get_rx_bandwidth(rx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }
    if (vm.count("tx-bw")) {
        std::cout << boost::format("Setting TX Bandwidth: %f MHz...") % (tx_bw / 1e6)
                  << std::endl;
        tx_radio_ctrl->set_tx_bandwidth(tx_bw, tx_chan);
        std::cout << boost::format("Actual TX Bandwidth: %f MHz...")
                         % (tx_radio_ctrl->get_tx_bandwidth(tx_chan) / 1e6)
                  << std::endl
                  << std::endl;
    }

    // set the antennas
    if (vm.count("rx-ant")) {
        rx_radio_ctrl->set_rx_antenna(rx_ant, rx_chan);
    }
    if (vm.count("tx-ant")) {
        tx_radio_ctrl->set_tx_antenna(tx_ant, tx_chan);
    }

    // check Ref and LO Lock detect
    if (not vm.count("skip-lo")) {
        // TODO
        // check_locked_sensor(usrp->get_rx_sensor_names(0), "lo_locked",
        // boost::bind(&uhd::usrp::multi_usrp::get_rx_sensor, usrp, _1, radio_id),
        // setup_time); if (ref == "external")
        // check_locked_sensor(usrp->get_mboard_sensor_names(0), "ref_locked",
        // boost::bind(&uhd::usrp::multi_usrp::get_mboard_sensor, usrp, _1, radio_id),
        // setup_time);
    }

    if (vm.count("spp")) {
        std::cout << "Setting samples per packet to: " << spp << std::endl;
        rx_radio_ctrl->set_property<int>("spp", spp, 0);
        spp = rx_radio_ctrl->get_property<int>("spp", 0);
        std::cout << "Actual samples per packet = " << spp << std::endl;
    }

    // Allow for some setup time
    std::this_thread::sleep_for(1s * setup_time);

    // Arm SIGINT handler
    std::signal(SIGINT, &sig_int_handler);

    // Calculate timeout and set timers
    // We just need to check is nsamps was set, otherwise we'll use the duration
    if (total_num_samps > 0) {
        total_time = total_num_samps / rate;
        std::cout << boost::format("Expected streaming time: %.3f") % total_time
                  << std::endl;
    }

    // Start streaming
    uhd::stream_cmd_t stream_cmd((total_num_samps == 0)
                                     ? uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS
                                     : uhd::stream_cmd_t::STREAM_MODE_NUM_SAMPS_AND_DONE);
    stream_cmd.num_samps  = size_t(total_num_samps);
    stream_cmd.stream_now = false;
    stream_cmd.time_spec =
        graph->get_mb_controller(rx_mb_idx)->get_timekeeper(rx_mb_idx)->get_time_now()
        + setup_time;
    std::cout << "Issuing start stream cmd..." << std::endl;
    rx_radio_ctrl->issue_stream_cmd(stream_cmd, rx_chan);
    std::cout << "Wait..." << std::endl;

    // Wait until we can exit
    uhd::time_spec_t elapsed_time = 0.0;
    while (not stop_signal_called) {
        std::this_thread::sleep_for(100ms);
        if (total_time > 0.0) {
            elapsed_time += 0.1;
            if (elapsed_time > total_time) {
                break;
            }
        }
    }

    // Stop radio
    stream_cmd.stream_mode = uhd::stream_cmd_t::STREAM_MODE_STOP_CONTINUOUS;
    std::cout << "Issuing stop stream cmd..." << std::endl;
    rx_radio_ctrl->issue_stream_cmd(stream_cmd, rx_chan);
    std::cout << "Done" << std::endl;
    // Allow for the samples and ACKs to propagate
    std::this_thread::sleep_for(100ms);

    return EXIT_SUCCESS;
}

Additional Information

Stacktrace of the location, where AccessError: Attempting to read ... before it was initialized! is thrown.

libuhd.so.4.1.0!uhd::rfnoc::property_t<double>::get(const uhd::rfnoc::property_t<double> * const this) (/home/user/ettus-project/uhd_debug_build/host/include/uhd/rfnoc/property.hpp:251)
libuhd.so.4.1.0!uhd::rfnoc::property_t<double>::equal(const uhd::rfnoc::property_t<double> * const this, uhd::rfnoc::property_base_t * rhs) (/home/user/ettus-project/uhd_debug_build/host/include/uhd/rfnoc/property.hpp:182)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::_assert_edge_props_consistent(uhd::rfnoc::detail::graph_t * const this, boost::adjacency_list<boost::vecS, boost::vecS, boost::bidirectionalS, boost::property<uhd::rfnoc::detail::graph_t::vertex_property_t, uhd::rfnoc::node_t*, boost::no_property>, boost::property<uhd::rfnoc::detail::graph_t::edge_property_t, uhd::rfnoc::graph_edge_t, boost::no_property>, boost::no_property, boost::listS>::edge_descriptor edge) (/home/user/ettus-project/uhd_debug_build/host/lib/rfnoc/graph.cpp:673)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::resolve_all_properties(uhd::rfnoc::detail::graph_t * const this, uhd::rfnoc::resolve_context context, boost::adjacency_list<boost::vecS, boost::vecS, boost::bidirectionalS, boost::property<uhd::rfnoc::detail::graph_t::vertex_property_t, uhd::rfnoc::node_t*, boost::no_property>, boost::property<uhd::rfnoc::detail::graph_t::edge_property_t, uhd::rfnoc::graph_edge_t, boost::no_property>, boost::no_property, boost::listS>::vertex_descriptor initial_node) (/usr/include/boost/graph/detail/edge.hpp:41)
libuhd.so.4.1.0!uhd::rfnoc::detail::graph_t::commit(uhd::rfnoc::detail::graph_t * const this) (/usr/include/boost/range/irange.hpp:89)
libuhd.so.4.1.0!rfnoc_graph_impl::commit(rfnoc_graph_impl * const this) (/usr/include/c++/9/bits/unique_ptr.h:360)
_main(int argc, char ** argv) (/home/user/ettus-project/devl/my-rfnoc-module/apps/rfnoc_radio_loopback_modified.cpp:201)
main(int argc, char ** argv) (/home/user/ettus-project/devl/my-rfnoc-module/apps/rfnoc_radio_loopback_modified.cpp:122)

Local variables of graph_t::_assert_edge_props_consistent(rfnoc_graph_t::edge_descriptor) then:

src_node->get_unique_id(): "0/DDC#0"
dst_node->get_unique_id(): "0/DUC#1"

[TSlivede]

I now have a workaround: I manually set the uninitialized scaling properties by hand. This is most certainly not the intended way, but it works.

To do this I copied node_accessor.hpp and prop_accessor.hpp to a place where they are accessible for my project (definitly not recommended...) and included them into my modified rfnoc_radio_loopback example. Then directly after the line I already modified (graph->connect(src_to_sep_id, src_to_sep_port, sep_to_dst_id, sep_to_dst_port, true);, 6th to last line in my_connect_through_blocks() in posted code above) I added:

        const uhd::rfnoc::res_source_info src_edge_info{uhd::rfnoc::res_source_info::OUTPUT_EDGE, src_to_sep_port};
        const uhd::rfnoc::res_source_info dst_edge_info{uhd::rfnoc::res_source_info::INPUT_EDGE, sep_to_dst_port};
        auto fix_scaling = [&](auto id, auto edge){
            auto props_ptrs = uhd::rfnoc::node_accessor_t{}.filter_props(
                graph->get_block(id).get(),
                [&](auto x) { return x->get_id() == "scaling"
                                  && x->get_src_info() == edge; });
            for (auto &&prop_ptr : props_ptrs)
            {
                auto prop = dynamic_cast<uhd::rfnoc::property_t<double> *>(prop_ptr);
                if(!prop->is_valid()){
                    uhd::rfnoc::prop_accessor_t{}.set_access(prop, uhd::rfnoc::property_base_t::RW);
                    prop->set(1.0);
                }
            }
        };
        fix_scaling(src_to_sep_id, src_edge_info);
        fix_scaling(sep_to_dst_id, dst_edge_info);

As I said: this works, but whatever the actual problem is should probably be properly fixed somewhere in the uhd library.

[andreaskuster]

@TSlivede
I ran into the same issue, but actually found a simpler work-round, which can be summarized as follow:

• switch from connect_through_blocks to the graph->connect function (all, even the static connections have to be setup manually)
• break the loop of property propagation, i.e. in my case, from my custom block to the DUC by setting skip_property_propagation=true

    /*
     * Connect blocks (blockname:port)
     */
    // radio0:0 -> ddc0:0
    graph->connect(rx_radio_id, rx_radio_chan, ddc_block_id, ddc_chan);
    // ddc0:0 -> custom0:0
    graph->connect(ddc_block_id, ddc_chan, custom_block_ctrl->get_block_id(), 0);
    // custom0:0 -> duc0:0
    graph->connect(custom_block_ctrl->get_block_id(), 1, duc_block_id, duc_chan, true);
    // duc0:0 -> radio0:0
    graph->connect(duc_block_id, duc_chan, tx_radio_id, tx_radio_chan);

    graph->commit();

Note that I did not test this example without my custom block in the chain, however, this should not change much.

[mbr0wn]

connect_through_blocks() now also has that argument, and that's the correct way to handle this.

[tomwts]

I experienced the same issue running the same example app on X310 and RFNOC 4.7 release. As long as the Rx and Tx radio block is different, the app ran fine, but it failed to initialize when Tx and Rx radio block Id is the same.
Is this issue still occurred on RFNOC 4.7 release?

[INFO] [UHD] linux; GNU C++ version 13.2.0; Boost_108300; UHD_4.7.0.0-0-ga5ed1872
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Radio 1x clock: 200 MHz
Using RX radio 0/Radio#1, channel 0
Using TX radio 0/Radio#1, channel 0
[ERROR] [RFNOC::GRAPH::DETAIL] Cannot create route from device:1/sep:2 and device:1/sep:2, no route was found!
Error: RuntimeError: Cannot create route from device:1/sep:2 and device:1/sep:2, no route was found!