Ettus X410 UHD 4,4 LMX2572 Unable to communicate with LMX2572 Expected R125==0x2288 got: FFFF

[jres-mil]

Issue Description

Trying to get any output from Ettus X410. Running uhd_usrp_probe returns error message. device is found using uhd_find_devices
I am able to login to stm32 microcontroller, and linux partition using screen. When trying to collect any data from x410 I receive the error. Any --args that are included does not fix the issue.

Setup Details

UHD 4.4.0.0
FPGA X4_200

Expected Behavior

Actual Behaviour

[ERROR] [LMX2572] Unable to communicate with LMX2572! Expected R125==0x2288, got: FFFF
[ERROR] [RFNOC::GRAPH] Error during initialization of block 0/Radio#0!
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: RuntimeError: Unable to communicate to LMX2572!
Error: RuntimeError: Failure to create rfnoc_graph.

[ERROR] [LMX2572] Unable to communicate with LMX2572! Expected R125==0x2288, got: 0000
[ERROR] [RFNOC::GRAPH] Error during initialization of block 0/Radio#0!
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: RuntimeError: Unable to communicate to LMX2572!
Error: RuntimeError: Failure to create rfnoc_graph.

These are two separate error messages I have received. One is got: 0000 and the other got: FFFF

Steps to reproduce the problem

uhd_usrp_probe

Additional Information

In the --help on the linux partition lists a way --interactive-iface to poke and peek registers, which may be a way to rewrite the R125 register but I cannot find any documentation about the command.

in uhd/host/lib/usrp/common/lmx2572.cpp starting at line 98 I see where the message is comming from.

How would I check the status of the PLL lock though a command on the x410 command line? Or to unlock the PLL?

    void reset() override
    {
        // Power-on Programming Sequence described in the datasheet,
        // Section 7.5.1.1
        _regs       = lmx2572_regs_t{};
        _regs.reset = lmx2572_regs_t::reset_t::RESET_RESET;
        _poke16(0, _regs.get_reg(0));
        // Reset bit is self-clearing, it does not need to be poked twice. We
        // manually reset it in the SW cache so we don't accidentally reset again
        _regs.reset = lmx2572_regs_t::reset_t::RESET_NORMAL_OPERATION;
        // Also enable register readback so we can read back the magic number
        // register
        _enable_register_readback(true);
        // If the LO was previously powered down, the reset above will power it up. On
        // power-up, we need to wait for the power up delay recommended by TI.
        _sleep(POWERUP_DELAY);
        // Check we can read back the last register, which always returns a
        // magic constant:
        const auto magic125 = _peek16(125);
        if (magic125 != 0x2288) {
            UHD_LOG_ERROR(LOG_ID,
                "Unable to communicate with LMX2572! Expected R125==0x2288, got: "
                    << std::hex << magic125 << std::dec);
            throw uhd::runtime_error("Unable to communicate to LMX2572!");
        }
        UHD_LOG_TRACE(LOG_ID, "Communication with LMX2572 successful.");
        // Now set _regs into a sensible state
        _set_defaults();
        // Now write the regs in reverse order, skipping RO regs
        const auto ro_regs = _regs.get_ro_regs();
        // Write R0 last for the double buffering
        for (int addr = _regs.get_num_regs() - 2; addr >= 0; addr--) {
            if (ro_regs.count(addr)) {
                continue;
            }
            _poke16(uhd::narrow_cast<uint8_t>(addr), _regs.get_reg(addr));
        }
        _regs.save_state();
    }

    bool get_lock_status() override
    {
        // Disable register readback which implicitly enables lock detect mode
        _enable_register_readback(false);
        // If the PLL is locked we expect to read 0xFFFF from any read
        return _peek16(0) == 0xFFFF;
    }

    uint16_t peek16(const uint8_t addr)
    {
        _enable_register_readback(true);
        return _peek16(addr);
    }

[manderseck]

@jres-mil In general it looks like a communication problem between the daughterboard CPLD and the LO synthesizer. We poke and peek some registers during session setup to ensure the communication works properly and that obviously fails with some random return value (0xFFFF and 0x0000) here. May I ask you for the following things?
Can you please login to the device and run eeprom-dump db0 and paste the output into here (you may remove the serial number). Since it may have to make with the CPLD not communicating properly you could try to rewrite the CPLD image by using the zbx_update_cpld utility which is available on the device, too, see here: https://files.ettus.com/manual/page_zbx.html#zbx_updating_cpld
Ensure to power-cycle the device afterwards. If that all doesn't help, a hardware failure may be possible, too, unfortunately.

[jres-mil]

@manderseck I am using a standalone development computer to run X410. I am not able to copy and paste outputs, I will type them out.

Output from eeprom-dump db0 as follows:
usrp_eeprom_board_info (0x10) pid: 0x4002, rev: 0x0005, compat_rev: 0x0002, serial: 328XXXX
usrp_eeprom_db_pwr_seq (0x12) (20, 0x07) (20, 0x18) (0, 0x00) (0, 0x00) (0, 0x00) (0, 0x00) (0, 0x00) (0, 0x00)

Running zbx_update_cpld I receive the error
NotImplementedError: The CPLD update Strategy for rev or CPLD model 5 is not available.

[manderseck]

@jres-mil Thanks for providing more information. You could still try to update the CPLD by installing a later file system version, UHD 4.5 or 4.6 should work, in UHD 4.7 I just discovered that the CPLD updater got broken, we're working on this. While the operation of that daughterboard revision (normally) works in older UHD versions, the CPLD image file for this particular component is only shipped from 4.5 on. However to me it looks like a case for RMA unfortunately, so I'd ask you to get in touch with NI's support/RMA. I'm sorry for that.

[jres-mil]

@manderseck Thank you for the update I will try a newer version. The platform we are building for requires 4.4, however I will be able to make sure the equipment is working or not.

[manderseck]

For now I'm going to close this as it doesn't seem to be a software issue. I expect the CPLD flashing to work properly in later UHD versions (except for 4.7) but at the same time I fear that doesn't fix your (hardware) issue. If anything new comes up in that issue feel free to re-open. Thanks!

[brt99]

@manderseck I have the same problem with my x410 as @jres-mil. I have installed UHD version 4.6 and flashed the daughterboard firmware with zbx_update_cpld as described. However, this did not solve the problem and the error message still exists. Is there already a solution? Thanks!

[manderseck]

@brt99 We are still looking into the issue internally. May I ask you to open a service request, so we can get more information from your device, too?
https://www.ni.com/en/support.html