Ubuntu and Mellanox OFED version that works with Moongen?

[Assar-Westman]

Thanks for a really nice high speed packet generator.
I used the Moongen packet generator a lot a year ago on Mellanox mlx5 100 Gbe cards. Now I needed to do some more benchmarks using Moongen using these 100 Gbe cards. During last year I updated the computer to latest version of ubuntu and also the Mellanox OFED was updated to latest version. Now when I try to use Moongen I get this error

PMD: net_mlx5: ibv_exp_query_device() failed

So I wonder what is the latest version of Ubuntu and Mellanox OFED that are supported by Moongen?

Best regards Assar W.

[emmericp]

sorry, our support for mellanox is somwhat bad due to their terrible driver model. Can you try to use the branch dpdk-19.05?

[Assar-Westman]

Thanks for the suggestion. When I using the 19.05 branch I get a lot of compile error like

libmoon/deps/dpdk/drivers/net/mlx5/mlx5_glue.c:754: undefined reference to `mlx5dv_open_device'
I have added the preferred deb packages, except
libmlx5[...].deb
libmlx5-dev[...].deb
which are missing in the used Mellanox OFED 5.2 distribution
It feels like some Mellanox library is missing maybe it is supplied with libmlx5...

Best regards Assar W.

[Assar-Westman]

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

#Add Mellanox libraries if options are specified
if(USE_MLX5 OR USE_MLX4)
SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

#Add Mellanox libraries if options are specified
if(USE_MLX5 OR USE_MLX4)
SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)
SET(DPDK_LIBS ${DPDK_LIBS} mnl)
SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

[michael-kaplan]

Did you succeed? I'm curious if you got DPDK to include the Mellanox drivers, --mlx5 flag to build.sh not seem to be working for me. have very similar setup.

[Assar-Westman]

My moongen setup now works with Mellanox OFED driver version 5.2-2.2.0.0, Ubuntu version 20.04.2 LTS and kernel 5.8.0-44-generic. For me the crucial thing was to use branch 19.05, install the whole Mellanox OFED driver package and also to install libmnl-dev package, and of course edit libmoon/CMakeLists.txt as seen above
AW

[michael-kaplan]

Fascinating. I am using Mellanox OFED driver 5.3-OFED.5.3.1.0.0.1, Ubuntu 20.04.2 LTS, and kernel 5.4.0-26-generic. I changed the MoonGen branch to 19.05 and built with ./build.sh --mlx5 (after updating CMAKELists.txt, built successfully).

Yet for whatever reason I continue to get a 0 device error. Arg! I have a MT28880, do you know which you have? I also find some of the complaints concerning, but do not know what to make of them. I suppose there's also updating the Ubuntu kernel.

[INFO]  Initializing DPDK. This will take a few seconds...
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/10002/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[INFO]  Found 0 usable devices:
[FATAL] Lua error in task master
...me/mkaplan/prod3/MoonGen/build/../libmoon/lua/device.lua:100: there are only 0 ports, tried to configure port id 0

[Assar-Westman]

I have a MT28800 Family ConnectX-5 PCie 4.0 I see in my notes that I only compile Moongen with GCC version 7.5
Of course you have run both bind-interfaces.sh and setup-hugetlbfs.sh
AW

[liqia]

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

#Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

#Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs) SET(DPDK_LIBS ${DPDK_LIBS} mnl) SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

thanks！！！！

[Ramla-I]

Hi, have you found a solution for this? I'm also able to to build MoonGen with the mlx5 flag, but when I try to run it I get the 0 usable devices error.

[tom-wegener]

I hope Ramla-I solved his problem but for people looking into this later:
because of the special driver model of the mlx-devices the devices do not have to bind by the script and the script reports 0 devices anyway. I could just my script after following the steps from Assar-Westman .