Data Discrepancy Issue Between Nebula Studio and Spark Connector

[vaibhavkgiradkar]

Summary:
There is an observed difference in data/count when fetching data from Nebula Studio compared to using the Spark connector. However, the count matches when reading data with the Spark connector. The data in question has been written with the help of the Spark connector.

Steps to Reproduce:

1. Write data using the Spark connector to Nebula Graph.
2. Query the data using Nebula Studio.
3. Query the same data using the Spark connector.
4. Compare the results and observe the discrepancy.

Expected Behavior:
The data and count retrieved from Nebula Studio should match the data and count obtained through the Spark connector.

Actual Behavior:
There is a discrepancy in the data or count between Nebula Studio and the Spark connector, even though the count matches when using the Spark connector alone.

Environment Details:

• Nebula Studio Version: 3.7.0
• Spark Connector Version: 3.6.0
• Nebula Graph Version: 3.6.0
• Spark Version: 3.4.1

[vaibhavkgiradkar]

@wey-gu any thoughts on this?

[wey-gu]

Could you please help provide what it's like to query the same data of the two?

Like pattern of query etc.

[vaibhavkgiradkar]

match (v1:vertex_a)-[:edge_a]->(v2:vertex_b) return count(*)

spark.read.format(
"com.vesoft.nebula.connector.NebulaDataSource").option(
"type", "edge").option(
"spaceName", "sample_space").option(
"label", "edge_a").option(
"returnCols", "").option(
'passwd', 'nebula').option(
'user', 'root').option(
"metaAddress", "").option(
"operateType", "read").option(
"partitionNumber", 10).load()

[wey-gu]

Could we assume all edge_a 's source tag is vertex_a and the dest tag is vertex_b?

If not, there are not equivalent.

In case yes, there are cases of dangling edge that lead to the difference between the two.

If there are edge_a edges that with only edges being inserted but the src/dest vertices were not inserted, they are dangling edges, which could be scanned from the storage side(with spark) but cannot be scanned in some queries like match (v1:vertex_a)-[:edge_a]->(v2:vertex_b).

Also, when possible. SHOW STATS is the recommended way to query the counts.

ref: https://docs.nebula-graph.io/3.6.0/8.service-tuning/2.graph-modeling/#about_dangling_edges