RUST-1899 Deserialization with UUID

[Bryson14]

Versions/Environment

1. What version of Rust are you using? 1.77
2. What operating system are you using? windows + wsl2
3. What versions of the driver and its dependencies are you using? (Run
   cargo pkgid mongodb & cargo pkgid bson) 2.9.0 & 2.8.2
4. What version of MongoDB are you using? (Check with the MongoDB shell using db.version()) 7.2.0
5. What is your MongoDB topology (standalone, replica set, sharded cluster, serverless)? replica set

Describe the bug

A clear and concise description of what the bug is.

I am migrating data from another database which includes UUIDs stored as strings. I understand that bson::uuid stores data as binary base-64 data with a subtype flag = 4, however my current database is using strings and the rust code interacting with it is expected deserialization to and from strings.

My current struct definition:

#[derive(Serialize, Deserialize, Debug, PartialEq, PartialOrd, Clone)]
pub struct UserMetadata {
    /// A unique Single Sign On at GEHC
    pub sso: String,

    /// A unique identifier for the user.
    /// If not provided during deserialization, a random UUID will be generated.
    #[serde(default = "Uuid::now_v7")]
    pub user_id: Uuid,

    // The time the user was created.
    /// If not provided during deserialization, the current UTC time will be used.
    #[serde(with = "bson::serde_helpers::chrono_datetime_as_bson_datetime")]
    #[serde(default = "Utc::now")]
    pub last_access_time: DateTime<Utc>,

    // The time the user was created.
    /// If not provided during deserialization, the current UTC time will be used.
    #[serde(with = "bson::serde_helpers::chrono_datetime_as_bson_datetime")]
    #[serde(default = "Utc::now")]
    pub create_time: DateTime<Utc>,

    /// Team ID at GEHC
    #[serde(skip_serializing_if = "Option::is_none")]
    pub team_id: Option<String>,

    /// Organization ID at GEHC
    #[serde(skip_serializing_if = "Option::is_none")]
    pub org_id: Option<String>,

    /// User Type
    #[serde(skip_serializing_if = "Option::is_none")]
    pub user_type: Option<String>,

    /// Content for Content Privacy
    #[serde(skip_serializing_if = "Option::is_none")]
    #[serde(default = "default_deny")]
    pub content_privacy_consent: Option<PrivacyConsent>,

    #[serde(skip_serializing_if = "Option::is_none")]
    pub content_privacy_consent_date: Option<DateTime<Utc>>,
}

When I run find() to get items from this collection I get,

called `Result::unwrap()` on an `Err` value: Error { kind: BsonDeserialization(DeserializationError { 
message: "invalid type: string \"018d6b0c-4a48-76e0-8d61-eb7977071904\", expected bytes" }), labels: {}, wire_version: None, source: None }

I tried to remedy this by changed uuid::Uuid in the struct definition to bson::Uuid but got a similar error:

called `Result::unwrap()` on an `Err` value: Error { kind: BsonDeserialization(DeserializationError { message: "expected Binary with subtype Uuid, instead got 
String" }), labels: {}, wire_version: None, source: None }

Trying to use the serde_with-3 flag didn't help and I actually got compiler errors. This was after adding the feature flag with cargo add bson -F serde_with-3. The documentation didn't say that I had to add other dependancies to get this to work.

error[E0433]: failed to resolve: use of undeclared crate or module `serde_with_3`
  --> src\models\user_metadata_bson.rs:27:3
   |
27 | #[serde_with_3::serde_as]
   |   ^^^^^^^^^^^^ use of undeclared crate or module `serde_with_3`

error: cannot find attribute `serde_as` in this scope
  --> src\models\user_metadata_bson.rs:36:7
   |
36 |     #[serde_as(as = "Option<bson::Uuid>")]
   |       ^^^^^^^^

error: cannot find attribute `serde_as` in this scope
  --> src\models\user_metadata_bson.rs:81:7
   |
81 |     #[serde_as(as = "Option<bson::Uuid>")]
   |       ^^^^^^^^

what to do:

So I'm thinking there are two fixes to this. Either I fix this issue in code and get the serialization and deserialization to play well with uuid::Uuid, or I migrate all the fields that contain string uuid in the db to Mongo's UUID format. In that case, I'd appreciate any tools to do that, but might just have to bite the bullet and write a small python scipt to do that.

To Reproduce

1. create a document in mongo with a field that contains a string
2. attempt to find from that collection into a uuid::UUid

Cargo.toml

[dependencies]
aws-config = { version = "1.1.5", features = ["behavior-version-latest"]}
aws-sdk-s3 = "1.15.0"
aws-smithy-types = "1.1.5"
axum = "0.7.4"
axum-test = "14.2.2"
bson = { version = "2.9.0", features = ["chrono-0_4", "uuid-1", "serde_with-3"] }
chrono = { version = "0.4.34", features = ["serde"] }
futures = "0.3.30"
http = "1.0.0"
lambda_http = "0.9.3"
lambda_runtime = "0.9.2"
mongodb = {version = "2.8.2" , features = ["tokio-runtime"]}
serde = { version = "1.0.196", features = ["derive"] }
serde_json = "1.0"
thiserror = "1.0.56"
tokio = { version = "1", features = ["macros"] }
tower-http = { version = "0.5.0", features = ["trace", "add-extension"] }
tracing = { version = "0.1", features = ["log"] }
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] }
uuid = { version = "1.7", features = ["v4", "v7", "serde"] }

[Bryson14]

example of data currently in mongodb

{
  "_id": {
    "$oid": "65fdf259bc497a1b87a5c9ce"
  },
  "sso": "123456789",
  "content_privacy_consent": null,
  "content_privacy_consent_date": null,
  "create_time": {
    "$date": "2024-02-11T02:29:05.302Z"
  },
  "last_access_time": {
    "$date": "2024-03-09T22:21:26.890Z"
  },
  "user_id": "018d95ff-2e96-7169-9d08-175b96d4e7e3"
}

[isabelatkinson]

Hi @Bryson14, thanks for this detailed report! We do indeed have a bug in our deserialization logic for UUIDs that are not in binary format. I made a PR to fix this (#468), so we'll put out a patch release once that's approved and merged. Once the fix is in, your UUID deserialization from strings when using the bson::Uuid type should work without needing to enable any feature flags.

I was unable to reproduce the compilation errors you're encountering. Are you seeing those errors with the chrono-0_4, uuid-1, and serde_with-3 feature flags enabled?

[abr-egn]

I think the serde_with error was fixed by 81a9895.

[Bryson14]

I'll give the feature flags a try again. For the bson::uuid, it was too much time dealing with it that I just went with treating the data from the database as strings (which they are strings from the migration). I would have to change the rust code and database UUIDs to deal with this right now and I don't see any good benefit from using bson::uuid at the moment. Perhaps performance?

[isabelatkinson]

I don't see any good benefit from using bson::uuid at the moment

It depends upon your use case. If you're just reading your data via the driver then it likely doesn't matter much which type you're using, but bson::Uuid provides utilities for constructing values that could be useful if you're inserting new data.

I'm going to close this out since the fix in #468 has been released, but feel free to create another issue if you have any further feedback or questions!

[FalkWoldmann]

Hi @isabelatkinson, we hit the same problem when trying to deserialize a string _id field using this newtype:

#[derive(Clone, PartialEq, Eq, Debug, Hash, Serialize, Ord, PartialOrd, Deserialize, Display, Default, Copy)]
pub struct UserId(uuid::Uuid);

We are inserting data like this:

doc! {
      "_id": "021afc11-2771-44c8-8b61-e17c3d4b2685",
      ...
    }

We receive this error:

ERROR Error while fetching from database: Error { kind: BsonDeserialization(DeserializationError { message: "invalid type: string \"021afc11-2771-44c8-8b61-e17c3d4b2685\", expected bytes" }), labels: {}, wire_version: None, source: None }

Unfortunately, we cannot easily use the bson::serde_helpers::HumanReadable since it lacks a lot of impl we need (like copy). We're on mongodb 3.2.3 and bson 2.14.0

Manually implementing Deserialize makes it work, but this feels kind of hacky imo:

impl<'de> Deserialize<'de> for UserId {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        struct UserIdVisitor;

        impl<'de> Visitor<'de> for UserIdVisitor {
            type Value = UserId;

            fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
                formatter.write_str("a UUID string")
            }

            fn visit_str<E>(self, value: &str) -> Result<Self::Value, E>
            where
                E: de::Error,
            {
                Uuid::parse_str(value).map(UserId).map_err(de::Error::custom)
            }
        }
        deserializer.deserialize_str(UserIdVisitor)
    }
}

or we could add #[serde(try_from = "&str", into = "String")], which isn't great as well because it incurs unnecessary allocations

[isabelatkinson]

Hi @FalkWoldmann, would it be possible for you to use the bson::Uuid type for deserialization rather than uuid::Uuid? That will deserialize from a string without issue, e.g.:

#[derive(Debug, Deserialize)]
struct UserId(bson::Uuid);
#[derive(Debug, Deserialize)]
struct Data {
    #[serde(rename = "_id")]
    id: UserId,
}

#[tokio::main]
async fn main() -> mongodb::error::Result<()> {
    let uuid = uuid::Uuid::new_v4();
    let document = doc! {
        "_id": uuid.to_string(),
    };

    let client = Client::with_uri_str("<uri str>").await?;
    let collection = client.database("db").collection("coll");

    collection.insert_one(document).await?;

    let collection = collection.clone_with_type::<Data>();
    let document = collection.find_one(doc! {}).await?.unwrap();

    // requires uuid-1 feature flag enabled for the bson crate
    assert_eq!(document.id.0.to_uuid_1(), uuid);

    Ok(())
}

We can also consider adding a helper to serde_helpers for conversion between uuid::Uuid and strings.

Unfortunately, we cannot easily use the bson::serde_helpers::HumanReadable since it lacks a lot of impl we need (like copy).

We can certainly add more trait impls to HumanReadable. Is there anything you'd need other than Copy?

[FalkWoldmann]

Unfortunately, we store our data _ids in String format and it is not feasible to migrate the field to bson anytime soon and as far as I understood (please correct me if I'm wrong), we cannot just use bson::Uuid in our code in this case.

Having some serde_helper for this case would be very nice!

For HumanReadable, we would need: #[derive(Clone, PartialEq, Eq, Debug, Hash, Serialize, Ord, PartialOrd, Deserialize, Display, Default, Copy)], which may be too much for many use cases