VARCHAR[] cannot be extracted as Rust Vec<String>

[aykut-bozkurt]

When I check the is_compatible_with function for String, we also check VARCHAROID. This way, we can map TEXT + VARCHAR datums to Rust String.

But it is not the case with VARCHAR[]. I get non-binary coercible types error when I try to extract Vec<String> from VARCHAR[] datum. I believe we lack a similar check at is_compatible_with for Vec<T>.

I wonder might my very naive approach below work out. It basically relies on if element types are compatible.

Looks like a related discussion. #1361

[eeeebbbbrrrr]

Put up a PR with a unit test or two and lets see what CI says. In general, this looks like a good change. AFAIK, the structure of an ArrayType doesn't change based on its element type.

[workingjubilee]

Reusing the IntoDatum::is_compatible_with logic will allow incorrect integer sizes to work. The integer sizes do in fact alter the structure of arrays, as integers are densely packed in the array, instead of being spaced to every eightbyte.

[workingjubilee]

So no, we have to figure out something different.

[eeeebbbbrrrr]

I was probably a little imprecise. What I mean is that an ArrayType will have the same shape as ArrayType so long as E and E2 are themselves binary compatible.

And yeah, I now see here in that diff that it's using our T::is_compatible_with() and not Postgres' "is_binary_compatible" function, which we'd need instead.

[workingjubilee]

I'm not aware of such a function, but I believe that you are referring to the answer of "binary coercible", and that is the question that is answered by IntoDatum::is_compatible_with. "If you put this type in as an argument, can you always also get this other type out?" The problem is that it's not the question we need to answer for Arrays.

[workingjubilee]

It's discussed a bit in https://www.postgresql.org/docs/current/sql-createcast.html

[eeeebbbbrrrr]

I'm not aware of such a function, but I believe that you are referring to the answer of "binary coercible", and that is the question that is answered by IntoDatum::is_compatible_with. "If you put this type in as an argument, can you always also get this other type out?" The problem is that it's not the question we need to answer for Arrays.

Yeah, digging through the postgres sources, IsBinaryCompatible() was replaced by IsBinaryCoercible sometime before pg12. hmm...

I guess we'd need to do something like lookup both types in the pg_type catalog and see if they have the same (at least) typstorage?

EDIT: Actually, I'm not sure what we'd check. typstorage isn't what I initially thought it was.

I'm not seeing anything in the Postgres sources that check for "array binary compatibility" -- at least not in the obvious places.

[eeeebbbbrrrr]

hmm. Maybe it's a combination of (typlen, typbyval, typalign).

EDIT: Maybe that's not enough either.

[workingjubilee]

We're gonna have to handroll it if we want it. It will be easy enough to write something that only works in a reasonable number of cases and doesn't work in all cases, and that will probably be fine.

[aykut-bozkurt]

My goal actually is to be able to represent different Postgres string in Rust and convert them back correctly, without losing the actual type, to Postgres types. Representing different Postgres strings via a single Rust struct String loses the actual type in that scenario (String cannot know what was the actual Postgres type). Hence, I internally created different structures for other string types.(mapping each Rust structure to a single Postgres type) e.g.

#[derive(Debug)]
pub(crate) struct Varchar(pub(crate) String);

impl IntoDatum for Varchar {
    fn into_datum(self) -> Option<pg_sys::Datum> {
        self.0.into_datum()
    }

    fn type_oid() -> pg_sys::Oid {
        VARCHAROID
    }
}

impl FromDatum for Varchar {
    unsafe fn from_polymorphic_datum(
        datum: pg_sys::Datum,
        is_null: bool,
        typoid: pg_sys::Oid,
    ) -> Option<Self>
    where
        Self: Sized,
    {
        let val = String::from_polymorphic_datum(datum, is_null, typoid);
        val.and_then(|val| Some(Self(val)))
    }
}

This way I can differentiate between different String types.

[workingjubilee]

Can you explain why you wish to preserve these types?

[aykut-bozkurt]

Data migration purposes. I serialize tuples in Postgres tables into a different data format and deserialize the format back into PG tuples.