Support for use cases other than backend applications

[Terrance]

I just starting looking at Prisma as an ORM for a new project. It took a bit of digging through the docs and playing with the model generation to come to the conclusion that Prisma isn't suitable outside of a backend application (yes, I realise it's stated on the docs' home page -- I managed to gloss over "any backend application" ruling out non-backend use cases).

As it stands, with models generated directly into site-packages, Prisma acts more like a deployment tool and only works with a single project's database. That said, what would it take to support packaging and distribution of models in a project package?

Naively, one can copy the prisma package into their project, and it seems to Just Work as internal references already use relative imports to stay within the package.

I'm aware from the architecture docs page that a local query engine web server is involved, which is certainly overkill for, say, a small CLI application, but it would be neat to expand the scope beyond just server applications.

Suggested solution

Models, actions, migrations, and other project-specific files would be generated to a (user-selected) location within the current Python project (e.g. for the spam project, spam.prisma containing models.py, enums.py etc.). This could be configured via an option of the generator section of schema.prisma.

In theory, this location could be ignored by version control of the project, with the schema file remaining the single source of truth and developers still needing to run the likes of prisma db pull to get a working local set of models. It would however be included in packaged releases of the project, so that end users obtain a copy of the models without having to interface with Prisma themselves.

Depending on the compatibility of models with the rest of the Prisma framework, there would probabl need to be a note on how strict a version requirement for the prisma package should be set once a set of models is generated.

[RobertCraigie]

Naively, one can copy the prisma package into their project, and it seems to Just Work as internal references already use relative imports to stay within the package.

You can also specify where you want the prisma package to be generated to with the output option in the schema, e.g.

generator py {
  provider = "prisma-client-py"
  output = "my_local_dir"
}

Upon generation this will copy the entire prisma package to my_local_dir along with the newly generated files.

I'm aware from the architecture docs page that a local query engine web server is involved, which is certainly overkill for, say, a small CLI application, but it would be neat to expand the scope beyond just server applications.

The fact that the query engine is internally implemented as a web server shouldn't matter to end users, the internal web server is not intended to be exposed publicly and is merely an implementation detail. However it should be noted that I am working on removing the web server and instead switching to using native FFI bindings instead, see #27.

In theory, this location could be ignored by version control of the project, with the schema file remaining the single source of truth and developers still needing to run the likes of prisma db pull to get a working local set of models. It would however be included in packaged releases of the project, so that end users obtain a copy of the models without having to interface with Prisma themselves.

Including the auto-generated prisma code in a packaged application should be possible today, albeit with some workarounds. The biggest challenges I can see with using Prisma in say a packaged CLI application is handling of the engine binaries and migrations.

Here's how I'd get around these issues:

from pathlib import Path

import click

# this assumes you're generating prisma to a local directory within
# the package named _prisma, e.g. pkg/_prisma
from ._prisma import Prisma
from ._prisma.cli import prisma as prisma_cli

@click.command()
def main() -> None:
    # use the packaged schema file
    schema = Path(__file__).parent / 'schema.prisma'

    # ensure the database is in the correct state
    # and ensure the engine binaries are downloaded
    prisma_cli.run(['db', 'push', '--skip-generate', f'--schema={schema}'])

    # as you've already packaged the generated prisma code you can
    # make any queries like you normally would
    client = Prisma()
    client.connect()

if __name__ == '__main__':
    main()

It should be noted that the application of migrations in this solution is naive and may cause issues down the line, a better solution would be to also package migration files as well and apply them serially although I do not know how difficult this would be in practice.

---

Prisma Client Python was designed with backend use cases in mind, where the developer is in control of the database and the machine. As such, use cases outside of that are more difficult to work with although I agree that it's a valid use case and we should figure out how to make it easy for people who want to use prisma within a packaged application as well.

Another use case I've considered before is using prisma within a library although I'm not sure what a solution for that would look like.

[Terrance]

Aha, good to know outputting the generated prisma module inside a project is already a feature. I'll have to give it a go and see how well it works in practice, but it sounds promising!

To touch on a few other points:

the internal web server

For what it's worth, this is a potential issue with shared hosting, both for backend servers and more general-purpose applications -- other users of the system can access the (unauthenticated?) database API if they stumble upon the port, whilst the owning user may not be aware of it running in the first place.

It looks like the engine has UNIX socket support, but the Python bindings don't currently make use of it (the code to spawn the engine hard-codes the port option).

If the HTTP server is going away then I guess this is a temporary concern, though if it's more of a long-term goal then exposing the option for UNIX sockets would be useful.

a better solution would be to also package migration files as well and apply them serially

Agreed, migrations should be distributed with a project, so that updates from previous or older versions can just apply any migrations they're missing. Django does this well, and I imagine the process here wouldn't be much different.

using prisma within a library

I had this in mind too when thinking of other use cases, though the combination of being a library and requiring your own database is perhaps quite niche, as I can't think of a good concrete example.

A somewhat-related use case that may struggle is a framework that can be extended with plugins -- consider an application that ships its own copy of the prisma module containing its own models, but also allows for plugins that need their own models on top. Ideally we want those models to integrate with the main project's Prisma client, which might be tricky. Not something I need for the use case I'm imaginging, but perhaps another angle to consider. 🙂

[RobertCraigie]

For what it's worth, this is a potential issue with shared hosting, both for backend servers and more general-purpose applications -- other users of the system can access the (unauthenticated?) database API if they stumble upon the port, whilst the owning user may not be aware of it running in the first place.

Yes, the database API is unauthenticated. I can see this being a security concern although I'm not clear on the situation you're describing? I was under the impression that in shared hosting services even though multiple users are granted access to the same machine, each user only gets access to their own "container" which means they wouldn't be able to access any resources running in other containers unless they were explicitly exposed.

In a more general backend server situation, you should only make ports publicly accessible if you intend on using them. I can imagine a case where you aren't using a port that you've exposed and the client unluckily picks that port to run the engine on. Although this would be highly unlikely, I admit it is still a possibility.

I'm actually not familiar with UNIX sockets, maybe that would be worth considering, thanks for pointing it out.

It should be noted that even though I intend to switch to FFI bindings (i.e. 'library' engine), the current behaviour with the binary engine will be preserved for backwards compatibility and to handle cases where native bindings are not possible for whatever reason.

so that updates from previous or older versions can just apply any migrations they're missing.

That might be as simple as using prisma migrate deploy. I think the difficulty with this is the potential performance issues as you would have to check if the database was up to date on every run, although this may not actually be that big of a deal in practice.

though the combination of being a library and requiring your own database is perhaps quite niche, as I can't think of a good concrete example.

In web applications I think it's quite common to use utility libraries that handle things like authorization, authentication etc for you. These libraries will most likely need to interface with a database.

A somewhat-related use case that may struggle is a framework that can be extended with plugins -- consider an application that ships its own copy of the prisma module containing its own models, but also allows for plugins that need their own models on top. Ideally we want those models to integrate with the main project's Prisma client, which might be tricky. Not something I need for the use case I'm imaginging, but perhaps another angle to consider. 🙂

Plugins would be difficult as I imagine it would be impossible to integrate them in a way that static type checkers can resolve. The alternative for the time being is for libraries to provide models in their documentation that users then have to copy into their own Prisma Schema file, this is what next-auth does.

A better solution would involve either prisma/prisma#2377 or #172. When #172 is implemented, libraries could provide models that users could then customise if they wanted to, e.g.

from mylibrary.schema import User

class CustomUser(User):
    points: int = Field(default=0)

[Terrance]

Yes, the database API is unauthenticated. I can see this being a security concern although I'm not clear on the situation you're describing?

In a more informal hosting setup, it may well just be multiple users sharing SSH access to a server. The database port wouldn't be accessible to site visitors as it's bound to localhost, but other users with access to the server would be able to poke it. In a setup like this, you'd either use a UNIX socket so that only you have permission to interact with the server, or you else you have to ensure your backend servers and APIs are authenticated.

[RobertCraigie]

I see, thanks for clarifying that.

I don't see that situation being that common but it is a valid concern and I'll speak to the Prisma team as well to see if this was a concern for them and if they have any additional information that would be helpful.