Streaming chat completion demo

[timathom]

It would be cool to have a Fore demo that featured integration with a GenAI streaming API (OpenAI, Claude, etc.). For example, OpenAI has its Node.js API. The goal would be to allow the user to maintain a chat session within a Fore app that could process structured API responses as Fore instance data (e.g., ask the AI model for a JSON response that will populate a Fore instance).

[JoernT]

@timathom while i get the idea and certainly in general this would be doable with Fore it's more a practical problem as such services like the OpenAI API usually need an authorization token to speak to them. The Fore demos are running on github pages and i see no immediate way to make such a demo public without exposing the auth token opening the door for abuse.

Further - we would need to make the use case more specific - do you have a more specific vision of how such a chat would look like? A chat session usually just returns text. We can of course try to parse a response and see if it's parseable as json and then push that into an instance but what would you like to do with that instance then?

In our applications of Fore we of course use json instances and speak to OpenAPI endpoints all the time but these applications then involve a server-side part to work against which in our case is usually an eXist-db app exposing an OpenAPI. Such more complex demos however can't be directly realized in a github environment and we would need to setup and run our own servers for that.

[timathom]

@JoernT Right, I was thinking that this could be a demo that might have a placeholder page on the website, but a user would need to run it locally in order to supply the authentication credentials.

[timathom]

It would be up to the user to decide what they wanted to do with the instance. My immediate use case is for data remediation. I have a prompt that I would like to set and then initiate a chat session in which data gets passed in for review and a human user evaluates the results for accuracy.

Initial prompt:

As a skilled geographer, assist in rectifying errors within a geographical dataset. I will present a set of locations for assessment. The locations represent a chain of transitive relations, where the first place is asserted to be contained within the subsequent places. The names should represent known and verified geographic entities (e.g., city, river, region)--whether past or present. Pay heed to the hierarchical relationships; if a location is part of another (e.g., city within a country within a continent), acknowledge this. Respond 'X' if a term is a concept or demographic term instead of a place (e.g., 'accounting', 'aerial photogrammetry', 'publication place', 'Afrikaners'). For mismatched relationships, reply 'F'; for matched relationships, 'T'. Include a brief rationale. JSON format required, with two keys: 'answer' and 'explanation'. Provide the correct relationship based on the given context. Additionally, ensure that the locations correspond to the appropriate level of granularity. For example, if the first location is a city, consider whether the second location should be a country, state, or county, depending on the context. Evaluate the following list of places:

Example data (what gets passed in for each turn):

1. Ontario is part of Malheur. 
2. Malheur is part of Oregon. 
3. Oregon is part of United States. 

Response (from Claude 3):

{
  "answer": "T",
  "explanation": "The hierarchical relationships are correct. Ontario is a city in Malheur County, Oregon, which is a state in the United States. The locations correspond to the appropriate levels of granularity, progressing from city to county to state to country."
}

This data would be pushed to a Fore instance, and the front-end user would check a box or something to indicate whether the assessment was correct. The flagged data would then get persisted to a back-end database (BaseX, etc.).

[JoernT]

wow - i call that a prompt ;) Thanks for the idea. I just built a chat interface anyway for developing the fx-connection element.

Maybe we can expand from there. Shouldn't actually be hard. It would however the great to have a few more data examples of questions and answers to spice the whole thing up a bit. - do you @timathom eventually have some example data to share?