-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Support for Multiple Data Sources #206
Comments
Analysis from the plugin side: Problem at hand: 1. Note: PebbloSafeLoader instantiation --> app:loader discovery, and it's .load() --> classifies the loaded data. In 1st case we will get to know the first loader and then the 2nd loader, and then both loaders will be loaded (and their data classified) one by one (generating reports twice, on every loading_end = true from both the loaders). As of today, the pebblo server will generate the report with every new loading_end = true. the 2ne case mentioned above is a kind of race condition where we won't be able to get a trigger point to make sure the loading is actually over in an RAG app. Let's wait for a few seconds before generating the report, a new loader discovery might just kick in. Also, we generate(overwrite) a new report with every new loading_end=True |
In order to support multiple data sources, we need to update out folder structure within scratch space as follows:
|
Pebblo (as of 0.1.12 version) supports single data source. Having support multiple data sources within single RAG application would be a good feature.
Description:
When I have multiple data sources to be used in my app, I should be able to see all those data sources and their details in the pebblo report.
As part of this feature, following changes would need to be done in the report:
The text was updated successfully, but these errors were encountered: