Enhance cli index command #391
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Integrates the JobManager into the dump method to leverage `dump_src` which is the same as the hub server endpoint. Adds support for the parallel uploader, as the previous iteration couldn't support plugins with a parallel uploader Update the index command to build a mapping if one isn't found from the uploader stage (likely because one doesn't exist or isn't hard-coded for the plugin)
|
Example usage for elasticsearch mapping inference (for reference I've never even used the command-line tooling on this plugin before). |
ctrl-schaff
requested review from
newgene,
everaldorodrigo,
jal347 and
DylanWelzel
…hings.api into enhance-cli-index-command
|
@newgene alright I think everything has been cleaned up from our last discussion. We now only have only the dataplugin command so try evaluating it as both a singular data-plugin and hub depending on the execution folder |
New layout for the |
…hings.api into enhance-cli-index-command
shuchenliu
pushed a commit
that referenced
this pull request
May 29, 2025
* Change build command name to index * Merge the manifest, dataplugin, and hub cli * Move the number truncation function to doc_inspect * Leverage the hub JobManager in the cli tooling * Add JobManager settings for cli tooling * Operations update ... Integrates the JobManager into the dump method to leverage `dump_src` which is the same as the hub server endpoint. Adds support for the parallel uploader, as the previous iteration couldn't support plugins with a parallel uploader Update the index command to build a mapping if one isn't found from the uploader stage (likely because one doesn't exist or isn't hard-coded for the plugin) * Breakup the `process_inspect` command * Add address and get_conn to the sqlite3 client * Fix the methods calls for inspection * Update the empty metadata check * Add console logging for index command * Update the docstrings and help messages for commands * Change the default log level from INFO to WARNING in cli * Improve consistency across CLI operation arguments * Change batch_limit default size * Improve batch_limit help text * Normalize the bounding box for the output messages * Remove the CLIJobManager in-place for the JobManager * Update the CLIAssistant for handling directories * Fix default uploader database address assignment * Update our dataplugin help message * Combine the validate and schema commands * Remove the cli sync/async function runner * Create async loop for manifest validation * Fix plugin_name generation with dump command * Push plugin_name resolution entirely to CLIAssistant * Add data_directory when building managers in CLIAssistant * Suppress import error for aiocron in builder * Simplify the BaseManager constructor arguments --------- Co-authored-by: jschaff <[email protected]> Co-authored-by: Chunlei Wu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates our CLI tooling index ability to automatically inspect mapping for plugins that don't have them. Allows for the creation of a lot more index testing for the pending.api and other plugins that don't hard-code the elasticsearch mapping