Skip to content

Commit

Permalink
Add maxcompute batch predictor (#626)
Browse files Browse the repository at this point in the history
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. Run unit tests and ensure that they are passing
2. If your change introduces any API changes, make sure to update the
e2e tests
3. Make sure documentation is updated for your PR!

-->
# Description
* This PR adds MaxCompute as a source and sink to batch prediction jobs.
This allows users to perform batch prediction on data stored in
MaxCompute and write the results back into MaxCompute.

# Modifications
* `python/batch-predictor` 
  * `merlinpyspark` package source code modified
  * Added missing proto spec for PredictionJob protobuf
  * Added MaxCompute Jar dependencies to base.Dockerfile
* `python/sdk`
* Modified python sdk to include MaxComputeSource and MaxComputeSink
classes
  * Modified swagger.yaml and regenerated python and go clients
  * Will add tests and readme in separate MR

# Tests
<!-- Besides the existing / updated automated tests, what specific
scenarios should be tested? Consider the backward compatibility of the
changes, whether corner cases are covered, etc. Please describe the
tests and check the ones that have been completed. Eg:
- [x] Deploying new and existing standard models
- [ ] Deploying PyFunc models
-->

# Checklist
- [x] Added PR label
- [ ] Added unit test, integration, and/or e2e tests
- [x] Tested locally
- [ ] Updated documentation
- [x] Update Swagger spec if the PR introduce API changes
- [x] Regenerated Golang and Python client if the PR introduces API
changes

# Release Notes
<!--
Does this PR introduce a user-facing change?
If no, just write "NONE" in the release-note block below.
If yes, a release note is required. Enter your extended release note in
the block below.
If the PR requires additional action from users switching to the new
release, include the string "action required".

For more information about release notes, see kubernetes' guide here:
http://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
- Support MaxCompute source and sink for batch prediction jobs
```
  • Loading branch information
shydefoo authored Feb 21, 2025
1 parent 1beb8b9 commit 75ead87
Show file tree
Hide file tree
Showing 27 changed files with 2,448 additions and 698 deletions.
268 changes: 268 additions & 0 deletions api/client/model_prediction_job_config_maxcompute_sink.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 75ead87

Please sign in to comment.