Skip to content

fix(video): resolve File API video understanding issue (#864) #916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

Adewale-1
Copy link

Description

This PR fixes issue #864 where video understanding fails for videos uploaded through the File API but works fine when the same video is provided as a YouTube URL.

Root Cause

After investigation, I found two issues contributing to the problem:

  1. File Processing Status: Files uploaded through the API need time to be processed before they can be analyzed. If a file is still in the PROCESSING state when it's used in a content generation request, the API rejects it.

  2. Error Handling: The error from the API when using a file in the PROCESSING state was not clearly indicating the issue, making it hard to diagnose.

Solution

The solution implemented in this PR:

  1. Added a _ensure_file_active function that:

    • Checks if a file is in the PROCESSING state
    • If it is, waits and polls the API until the file becomes ACTIVE
    • Has configurable retry count and delay between retries
  2. Enhanced the _process_contents_for_generation function to:

    • Recursively find all File objects in the content
    • Apply the _ensure_file_active function to each file
    • Handle files in different parts of the content structure
  3. Added a FileProcessingError class to provide better error messages when file processing fails

  4. Added comprehensive unit tests to verify the solution works correctly

Testing

The solution was tested with both:

  • YouTube video links (which continue to work as before)
  • Uploaded video files (which now wait for ACTIVE state before proceeding)

Note for Reviewers

This approach is more robust than simply adding a fixed delay because it:

  1. Only adds delays when necessary (when files are in PROCESSING state)
  2. Adapts to varying processing times by polling until the file is ready
  3. Has a configurable maximum wait time to prevent indefinite waiting

@Adewale-1 Adewale-1 changed the title Fix Issue #864 video file api fix(video): resolve File API video understanding issue (#864) Jun 2, 2025
@Adewale-1
Copy link
Author

Adewale-1 commented Jun 5, 2025

@hkt74 I've resolved the merge conflicts in models.py by:

  • Accepting the upstream implementation for enhance_prompt handling
  • Keeping both google_search_retrieval and url_context functionality
  • Incorporating both urlContextMetadata and avgLogprobs handling

All conflicts have been resolved and the PR is ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant