fix(oauth-ingestion): Fix 401 error (invalid credentials) during OAuth ingestion

[kalpadhwaryu]

Description

A 401 error (invalid credentials) was encountered when the Google OAuth access token expired during OAuth ingestion. Now, upon encountering this error, new credentials are generated, and the oauthClient uses those newly generated credentials.

Testing

Tested locally.

[gemini-code-assist]

Hello @kalpadhwaryu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request addresses a 401 error (invalid credentials) encountered during Google OAuth ingestion when the access token expires. To resolve this, the code now generates new credentials upon encountering this error and uses them to re-initialize the oauthClient. The changes involve modifying the OAuthCallback function to set an expiration time for the SaaS queue job and updating the handleGoogleOAuthIngestion function to refresh the access token when a 401 error occurs. Additionally, the retryWithBackoff utility function is modified to handle 401 errors and refresh the OAuth access token. The Google API calls are also updated to pass the oauth2Client to the retryWithBackoff function.

Highlights

• Error Handling: Implements a mechanism to handle 401 errors during OAuth ingestion by refreshing the access token.
• OAuth Client Refresh: Modifies the retryWithBackoff function to refresh the OAuth access token when a 401 error is encountered.
• Function Signature Updates: Updates function signatures to include the GoogleClient (oauth2Client) for use in retryWithBackoff.
• SaaS Queue Job Expiration: Sets an expiration time for the SaaS queue job to avoid indefinite retries.

Changelog

Click here to see the changelog

• server/api/oauth.ts
  • Added expireInHours option when sending the SaaS queue job to expire the job after 23 hours.
• server/integrations/google/gmail-worker.ts
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail user profile (line 113).
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail messages list (line 132).
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail message (line 151).
  • Modified parseMail to accept client as an argument (line 228).
  • Modified parseMail to pass the client to getGmailAttachmentChunks (line 317).
• server/integrations/google/gmail/index.ts
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail user profile (line 49).
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail messages list (line 68).
  • Modified handleGmailIngestion to pass the client to retryWithBackoff when fetching Gmail message (line 87).
  • Modified parseMail to accept client as an argument (line 153).
  • Modified parseMail to pass the client to getGmailAttachmentChunks (line 248).
• server/integrations/google/index.ts
  • Added imports for oauthProviders, selectConnectorSchema, type SelectOAuthProvider from '@/db/schema' (lines 43-47).
  • Added imports for NoOauthConnectorFound, MissingOauthConnectorCredentialsError, and FetchProviderFailed from '@/errors' (lines 86-88).
  • Modified handleGoogleOAuthIngestion to fetch OAuth provider details and set up the oauth2Client with clientId, clientSecret, and redirectUri (lines 581-623).
  • Modified handleGoogleOAuthIngestion to set the refresh_token when setting credentials (line 640).
  • Modified getPresentationToBeIngested to pass the client to retryWithBackoff (line 1053).
  • Modified getAllSheetsFromSpreadSheet to accept client as an argument (line 1310).
  • Modified getAllSheetsFromSpreadSheet to pass the client to retryWithBackoff (line 1333).
  • Modified getSpreadsheet to accept client as an argument (line 1369).
  • Modified getSpreadsheet to pass the client to retryWithBackoff (line 1375).
  • Modified getSheetsListFromOneSpreadsheet to pass the client to getSpreadsheet and getAllSheetsFromSpreadSheet (lines 1455, 1464).
  • Modified downloadPDF to accept client as an argument (line 1613).
  • Modified downloadPDF to pass the client to retryWithBackoff (line 1625).
  • Modified googlePDFsVespa to pass the client to downloadPDF (line 1683).
  • Modified listAllContacts to pass the client to retryWithBackoff (line 1832).
  • Modified listAllContacts to pass the client to retryWithBackoff (line 1858).
  • Modified listFiles to pass the client to retryWithBackoff (line 2032).
  • Modified googleDocsVespa to pass the client to retryWithBackoff (line 2073).
• server/integrations/google/sync.ts
  • Modified getSpreadsheet to pass the client to getSpreadsheet (line 115).
  • Modified getDriveChanges to accept oauth2Client as an argument (line 403).
  • Modified getDriveChanges to pass the oauth2Client to retryWithBackoff (line 413).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to getDriveChanges (line 468).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to retryWithBackoff (line 519).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to syncContacts (line 529).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to retryWithBackoff (line 563).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to handleGmailChanges (line 681).
  • Modified handleGoogleOAuthChanges to pass the oauth2Client to handleGoogleCalendarEventsChanges (line 776).
  • Modified handleGoogleCalendarEventsChanges to accept oauth2Client as an argument (line 905).
  • Modified handleGoogleCalendarEventsChanges to pass the oauth2Client to retryWithBackoff (line 927).
  • Modified handleGmailChanges to accept client as an argument (line 1106).
  • Modified handleGmailChanges to pass the client to retryWithBackoff (line 1128).
  • Modified handleGmailChanges to pass the client to parseMail (line 1164).
  • Modified syncContacts to accept oauth2Client as an argument (line 1290).
  • Modified syncContacts to pass the oauth2Client to retryWithBackoff (line 1313).
• server/integrations/google/utils.ts
  • Modified getFile to pass the client to retryWithBackoff (line 104).
  • Modified getFileContent to pass the client to retryWithBackoff (line 141).
  • Modified getPDFContent to pass the client to downloadPDF (line 205).
• server/integrations/google/worker-utils.ts
  • Modified getGmailAttachmentChunks to accept client as an argument (line 47).
  • Modified getGmailAttachmentChunks to pass the client to retryWithBackoff (line 72).
• server/utils.ts
  • Modified retryWithBackoff to accept oauth2Client as an argument (line 109).
  • Modified retryWithBackoff to refresh the OAuth access token when a 401 error is encountered (lines 139-142).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Did you know?

The term 'computer bug' originated in 1947 when a moth was found trapped in a relay of the Harvard Mark II computer, causing it to malfunction.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request addresses a 401 error encountered during OAuth ingestion by generating new credentials upon token expiration. The changes involve modifying several files to pass the client object to various Google API calls, enabling token refreshing. Overall, the changes seem reasonable and address the described issue. However, there are a few areas that could be improved for clarity and maintainability.

Summary of Findings

• Missing Error Handling: The code adds a check for providers.length but only logs an error and throws an exception. It would be beneficial to handle this case more gracefully, potentially by retrying or notifying the user.
• Inconsistent Parameter Passing: The client parameter is added to many functions, but it's not always clear why it's needed in each specific case. Adding comments to explain the purpose of the client parameter in each function would improve readability.
• Job Expiration: The expireInHours option is set to 23 hours. Consider making this configurable or using a constant to improve maintainability.

Merge Readiness

The pull request addresses an important issue and seems to resolve the 401 error. However, the missing error handling and inconsistent parameter passing should be addressed before merging. I am unable to directly approve this pull request, and recommend that others review and approve this code before merging. At a minimum, the high severity issues should be addressed before merging.