Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audio transcription support #781

Merged
merged 24 commits into from
Jun 4, 2024

Conversation

andrewfrench
Copy link
Member

@andrewfrench andrewfrench commented May 15, 2024

  • Introduces an AudioLoader, responsible for producing an AudioArtifact from binary audio data
  • Add AudioTranscriptionTask, accepting an AudioArtifact and responding with a TextArtifact containing the transcribed text
  • Add AudioTranscriptionClient allowing an Agent to generate a transcription from audio in memory or on the filesystem
  • Add OpenAiAudioTranscriptionDriver to support integration with OpenAI's speech-to-text models
  • Update structure configs to reference either OpenAiAudioTranscriptionDriver or DummyAudioTranscriptionDriver
  • Add tests and documentation for added constructs

馃摎 Documentation preview 馃摎: https://griptape--781.org.readthedocs.build//781/

@andrewfrench andrewfrench marked this pull request as draft May 15, 2024 00:05
@andrewfrench andrewfrench changed the title French/240514/transcription transcription Audio transcription support May 15, 2024
@andrewfrench andrewfrench force-pushed the french/240514/transcription--transcription branch from cfa71d8 to a4d2383 Compare May 15, 2024 22:19
@andrewfrench andrewfrench marked this pull request as ready for review May 21, 2024 22:58
@andrewfrench andrewfrench force-pushed the french/240514/transcription--transcription branch from 6d9af68 to c055935 Compare May 21, 2024 23:07
@andrewfrench andrewfrench requested a review from a team May 22, 2024 22:08
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outside the scope of this PR but I'm starting to doubt our pattern of creating Engines that are just thin wrappers over Drivers. Engines should augment 1 or more Drivers for a specific use-case that a single Driver cannot do on its' own.

What do you think?
CC @vasinov

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly agree, this layer is way too thin, but I also did not want to create a weird single audio-to-text modality that breaks our established pattern. Optimistically, they're a layer where we can add additional functionality. For now, they're mostly an additional initialization step.

Copy link
Member

@collindutter collindutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, just naming comments

griptape/tools/transcription_client/tool.py Outdated Show resolved Hide resolved


@define
class BaseAudioTranscriptionDriver(SerializableMixin, ExponentialBackoffMixin, ABC):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we name these Drivers BaseSpeechToTextDriver for consistency with the inverse Drivers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've steered a bit too far in the direction of naming drivers based on their artifact interfaces. I think the specificity of this name is helpful, what do you think about adding a similarly specific name to the BaseTextToSpeechDriver? BaseSpeechGenerationDriver?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thinking about it more I think this current convention is the more "correct" one.

Down to do a rename of BaseTextToSpeechDriver (though maybe in a separate PR), what do you think about BaseAudioGenerationDriver? This may extend beyond speech in the future?

collindutter
collindutter previously approved these changes May 31, 2024
collindutter
collindutter previously approved these changes Jun 3, 2024
Copy link
Member

@collindutter collindutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

collindutter
collindutter previously approved these changes Jun 3, 2024
@andrewfrench andrewfrench merged commit ff008c0 into dev Jun 4, 2024
11 checks passed
@andrewfrench andrewfrench deleted the french/240514/transcription--transcription branch June 4, 2024 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants