Transcripter is a tool for indexing and searching YouTube video transcripts. It makes use of the YouTube Data API to fetch video details and transcripts, and collates and stores the transcript chunks in Redis for efficient full-text search.
Transcripter's main purpose is to provide a simple way to add transcript-search functionality to your application. Inspired by (but unrelated to) https://ippsec.rocks/.
transcripter/
: Main packagecore/
: Core functionalityredis_manager.py
: Manages Redis operationsyoutube_manager.py
: Handles YouTube API interactions
services/
: Service layerindexing_service.py
: Manages the indexing processsearch_service.py
: Manages the search process
config.py
: Configuration managementlogs.py
: Logging management
examples/
: Example applicationsbasic/
: Basic exampleapp.py
: Simple application demonstrating usage
index-config.yml
: Configuration file for indexing. Used by the example app.docker-compose.yml
: Docker compose file for the example app.
- The
IndexingService
periodically fetches video details and transcripts from configured YouTube playlists, channels, or individual videos. - Transcripts are processed and stored in Redis using the
RedisManager
. - The
YouTubeManager
handles interactions with the YouTube API. - Indexed data can be searched efficiently using Redis's full-text search capabilities.
You can install from PyPI:
pip install transcripter
make install
or
pip3 install git+https://github.com/0xRy4n/transcripter
Note: Transcripter requires a Redis Stack server to run. It's recommended to use the official Docker image like so:
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`
You can run the Redis Stack either locally or remote, depending on your preference.
Set the following environment variables:
YOUTUBE_API_KEY
: Your YouTube API keyREDIS_HOST
: Redis server hostREDIS_PORT
: Redis server portREDIS_PASSWORD
: Redis server password (if applicable)REDIS_INDEX
: Name of the Redis index to useTRANSCRIPTER_LOG_LEVEL
: Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
Configure the indexing process in your application by specifying a indexing-config.yml
file like so:
# Change me
sources:
playlists:
# playlist id
- PLlkHl5i0GaAlBvAhi9R7S4vQ0IhR3ePCU
channels:
- # channel id
videos:
- # video id
indexing:
interval: 3600 # Reindex every hour
You can then pass the path to this file in to the Config
object:
config = Config(config_path='path/to/indexing-config.yml')
By default, it will search for a file called indexing-config.yml
in the root directory of the repository (one directory up from the transcripter
directory).
Alternatively, you can simply instantiate Config
without any arguments, and set each property directly:
from transcripter.config import Config
config = Config()
config.playlists = ['playlist_id1', 'playlist_id2']
config.channels = ['channel_id1', 'channel_id2']
config.videos = ['video_id1', 'video_id2']
config.indexing_interval = 3600 # Indexing interval in seconds
from transcripter.services.indexing_service import IndexingService
indexing_service = IndexingService(config)
Indexing your videos is easy.
indexing_service.index_all()
Once a video has been indexed, it will not be indexed again, and it's transcript won't be pulled on successive runs.
You'll likely want to run the indexing service periodically as a background job. It is up to you to decide how you would like to implement this.
You can search the indexed data using the SearchService
class.
from transcripter.services.search_service import SearchService
search_service = SearchService(config)
You can then search for videos by title:
search_service.search('search query')
It will return a list of stored transcript chunks in the following format:
[
{
"snippet": "directory don't worry about it too much it's not going to be on the test there's not going to be a test but if there were",
"start_time": 300.56,
"timecode": "00:05:00",
"video_id": "_haJQY-Y70E",
"video_title": "Linux Structure and Commands - Intro to Hacking w/ HTB Academy #4"
},
{
"snippet": "directory don't worry about it too much it's not going to be on the test there's not going to be a test but if there were",
"start_time": 300.56,
"timecode": "00:05:00",
"video_id": "_haJQY-Y70E",
"video_title": "Linux Structure and Commands - Intro to Hacking w/ HTB Academy #4"
}
]
A fully functional example web app is included in the examples/basic
directory. It is a simple web app that allows you to search for videos by title. It is built with Flask and uses Bootstrap for the frontend.
You can change the index-config.yml
file in the root of the repository to configure the playlists, channels, and videos you want to index.
Additionally, you can opt to either create a .env
file in the root of the repository and set the environment variables directly, or set them in your operating system's environment variables.
The example app is containerized with Docker, making it simple to run:
docker compose up
Alternatively, you can run the app without Docker by installing the dependencies and running the app.py
directly:
First you'll need to run Redis Stack in a docker container:
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest
Then install the dependencies and run the app:
pip3 install -e .
python3 examples/basic/app.py