avatar/video/create-avatar-video.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use the TTS Avatar Service to Create Engaging Videos\n",
    "We guide you through creating avatar-based videos using the Azure TTS Avatar API and your own scripts. You can use these videos for various purposes, such as education or communication with customers and partners.\n",
    "\n",
    "First, we'll walk you through creating the script for your avatar using the 'Audio Content Creation' tool. Next, we'll use the Azure TTS Avatar service to turn your script into a video with a custom avatar.\n",
    "\n",
    "We'll then show you how to combine the avatar video with a content image or video created using PowerPoint. To finalize your video, you can use the FFMpeg command line tool for simplicity or the ClipChamp video editor for more advanced options.  \n",
    "\n",
    "__Additional resources:__\n",
    "\n",
    "- Available neural voices and languages: https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support?tabs=tts\n",
    "- SSML overview: https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-synthesis-markup\n",
    "- Export PowerPoint slides as 1920x1080 resolution PNGs: https://youtu.be/Cv7vGce25rs\n",
    "- FFmpeg command line tool for post processing of videos: https://ffmpeg.org/download.html\n",
    "- Free video editing tool Clipchamp: https://clipchamp.com/\n",
    "\n",
    "__Prerequisites:__  \n",
    "You need an Azure subscription with a Speech resource to use the service. Add the `SPEECH_SERVICEE_REGION` and `SPEECH_SERVICE_API_KEY` to the `.env-avatar-video` file in this folder. You can find the values in the Azure portal under 'Keys and Endpoint' in the Speech resource. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install python-dotenv --quiet"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import time\n",
    "import os\n",
    "from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not load_dotenv('./.env-avatar-video'):\n",
    "    raise Exception(\"env file not found\")\n",
    "\n",
    "service_region = os.getenv(\"SPEECH_SERVICE_REGION\")\n",
    "subscription_key = os.getenv(\"SPEECH_SERVICE_API_KEY\")\n",
    "url_base = f\"https://{service_region}.api.cognitive.microsoft.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_folder = './my-project'  # your project folder\n",
    "os.makedirs(project_folder, exist_ok=True)\n",
    "\n",
    "texttype = 'Plaintext'  # SSML or Plaintext\n",
    "ssml_path = os.path.join(project_folder, 'ssml.txt')  # if your avatar text input is in SSML format\n",
    "plaintext_path = os.path.join(project_folder, 'plaintext.txt')  # if your avatar text input is plaintext format\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper functions\n",
    "\n",
    "def download_file(url, local_path):\n",
    "    \"\"\"\n",
    "    Download a file from a given URL to a local path. This function streams the file from the URL and writes it in chunks to the local\n",
    "    file system. This allows it to handle large files that might not fit in memory.\n",
    "\n",
    "    Parameters:\n",
    "    url (str): The URL of the file to download.\n",
    "    local_path (str): The local path where the file should be saved.\n",
    "\n",
    "    Returns:\n",
    "    str: The local path to the downloaded file.\n",
    "    \"\"\"\n",
    "    with requests.get(url, stream=True) as r:\n",
    "        r.raise_for_status()\n",
    "\n",
    "        # Extract filename from URL\n",
    "        filename = url.split(\"/\")[-1].split(\"?\")[0]\n",
    "\n",
    "        local_filename = os.path.join(local_path, filename)\n",
    "\n",
    "        with open(local_filename, 'wb') as f:\n",
    "            for chunk in r.iter_content(chunk_size=8192): \n",
    "                if chunk:\n",
    "                    f.write(chunk)\n",
    "\n",
    "    return local_filename"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provide text input for the avatar\n",
    "Your avatar's speaking text can be provided as plain text file `plaintext.txt` or in the Spech Synthesis Markup Language (SSML) format using the file `ssml.txt`. SSML provides more flexibility to adjust neural voices, control pronounciation and add gestures to your avatar. You can use the __Audio Content Generation__ option in the [Speech Studio](https://speech.microsoft.com/portal) to easily generate SSML for various neural voices.\n",
    "\n",
    "<img src=\"./media/audio-content-creation.png\" alt=\"drawing\" style=\"width:800px;\"/>\n",
    "\n",
    "Switch to the SSML view when you are satisfied with the results and copy the SSML content into the `ssml.txt` file of your project folder before you execute the next cell.\n",
    "\n",
    "## Working with SSML directly\n",
    "Alternatvely, you can create or edit your SSML file manually. Here is an example that illustrates a few customization options. Select the `ssml-example` project folder to try it out.\n",
    "\n",
    "```xml\n",
    "<speak xmlns=\"http://www.w3.org/2001/10/synthesis\" xmlns:mstts=\"http://www.w3.org/2001/mstts\" xmlns:emo=\"http://www.w3.org/2009/10/emotionml\" version=\"1.0\" xml:lang=\"en-US\">\n",
    "  <voice name=\"en-US-JennyMultilingualNeural\">\n",
    "    This example shows what you can do with SSML to let your avatar speak in various languages and how to add effects. \n",
    "    We are using the voice Jenny Multilangual which is able to speak in several languages.\n",
    "    <lang xml:lang=\"fr-FR\">Voici le talent linguistique Jenny parlant français. Ma voix est bien reconnaissable.</lang>\n",
    "  </voice>\n",
    "  <voice name=\"en-US-JennyNeural\">\n",
    "    I can speak in 15 different styles.\n",
    "    <s />\n",
    "    <mstts:express-as style=\"shouting\">If you want a shouting Avatar, no problem for me!</mstts:express-as>\n",
    "    <s />\n",
    "    <mstts:express-as style=\"whispering\">Or what about some whispered dialog?</mstts:express-as>\n",
    "    You can insert a pause <break strength=\"medium\" /> in my speech with a break tag.\n",
    "    Use a phoneme to pronounce specific words correctly like OpenAI's <phoneme alphabet=\"ipa\" ph=\"ˈdɑli\">DALLE</phoneme> model. Without the phoneme, the neural voice would say DALLE, which is not correct.\n",
    "    Adjust the speaking speed <prosody rate=\"+30.00%\">so I can talk really fast if that's what you want</prosody><prosody rate=\"-40.00%\"> or talk quite slowly.</prosody>\n",
    "    <prosody pitch=\"+10.00%\">Feel free to adjust the pitch for more highness of the sound.</prosody><prosody pitch=\"-20.00%\"> Or the opposite if you like that better.</prosody>\n",
    "    <prosody volume=\"-80.00%\">Finally, here is how you reduce the volume.</prosody>\n",
    "    <prosody volume=\"+40.00%\">Or make me speak louder.</prosody>\n",
    "    We hope that these examples help you to customize your avatar's communication for more engaging experiences.\n",
    "    <s />\n",
    "    <mstts:express-as style=\"friendly\">Have fun!</mstts:express-as>\n",
    "    <s />\n",
    "  </voice>\n",
    "</speak>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load speaking script\n",
    "if texttype == 'SSML':\n",
    "    with open(ssml_path, 'r') as file:\n",
    "        content = file.read()\n",
    "elif texttype == 'Plaintext':\n",
    "    with open(plaintext_path, 'r') as file:\n",
    "        content = file.read()\n",
    "else:\n",
    "    print(f'Error: Texttype needs to be either \"ssml\" or \"Plaintext\". Got {texttype} instead.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate avatar video\n",
    "You can specify the avatar character, style and further settings below. Note that video synthesis will take a few minutes depending on the length of your talking script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# True for custom avatar and False for Prebuilt Avatars\n",
    "\n",
    "customized = False\n",
    "\n",
    "\n",
    "payload = {\n",
    "    'synthesisConfig': {\n",
    "            \"voice\": 'en-US-JennyMultilingualNeural',\n",
    "        },\n",
    "        # Replace with your custom voice name and deployment ID if you want to use custom voice.\n",
    "        # Multiple voices are supported, the mixture of custom voices and platform voices is allowed.\n",
    "        # Invalid voice name or deployment ID will be rejected.\n",
    "        'customVoices': {\n",
    "            # \"YOUR_CUSTOM_VOICE_NAME\": \"YOUR_CUSTOM_VOICE_ID\"\n",
    "        },\n",
    "        \"inputKind\": texttype,\n",
    "        \"inputs\": [\n",
    "            {\n",
    "                \"content\": content,\n",
    "            },\n",
    "        ],\n",
    "}\n",
    "\n",
    "\n",
    "if customized:\n",
    "    payload['avatarConfig'] = {\n",
    "        \"customized\": customized,\n",
    "        \"talkingAvatarCharacter\": 'Lisa-casual-sitting',  # talking avatar character\n",
    "        \"videoFormat\": \"mp4\",  # mp4 or webm, webm is required for transparent background\n",
    "        \"videoCodec\": \"h264\",  # hevc, h264 or vp9, vp9 is required for transparent background; default is hevc\n",
    "        \"subtitleType\": \"soft_embedded\",\n",
    "        \"backgroundColor\": \"#FFFFFFFF\", # background color in RGBA format, default is white; can be set to 'transparent' for transparent background\n",
    "        # \"backgroundImage\": \"https://samples-files.com/samples/Images/jpg/1920-1080-sample.jpg\", # background image URL, only support https, either backgroundImage or backgroundColor can be set\n",
    "    }\n",
    "else:\n",
    "    payload['avatarConfig'] = {\n",
    "        \"customized\": customized,\n",
    "        \"talkingAvatarCharacter\": 'Lisa',  # talking avatar character\n",
    "        \"talkingAvatarStyle\": 'casual-sitting',  # talking avatar style, required for prebuilt avatar, optional for custom avatar\n",
    "        \"videoFormat\": \"mp4\",  # mp4 or webm, webm is required for transparent background\n",
    "        \"videoCodec\": \"h264\",  # hevc, h264 or vp9, vp9 is required for transparent background; default is hevc\n",
    "        \"subtitleType\": \"soft_embedded\",\n",
    "        \"backgroundColor\": \"#FFFFFFFF\", # background color in RGBA format, default is white; can be set to 'transparent' for transparent background\n",
    "        # \"backgroundImage\": \"https://samples-files.com/samples/Images/jpg/1920-1080-sample.jpg\", # background image URL, only support https, either backgroundImage or backgroundColor can be set\n",
    "    }\n",
    "\n",
    "\n",
    "payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import uuid\n",
    "import requests\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO,  # set to logging.DEBUG for verbose output\n",
    "        format=\"[%(asctime)s] %(message)s\", datefmt=\"%m/%d/%Y %I:%M:%S %p %Z\")\n",
    "logger = logging.getLogger(__name__)\n",
    "\n",
    "\n",
    "\n",
    "API_VERSION = \"2024-04-15-preview\"\n",
    "\n",
    "\n",
    "def _create_job_id():\n",
    "    # the job ID must be unique in current speech resource\n",
    "    # you can use a GUID or a self-increasing number\n",
    "    return uuid.uuid4()\n",
    "\n",
    "\n",
    "def _authenticate():\n",
    "    return {'Ocp-Apim-Subscription-Key': subscription_key}\n",
    "\n",
    "\n",
    "def submit_synthesis(job_id: str):\n",
    "    \"\"\"\n",
    "    Submits a batch avatar synthesis job to the specified URL.\n",
    "    Args:\n",
    "        job_id (str): The ID of the job to be submitted.\n",
    "    Returns:\n",
    "        bool: True if the job was submitted successfully, False otherwise.\n",
    "    Logs:\n",
    "        - Info: When the job is submitted successfully, including the job ID.\n",
    "        - Error: When the job submission fails, including the status code and error message.\n",
    "    \"\"\"\n",
    "    url = f'{url_base}/avatar/batchsyntheses/{job_id}?api-version={API_VERSION}'\n",
    "    header = {\n",
    "        'Content-Type': 'application/json'\n",
    "    }\n",
    "    header.update(_authenticate())\n",
    "    \n",
    "\n",
    "    response = requests.put(url, json.dumps(payload), headers=header)\n",
    "    if response.status_code < 400:\n",
    "        logger.info('Batch avatar synthesis job submitted successfully')\n",
    "        logger.info(f'Job ID: {response.json()[\"id\"]}')\n",
    "        return True\n",
    "    else:\n",
    "        logger.error(f'Failed to submit batch avatar synthesis job: [{response.status_code}], {response.text}')\n",
    "\n",
    "\n",
    "def get_synthesis(job_id):\n",
    "    \"\"\"\n",
    "    Retrieve the status and result of a batch synthesis job.\n",
    "\n",
    "    This function sends a GET request to the batch synthesis endpoint to check the status of a job.\n",
    "    If the job is successful, it downloads the resulting video file to the specified project folder.\n",
    "\n",
    "    Args:\n",
    "        job_id (str): The unique identifier of the batch synthesis job.\n",
    "\n",
    "    Returns:\n",
    "        str: The status of the batch synthesis job. Possible values include 'Succeeded', 'Failed', etc.\n",
    "\n",
    "    Logs:\n",
    "        - Debug: When the batch synthesis job is successfully retrieved.\n",
    "        - Info: When the batch synthesis job succeeds and the video is downloaded.\n",
    "        - Error: When the request to retrieve the batch synthesis job fails.\n",
    "    \"\"\"\n",
    "    url = f'{url_base}/avatar/batchsyntheses/{job_id}?api-version={API_VERSION}'\n",
    "    header = _authenticate()\n",
    "\n",
    "    response = requests.get(url, headers=header)\n",
    "    if response.status_code < 400:\n",
    "        logger.debug('Get batch synthesis job successfully')\n",
    "        logger.debug(response.json())\n",
    "        if response.json()['status'] == 'Succeeded':\n",
    "            logger.info(f'Batch synthesis job succeeded, download URL: {response.json()[\"outputs\"][\"result\"]}')\n",
    "            download_file(response.json()[\"outputs\"][\"result\"], project_folder)\n",
    "            logger.info(f'Video downloaded to {project_folder}')\n",
    "        return response.json()['status']\n",
    "    else:\n",
    "        logger.error(f'Failed to get batch synthesis job: {response.text}')\n",
    "\n",
    "\n",
    "def list_synthesis_jobs(skip: int = 0, max_page_size: int = 100):\n",
    "    \"\"\"List all batch synthesis jobs in the subscription\"\"\"\n",
    "    url = f'{url_base}/avatar/batchsyntheses?api-version={API_VERSION}&skip={skip}&maxpagesize={max_page_size}'\n",
    "    header = _authenticate()\n",
    "\n",
    "    response = requests.get(url, headers=header)\n",
    "    if response.status_code < 400:\n",
    "        logger.info(f'List batch synthesis jobs successfully, got {len(response.json()[\"values\"])} jobs')\n",
    "        logger.info(response.json())\n",
    "    else:\n",
    "        logger.error(f'Failed to list batch synthesis jobs: {response.text}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_id = _create_job_id()\n",
    "if submit_synthesis(job_id):\n",
    "    while True:\n",
    "        status = get_synthesis(job_id)\n",
    "        if status == 'Succeeded':\n",
    "            logger.info('batch avatar synthesis job succeeded')\n",
    "            break\n",
    "        elif status == 'Failed':\n",
    "            logger.error('batch avatar synthesis job failed')\n",
    "            break\n",
    "        else:\n",
    "            logger.info(f'batch avatar synthesis job is still running, status [{status}]')\n",
    "            time.sleep(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add content to your avatar video\n",
    "In addition to the avatar video, we require another asset to represent the content we wish to display alongside the avatar. This could be either high-resolution images or a separate video, depending on your specific needs. \n",
    "\n",
    "<img src=\"./media/content-avatar.png\" alt=\"drawing\" style=\"width:1200px;\"/>\n",
    "\n",
    "PowerPoint is an excellent tool for creating this supplemental content, and here's how you can do it:\n",
    "\n",
    "1. __Export slides as high-resolution images__: PowerPoint allows you to export slides as images. For optimal results, we recommend exporting the slides as 1920x1080 PNG files. You can follow this brief tutorial on how to do so: [Export slides as 1920x1080 PNGs](https://youtu.be/Cv7vGce25rs).\n",
    "\n",
    "2. Alternatively __export the entire presentation as an MP4 video__: If you prefer to use a video instead of static images, PowerPoint offers the option to export your whole animated presentation as an MP4 file.  Here's how you can do it:  \n",
    "Go to __File__ > __Export__ > __Create a Video__ in PowerPoint. Use slide transition and animation durations to adjust timings. The best way to align avatar and content video timing is to use a video editor (Option 2)\n",
    "\n",
    "> **Tip:** For smooth synchronization, match the PowerPoint slide transition durations with the timestamps from your avatar video. This preemptive alignment minimizes the need for later timing adjustments.\n",
    "\n",
    "Lastly, there are various methods available for integrating the avatar video and the content assets to produce your final video output.\n",
    "\n",
    "### Option 1: Use the FFmpeg command line \n",
    "\n",
    "Install FFmpeg on a Linux system (Check out the [FFMpeg website](https://ffmpeg.org/download.html) for Windows and Mac options):\n",
    "```bash\n",
    "sudo apt update\n",
    "sudo apt install ffmpeg\n",
    "```\n",
    "#### Examples\n",
    "Start your terminal and navigate to the project folder.\n",
    "\n",
    ">Add a content image as background to transparent avatar video:\n",
    ">\n",
    ">```bash\n",
    ">ffmpeg -i content.png -vcodec libvpx-vp9 -i 0001.webm -filter_complex \"overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2\" -map 1:a output.mp4\n",
    ">```\n",
    "\n",
    ">Add a content video as background to transparent avatar video:\n",
    ">\n",
    ">```bash\n",
    ">ffmpeg -i content.mp4 -vcodec libvpx-vp9 -i 0001.webm -filter_complex \"overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2\" -map 1:a output.mp4\n",
    ">```\n",
    "\n",
    ">Add a content video and an audio background music file with reduced volume:\n",
    ">```bash\n",
    ">ffmpeg -i content.mp4 -vcodec libvpx-vp9 -i 0001.webm -i background.wav -filter_complex \"[2:a]volume=0.3[bg]; [1:a][bg]amix=inputs=2:duration=first[a]; overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2[v]\" -map \"[v]\" -map \"[a]\" output.mp4\n",
    ">```\n",
    "\n",
    ">Crop avatar and move it to the right:\n",
    ">```bash\n",
    ">ffmpeg -i content.mp4 -vcodec libvpx-vp9 -i 0001.webm -filter_complex \"[1:v]crop=440:1042:740:38[webm];[0:v][webm]overlay=W-w-160:38[outv]\" -map \"[outv]\" -map 1:a output.mp4\n",
    ">```\n",
    ">Above command was adjusted to the default size of the Lisa avatar in \"technical standing\" style. The general pattern of croping and repositioning is as follows:\n",
    ">\n",
    ">```bash\n",
    ">ffmpeg -i content.mp4 -vcodec libvpx-vp9 -i 0001.webm -filter_complex \"[1:v]crop=w:h:x:y[webm];[0:v][webm]overlay=W-w-10:10[outv]\" -map \"[outv]\" -map 1:a output.mp4\n",
    ">```\n",
    ">- __[1:v]crop=w:h:x:y[webm]__ is the cropping filter. Replace w, h, x, and y with the width, height, and the x, y coordinates of the top-left corner of the crop rectangle.\n",
    ">- __[0:v][webm]overlay=W-w-10:10[outv]__ is the overlay filter. The overlay is positioned 10 pixels from the right edge and 10 pixels from the top.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Option 2: Use a video editing tool\n",
    "\n",
    "Using a video editing tool provides more intuitive and flexible options for generating your final output video. The following screenshot illustrates how to use the free edition of Microsoft Clipchamp.\n",
    "\n",
    "<img src=\"./media/Clipchamp.png\" alt=\"drawing\" style=\"width:800px;\"/>\n",
    "\n",
    "You can multi-select the required files in Windows Explorer and select \"Edit with Clipchamp\" in the context menu. In our example, we have selected the avatar video `0001.webm`, the content video `content.mp4`, and an audio file for background music `background.wav`.\n",
    "\n",
    "Ensure that you add the content as tracks in the video editor, positioning the avatar track with the transparent background on top. Then, align the timing of your tracks using the video cutting option. You can also use the tool to reposition your avatar, generate captions for subtitles, and add further content and special effects.\n",
    "Lastly, use the export button to create your final video.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}