Upload 113 files

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data/call_center_prompt_with_intents_categories_context.csv filter=lfs diff=lfs merge=lfs -text
+libs/ffmpeg.exe filter=lfs diff=lfs merge=lfs -text
+libs/ffprobe.exe filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,12 +1,176 @@
 ---
-title: Solution 2
-emoji: 🐢
-colorFrom: gray
-colorTo: red
-sdk: gradio
-sdk_version: 5.13.1
+title: Solution_2
 app_file: app.py
-pinned: false
+sdk: gradio
+sdk_version: 4.44.1
 ---
+## Prepare Development Environment
+
+We will be using `Poetry`, a versatile tool for Python projects that simplifying dependency management and packaging. It automates the process of declaring, installing, and updating the libraries your project relies on. By using a lockfile, `Poetry` guarantees consistent and reproducible builds, ensuring that your project's dependencies are always installed in the specified versions. Additionally, `Poetry` provides convenient features for building your project into distributable formats, making it easier to share and deploy your work.
+
+1. Install [Python](https://www.python.org/downloads/) (both 3.10.12 and 3.12.6 were used in testing to offer a wide range of diversity.) 
+
+1. Install [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer).The flexibility of poetry makes it best of breed as it plays well with `pip` and `conda` alike. 
+
+1. Install [ffmpeg](https://www.ffmpeg.org/download.html) 
+
+   - Windows
+
+        To simplify setup for windows users, `ffmpeg.exe` and `ffprode.exe` have been included in this repo under `libs` directory. For other operating systems, please reference below.
+    
+   - Mac 
+     - **For MacOS Monterey v12 and below:** We will be using MacPorts. Homebrew is no longer supported for these MacOSx versions. 
+       - Install Apple's CLI Developer Tools (If required): `xcode-select --install`
+       - Download and install [**MacPorts**](https://www.macports.org/install.php) for the version of your Mac operating system.  
+     
+     - **For Mac OSX Ventura and above:** We will be using `brew` but first we must make sure that it's up-to-date.  
+
+       1. First update `brew`.  
+
+             ```bash
+             brew update
+             ``` 
+             > **NOTE:** if you get an error like `fatal: couldn't find remote ref refs/heads/master` when trying to run `brew update`, the possible culprit is `dart-lang` changing it's default branch from `master` to `main`. To resolve this run the following set of commands:
+
+             ```bash
+             brew tap --repair && brew cleanup && brew update-reset
+             ```
+
+             Now you should be able to run `brew update`.
+
+       2.  Next, we can upgrade the outdated `brew` formulae. 
+
+             ```bash
+             brew upgrade
+             ``` 
+             > **NOTE:** You might need to run this a few times to get all outdated formulae successfully updated. Depending on how outdated packages are, this may take some time to complete. Please be patient. This is a good time to grab a `brew` :) 
+
+         1.  We can now install the latest/stable version of `ffmpeg` in Homebrew.  
+             ```bash
+             brew install ffmpeg
+             ```
+      
+
+
+   - Linux  
+      Install using the following command:
+
+      ```bash
+      apt install libasound2-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg
+      ```
+
+3. Create a Virtual Environment, and install libraries needed, using a Linux shell or **git bash** in Windows
+
+    - Windows
+
+        ```bash
+        python -m venv .venv
+        .venv/Scripts/activate
+        ```
+
+    - Linux/Mac
+
+        ``` bash
+        python3 -m venv .venv
+        source ./.venv/bin/activate
+        ```
+
+    - Mac Only
+
+        **Prerequisites**  
+
+        One of the requirements is [PyAudio](https://pypi.org/project/PyAudio/). According to their instructions we need to use Homebrew to install the prerequisite `portaudio` library before we can install `PyAudio`.  
+
+        ```bash
+        brew install portaudio
+        ```
+1. Install the code, from the root directory that contains `pyproject.toml` file, using `poetry`.
+    ```bash
+    poetry lock 
+    poetry install
+    ```
+
+4. Install `playwright`, an open-source tool for auomating web testing in python. We'll use it to get some data for our LLM.
+    ```bash
+    playwright install
+    ```
+
+5. Test Environment
+    
+    Run the command below to make sure the virtual environment is activated.
+    
+    ``` bash
+    python -V
+    ```
+
+    Create an OpenAI Account and Obtain a Key
+    
+    1. Follow the instructions [here](https://platform.openai.com/docs/quickstart) to create your key.
+    2. Make a copy of [.env_template](.env_template) and rename it to `.env`. Then add your key in the `.env` file as shown:
+    ```bash
+    OPENAI_API_KEY="<YOUR_KEY_GOES_HERE>"
+    ```
+
+## Project Directory Overview  
+
+The project is structured in a modularized manner, focusing on building and running chatbots with voice capabilities. It has a good separation of concerns with well-defined directories for different functionalities. Such as `bots`, `data_utils`, and `models`, promoting code reusability and maintainability. It offers various voice capabilities, including text-to-speech, speech-to-text, and voice-based interactions, making it suitable for creating interactive chatbot applications.
+
+#### **Key Directories:**
+
+* **app:** Contains the main application files:
+    * `chatbot_gradio_runner.ipynb`: Jupyter notebook for running the chatbot interactively.
+
+    * `chatbot_gradio_runner.py`: Python script for running the chatbot with Gradio for a web interface.
+
+* **data:** Stores various types of data for chatbots. This is primarily used for LLM context for different bots:
+
+    * `travel_bot_context.txt`: This is the context we'll use for the chat assistant today.
+
+    * Other files with specific chatbot contexts (financial, call center, etc.).
+
+    * CSV and JSON files formats to allow flexibility.
+* **genai_voice:** Core project code:
+    * **bots:** Code specific to implementing chatbots (`chatbot.py`).
+    * **config:** Configuration files (`defaults.py`).
+    * **data_utils:** Utilities for data handling and gathering data from websites (e.g., `extract_web_data.py`).
+    * **defintions:** Defines response formats and prompts for chatbots.
+
+    * **logger:** Custom utility for logging information.
+
+    * **models:** Code for managing and interacting with language models (`open_ai.py`).
+
+    * **moderation:** Code for handling and filtering chatbot responses.
+
+    * **processing:** Functions for processing audio data (`audio.py`).
+
+* **libs:** External libraries used by the project (ffmpeg binaries for Windows only).
+
+* **poetry.lock** and **pyproject.toml**: Poetry-related files for dependency management.
+
+    
+## Launch Notebook
+
+1. We need to be able to point Jupyter notebook to our virtual environment that has the right packages and libraries.  
+
+    ```bash
+    ipython kernel install --user --name=venv
+    ```
+
+1. Run the command below from the virtual environment to launch the [notebook](app/chatbot_gradio_runner.ipynb) in a browser. Once ran, select the `venv` kernel and continue executing the cells.
+![selecting a kernel](assets/image.png)
+
+
+    ``` bash
+    jupyter notebook app/chatbot_gradio_runner.ipynb
+    ```
+2. You can also run the same chatbot directly from a python script using `poetry`. 
+    ```bash
+    poetry run RunChatBotScript
+    ```
+
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## Troubleshooting
+1. Try and use a headset microphone
+2. Record in a quiet room 
+3. Make sure that you have granted microphone permissions
+4. Ensure the required audio libraries are installed

app.py

@@ -0,0 +1,74 @@
+"""Script to run the Gradio chatbot application"""
+
+import sys
+import os
+from dotenv import load_dotenv
+
+# Load environment variables from .env file if available
+load_dotenv()
+
+# Ensure the correct path is added to the system to avoid import issues
+sys.path.append(os.path.abspath(os.path.dirname(__file__)))
+
+import gradio as gr
+from genai_voice.bots.chatbot import ChatBot
+from genai_voice.logger.log_utils import log, LogLevels
+
+# Ensure API key is set
+api_key = os.getenv("OPENAI_API_KEY")
+if not api_key:
+    raise ValueError("Missing OpenAI API key. Please set the environment variable OPENAI_API_KEY.")
+
+# Function to run the chatbot in microphone mode
+def run():
+    """Run Chatbot app"""
+    chatbot = ChatBot(enable_speakers=True, threaded=True)
+    history = []
+
+    def get_response(audio):
+        """Get audio response from Chatbot"""
+        if not audio:
+            raise ValueError("No audio file provided.")
+        prompt = chatbot.get_prompt_from_gradio_audio(audio)
+        log(f"Transcribed prompt: {prompt}", log_level=LogLevels.ON)
+        response = chatbot.respond(prompt, history)
+        history.append([prompt, response])
+        return response
+
+    demo = gr.Interface(
+        fn=get_response,
+        inputs=gr.Audio(sources="microphone"),
+        outputs="text",
+        title="Wanderwise Travel Assistant",
+    )
+    demo.launch(share=True)
+
+
+# Function to run chatbot with file support
+def run_with_file_support():
+    """Run Chatbot app and save files to disk"""
+    chatbot = ChatBot(enable_speakers=True, threaded=True)
+    history = []
+
+    def get_response_from_file(file):
+        """Process file input and generate response"""
+        prompt = chatbot.get_prompt_from_file(file)
+        response = chatbot.respond(prompt, history)
+        history.append([prompt, response])
+        return response
+
+    demo = gr.Interface(
+        fn=get_response_from_file,
+        inputs=gr.Audio(sources="microphone", type="filepath"),
+        outputs="text",
+    )
+    demo.launch(share=True)
+
+
+# Entry point to run the chatbot
+if __name__ == "__main__":
+    mode = os.getenv("RUN_MODE", "mic")  # Set RUN_MODE=microphone or file in environment
+    if mode == "file":
+        run_with_file_support()
+    else:
+        run()

app/.ipynb_checkpoints/chatbot_gradio_runner-checkpoint.ipynb

@@ -0,0 +1,325 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Generative AI for Audio Application"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Restart Kernel (If needed)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'status': 'ok', 'restart': True}"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "ename": "",
+     "evalue": "",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31mThe Kernel crashed while executing code in the current cell or a previous cell. \n",
+      "\u001b[1;31mPlease review the code in the cell(s) to identify a possible cause of the failure. \n",
+      "\u001b[1;31mClick <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. \n",
+      "\u001b[1;31mView Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
+     ]
+    }
+   ],
+   "source": [
+    "import IPython\n",
+    "\n",
+    "app = IPython.Application.instance()\n",
+    "app.kernel.do_shutdown(True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Import Libraries\n",
+    "Most of the complexity for the chatbot is in [customizable_chatbot.py](./customizable_chatbot.py) that uses [audio.py](./audio.py) internally for the audio capabilities."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import gradio as gr\n",
+    "from genai_voice.bots.chatbot import ChatBot\n",
+    "from genai_voice.config.defaults import Config\n",
+    "from genai_voice.logger.log_utils import log, LogLevels"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "19:22:35 Configuration: \t\n",
+      "        Default(MODEL_GPT_TURBO_NAME='gpt-4-turbo', \n",
+      "        OPENAI_API_KEY='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', \n",
+      "        TEMPERATURE=0.0, \n",
+      "        TOP_P=0.97, \n",
+      "        TOP_K=40, \n",
+      "        MAX_OUTPUT_TOKENS=2048, \n",
+      "        WEB_SCRAP_OUTPUT_DIR=data/context.txt)\n"
+     ]
+    }
+   ],
+   "source": [
+    "log(f'Configuration: {Config()}', log_level=LogLevels.ON)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set Current Working Directory  \n",
+    "\n",
+    "We want to simulate running this notebook from the project root just as it would work when using `Poetry` scripts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "19:22:58 Before directory change: /Users/capturedbyace/sources/genai_voice/app\n",
+      "19:22:58 Changing directory to root\n",
+      "19:22:58 Before directory change: /Users/capturedbyace/sources/genai_voice\n"
+     ]
+    }
+   ],
+   "source": [
+    "curr_wrk_dir = os.getcwd()\n",
+    "log(f'Before directory change: {curr_wrk_dir}')\n",
+    "if curr_wrk_dir.endswith('app'):\n",
+    "    log(f'Changing directory to root')\n",
+    "    os.chdir('..')\n",
+    "    curr_wrk_dir = os.getcwd()\n",
+    "log(f'Before directory change: {curr_wrk_dir}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create Data for LLM Context  \n",
+    "\n",
+    "`Poetry` scripts allow us to install our code as a package and run functions executables. \n",
+    "\n",
+    "We will use the `ExtractWebPagesAndSaveData` script, that is defined in `pyproject.toml` to scrape, extract and generate the file that the LLM will use as context data.\n",
+    "\n",
+    "`SAMPLE_URLS` have been defined under provided under `genai_voice.data_utils.urls.py`. Feel free to modify the links in that file to customize the source of data. \n",
+    "\n",
+    "> **DISCLAIMER:** Be responsible when scraping data that is not yours, complying with the EULA of the sites and conducting it in a legal fashion. Also remember that most sites will throttle scrapes, so do this with caution.  \n",
+    "\n",
+    "> **NOTE:** This is an optional step. It just shows you have you can get custom data for your LLM context. We have provided the data for this project. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "19:23:25 Starting the web scraper...\n",
+      "19:23:25 Creating the AsyncChromiumLoader with #10 urls...\n",
+      "19:23:25 Starting scraping...\n",
+      "19:23:28 Content scraped\n",
+      "19:23:29 Starting scraping...\n",
+      "19:23:33 Content scraped\n",
+      "19:23:33 Starting scraping...\n",
+      "19:23:36 Content scraped\n",
+      "19:23:36 Starting scraping...\n",
+      "19:23:39 Content scraped\n",
+      "19:23:39 Starting scraping...\n",
+      "19:23:41 Content scraped\n",
+      "19:23:41 Starting scraping...\n",
+      "19:23:45 Content scraped\n",
+      "19:23:45 Starting scraping...\n",
+      "19:23:49 Content scraped\n",
+      "19:23:49 Starting scraping...\n",
+      "19:23:53 Content scraped\n",
+      "19:23:53 Starting scraping...\n",
+      "19:23:57 Content scraped\n",
+      "19:23:57 Starting scraping...\n",
+      "19:24:00 Content scraped\n",
+      "19:24:00 Documents scraped.\n",
+      "19:24:00 Using BeautifulSoutTransformer to extract ['h1', 'h2', 'h3', 'p'].\n",
+      "19:24:02 Transformed #10 urls.\n",
+      "19:24:02 Writing to output file.\n",
+      "19:24:02 Successfully written data to data/context.txt\n"
+     ]
+    }
+   ],
+   "source": [
+    "!poetry run ExtractWebPagesAndSaveData"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gradio Interface\n",
+    "This launches the UI, you will probably need to allow the browser to use the microphone to enable the audio functions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "19:27:45 Context file: travel_bot_context.txt\n",
+      "19:27:45 Creating the OpenAI Model Client.\n",
+      "19:27:45 Initialized OpenAI model: gpt-4-turbo\n",
+      "19:27:45 HTTP Request: GET http://127.0.0.1:7861/startup-events \"HTTP/1.1 200 OK\"\n",
+      "19:27:45 HTTP Request: HEAD http://127.0.0.1:7861/ \"HTTP/1.1 200 OK\"\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running on local URL:  http://127.0.0.1:7861\n",
+      "\n",
+      "To create a public link, set `share=True` in `launch()`.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div><iframe src=\"http://127.0.0.1:7861/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "19:27:46 HTTP Request: GET https://api.gradio.app/pkg-version \"HTTP/1.1 200 OK\"\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": []
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Getting prompt from audio device: (44100, array([ 9, 12, 10, ...,  0,  0,  0], dtype=int16))\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/capturedbyace/sources/genai_voice/venv/lib/python3.10/site-packages/transformers/models/whisper/generation_whisper.py:496: FutureWarning: The input name `inputs` is deprecated. Please make sure to use `input_features` instead.\n",
+      "  warnings.warn(\n",
+      "19:28:23 Transcribed prompt:  Hello, how are you?\n",
+      "19:28:23 Empty history. Creating a state list to track histories.\n",
+      "19:28:23 {'temperature': 0.0, 'top_p': 0.97, 'top_k': 40, 'max_output_tokens': 2048, 'seed': 0, 'response_format': {'type': 'text'}}\n",
+      "19:28:25 HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
+      "19:28:25 Chatbot response: Hello! I'm just a virtual assistant, but thank you for asking. How can I assist you with your travel plans today?\n"
+     ]
+    }
+   ],
+   "source": [
+    "\"\"\"Run Chatbot app\"\"\"\n",
+    "chatbot = ChatBot(enable_speakers=True, threaded=True)\n",
+    "history = []\n",
+    "\n",
+    "def get_response(audio):\n",
+    "    \"\"\"Get Audio Response From Chatbot\"\"\"\n",
+    "    if not audio:\n",
+    "        raise ValueError(\"No audio file provided.\")\n",
+    "    prompt = chatbot.get_prompt_from_gradio_audio(audio)\n",
+    "    log(f\"Transcribed prompt: {prompt}\", log_level=LogLevels.ON)\n",
+    "    response = chatbot.respond(prompt, history)\n",
+    "    log(f\"Chatbot response: {response}\", log_level=LogLevels.ON)\n",
+    "    history.append([prompt, response])\n",
+    "    return response\n",
+    "\n",
+    "demo = gr.Interface(\n",
+    "        get_response,\n",
+    "        gr.Audio(sources=\"microphone\"),\n",
+    "        None,\n",
+    "        title=\"Wanderwise Travel Assistant\"\n",
+    ")\n",
+    "demo.launch()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

app/chatbot_gradio_runner.ipynb

@@ -0,0 +1,349 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Generative AI for Audio Application"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Restart Kernel (If needed)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'status': 'ok', 'restart': True}"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import IPython\n",
+    "\n",
+    "app = IPython.Application.instance()\n",
+    "app.kernel.do_shutdown(True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Collecting gradio\n",
+      "  Using cached gradio-4.44.1-py3-none-any.whl (18.1 MB)\n",
+      "Collecting aiofiles<24.0,>=22.0\n",
+      "  Using cached aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
+      "Collecting uvicorn>=0.14.0\n",
+      "  Using cached uvicorn-0.34.0-py3-none-any.whl (62 kB)\n",
+      "Collecting typing-extensions~=4.0\n",
+      "  Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)\n",
+      "Collecting jinja2<4.0\n",
+      "  Using cached jinja2-3.1.5-py3-none-any.whl (134 kB)\n",
+      "Collecting huggingface-hub>=0.19.3\n",
+      "  Using cached huggingface_hub-0.27.1-py3-none-any.whl (450 kB)\n",
+      "Collecting pydub\n",
+      "  Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
+      "Collecting markupsafe~=2.0\n",
+      "  Using cached MarkupSafe-2.1.5-cp39-cp39-macosx_10_9_x86_64.whl (14 kB)\n",
+      "Collecting numpy<3.0,>=1.0\n",
+      "  Using cached numpy-2.0.2-cp39-cp39-macosx_10_9_x86_64.whl (21.2 MB)\n",
+      "Collecting tomlkit==0.12.0\n",
+      "  Using cached tomlkit-0.12.0-py3-none-any.whl (37 kB)\n",
+      "Collecting pandas<3.0,>=1.0\n",
+      "  Using cached pandas-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl (12.6 MB)\n",
+      "Collecting ffmpy\n",
+      "  Using cached ffmpy-0.5.0-py3-none-any.whl (6.0 kB)\n",
+      "Collecting importlib-resources<7.0,>=1.3\n",
+      "  Using cached importlib_resources-6.5.2-py3-none-any.whl (37 kB)\n",
+      "Collecting fastapi<1.0\n",
+      "  Using cached fastapi-0.115.7-py3-none-any.whl (94 kB)\n",
+      "Collecting pillow<11.0,>=8.0\n",
+      "  Using cached pillow-10.4.0-cp39-cp39-macosx_10_10_x86_64.whl (3.5 MB)\n",
+      "Collecting gradio-client==1.3.0\n",
+      "  Using cached gradio_client-1.3.0-py3-none-any.whl (318 kB)\n",
+      "Collecting httpx>=0.24.1\n",
+      "  Using cached httpx-0.28.1-py3-none-any.whl (73 kB)\n",
+      "Collecting anyio<5.0,>=3.0\n",
+      "  Using cached anyio-4.8.0-py3-none-any.whl (96 kB)\n",
+      "Collecting pydantic>=2.0\n",
+      "  Using cached pydantic-2.10.6-py3-none-any.whl (431 kB)\n",
+      "Collecting pyyaml<7.0,>=5.0\n",
+      "  Using cached PyYAML-6.0.2-cp39-cp39-macosx_10_9_x86_64.whl (184 kB)\n",
+      "Collecting semantic-version~=2.0\n",
+      "  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
+      "Collecting orjson~=3.0\n",
+      "  Using cached orjson-3.10.15-cp39-cp39-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl (250 kB)\n",
+      "Collecting python-multipart>=0.0.9\n",
+      "  Using cached python_multipart-0.0.20-py3-none-any.whl (24 kB)\n",
+      "Collecting ruff>=0.2.2\n",
+      "  Using cached ruff-0.9.3-py3-none-macosx_10_12_x86_64.whl (11.6 MB)\n",
+      "Collecting matplotlib~=3.0\n",
+      "  Using cached matplotlib-3.9.4-cp39-cp39-macosx_10_12_x86_64.whl (7.9 MB)\n",
+      "Collecting urllib3~=2.0\n",
+      "  Using cached urllib3-2.3.0-py3-none-any.whl (128 kB)\n",
+      "Collecting typer<1.0,>=0.12\n",
+      "  Using cached typer-0.15.1-py3-none-any.whl (44 kB)\n",
+      "Collecting packaging\n",
+      "  Using cached packaging-24.2-py3-none-any.whl (65 kB)\n",
+      "Collecting websockets<13.0,>=10.0\n",
+      "  Using cached websockets-12.0-cp39-cp39-macosx_10_9_x86_64.whl (121 kB)\n",
+      "Collecting fsspec\n",
+      "  Using cached fsspec-2024.12.0-py3-none-any.whl (183 kB)\n",
+      "Collecting exceptiongroup>=1.0.2\n",
+      "  Using cached exceptiongroup-1.2.2-py3-none-any.whl (16 kB)\n",
+      "Collecting idna>=2.8\n",
+      "  Using cached idna-3.10-py3-none-any.whl (70 kB)\n",
+      "Collecting sniffio>=1.1\n",
+      "  Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)\n",
+      "Collecting starlette<0.46.0,>=0.40.0\n",
+      "  Using cached starlette-0.45.3-py3-none-any.whl (71 kB)\n",
+      "Collecting httpcore==1.*\n",
+      "  Using cached httpcore-1.0.7-py3-none-any.whl (78 kB)\n",
+      "Collecting certifi\n",
+      "  Using cached certifi-2024.12.14-py3-none-any.whl (164 kB)\n",
+      "Collecting h11<0.15,>=0.13\n",
+      "  Using cached h11-0.14.0-py3-none-any.whl (58 kB)\n",
+      "Collecting requests\n",
+      "  Using cached requests-2.32.3-py3-none-any.whl (64 kB)\n",
+      "Collecting tqdm>=4.42.1\n",
+      "  Using cached tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
+      "Collecting filelock\n",
+      "  Using cached filelock-3.17.0-py3-none-any.whl (16 kB)\n",
+      "Collecting zipp>=3.1.0\n",
+      "  Using cached zipp-3.21.0-py3-none-any.whl (9.6 kB)\n",
+      "Collecting fonttools>=4.22.0\n",
+      "  Using cached fonttools-4.55.6-cp39-cp39-macosx_10_9_x86_64.whl (2.3 MB)\n",
+      "Collecting pyparsing>=2.3.1\n",
+      "  Using cached pyparsing-3.2.1-py3-none-any.whl (107 kB)\n",
+      "Collecting python-dateutil>=2.7\n",
+      "  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)\n",
+      "Collecting contourpy>=1.0.1\n",
+      "  Using cached contourpy-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl (265 kB)\n",
+      "Collecting cycler>=0.10\n",
+      "  Using cached cycler-0.12.1-py3-none-any.whl (8.3 kB)\n",
+      "Collecting kiwisolver>=1.3.1\n",
+      "  Using cached kiwisolver-1.4.7-cp39-cp39-macosx_10_9_x86_64.whl (65 kB)\n",
+      "Collecting pytz>=2020.1\n",
+      "  Using cached pytz-2024.2-py2.py3-none-any.whl (508 kB)\n",
+      "Collecting tzdata>=2022.7\n",
+      "  Using cached tzdata-2025.1-py2.py3-none-any.whl (346 kB)\n",
+      "Collecting pydantic-core==2.27.2\n",
+      "  Using cached pydantic_core-2.27.2-cp39-cp39-macosx_10_12_x86_64.whl (1.9 MB)\n",
+      "Collecting annotated-types>=0.6.0\n",
+      "  Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)\n",
+      "Collecting six>=1.5\n",
+      "  Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)\n",
+      "Collecting shellingham>=1.3.0\n",
+      "  Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)\n",
+      "Collecting rich>=10.11.0\n",
+      "  Using cached rich-13.9.4-py3-none-any.whl (242 kB)\n",
+      "Collecting click>=8.0.0\n",
+      "  Using cached click-8.1.8-py3-none-any.whl (98 kB)\n",
+      "Collecting pygments<3.0.0,>=2.13.0\n",
+      "  Using cached pygments-2.19.1-py3-none-any.whl (1.2 MB)\n",
+      "Collecting markdown-it-py>=2.2.0\n",
+      "  Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB)\n",
+      "Collecting mdurl~=0.1\n",
+      "  Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)\n",
+      "Collecting charset-normalizer<4,>=2\n",
+      "  Using cached charset_normalizer-3.4.1-cp39-cp39-macosx_10_9_universal2.whl (197 kB)\n",
+      "Installing collected packages: urllib3, typing-extensions, sniffio, mdurl, idna, h11, exceptiongroup, charset-normalizer, certifi, zipp, tqdm, six, requests, pyyaml, pygments, pydantic-core, packaging, numpy, markdown-it-py, httpcore, fsspec, filelock, anyio, annotated-types, websockets, tzdata, starlette, shellingham, rich, pytz, python-dateutil, pyparsing, pydantic, pillow, markupsafe, kiwisolver, importlib-resources, huggingface-hub, httpx, fonttools, cycler, contourpy, click, uvicorn, typer, tomlkit, semantic-version, ruff, python-multipart, pydub, pandas, orjson, matplotlib, jinja2, gradio-client, ffmpy, fastapi, aiofiles, gradio\n",
+      "Successfully installed aiofiles-23.2.1 annotated-types-0.7.0 anyio-4.8.0 certifi-2024.12.14 charset-normalizer-3.4.1 click-8.1.8 contourpy-1.3.0 cycler-0.12.1 exceptiongroup-1.2.2 fastapi-0.115.7 ffmpy-0.5.0 filelock-3.17.0 fonttools-4.55.6 fsspec-2024.12.0 gradio-4.44.1 gradio-client-1.3.0 h11-0.14.0 httpcore-1.0.7 httpx-0.28.1 huggingface-hub-0.27.1 idna-3.10 importlib-resources-6.5.2 jinja2-3.1.5 kiwisolver-1.4.7 markdown-it-py-3.0.0 markupsafe-2.1.5 matplotlib-3.9.4 mdurl-0.1.2 numpy-2.0.2 orjson-3.10.15 packaging-24.2 pandas-2.2.3 pillow-10.4.0 pydantic-2.10.6 pydantic-core-2.27.2 pydub-0.25.1 pygments-2.19.1 pyparsing-3.2.1 python-dateutil-2.9.0.post0 python-multipart-0.0.20 pytz-2024.2 pyyaml-6.0.2 requests-2.32.3 rich-13.9.4 ruff-0.9.3 semantic-version-2.10.0 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 starlette-0.45.3 tomlkit-0.12.0 tqdm-4.67.1 typer-0.15.1 typing-extensions-4.12.2 tzdata-2025.1 urllib3-2.3.0 uvicorn-0.34.0 websockets-12.0 zipp-3.21.0\n",
+      "\u001b[33mWARNING: You are using pip version 21.2.3; however, version 24.3.1 is available.\n",
+      "You should consider upgrading via the '/Users/voldemort/Downloads/Solution 2/venv/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!pip install gradio"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Import Libraries\n",
+    "Most of the complexity for the chatbot is in [customizable_chatbot.py](./customizable_chatbot.py) that uses [audio.py](./audio.py) internally for the audio capabilities."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "ModuleNotFoundError",
+     "evalue": "No module named 'gradio'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[4], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mos\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mgradio\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mgr\u001b[39;00m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mgenai_voice\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mbots\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchatbot\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ChatBot\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mgenai_voice\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mconfig\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdefaults\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Config\n",
+      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'gradio'"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import gradio as gr\n",
+    "from genai_voice.bots.chatbot import ChatBot\n",
+    "from genai_voice.config.defaults import Config\n",
+    "from genai_voice.logger.log_utils import log, LogLevels"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log(f'Configuration: {Config()}', log_level=LogLevels.ON)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set Current Working Directory  \n",
+    "\n",
+    "We want to simulate running this notebook from the project root just as it would work when using `Poetry` scripts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "curr_wrk_dir = os.getcwd()\n",
+    "log(f'Before directory change: {curr_wrk_dir}')\n",
+    "if curr_wrk_dir.endswith('app'):\n",
+    "    log(f'Changing directory to root')\n",
+    "    os.chdir('..')\n",
+    "    curr_wrk_dir = os.getcwd()\n",
+    "log(f'Before directory change: {curr_wrk_dir}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create Data for LLM Context  \n",
+    "\n",
+    "`Poetry` scripts allow us to install our code as a package and run functions executables. \n",
+    "\n",
+    "We will use the `ExtractWebPagesAndSaveData` script, that is defined in `pyproject.toml` to scrape, extract and generate the file that the LLM will use as context data.\n",
+    "\n",
+    "`SAMPLE_URLS` have been defined under provided under `genai_voice.data_utils.urls.py`. Feel free to modify the links in that file to customize the source of data. \n",
+    "\n",
+    "> **DISCLAIMER:** Be responsible when scraping data that is not yours, complying with the EULA of the sites and conducting it in a legal fashion. Also remember that most sites will throttle scrapes, so do this with caution.  \n",
+    "\n",
+    "> **NOTE:** This is an optional step. It just shows you have you can get custom data for your LLM context. We have provided the data for this project. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!poetry run ExtractWebPagesAndSaveData"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gradio Interface\n",
+    "This launches the UI, you will probably need to allow the browser to use the microphone to enable the audio functions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "NameError",
+     "evalue": "name 'ChatBot' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[5], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;124;03m\"\"\"Run Chatbot app\"\"\"\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m chatbot \u001b[38;5;241m=\u001b[39m \u001b[43mChatBot\u001b[49m(enable_speakers\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, threaded\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m      3\u001b[0m history \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m      5\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_response\u001b[39m(audio):\n",
+      "\u001b[0;31mNameError\u001b[0m: name 'ChatBot' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "\"\"\"Run Chatbot app\"\"\"\n",
+    "chatbot = ChatBot(enable_speakers=True, threaded=True)\n",
+    "history = []\n",
+    "\n",
+    "def get_response(audio):\n",
+    "    \"\"\"Get Audio Response From Chatbot\"\"\"\n",
+    "    if not audio:\n",
+    "        raise ValueError(\"No audio file provided.\")\n",
+    "    prompt = chatbot.get_prompt_from_gradio_audio(audio)\n",
+    "    log(f\"Transcribed prompt: {prompt}\", log_level=LogLevels.ON)\n",
+    "    response = chatbot.respond(prompt, history)\n",
+    "    log(f\"Chatbot response: {response}\", log_level=LogLevels.ON)\n",
+    "    history.append([prompt, response])\n",
+    "    return response\n",
+    "\n",
+    "demo = gr.Interface(\n",
+    "        get_response,\n",
+    "        gr.Audio(sources=\"microphone\"),\n",
+    "        None,\n",
+    "        title=\"Wanderwise Travel Assistant\"\n",
+    ")\n",
+    "demo.launch()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

assets/file_map.txt

@@ -0,0 +1,44 @@
+.
+├── README.md
+├── app
+│   ├── chatbot_gradio_runner.ipynb
+│   ├── chatbot_gradio_runner.py
+│   └── data
+│       └── context.txt
+├── data
+│   ├── audio_bot_context.txt
+│   ├── audio_bot_context_extended.txt
+│   ├── call_center_prompt_with_intents_categories_context.csv
+│   ├── call_center_prompt_with_intents_categories_context.json
+│   ├── call_center_prompt_with_intents_categories_context.txt
+│   ├── financial_bot_context.txt
+│   └── travel_bot_context.txt
+├── file_map.txt
+├── genai_voice
+│   ├── bots
+│   │   └── chatbot.py
+│   ├── config
+│   │   └── defaults.py
+│   ├── data_utils
+│   │   ├── dataset_converter.py
+│   │   ├── extract_web_data.py
+│   │   └── urls.py
+│   ├── defintions
+│   │   ├── model_response_formats.py
+│   │   └── prompts.py
+│   ├── logger
+│   │   └── log_utils.py
+│   ├── models
+│   │   ├── model_config.py
+│   │   └── open_ai.py
+│   ├── moderation
+│   │   └── responses.py
+│   └── processing
+│       └── audio.py
+├── libs
+│   ├── ffmpeg.exe
+│   └── ffprobe.exe
+├── poetry.lock
+└── pyproject.toml
+
+14 directories, 28 files

data/audio_bot_context.txt

@@ -0,0 +1,86 @@
+
+Audio-Based Call Center Bot - Contextual Data
+
+1. Greeting and Introduction
+- Standard Greeting: "Hello! Thank you for calling [Company Name]. How can I assist you today?"
+- Personalized Greeting: If customer data is available, use the name: "Hello [Customer Name], welcome back to [Company Name]. How may I help you today?"
+
+2. Common Customer Queries
+- Account Information:
+  - Query: "Can you provide me with my account balance?"
+  - Response: "I can help with that. Please provide your account ID or phone number for verification."
+- Billing and Payments:
+  - Query: "I have a question about my recent bill."
+  - Response: "I'd be happy to help. Could you please provide your billing date or invoice number?"
+- Technical Support:
+  - Query: "I'm having trouble accessing my account."
+  - Response: "I'm sorry to hear that. Let me walk you through some steps to troubleshoot the issue."
+
+3. Verification and Security
+- Verification Steps:
+  - "For security purposes, could you please verify your account with your registered phone number or email address?"
+  - If incorrect information is provided: "I'm sorry, that doesn't match our records. Could you try again or provide an alternate detail?"
+
+4. Handling Wait Times and Transfers
+- Wait Time: "It may take a moment to retrieve that information. Please hold on while I process your request."
+- Transfer to Agent: "I'm transferring you to a human agent for further assistance. Please hold the line."
+
+5. Contextual Awareness and Memory
+- Remembering Previous Interactions: "I see you called last week about your [specific issue]. Have you been able to resolve that, or do you need further assistance?"
+- Follow-Up Questions: "Based on our previous conversation, it seems like you were facing [issue]. Is this still a concern, or would you like to discuss something else?"
+
+6. Common Problem-Solving Scenarios
+- Troubleshooting: 
+  - Customer: "My internet is not working."
+  - Bot: "Let's try restarting your modem. Please unplug it, wait for 10 seconds, and plug it back in. Let me know when you're ready."
+- Refunds and Cancellations: 
+  - Customer: "I want to cancel my subscription."
+  - Bot: "I'm here to help. Could you please share your subscription ID or email associated with the account for processing the cancellation?"
+
+7. Providing Information and Advice
+- Product or Service Information: "Our premium plan includes unlimited access to all features for just $29.99 per month. Would you like to know more details?"
+- Guidance: "If you want to change your password, go to the settings page on our website, select 'Account Settings,' and then click on 'Change Password.'"
+
+8. Sentiment Analysis Responses
+- If the customer seems frustrated: "I understand that this can be frustrating. I'm here to help you resolve this as quickly as possible."
+- If the customer seems satisfied: "I'm glad I could assist you! Is there anything else you need help with today?"
+
+9. Closing the Interaction
+- Standard Closing: "Thank you for contacting [Company Name]. If you need further assistance, feel free to reach out again. Have a great day!"
+- Issue Escalation: "I'm escalating this issue to a higher support level. You will receive a follow-up call shortly."
+
+10. General FAQ Responses
+- Query: "How do I reset my password?"
+  - Response: "You can reset your password by visiting our website and selecting 'Forgot Password' on the login page. You'll receive an email with a reset link."
+- Query: "What are your working hours?"
+  - Response: "Our support team is available Monday through Friday, from 9 AM to 6 PM."
+
+11. Possible Errors and Recovery
+- Inaudible Input: "I'm sorry, I couldn't hear that. Could you please repeat your question?"
+- Out-of-Scope Questions: "I'm here to assist with account and service-related queries. For other matters, please contact our main support line."
+
+Additional Contextual Examples
+- Handling Product Returns:
+  - Query: "I want to return a product I purchased."
+  - Response: "I'm here to help with that. Could you please provide the order ID and the reason for the return?"
+- Subscription Upgrades:
+  - Query: "How can I upgrade my current subscription?"
+  - Response: "To upgrade your subscription, please log in to your account and go to the 'Subscriptions' section. Would you like me to guide you through the process?"
+- Account Settings and Personal Information Update:
+  - Query: "How can I change my contact information?"
+  - Response: "You can change your contact information by visiting the 'Account Settings' page. Would you like me to send you a link to update your details?"
+
+Guidelines for Effective Conversations
+1. Always use polite language and maintain a friendly tone.
+2. Confirm information with the customer to ensure accuracy.
+3. If the bot cannot handle the request, provide a clear message and suggest transferring to a human agent.
+4. Use follow-up questions to keep the conversation natural and gather more information when needed.
+
+Context Handling and Memory Implementation
+- Use conversation memory to store the context of the current interaction, allowing the bot to provide more tailored and relevant responses.
+- Retain customer preferences and recent interactions for a more personalized experience in future calls.
+
+Future Enhancements
+- Multilingual Support: Add support for multiple languages to cater to a wider audience.
+- Advanced Sentiment Analysis: Implement sentiment analysis for better understanding of customer emotions and to adjust responses accordingly.
+- Integration with CRM Systems: Connect with CRM databases to automatically fetch customer information, thus streamlining interactions.

data/audio_bot_context_extended.txt

@@ -0,0 +1,278 @@
+
+Audio-Based Call Center Bot - Contextual Data
+
+1. Introduction
+The rise of automated systems in customer service has led to the development of voice-based bots for call centers. This document aims to provide detailed context for implementing an audio-based chatbot, covering various interaction scenarios, best practices, and future enhancements to ensure a smooth customer experience.
+
+2. Greeting and Introduction
+- Standard Greeting: "Hello! Thank you for calling [Company Name]. How can I assist you today?"
+- Personalized Greeting: If customer data is available, use the name: "Hello [Customer Name], welcome back to [Company Name]. How may I help you today? I see that you contacted us last week; is this call related to the same issue, or is it a new inquiry?"
+- Warm Introduction: "Hi there! This is [Bot Name], your virtual assistant. I’m here to help with any questions or issues you may have. What can I do for you today?"
+
+3. Common Customer Queries
+3.1 Account Information:
+- Query: "Can you provide me with my account balance?"
+- Response: "Certainly! For security purposes, could you please provide your account ID or registered phone number?"
+- Follow-Up: If the customer provides the necessary information, proceed to validate. "Thank you! Let me quickly verify your details... Your current balance is [balance amount]. Would you like to know anything else related to your account?"
+
+3.2 Billing and Payments:
+- Query: "I have a question about my recent bill."
+- Response: "I'd be happy to help. Could you please provide your billing date or invoice number so I can locate the details for you?"
+- Further Assistance: "I see that there was an additional charge on your recent bill for [service/product]. This was due to [reason]. If you need more clarification or wish to dispute this charge, I can connect you to a billing specialist. Would you like me to do that?"
+
+3.3 Technical Support:
+- Query: "I'm having trouble accessing my account."
+- Response: "I'm sorry to hear that. Let's go through some steps to resolve this. Are you trying to log in via our website or mobile app?"
+- Step-by-Step Troubleshooting: 
+  1. "Please ensure that you are using the latest version of the app. Would you like me to guide you through updating it?"
+  2. "If you are on the website, try clearing your browser cache and cookies. If that doesn't work, we can reset your password. Would you like to proceed with a password reset?"
+
+4. Verification and Security
+4.1 Verification Steps:
+- "For security purposes, could you please verify your account with your registered phone number or email address?"
+- Incorrect Information Handling: If the customer provides incorrect information, reply with: "I'm sorry, the information you provided does not match our records. Could you please try again or use an alternative verification method, like your registered address?"
+
+4.2 Handling Sensitive Data:
+- "Please note that for your security, I cannot process sensitive information like credit card details. You can update this information securely through our website."
+
+5. Handling Wait Times and Transfers
+- Wait Time Management: "It may take a moment to retrieve that information. Please hold on while I process your request. Your patience is greatly appreciated."
+- Music or Notification During Wait: "While I fetch that information for you, you'll hear a short tone. This won't take more than a few seconds."
+- Transfer to Agent: "I will now connect you to one of our customer service representatives. Please hold the line while I transfer your call. If you encounter any issues during this transfer, feel free to call back, and I’ll be here to assist you."
+
+6. Contextual Awareness and Memory
+6.1 Remembering Previous Interactions:
+- "I see you called last week about your subscription renewal. Have you been able to resolve that, or do you need further assistance on the same matter?"
+
+6.2 Follow-Up Questions:
+- "Based on our previous conversation, you mentioned experiencing connectivity issues with our service. Is this still a concern, or would you like to discuss a new issue today?"
+- "I noticed you were interested in upgrading your plan during our last conversation. Would you like to continue with that today, or is there something else I can help you with?"
+
+7. Common Problem-Solving Scenarios
+7.1 Troubleshooting: 
+- Customer: "My internet is not working."
+- Bot: "Let's try a quick solution. Could you please restart your modem by unplugging it, waiting for 10 seconds, and then plugging it back in? Let me know once you've done that."
+- Advanced Troubleshooting: "If restarting the modem did not work, we might need to check for outages in your area. Would you like me to do that for you?"
+
+7.2 Refunds and Cancellations: 
+- Customer: "I want to cancel my subscription."
+- Bot: "I'm here to help. Could you please share your subscription ID or email associated with the account for processing the cancellation?"
+- Clarification: "Before we proceed, may I ask the reason for the cancellation? Your feedback is important to us and helps us improve our services."
+
+8. Providing Information and Advice
+8.1 Product or Service Information:
+- "Our premium plan includes unlimited access to all features for just $29.99 per month, and it comes with a 30-day free trial. Would you like to activate this plan now, or would you prefer more information on its benefits?"
+
+8.2 Guidance:
+- "If you want to change your password, visit the settings page on our website, select 'Account Settings,' and then click on 'Change Password.' I can also send you a direct link to the page if that would be more convenient."
+
+8.3 Policy Information:
+- "For returns and refunds, our policy states that products must be returned within 30 days of purchase in their original condition. Would you like me to guide you through the return process?"
+
+9. Sentiment Analysis Responses
+9.1 Handling Frustration:
+- If the customer seems frustrated: "I understand that this can be frustrating. Your satisfaction is important to us, and I’m here to resolve this issue as quickly as possible. Let's work together to find a solution. Could you please provide more details on the issue you’re facing? This will help me assist you better."
+
+9.2 Handling Positive Feedback:
+- If the customer seems satisfied: "I'm glad I could assist you! Is there anything else you need help with today? Your feedback is greatly appreciated, and it helps us serve you better. Feel free to reach out anytime."
+
+9.3 Handling Neutral Feedback:
+- If the customer seems indifferent: "I hope I have addressed your concerns adequately. If you have any further questions or need more assistance in the future, please don’t hesitate to contact us."
+
+9.4 Handling Multiple Emotions:
+- If the customer seems both confused and frustrated: "I can sense that this situation might be confusing. Let’s break it down step-by-step. I’m here to help clear things up and find the best possible solution for you."
+
+10. Closing the Interaction
+10.1 Standard Closing:
+- "Thank you for contacting [Company Name]. If you need further assistance, feel free to reach out again. Have a great day!"
+
+10.2 Personalized Closing:
+- "It was a pleasure assisting you, [Customer Name]. If there’s anything else you need, just let us know. Enjoy the rest of your day!"
+
+10.3 Issue Escalation:
+- "I’m escalating this issue to a higher support level for a more in-depth review. You will receive a follow-up call shortly, and our team will ensure that this matter is resolved to your satisfaction."
+
+11. General FAQ Responses
+11.1 Password Reset:
+- Query: "How do I reset my password?"
+- Response: "You can reset your password by visiting our website and selecting 'Forgot Password' on the login page. A reset link will be sent to your registered email address. If you encounter any issues during this process, feel free to call us back."
+
+11.2 Working Hours:
+- Query: "What are your working hours?"
+- Response: "Our support team is available Monday through Friday, from 9 AM to 6 PM. For urgent matters outside these hours, you can access our self-service portal online."
+
+11.3 Subscription and Services:
+- Query: "What services are included in my subscription?"
+- Response: "Your current subscription includes [list of services]. If you are interested in additional services, I can provide you with details about our premium packages."
+
+11.4 Upgrading Services:
+- Query: "How do I upgrade my service?"
+- Response: "To upgrade your service, log in to your account and go to the 'Account Settings' section. Click on 'Upgrade Plan' to view the available options. Would you like a step-by-step guide sent to your email?"
+
+12. Possible Errors and Recovery
+12.1 Inaudible Input Handling:
+- "I'm sorry, I couldn't hear that clearly. Could you please repeat your question? Make sure you’re in a quiet place with minimal background noise."
+
+12.2 Out-of-Scope Questions:
+- "I'm here to assist with account and service-related queries. For other matters, please contact our main support line or visit our website for more information."
+
+12.3 Error Recovery Mechanism:
+- "It seems like there was an error processing your request. Let's try that again. If the issue persists, I will connect you with a human representative."
+
+12.4 Invalid Input Handling:
+- "It seems the information you entered is not recognized. Could you please double-check and provide the correct details?"
+
+12.5 Timeout Scenarios:
+- "It looks like we haven’t received any input for a while. Are you still there? Please respond so I can continue assisting you."
+
+13. Additional Contextual Examples
+13.1 Handling Product Returns:
+- Query: "I want to return a product I purchased."
+- Response: "I'm here to help with that. Could you please provide the order ID and the reason for the return? Once I have this information, I’ll guide you through the return process."
+
+13.2 Subscription Upgrades:
+- Query: "How can I upgrade my current subscription?"
+- Response: "To upgrade your subscription, please log in to your account, navigate to the 'Subscriptions' section, and select 'Upgrade Plan.' Would you like me to email you a detailed guide on how to do this?"
+
+13.3 Providing Instructions:
+- Customer: "I’m having trouble changing my password."
+- Response: "No problem! To change your password, go to the 'Account Settings' page, then select 'Change Password.' You'll need to enter your current password and a new one. I can send a step-by-step guide to your email if you prefer."
+
+13.4 Multi-Turn Conversations:
+- Customer: "I'm interested in upgrading my plan."
+- Bot: "Great! We have several plans available. Would you like to upgrade to the 'Pro Plan' or the 'Premium Plan'? The Pro Plan offers more storage, while the Premium Plan includes unlimited access to all features."
+
+- Customer: "Tell me more about the Premium Plan."
+- Bot: "The Premium Plan includes unlimited access to all features, 24/7 priority support, and a 30-day free trial. Would you like to proceed with upgrading to the Premium Plan?"
+
+14. Guidelines for Effective Conversations
+1. Always use polite language and maintain a friendly tone, even when dealing with complex or sensitive topics.
+2. Confirm information with the customer to ensure accuracy before proceeding with any actions.
+3. If the bot cannot handle the request, provide a clear message and suggest transferring to a human agent.
+4. Use follow-up questions to keep the conversation natural and gather more information when needed.
+5. Acknowledge the customer's concern, particularly in cases of frustration, to demonstrate empathy and a willingness to assist.
+6. Provide concise instructions and confirm customer understanding at each step of troubleshooting.
+7. Repeat key information such as order numbers or confirmation IDs to ensure the customer has received the necessary details.
+8. Personalize responses based on the customer's interaction history to make the conversation more relevant and efficient.
+
+15. Context Handling and Memory Implementation
+- Utilize conversation memory to store the context of current interactions. This allows the bot to reference previous discussions, providing a more tailored and relevant experience for the customer.
+- Example: If the customer previously inquired about a subscription upgrade, the bot can follow up in future interactions with, "I noticed you were interested in upgrading your subscription during our last conversation. Would you like to continue with that today?"
+
+- Maintain history for multi-turn conversations, ensuring that the bot can handle ongoing queries and provide consistent responses.
+
+16. Advanced Troubleshooting Scenarios
+16.1 Internet Connectivity Issues:
+- Customer: "My internet is still not working after restarting the modem."
+- Bot: "I’m sorry to hear that. Let's try another step. Could you please check if all the cables are securely connected to the modem and router? If everything is connected properly, we might need to reset the modem to factory settings. Would you like to proceed with that?"
+
+16.2 Payment Dispute:
+- Customer: "I was charged twice for my subscription."
+- Bot: "I understand your concern. Let’s take a closer look. Please provide the invoice numbers or the dates of the charges, and I will verify them for you. If there's an error, I will initiate the refund process immediately."
+
+16.3 Software Installation Issues:
+- Customer: "I can't install your software on my computer."
+- Bot: "I’m here to help. Could you please let me know which operating system you’re using (Windows, macOS, or Linux)? Also, have you tried running the installer as an administrator?"
+
+17. Technical Implementation Guidance
+- Implement Voice Activity Detection (VAD) to determine when the customer is speaking and when to pause for responses.
+- Use OpenAI’s Whisper for converting audio input to text, ensuring accurate transcription of customer queries.
+- Integrate with a Large Language Model (LLM) to generate responses based on the conversation's context and the stored customer data.
+- Develop a conversation memory module using a tool like LangChain to maintain context during multi-turn interactions, allowing the bot to recall previous customer inputs and inquiries.
+
+24. Expanded Edge Cases and Error Handling
+24.1 Handling Multiple Inputs:
+- If the customer provides multiple requests in one input: "I noticed you mentioned both a billing issue and a subscription upgrade. Let’s address each one. First, could you please provide more details about the billing issue?"
+
+24.2 Managing Incomplete Information:
+- If the customer provides partial information: "It looks like we need a bit more information to proceed. Could you please provide the invoice number or the date of the charge you’re inquiring about?"
+
+24.3 Dealing with Silent Calls:
+- If there is no input for an extended period: "I haven't heard from you in a while. Are you still there? If you need more time, feel free to take it, and I’ll be here when you’re ready."
+
+24.4 Handling Disconnected Calls:
+- If a call is disconnected abruptly: "It seems that the call has been disconnected. I will log this interaction and keep it on hold for when you reconnect. Please call back at your convenience, and we can continue from where we left off."
+
+24.5 Managing Repeated Requests:
+- If the customer repeats the same question: "I noticed you’ve mentioned this before. Just to clarify, are you asking for more details, or would you like to address a different aspect of the issue?"
+
+25. Technical Documentation for Developers
+25.1 Setting Up the Development Environment
+- Install Python: Download Python 3.10 or above from Python.org. Ensure the Python installation is added to the system path.
+- Create a Virtual Environment: Use the `venv` module to create a virtual environment for the project.
+```bash
+python3 -m venv call_center_bot_env
+source call_center_bot_env/bin/activate
+```
+
+25.2 Installing Required Libraries
+- Use Poetry for Dependency Management: Poetry is a versatile tool for managing project dependencies. To install Poetry, follow the instructions at https://python-poetry.org/docs/#installation.
+- Install Dependencies: Run the following command to install all necessary libraries including OpenAI, Pyttsx3, Streamlit, and more.
+```bash
+poetry install
+```
+
+25.3 Audio Processing Setup
+- Install FFmpeg: FFmpeg is required for handling audio files in different formats.
+  - **Windows:** Include `ffmpeg.exe` in the `libs` directory of your project.
+  - **Mac:** Use Homebrew to install FFmpeg.
+  ```bash
+  brew install ffmpeg
+  ```
+
+25.4 Connecting with OpenAI API
+- Obtain an OpenAI API Key: Sign up at https://openai.com and create an API key.
+- Store the API Key securely in an environment file.
+```env
+OPENAI_API_KEY="<YOUR_API_KEY>"
+```
+
+25.5 Sample Code for Processing Audio Input
+- Using OpenAI Whisper for Transcription
+```python
+import openai
+
+audio_file = open("path/to/audio/file.mp3", "rb")
+transcription = openai.Audio.transcribe("whisper-1", audio_file)
+
+print(transcription["text"])
+```
+- Converting the transcription to a conversational response using GPT-4
+```python
+response = openai.Completion.create(
+    engine="text-davinci-003",
+    prompt=transcription["text"],
+    max_tokens=150
+)
+
+print(response.choices[0].text.strip())
+```
+
+25.6 Building the User Interface with Streamlit
+- Create a basic Streamlit app to handle user interactions.
+```python
+import streamlit as st
+
+st.title("Call Center Bot")
+audio_input = st.file_uploader("Upload an audio file", type=["mp3", "wav"])
+user_query = st.text_input("Your query:")
+if st.button("Submit"):
+    st.write("Processing your input...")
+    # Add audio processing and response generation code here
+```
+
+26. Future Enhancements and Scalability
+26.1 Advanced AI Model Integration
+- Explore integrating additional AI models like BERT for more nuanced language understanding. This would allow the bot to handle more complex customer queries with increased accuracy.
+
+26.2 Multilingual and Cultural Context
+- Enable multilingual support with automatic language detection. Include cultural context in responses for international customers to ensure a more personalized and relevant interaction.
+
+26.3 Continuous Learning from Conversations
+- Implement a feedback loop where customer interactions are anonymized and used to retrain the bot's language models, improving response quality over time.
+
+26.4 Integration with External Services
+- Explore integrating with payment gateways, customer databases, and other third-party services to provide real-time account updates, payments, and order processing.
+
+26.5 Sentiment-Based Routing
+- Use sentiment analysis to route customers to specialized agents when the bot detects high frustration or negative sentiment, ensuring that sensitive issues receive appropriate human intervention.

data/call_center_prompt_with_intents_categories_context.csv

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8ddbd7dc83243ac2d4f5f7d6aa6ec0a249d99dd311e9a7dc8d7598f1fc0537c
+size 19087267

data/context.txt

@@ -0,0 +1,20 @@
+15. Supplementary note – The 20 market depth or level 3 data (https://zerodha.com/varsity/chapter/supplementary-note-the-20-market-depth/) The 20 Market Depth (level 3 data) Window I’ve driven a car for many years and I’ve even changed my car a few times now. Each time I changed my car, the engine remained more or less the same, but the features within the vehicle and its aesthetics continuously changed. Air conditioner, power steering, and power windows were all luxury features in the car at one point, but today, I guess no one buys a car without these essential features. The game-changer for me though was parking assist. The little camera at the back of the car gave me complete visibility of the parking space available. I was no longer required to pop and twist my head out and struggle to park the car, nor did I have to bug my co-passenger to get down and help me navigate my way into a parking spot. The parking assist feature did everything and helped me execute a perfect parallel park. The parking assist feature was my edge for hassle-free car parking. I feel the same edge while trading the markets with the level 3 data 🙂 Level 3 or the 20 market depth feature is unique and has multiple uses. You’ll probably appreciate the level 3 market window if you have traded at an institutional desk. A regular retail trader would not understand this feature anytime soon, simply because this feature was unavailable all these years until we introduced it for the very first time to the Indian retail traders. The purpose of this chapter is to help you understand how useful this feature is and get you started on building trading strategies around this feature. If you are entirely new to this, I’d suggest you read this blog (https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite) to understand what the level 3 data is all about. Assuming you know what it is, this chapter will help you understand the multiple uses of this feature. Contract availability For the option traders, the 20-depth order book gives great visibility into the availability of contracts to trade and help identify better price points to execute these trade. Without this visibility, it becomes really hard to trade illiquid contracts. While I’m specifically talking about options here, you can extend this to Futures contracts as well, especially the illiquid ones. Let us put this in context, have a look at the regular market depth (i.e. the top 5 bid-ask) of the 13000 CE expiring in Jan 2020. We can see narrow bids on the left and a notch better offer on the right. You’d probably hesitate to trade this contract if you are someone looking at trading a few lots of Nifty. But check what’s hiding under the hood here by opening the level 3 data –  As you can see, there are many contracts available, but they are not visible in the regular market depth. In fact, the bid and offer quantities are heavily concentrated below the 8 th row respectively. Given the availability of the contracts in this strike, the perspective to trade or not completely changes and will now depend upon your trading strategy. Execution control Level 3 data gives you full visibility of the approximate execution price for your trade. This is particularly useful when you decide to scalp the market. When you scalp the market — Let us say you want to buy and sell 5000 shares of Hindustan Zinc; the regular market depth window gives you the following information —  As you can see, there is no visibility on how these 5000 shares will get filled. Now, take a look at the 20 depth window —  The 20 depth window paints an entirely different picture. It not only tells me that I’ll get the 5000 shares, but it also gives me information about the approximate buy price. If I were to place a market order for 5000 shares, I’d be buying this order book from 210.5 to 211.25. I also see at 211; there are 2425 shares available, so I can expect the average price is at or around 211. Now, my decision to scalp the stock should depend on the pop I’d expect over and above 211. Maybe 211.5 or so. Of course, you’ll get the exact breakeven (post charges) if you were to use a brokerage calculator. Position sizing Level 3 market window plays a critical role in ‘guesstimating’ the number of shares to trade, given the liquidity of the stock. For the sake of this discussion, we will assume that the availability of capital is not an issue. Now, have a look at the regular market depth — You expect Siemens to move from 1675 to about 1690 over the next hour. So, given the fact that you are not constrained by capital, how many shares will you buy for this intraday trade? The regular market depth window suggests that you can buy close to 175 shares. However, the 20 depth opens up a different perspective altogether — In fact, the liquidity in this stock lies below the best five bid and ask, and the impact cost is reasonable. The regular market depth window fails to capture this information. Assuming you intend to buy about 1500 shares, the buy price will lie somewhere within 1675.5 to 1678, which is spread of 0.149%. In this case, assuming you are sure about the target price (1690), you can go all in and buy through whatever is available at that moment. Order placement You can extend the position sizing concept and use the 20 depth market watch to place a stop loss or a limit order. Assume you have an intraday buy position in VST Tillers at 1313.8. The question is, where you would place the stop loss for this trade? Can the 20 market depth help us with this? Of course. Have a look at the 20 depth window for VST Tillers. As you can see, there is a concentration of bids in 1290. The good part is that the number of order count is also the highest (35) in 1290. This implies that several traders have placed an order at 1290, indicating some sort of price action at that level. This perhaps builds a case for placing the stop-loss. A prudent trader would probably place a stoploss not at 1290, but maybe at a price just below it. So I was a buyer in this stock, then purely based on 20 depth I’d probably place my SL at 1290 or below, maybe at 1287 and by the same logic, set my target at 1340 or at 1338.8. Validate the support and resistance level I find this extremely interesting. In the example above, we identified 1290 as the stoploss price, simply because there was a concentration of bids. In other words, we expect 1290 as a support price. If this is indeed true, then it should show up on the charts as well, right? Have a look at the chart below – Clearly, there is some price action around 1296. Remember, support and resistance is not one price point, but rather a range. Therefore 1290 – 1300 marks as an intraday support for this stock. This is a perfect example of seeing the price action concept play out in the market. Another way to look at this is first to identify the S&R level and then check the 20 depth to figure if there is a concentration of bids/offers in that zone.  Hopefully, by now you’ve started to appreciate the immeasurable value 20 depth order book brings to you while trading. Remember, irrespective of which technique you use to develop a point of view (technical or quantitative analysis), things boil down to price, and the action trades take at that price. The 20 depth market window is essentially your ticket to validate the truth of this price action. Make sure you use your card wisely! Do post your comments and tell us how differently you will use the 20 depth window for identifying trading opportunities. Good luck!   608 comments cannot see market depth You need to have generated at least 100 Rupees in brokerage to see the 20 depth. Please check this – https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite (https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite) Very nicely written, thank you for your incredible insight. Salman, thank you 🙂 Sir, how will the market depth data be relied upon? When clients or trading systems use the disclosed quantity feature, not the entire quantity is shown. In such a case, isn’t there a chance that the wrong assumptions are made looking at just the market depth? Valid point, but there is no way one can identify disclosed quantity. By the way, level 3 helps you identify contract availability. Given this, the disclosed quantity, if any, will only make the liquidity situation better. Hi Karthik! Newbie here, under the ‘The 20 Market Depth (level 3 data) Window’ section, you’ve said – If you are entirely new to this, I’d suggest you read this blog to understand what the level 3 data is all about. Are you referring to this chapter as the blog or have a separate blog elsewhere, in which case I do not see a link to it. Can you help? ‘Read this blog’ is hyperlinked 🙂 Amazing featute for the reatil traders. Way to go team for bringing such innovative and pro Tools to the world at such decent costs. Happy learning and trading 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+12. Key Events and Their Impact on Markets (https://zerodha.com/varsity/chapter/key-events-and-their-impact-on-markets/) 12.1 – Events Trading or investing based on just company-specific information may not be sufficient. Outside events, both economic and/or non-economic, impact stocks and the market’s performance in general. It is also important to understand the events that influence the markets. In this chapter, we will try to understand some common events and how the stock market reacts to these events. 12.2 – Monetary Policy The monetary policy is a tool through which the Reserve Bank of India (RBI) controls the money supply by controlling the interest rates. RBI is India’s central bank. Likewise, every country’s central bank is responsible for setting interest rates. For example, the European Central Bank in Europe and Federal Reserves in the US. Central Banks tweak the interest rates to control the money supply in the mainstream economy. While setting the interest rates, the RBI has to strike a balance between growth and inflation. In a nutshell – if the interest rates are high, the borrowing rates are high (particularly for corporations). If corporate can’t borrow easily, they cannot grow. If corporations don’t grow, the economy slows down. On the other hand, borrowing becomes easier when the interest rates are low. This translates to more money in the hands of corporations and consumers. With more money, there is increased spending which means the sellers tend to increase the prices of goods and services, leading to inflation. I’d encourage you to watch this YouTube video where I’ve tried to explain what causes inflation and the means through which RBI controls inflation. To strike a balance, the RBI has to consider all economic factors and carefully set the key rates. Any imbalance in these rates can lead to economic chaos. The key RBI rates that you need to track are as follows: Repo Rate – Banks can borrow from the RBI. The rate at which RBI lends money to other banks is called the Repo Rate. If the repo rate is high, the cost of borrowing is high, leading to slow economic growth. You can check the latest repo rate (And other rates, too) on RBI’s website (https://www.rbi.org.in/) . Markets don’t like the RBI increasing the repo rates because it slows down economic growth. Reverse repo rate – Reverse Repo rate is the rate at which RBI borrows money from banks. Or in other words, Reverse Repo is the deposit rate RBI offers to other banks when the banks park funds with RBI. When banks deposit money to RBI, they are certain that RBI will not default, so the rate RBI offers is relatively low. However, the banking system’s money supply reduces when banks deposit money with RBI (at a lower rate) instead of the corporate entity. An increase in the reverse repo rate is not great for the economy as it tightens the money supply. Sometimes via the central bank’s policy, the central bank mandates higher deposits by banks; again, this is a way to curtail excess money supply in the mainstream economy. Cash reserve ratio (CRR) – Every bank must maintain funds with RBI. The amount that they maintain is dependent on the CRR. If CRR increases, more money is sucked out of the mainstream economy, which is not good for the economy. The monetary policy committee members meet regularly to review the economic situation and decide upon these key rates; hence keeping track of the monetary policy event is a must for any active trader. The first to react to rate decisions would be interest-rate sensitive stocks across various sectors such as – banks, automobiles, housing finance, real estate, metals, etc. 12.3 – Inflation Inflation is a sustained increase in the general prices of goods and services. Increasing inflation erodes the purchasing power of money. All things being equal, if the cost of 1 KG of onion has increased from Rs.15 to Rs.20, this price increase is attributed to inflation. Inflation is inevitable, but a high inflation rate is not desirable as it could lead to economic uneasiness. A high level of inflation tends to send a bad signal to markets. Both the Government and RBI work towards reducing inflation to a manageable level. Inflation is generally measured using an index. If the inflation index increases by certain percentage points, it indicates rising inflation. Likewise, an index falling indicates inflation cooling off. There are two inflation indices – The Wholesale Price Index (WPI) and Consumer Price Index (CPI). Wholesale Price Index (WPI) – The WPI indicates the movement in prices at the wholesale level. It captures the price change when goods are bought and sold wholesale. WPI is an easy and convenient method to calculate inflation. The inflation measured here is at an institutional level and does not necessarily capture the consumer’s inflation. Consumer Price Index  (CPI) – The CPI, on the other hand, captures the effect of the change in prices at a retail level. As a consumer, CPI inflation is what matters. The calculation of CPI is quite detailed as it involves classifying consumption into various categories and subcategories across urban and rural regions. Each of these categories is made into an index, the final CPI index is a composition of several internal indices. The CPI captures the effect of inflation on daily household consumables like fruits, vegetables, cereals, and even fuels like petrol and diesel. The computation of CPI is quite rigorous and detailed. It is one of the most critical metrics for studying the economy.  A national statistical agency, the Ministry of Statistics and Programme Implementation (MOSPI), publishes the CPI numbers around the 2 nd week of every month. The RBI’s challenge is to strike a balance between inflation and interest rates. Usually, a low-interest rate tends to increase inflation, and a high-interest rate tends to arrest inflation. 12.4 – Index of Industrial Production (IIP) The Index of Industrial Production (IIP)  is a short-term indicator of the country’s industrial sector’s progress. The data is released every month (along with inflation data) by the Ministry of Statistics and Programme Implementation (MOSPI). As the name suggests, the IIP measures the Indian industrial sectors’ production, keeping a fixed reference point. As of today, India uses the reference point of 2004-05. The reference point is also called the base year. Roughly about 15 different industries submit their production data to the ministry, which collates the data and releases it as an index number. If the IIP increases, it indicates a vibrant industrial environment (as the production is going up) and hence a positive sign for the economy and markets. A decreasing IIP indicates a sluggish production environment, hence a negative sign for the economy and markets. To sum up, an upswing in industrial production is good for the economy, and a downswing rings an alarm. As India is getting more industrialized, the relative importance of the Index of Industrial Production is increasing. A lower IIP number puts pressure on the RBI to lower the interest rates and aid industrial credit with cheaper credit. 12.5 – Purchasing Managers Index (PMI) The Purchasing managers’ index (PMI) is an economic indicator that tries to capture business activity across the country’s manufacturing and service sectors. This is a survey-based indicator where the respondents – usually the purchasing managers- indicate their business perception change concerning the previous month. A separate survey is conducted for the service and manufacturing sectors. The data from the survey are consolidated on a single index. Typical areas covered in the survey include new orders, output, business expectations, and employment. The PMI number usually oscillates around 50. A reading above 50 indicates expansion, and below 50 indicates a contraction in the economy. And reading at 50 indicates no change in the economy. 12.6 – Budget A Budget is an event during which the Ministry of Finance discusses the country’s finance in detail. The Finance Minister, on behalf of the ministry, makes a budget presentation to the entire country. During the budget, major policy announcements and economic reforms are announced, which impacts various industries across the markets. Therefore the budget plays a vital role in the economy. To illustrate this further, in one of the recent budgets, the expectation was to increase the duties on a cigarette. As expected, during the budget, the Finance Minister raised the duties on a cigarette, so the prices increased. An increased cigarette price has a few implications: In reaction to the budget announcement, ITC traded 3.5% lower for this precise reason. A budget is an annual event, and it is announced during the last week of February. However, the budget announcement could be delayed under certain special circumstances, such as a new government formation. 12.7 – Corporate Earnings Announcement Corporate earning season is perhaps one of the important events to which the stocks react. The listed companies (trading on the stock exchange) must declare their earnings once every quarter, also called the quarterly earnings numbers. During an earnings announcement, the corporate gives out details on various operational activities, including: Besides, some companies give an overview of what to expect from the upcoming quarters. This forecast is called ‘corporate guidance.’ Invariably every quarter, the first blue-chip company to make the quarterly announcement is Infosys Limited. They also give out guidance regularly. Market participants follow what Infosys has to say regarding guidance as it impacts the markets. The table below gives you an overview of the earning season in India: Do note that the 1st of April in India marks the beginning of the financial year. In the US, the financial year starts on 1st Jan, so the first quarter starts from January through March, and so forth. Every quarter when the company declares its earnings, the market participants match the earnings with their expectations of how much the company should have earned. The market participant’s expectation is called the ‘street expectation.’ The stock price will react positively if the company’s earnings are better than the street expectations. The stock price will react negatively if the actual numbers are lower than the street expectation. If the street expectation and actual numbers match, the stock price tends to trade flat with a negative bias more often than not. This is mainly because the company could not give any positive surprises. 12.8 – Non Financial events Apart from the events we discussed above, it would be best to watch out for other non-financial events to understand their impact on markets. For example, the Covid crisis of 2020 had a significant effect on economies around the world, disrupting the world economic order. The supply chain took a hit across the globe leading to an inflation spike. That said, there were select pockets of the economy that did very well, mainly the online services industry. Events like the Russia – Ukraine war or the tension between China and Taiwan have impacted world markets. Geo-political affairs such as these impact various connected economies. For instance, the war between Russia and Ukraine affects the supply of natural gas and crude oil, which significantly impacts the energy costs in Europe. As an active trader or a market participant, you need to watch out for these events and understand how these events can impact the markets. While the world economies are interconnected, isolated events (Country specific) impact the local economy. For example, the elections in India impact only the Indian economy. So, keep an eye on these non-financial events and how they can impact the stock markets or sometimes specific industries. Key takeaways from this chapter 329 comments 1. From the close of a quarter/financial year, what are the time limits within which companies must declare the results? 2. In case of insider trading (SAST), what is the time limit to inform the exchanges about it ? 3. Is there any rule/law making it mandatory for companies to report events with financial significance? Answers in the same order – 1) If I’m not wrong companies must declare results  within 30 days 2) In order to make it a level playing field SEBI mandates the companies to disclose insider trading ‘at the earliest’, which pretty much means the company has to declare this information within the same trading day 3) Yes companies have to declare this information. Hi, What is insider trading and SAST ? How it works? Is their any facility into Versity to get notification once get solution of query? You should get an email notification when this gets answered. Anyway, insider trading is basically trading on information not known to the public. For example, an employee of TCS can buy the shares of TCS based on his knowledge of the quarterly results yet to be announced. Thanks Karthik 🙂 Cheers! By 8% & 7% in Repo and Reverse repo rate, do you mean it is same as we borrow from banks at 11 % ? No, repo & reverse repo rate are rate at which banks transact with RBI. The lending rates to us are different. Sir, i am not able to understand “Purchasing managers index(PMI)” PMI or the Purchasing Manager’s Index is an index which measures the activity of a purchasing manager (PM) of a company. This is measured by running a survey across many PM’s across sectors and industries. If the PM is sheen to be purchasing things for the company then the perception of growth and industrial activity is created, which is good for the markets. Else its considered not so good. Thank you sir Welcome 🙂 Important & better for new trader You bet, it is.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+16. Momentum Portfolios (https://zerodha.com/varsity/chapter/momentum-portfolios/) 16.1 – Defining Momentum If you have spent some time in the market, then I’m certain you’ve been bombarded with market jargon of all sorts. Most of us get used to these jargon and start using them without actually understanding what they mean. I’m guilty of using a few jargon without understanding the true meaning of it, and I get a feeling that some of you reading this may have experienced the same. One such jargon is – momentum. I’m sure we have used momentum in our daily conversations related to the markets, but what exactly is momentum, and how is it measured? When asked, traders loosely define momentum as the speed at which the markets move. This is correct to some extent, but that’s not all, and we should certainly not limit our understanding to just that. ‘Momentum’ is a physics term. It refers to the quantity of motion an object has. If you look at this definition in the context of stock markets, everything remains the same, except that you will have to replace ‘object’ with stocks or the index. Momentum is the rate of change of stock returns or the index. If the rate of change of returns is high, then the momentum is considered high; if the rate of change of returns is low, the momentum is considered low. This leads us to the next obvious question i.e. what is the rate of change of returns? The rate of change of return, as it states the return generated  (or eroded) between two reference periods. For the sake of this discussion, let’s stick to the rate of change of return on an end-of-day basis. So in this context, the rate of change of returns means the speed at which the daily return of the stock varies. To understand this better, consider this example – The table above shows an arbitrary stock’s daily closing price for six days. Two things to note here – Consider another example – Two things need to note – Given the behavior of these two stocks, I have two questions for you – To answer these questions, you can look at either the absolute change in the Rupee value or the percentage change from a close-to-close perspective. If you look at the absolute Rupee change, the change in Stock A is higher than in Stock B. However, this is not the right way to look at the change in daily return. For instance, in absolute Rupee terms, stocks in the range of, say, 2000 or 3000 will always have a higher change compared to stocks in the range of 1000 or lower. Hence, evaluating absolute Rupee change will not suffice, and therefore we need to look at the percentage change. In terms of percentage change, Stock B’s daily change is higher, and therefore we can conclude that Stock B has a higher momentum. Here is another situation, consider this – Stock A has trended up consistently daily, while stock B has been quite a dud all along except for the last two days. On an overall basis, if you check the percentage change over the 7 days, then both have delivered similar results. Given this, which of these two stocks is considered to have good momentum? Well, Stock A is consistent in terms of daily returns, exhibits a good uptrend, and, therefore, can be considered to have continuity in showcasing momentum. Now, what if I decide to measure momentum slightly differently? Instead of daily returns, what if we were to look at the return on a 7 days basis? If we do that, stocks A and B qualify as momentum stocks. The point I’m trying to make here is that traders generally look at momentum in terms of daily returns, which is perfectly valid, but this is not necessarily the only way to look at momentum. The momentum strategy we will discuss later in this chapter looks at momentum on a larger time frame, not daily. More on this later. I hope by now; you do have a sense of what momentum means and understand that momentum can be measured not just in terms of daily returns but also in terms of larger time frames. High-frequency traders measure momentum on a minute-to-minute or hourly basis. 16.2 – Momentum Strategy Among the many trading strategies traders use, momentum is one of the most popular strategies. Traders measure momentum in many different ways to identify opportunity pockets. The core idea across all these strategies remains the same, i.e., to identify momentum and ride the wave. Momentum strategies can be developed on a single-stock basis. The idea is to measure momentum across all the stocks in the tracking universe and trade the ones that showcase the highest momentum. Remember, momentum can be either long or short, so a trader following a single stock momentum strategy will get both long and short trading opportunities. Traders also develop momentum strategies on a sector-specific basis and set up sector-specific trades. The idea here is to identify a sector that exhibits strong momentum; this can be done by checking momentum in sector-specific indices. Once the sector is identified, look for the stocks that display maximum strength in terms of momentum. Momentum can also be applied on a portfolio basis. This involves portfolio creation with, say ‘n’ number of stocks, with each stock showcasing momentum. In my opinion, this is an excellent strategy as it is not just a plain vanilla momentum strategy but also offers safety in diversification. We will discuss one such strategy wherein the idea is to create a stock basket, aka a portfolio of 10 momentum stocks. Once created, the portfolio is held until the momentum lasts and then re-balanced. 16.3 – Momentum Portfolio Before we discuss this strategy, I want you to note a few things – Given the above, here is a systematic guide to building a ‘Momentum Portfolio’. Step 1  – Define your stock universe As you may know, there are close to 4000 stocks on BSE and about 1800 on NSE. This includes highly valuable companies like TCS and absolute thuds such as almost all the Z category stocks on BSE. Companies such as these form the two extreme ends of the spectrum.  Do you have to track all these stocks to build a momentum portfolio? Not really. Doing so would be a waste of time. One has to filter out the stocks and create the ‘tracking universe.’ The tracking universe will consist of a large basket of stocks within which we will pick stocks to constitute the momentum portfolio. The momentum portfolio will always be a subset of the tracking universe. Think of the tracking universe as a collection of your favorite shopping malls. Maybe out of the 100s of malls in your city, you may go to 2-3 shopping malls repeatedly. Clothes bought from these 2-3 malls comprise your entire wardrobe (read portfolio). Hence, these 2-3 malls form your tracking universe out of the 100s available in your city. The tracking universe can be pretty straightforward – the Nifty 50 or BSE 500 stocks. Therefore, the momentum portfolio will always be a subset of the Nifty 50 or BSE 500 stocks. Keeping the BSE 500 stocks as your tracking universe is an excellent way to start. However, if you feel adventurous, you can custom-create your tracking universe. Custom creation can be on any parameter – for example, out of the entire 1800 stocks on NSE, I could use a filter to weed out stocks that have a market cap of at least 1000 Crs. This filter alone will shrink the list to a much smaller, manageable set. Further, I may add other criteria, such as the stock price should be less than 2000. So on and so forth. I have randomly shared a few filter ideas, but you get the point. Using custom creation techniques helps you filter out and build a tracking universe that matches your requirement. Lastly, from my personal experience, I would suggest you have at least 150-200 stocks in your tracking universe if you wish to build a momentum portfolio of 12-15 stocks. Step 2  – Set up the data Assuming your tracking universe is set up, you can proceed to the 2 nd step. In this step, you must ensure you get the closing prices of all the stocks in your tracking universe. Ensure your data set is clean and adjusted for corporate actions like the bonus issue, splits, special dividends, and other corporate actions. Clean data is the crucial building block to any trading strategy. There are plenty of data sources from where you can download the data for free, including the NSE/BSE websites. The question is – what is the lookback period? How many historical data points are required? To run this strategy, you only need 1-year data point. For example, today is 2 nd March 2019; then I’d need data points from 1 st March 2018 to 2 nd March 2019. Please note once you have the data points for the last one-year set, you can update this daily, which means the daily closing prices are recorded. Step 3  – Calculate returns This is a crucial part of the strategy; in this step, we calculate the returns of all the stocks in the tracking universe. As you may have already guessed, we calculate the return to get a sense of the momentum in each stock. As discussed earlier in this chapter, one can calculate the returns on any frequency, be it daily/weekly/monthly, or even yearly returns. We will stick to yearly returns for the sake of this discussion; however, please note; you can add your own twist to the entire strategy and calculate the returns for any time frame you wish. Instead of yearly, you could calculate the half-yearly, monthly, or even fortnightly returns. So, you should have a tracking universe of about 150-200 stocks at this stage. All these stocks should have historical data for at least 1 year. Further, you need to calculate the yearly return for each stock in your tracking universe. To help you understand this better, I’ve created a sample tracking universe with just about ten stocks in it. The tracking universe contains the data for the last 365 days. The 1-year returns are calculated as well – If you are wondering how the returns are calculated, then this is quite straightforward, let us take the example of ABB – Return = [ending value/starting value]-1 = [1244.55/1435.55]-1 = -13.31% Relatively straightforward, I guess. Step 4 – Rank the returns Once the returns are calculated, you need to rank the returns from the highest to the lowest returns. For example, Asian paints has generated a return of 25.87%, the highest in the list. Hence, the rank of Asian paints is 1. The second highest is HDFC Bank, which will get the 2 nd rank.  Infosys’s return, on the other hand, is -35.98%, the lowest in the list; hence the rank is 10. So on and so forth. Here is the ‘return ranking’ for this portfolio – If you are wondering why the returns are negative for most of the stocks, that’s how stocks behave when deep corrections hit the market. I wish I had opted to discuss this strategy at a better point. So what does this ranking tell us? If you think about it, the ranking reorders our tracking universe to give us a list of stocks from the highest return stock to the lowest. For example, from this list, I know that Asian Paints has been the best performer (in terms of returns) over the last 12 months. Likewise, Infy has been the worst. Step 5 – Create the portfolio A typical tracking universe will have about 150-200 stocks, and with the help of the previous step, we would have reordered the tracking universe. Now, we can create a momentum portfolio with the reordered tracking universe. Remember, momentum is the rate of change of return, and the return itself is measured yearly. A good momentum portfolio contains about 10-12 stocks. I’m comfortable with up to 15 stocks in the portfolio, not more than that. For the sake of this discussion, let us assume that we are building a 12 stocks momentum portfolio. The momentum portfolio is now the top 12 stocks in the reordered tracking universe. In other words, we buy all the stocks starting from rank 1 to rank 12. In the example we were dealing with, if I were to build a 5-stock momentum portfolio, then it would contain – The rest of the stocks would not constitute the portfolio but will remain in the tracking universe. You may ask what is the logic of selecting this subset of stocks within the tracking universe? Well, read this carefully – if the stock has done well (in terms of returns generated) for the last 12 months, it implies that it has good momentum for the defined time frame. The expectation is that this momentum will continue onto the 13 th month, and therefore the stock will continue to generate higher returns.  So if you were to buy such stocks, you are to benefit from the expected momentum in the stock. This is a claim. I do not have data to back this up, but I have successfully used this technique for several years. It is easy to back-test this strategy, and I encourage you to do so. Back in the day, my trading partner and I were encouraged to build this momentum portfolio after reading this ‘Economist’ article. You need to read this article before implementing this strategy. Once the momentum portfolio stocks are identified, the idea is to buy all the momentum stocks in equal proportion. So if the capital available is Rs.200,000/- and there are 12 stocks, the idea is to buy Rs.16,666/- worth of each stock (200,000/12). By doing so, you create an equally weighted momentum portfolio. Of course, you can tweak the weights to create a skewed portfolio, there is no problem with it, but then you need a solid reason for doing so.  This reason should come from backtested results. If you like to experiment with skewed portfolios, here are few ideas – So on and so forth. Ideally, the approach to capital allocation should come from your backtesting process, this also means you will have to backtest various capital allocation techniques to figure out which works well for you. Step 6 – Rebalance the portfolio So far, we have created a tracking universe, calculated the 12-month returns, ranked the stocks in terms of the 12-month returns, and created a momentum portfolio by buying the top 12 stocks. The momentum portfolio was built based on the 12-month performance, hoping to continue to showcase the same performance for the 13 th month. There are a few assumptions here – Now the question is, what happens at the end of the month? At the end of the month, you re-run the ranking engine and figure out the top 10 or 12 stocks which have performed well over the last 12 months. Do note at any point, we consider the latest 12 months of data. So, we now buy the stocks from rank 1 to 12, just like we did in the previous month. From my experience, chances are that out of the initial portfolio, only a hand full of stocks would have changed positions. So based on the list, you sell the stocks that no longer belong in the portfolio and buy the new stocks featured in the latest momentum portfolio. In essence, you rebalance the portfolio and you do this at the end of every month. So on and so forth. 16.4 – Momentum Portfolio variations Before we close this chapter (and this module), I’d like to touch upon a few variations to this strategy. The returns have been calculated on a 12-month portfolio and the stocks are held for a month. However, you don’t have to stick to this. You can try out various options, like – As you can see, the options are plenty, and your imagination only restricts it. If you think about what we have discussed so far, the momentum portfolio is price based. However, you can build a fundamental-based momentum strategy as well. Here are a few ideas – You can do this on any fundamental parameter – EPS growth, profit margin, EBITDA margin etc. The beauty of these strategies is that the data is available, hence backtesting gets a lot easier. 16.5 – Word of caution As good as it may seem, the price-based momentum strategy works well only when the market is trending up. When the markets turn choppy, the momentum strategy performs poorly, and when the markets go down, the momentum portfolio bleeds heavier than the markets itself. Understanding the strategy’s behavior with respect to the market cycle is crucial to this portfolio’s eventual success. I learned it the hard way. I had a great run with this strategy in 2009 and ’10 but took a bad hit in 2011. So before you execute this strategy, do your homework (backtesting) right. Having said all of that, let me reassure you – a price-based momentum strategy, if implemented in the proper market cycle can give you great returns, in fact, better more often than not, better than the market returns. Good luck and happy trading. Key takeaways from this chapter 240 comments Dear sir,What about volatility based delta hedging strategy? Kehav, perhaps I’ll add that sometime as an addendum. Moving forward onto the next module. When will Varsity Android app launch.. waiting for it.. Here you go, Ankit – https://play.google.com/store/apps/details?id=com.zerodha.varsity&showAllReviews=true (https://play.google.com/store/apps/details?id=com.zerodha.varsity&showAllReviews=true) , please don’t forget to rate us on play store 🙂 Hey thanks sir.. app is awsum.. hopeing for all the modules to be added soon.. and even the new app of kite 3 is awsum.. thanks alot… I’m happy to note that, Ankit! Lots of efforts towards building these apps 🙂 Yes, eventually all the modules will be added to the app. Did you check out the wall feature? h1 Sir, Thanks for an this topic was waiting for it to be covered.. According to you which is the best source to get eod data which is adjusted of splits and bonus for bse 500 stocks apart from nse/bse websites. And secondly you said u need a coder to get a this strategy working can u share excel sheets to calculate momentum score as you did for pair trading chapter Thanks Nick, there are plenty of data vendors who provide you clean data. I’d suggest you subscribe to any of them. Unfortunately, I was unable to produce the excel, hence I took up the example of 10 stocks. At present I am investing on long-term basis and good results and very much Happy with the returns. I want to start trading shortly and watching articles related… thanks… waiting further. Good luck, Shakeel! Sir correct me if I’m wrong but this is one of the longest chapters in varsity and I enjoyed reading it in one sitting. I’m currently in the process of back testing the portfolio based on momentum. What would the next chapter be about sir? I’m glad you could read through it in one sitting, do share the backtested results if you are comfortable with it 🙂 This module ends with this chapter. We are moving ahead with the next one.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+1. Sector analysis overview (https://zerodha.com/varsity/chapter/sector-analysis-overview/) 1.1 – Introduction Welcome to another Varsity Module! This module on Varsity explores sectors as an investment avenue. We all know that a well-diversified portfolio is the key to building a successful stock portfolio. The diversification should be across sectors and market capitalizations. If we are looking at picking stocks from different sectors, we need to understand sectors from a stock-picking perspective. The objective of this module is just that, i.e., to help you understand what to look for in each sector as a stock picker or an equity investor. Each chapter will focus on one sector. So after reading this chapter, you can jump across to any sector you are curious to learn about without worrying about losing the chain of thought. By the way, my name is Vineet Rajani; I hold a CFA charter and have four years of experience in researching equities. I recently joined the Zerodha Varsity team to help Karthik develop content, and the sector analysis module is the first module I’ll be working on. I hope you all will like this module as much as you’ve liked all the previous modules 🙂 1.2 – What is a sector, and what is an industry? A sector is a set of companies engaged in similar business activities. For example, Infosys, TCS, and HCL are similar businesses, forming the Information Technology sector. HDFC Bank, ICIC Bank, and SBI are banks, and these companies belong to the banking sector. Sun Pharmaceuticals, Apollo Hospitals, and Dr. Lal Path Labs are all companies belonging to the healthcare sector. Sectors can have sub-sectors or industries. Sectors and industries are often used interchangeably. Each industry has a distinct business at a granular level, but at a broader level, many industries can make one sector. While banking, insurance, and mutual funds are all distinct industries, collectively, they are financial services and make up the financial services sector. Similarly, the healthcare sector comprises sub-sectors such as hospitals, diagnostics, pharmaceuticals, pharmacies, preventive healthcare, and wellness. The following table classifies various industries in a limited set of sectors. There are about 3-4 globally accepted standards for industry classification. This table shows how Refinitiv (https://www.refinitiv.com/content/dam/marketing/en_us/documents/quick-reference-guides/trbc-business-classification-quick-guide.pdf) classifies the sectors and industries. An investor would want to analyze a particular sector when they see a factor or phenomenon influencing it. For instance, an investor investing in the fertilizer or packaged foods sector would want to take note of the monsoon season’s data as it tends to impact those sectors as a whole. Or an investor in the IT sector may want to keep an eye on the USD-INR exchange rate as the sector makes a significant part of revenues in USD. Investors also take an interest in the sector of their occupation – a doctor might want to invest in healthcare stocks, or a software engineer might be confident about their understanding of the technology sector. Investor frenzy in a particular sector also stimulates the interest of other investors in that sector. 1.3 –  What is sector analysis? Sector analysis involves looking for factors, features, events, and metrics that impact the businesses in a given sector. A factor might positively impact one sector while a negative impact on another. The idea of sector analysis hinges on the fact that certain aspects or events are specific to sectors and do not impact the overall market. Sector analysis is a part of fundamental analysis. While our module on fundamental analysis explains the approach to researching a particular company, sector analysis focuses on the features and operational or performance metrics unique to a sector. The exercise becomes meaningful when several companies within a sector are compared based on these metrics. 1.4 -Different Approaches for Different Sectors No two sectors are the same; therefore, no two sectors can be analyzed the same way. Banks, for instance, are analyzed using key performance indicators such as NPAs, capital adequacy ratios, and interest margins. Insurance companies are analyzed for solvency ratio, claims settlement ratio, expense ratio, persistence ratio, etc. Airlines look at revenue per seat kilometer, cost per seat kilometer, fuel costs, and occupancy rates to understand performance. These metrics depict the operational efficiency of the players in an industry and how those players stack against each other. For industries in the heavy manufacturing space – cement, steel, aluminum, and the like – production capacity, production volume, and sales volume are important comparables. Volume metrics are significant for automobiles and electronics too. Companies in FMCG, or Fast Moving Consumer Goods, focus significantly on distribution, brand awareness, packaging, etc. When an investor begins studying a sector, understanding the value chain could be a good starting point (I will explain the concept of “value chain” soon). A study of the value chain provides more insights into a particular sector’s unique dynamics. The exercise could also unearth certain industry players’ competitive advantages or disadvantages. 1.5 – What is a value chain? Simply put, a value chain begins with the sourcing of raw materials and goes up to the point of end consumption. For example, the textile industry’s value chain would include fiber production, spinning yarn, fabric production, dying and printing, garment manufacturing, packaging, distribution, and retail. Cement’s value chain starts with limestone mining, followed by clinkerization, blending, grinding, packaging, and distribution. This value chain might be extended if the cement manufacturer processes it further into ready-mix concrete (RMC) before selling it in the market. Dissecting the value chain in this manner enables an investor to identify which steps drive costs or improve or hamper productivity. A value chain typically has many steps. Here, the investor must put on a business owner’s hat to understand what steps along the value chain add value to the business and what do not. Cement companies generally own the limestone mines and all the processes up to distribution. The cement industry is predominantly vertically integrated. Let me introduce three new concepts: Vertical Integration, Backward Integration, and Forward Integration. Although a rare phenomenon, a fully vertically integrated company owns all the processes from raw material production to end consumer retail sales. Samsung, a global leader in consumer electronics, is essentially a vertically integrated business. It produces semiconductors, memory chips, and screens that go into making mobile phones, then assembles mobile phones at its plants and even sells them through its own retail stores. While Samsung also sells through other retailers and e-commerce channels, its retail stores symbolize some level of vertical integration. Most companies within an industry exhibit similar business structures across the value chain. FMCG companies in India can hardly be backward integrated. Palm oil, a key ingredient in many food products, personal care, and cosmetic goods, is primarily sourced from Malaysia and Indonesia. Packaging for these products uses petroleum derivatives which have their source in oil-producing countries. It does not make business sense for an FMCG company to own supply-side processes. The case of the FMCG sector suggests that the geography of the value chain also influences a business’ vertical integration, costs, and material availability. Automobile companies are mostly assembly companies. The thousands of parts that go into making a car are all sourced from vendors that may be geographically located far apart. The significance of packaging in a value chain depends on the nature of a product, its application, size, shape, and perishability. It also depends on where the end consumer is located and what modes of transport will be used to deliver the product. Regulatory requirements must also be met concerning packaging material, dimensions, eco-friendliness, etc. 1.6 – Framework  These concepts above can be put into a framework for an investor researching a particular sector. This framework can act as a checklist for the investor to ensure that adequate efforts have been dedicated to comprehensively understanding a sector. Political Factors: The political will of the ruling government can influence the overall business sentiment in an economy as a whole. Businesses engaged in sin goods such as liquor and tobacco and necessities such as grains and oil often carry large political overhangs. Economic Factors : The strength of an economy and the stage of economic development can present different opportunities for different sectors. High-interest rates could limit corporate growth, while low-interest rates could enable easy borrowing and faster growth. However, low-interest rates could encourage excessive retail borrowing and spur inflation. The ability of the economic authorities to balance growth and control inflation is of the essence here. Economic factors such as inflation, exchange rates, FDI, and money supply play along with interest rates to influence the business environment. Socio-cultural Factors : Festivals can stimulate seasonal buying for specific sectors. Socio-cultural shifts can create long-term trend growth for certain products while a decline in others. The gradual move from coal-fired stoves to gas stoves results from economic development and social acceptance of a new cooking method. Demographic features and changes are a major influence on demand creation. The vast size of India’s youth population, in both absolute and percentage terms, has led to the creation of various products and brands meant to attract the young crowd. A decline in the average fertility rate suggests a decline in population growth. The aging of today’s young population could boost the demand for old-age products after three-four decades. Technological Factors : The stories of Nokia and Blackberry losing market share to Apple and Samsung have been discussed every time the impact of technological change has to be depicted. Innovations in technology can create new sectors and even wipe out some. All industries related to e-commerce were built upon the Internet. The Internet was a technological breakthrough. On the other hand, typewriters and Telegram are technologies that died as better ones replaced them. Legal Factors : The duties, tariffs, quotas, and other restrictions have an impact on the import-export trade of a country. Legal factors often create entry barriers for new players to enter a particular industry. For example, the heavy licensing requirements and different state-wise laws make liquor manufacturing a capital-intensive and difficult industry to get into. Pollution control, labor laws, and regulators like SEBI, RBI, and IRDAI constitute an economy’s legal machinery. Environmental Factors : The natural environment of a country can present opportunities for some sectors while discouraging others. Mining businesses are possible only if a country has minerals under its earth. Pollution and environmental deterioration result in imposing restrictions and requirements on industries. Water and effluent treatment plants have become a regulatory mandate for various chemical and industrial goods companies. Natural calamities could purge industries while displacing civilizations. Insurance companies carry a huge risk on account of natural disasters. These factors also influence each other. Economic development could lead to social development and shifts in culture. Issues related to society and the environment could be behind the development of certain legal barriers. Some legal factors can also be politically influenced. These factors make the PESTLE (https://pestleanalysis.com/what-is-pestle-analysis/) Analysis (Political-Economic-Social-Technological- Legal-Environmental), a framework commonly taught in colleges and business schools. 1.7 – How companies within a sector can differ? Beyond the PESTLE framework, as an investor, you must also study how a particular company competes with its peers. By the way, competition is much more than just outselling each other. A company also competes during the sourcing of raw materials. Two dairy companies, for example, will compete with each other to source milk from farmers. Companies also compete with new players in the industry. New players can disrupt the industry. A large influx of new players can change the industry’s competitive landscape. Industries also have to worry about being replaced by substitutes. A company’s ability to outrun its competition and negotiate with external factors depends on what differentiates it from its peers. Let us look at a few differentiating factors. Size : Size is essentially about the capital of the company. Being big or small comes with its advantages and disadvantages. Larger companies managed to stay afloat through the Covid-related lockdowns while many smaller companies shut down. Why did this happen? Business activity was mostly dull through the lockdowns. So regular operations were not yielding profit. Larger companies survived by using capital reserves. Remember how large retailers survived while many smaller retailers closed shops? Huge capital reserves also open opportunities that are otherwise out of reach. For example, the huge capital and licensing requirements to set up a telecom business are a barrier. Smaller businesses or individuals with small capital do not even consider starting a telecom business. But the deep pockets of Reliance Industries enabled it to pump in loads of capital to set up Jio. Its war chest was huge enough to wipe out some existing players (remember Aircel, Uninor) while others were forced to merge (Vodafone and Idea). Larger companies also enjoy economies of scale . Ever wonder why a readymade shirt generally costs lower than a tailor-made one? Makers of readymade shirts source everything in bulk to get deep discounts. These discounts can be passed down to customers in the form of lower prices. Conversely, a tailor cannot compete with the quick turnaround of automated sewing machines. A tailor charges a higher price to make up for the hours dedicated to a single shirt. Being small also has its advantages and disadvantages. A newly set-up business generally has limited funds, workers, and resources. But the ability to disrupt is high mostly because larger companies do not consider smaller companies a threat, and smaller companies do not have legacy issues. Let me break this down. For example, the Brushless DC (BLDC) technology has been used for 50 years in electricals to save electricity. However, it was first used in ceiling fans in the US only in 2009. And a few years later, smaller players like Versa Drives and Atomberg (https://atomberg.com/blog/post/ceiling-fans-a-brief-history) adopted the technology and made it big. The incumbent players had been improving upon the existing technology but did not consider an alternative technology as an energy-saving solution. Once the smaller players disrupted the space, all larger players introduced the BLDC range of fans under their brand names. Why didn’t the larger players act first? Introducing a new technology could mean re-training the production staff and introducing new machinery. Existing skills and systems make the management averse to changes. These are known as legacy issues . Decision-making often slows down in larger organizations. So even if they were aware of better technology, its adoption took time. Small companies are not considered a threat because even if they have a superior product, their ability to sell it and scale it up is limited. A few that do manage to scale up become the disruptors. Age : With age comes experience. It is a commonly known concept. It is applicable to businesses too. Sometimes experience helps businesses avoid the same mistakes. Sometimes, experience makes them averse to trying new methods, techniques and technologies. Just by the virtue of being around for years, businesses have a network of suppliers, distributors, and allies that a new business will take some time to build. However, new businesses with huge capital can overcome this obstacle. Ola and Uber disrupted other cab services mainly due to the large venture capital that they spent on technology, networking, and marketing. Long-standing relationships with vendors are useful when the supply of inputs is limited. They could also allow easier payment terms. A long-standing distribution network can be used to launch new products. For example, Polycab used its wide network of wire distributors and retailers to launch its electrical goods. Tata Consumer Products has been launching several new products that it can distribute using the existing Tata Tea and Tata Salt network. These long-standing relationships can also become a hurdle in making objective business decisions. For example, HUL and Colgate-Palmolive had to face disputes (https://www.moneycontrol.com/news/business/companies/why-hul-colgate-palmolive-are-in-a-tussle-with-distributors-7897591.html) with their long term distributors for offering differential pricing to B2B e-commerce platforms. HUL also faced boycott threats (https://www.thehindubusinessline.com/companies/hul-faces-distributors-ire-for-exploring-new-distributors-in-mp/article65822837.ece) in Madhya Pradesh when it was looking for distributors in addition to the existing ones. Newer consumer brands established their online channels before exploring physical retailing. They do not have long-standing relationships with any distributor. Therefore, they are unlikely to face protests like HUL and Colgate-Palmolive did. Focus : Focus can be related to products, target market, costs, or pricing. The product focus sounds very basic. Every business is focused on its product. However, some businesses have multiple products. Some organizations have multiple businesses. Reliance Industries has three large businesses – oil, retail, and telecom. Its next bet is going to be on financial services. Similarly, ITC is a large FMCG company with a presence in hospitality, technology, packaging, and agri-exports. So when you study conglomerates like these as part of a sector, you must consider what drives the business and profitability. You might even want to consider each business as a separate organization to draw proper comparisons with relevant peers. A company operating in a single sector might be better equipped to innovate and scale up than a peer conglomerate with other business interests. The difference is mainly about focus. A conglomerate has diluted focus across divisions. This is also why some analysts assign a conglomerate discount when valuing a conglomerate business. The target market is the target customers. Both Maruti Suzuki and Mercedes are automobile companies but are incomparable because they focus on different markets. Maruti will not be able to charge as high for its cars as Mercedes does. Similarly, Mercedes will not be able to sell as many cars as Maruti does. Their different market focus is also visible in their marketing tools. Maruti uses TV commercials to spread the word about its car. It wants the maximum population to buy its cars. Mercedes understands that millions may watch TV commercials, but only some can afford their cars. Therefore, it does not spend on TV commercials. Some businesses focus on attaining cost leadership . Indigo Airlines has managed to stay afloat while so many airlines are struggling or have perished. Indigo’s flight tickets may be priced similarly to its competitors, but its focus on cost controls has led to profitability. Other businesses focus on pricing . FMCG companies often price their products competitively, meaning they keep the prices low to maintain and increase the customer base. Ever wondered why many biscuits and chocolates still sell in ₹5-packs despite all the inflation after so many decades? It is because a consumer with limited means finds the ₹5 price point psychologically comforting. Round figures such as ₹5 or ₹10 sell more than odd figures such ₹6 or ₹7. Also, they might not be willing to spend ₹10 or higher when their budget is ₹5. Price leadership is not always about the lowest prices. Apple has become the world’s most valuable smartphone brand by always pricing its phones at a premium. Such pricing power is achieved with a heavy focus on the product and branding. Regulations also impact a company or sector’s pricing power. One can argue that the healthcare sector has strong pricing power. But because of its essential nature, the government regulates medicine prices. Substitutes : Substitutes come in various forms. Tea and coffee are each other’s substitutes. Mobile phones have substituted personal cameras, computers, diaries, and watches. In the case of luxury goods, a vacation, watch, car, handbag, and chandelier are all substitutes for each other. So when I pointed out earlier that Mercedes and Maruti may not be comparable, Mercedes could compare with other luxury goods. Substitutes are a challenge because they can come from a different sector or even create a new one altogether. There can also be regulatory support for substitutes in certain industries – solar and wind energy getting favorable policies over coal. Electric vehicles are also getting incentives over combustion engine (conventional) vehicles. Certain businesses, mostly large ones, can identify substitutes and even own them. For example, petrol and diesel pumps are now setting up EV charging and gas stations. Since substitutes can render an industry obsolete, an analyst must try to ascertain the magnitude of the threat from substitutes. Competition from substitutes pushes businesses to expand into the substitute business as well. New entrants in a sector often increase the competitive intensity in the market. This causes businesses to sell at lower prices. The advent of e-commerce has forced physical stores to give comparable discounts. Competition from Jio forced telecom operators to offer 1 GB of data per day for a price point they would otherwise charge for a monthly limit of 1 GB. 1.8 – Conclusion Sector analysis could help you as an investor understand whether a sector appears attractive or not. Accordingly, you may bet on the whole sector or a few selected stocks. Either way, valuations must be justified. A good business at a high price is likely a bad investment. Therefore, you must combine sector analysis with a proper valuation analysis to improve the likelihood of investment returns. A comprehensive study would also include fundamental analysis. I shall delve into understanding each sector in depth in the subsequent chapters of this module. The first sector I will cover is Cement. Stay tuned! Key Takeaways: 66 comments Hello VIneet, Welcome to varsity. I have been a long-time reader of Varsity and I think Karthik would remember me. I was disheartened when there were no new write-ups in varsity for a long time. All that is over now and thank you for that. My question is, you have written about valuations, and I am searching for a good book on the topic. Could you suggest some apart from Damodaran? Thank you, Hi Sundeep, Thanks for the warm welcome! About your question on Valuations, I have learned only from Damodaran’s books. Wouldn’t be too sure of others. Regards, sectoral analysis of steel & metal are highly solicited.also explain the forthcoming changes in relation to C emmission we may seee in next decade. regards. Great read. Especially the value chain part is extremely important for an investor to understand. Thanks team. Akshat. Thank you 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+2. Cement (https://zerodha.com/varsity/chapter/cement/) 2.1 – Cement – An Introduction We laid down a basic foundation for sector analysis in the previous chapter. In this chapter, we will discuss the Cement industry. I have picked cement as the first sector because cement is an inherent part of modern civilization. The concrete jungle, as we call our cities, is made of cement. India is the world’s second-largest cement producer, with a 7% global market share. Surprisingly, the largest producer, China , accounts for over half of the total global cement production. China produced 2.1 billion tonnes in 2022, while India produced ~370 million tonnes . The vast difference between China’s and India’s share in cement production perhaps shows the long runway ahead that India’s infrastructure has to cover and the opportunity for the cement sector. India has an installed capacity of over 570 million tonnes per annum (mtpa). Another ~150 mtpa in capacity is expected to be added by 2027. Owing to the housing and infrastructure boom in Eastern India, the region is expected to get a third of the total capacity additions. It also accounts for 80% of houses constructed under the PMAY-G scheme. Pradhan Mantri Awas Yojana (Gramin), or PMAY-G, is the central government’s scheme to subsidize the construction of pucca houses with basic amenities in villages. Generally speaking, there are two types of cement – Portland Cement and Non-Portland Cement. All cement we generally see being used around is Portland Cement and its various blended forms. Non-Portland cement is not commonly used due to its corrosive nature. Although a value-added product in itself, cement is essentially a commodity. Its all-pervasive use has made the cement an industry of its own, and its peculiarities warrant an analysis unique to the sector. 2.2 – How is cement manufactured? Let us have a glimpse of the Portland cement manufacturing process to identify the various sources or steps that could impact the cost of production, its selling price, and profitability. You can also understand cement production in this video by Vox . It is interesting for two reasons: One, it graphically explains the cement manufacturing process, and two, it discusses the possible solutions to the environmental concerns around cement manufacturing. But more on the environmental concerns later. As you can see, the process of manufacturing cement remains the same across companies. As an investor in the sector, you should analyze which steps in the manufacturing process consume more cash and where a company can save costs. Also, remember, although belonging to the same sector, no two companies are the same. You need to analyze if a factor impacts the whole sector or is specific to a company. In fact, this is the end objective of a typical sector analysis, i.e. to understand a sector and its nuances and eventually zero in on companies that thrive within the sector so that you can make wise investment decisions. It may sound complex at this stage, but eventually, you will realize this is a common sense approach without rocket science. 2.3 – The Three Major Cost Centres This is basically the Cost of Goods Sold, a concept we learned in the Fundamental Analysis module. Limestone is the most important ingredient in cement. Cement companies own limestone quarries to control costs. They mine limestone from their quarries and process it to make clinker at plants that are generally set up close to the quarries. Water is another major input in the process. Consistent water supply is also a major challenge in many parts of our country. Therefore, many cement companies have water recycling, rainwater harvesting, and groundwater recharging systems in place. This ensures regular water availability and better visibility of costs. Most cement manufacturers maintain captive power plants to bring down fuel costs. Let me introduce the concept of captive plants or units. If a manufacturing company can produce power for its own use, it is said to have a captive power plant. Another example would be a pizza chain owning a small tomato farm to produce fresh organic tomatoes for its pizzas. In this case, the tomato farm is called a captive farm. A captive unit’s product is not sold but used in-house for producing another product. Larger players like Ultratech Cement, Ambuja Cement, and ACC have captive thermal, wind, and solar power plants. Despite captive power plants, fuel costs for these major players can be as high as 25% of the revenues. Some cement makers even own coal mines to insulate from the impact of coal price fluctuations. Ultratech and Ambuja both have captive coal mines to support their requirements partially. Sourcing coal can often be a challenge. For instance, whenever there is a coal shortage, the government could ask domestic coal producers to sell coal only to power generation companies. Cement producers that own coal mines or other forms of captive power plants are at an advantage here. Others will have to import coal from international markets. Steel and aluminium industries are also heavy coal users and compete with cement in the international market to import coal, pushing prices up. This is not to say that coal mine owners are always better off. During a down cycle when the demand for cement is low, owned coal mines are a fixed cost that the manufacturer must bear amid slow production and sales. Or if there came a time when coal prices were abysmally low, buying coal from the market would become cheaper than mining at owned quarries. Given that coal is a dirty fuel, the possibility of a regulatory ban on its industrial use will only increase. Owned mines would become a dead cost if that were to happen. “ Controlling wastage and climate change All cement producers have waste heat recovery system (WHRS) plants in place. WHRS is good for two reasons – fuel cost savings and reducing carbon footprint. The tremendous heat generated in clinker production is channelled to generate steam. This steam is passed through turbines to generate electricity. This electricity can cater to 25-30% of the cement plant’s power requirements. The increasing focus on reducing fuel costs and carbon footprint has given rise to an interesting metric known as the Clinker factor. It represents the proportion of clinker in a cement recipe. The lower the clinker factor, the better. This may seem paradoxical – how can having less of the primary component be better? It is because clinker is also the most fuel-consuming step in the process of cement production. But wouldn’t that hurt the quality of cement? Manufacturers have been innovating recipes that augment the features of cement while retaining its strength. Fly ash, gypsum, silica fume, volcano ash, and other industrial by-products are common ingredients of blended cement. The Vox video after the manufacturing process above explains this well. Cement is a perishable product. It typically has a shelf-life of just 90 days. Therefore, the distribution of cement has to be fast and efficient. It is also perhaps why cement producers own the entire value chain beginning with limestone quarries. It enables them to be in control of the inventory and costs. Since perishability is a concern, cement companies might also own warehouses and trucks to monitor costs, time, and wastage. Another industry hack is to have grinding facilities closer to the market. According to Ultratech Cement’s Annual Report for 2021-22, it has 23 integrated plants, 27 grinding plants, and over 175 Ready-mix Concrete Plants. Let me break this down. An integrated plant crushes limestone, makes clinker, and grinds it into cement. Such plants are mostly close to the raw material source – the limestone quarries, in cement’s case. To solve the perishability problem, grinding units are set up closer to the market. The grinding unit will cater to a market which is more likely a state or comparable region. These facilities receive clinkers from the clinker plants that are set up near the quarries. Other components to be blended into the cement are procured directly at this grinding facility. The clinker with other components is ground to make the final cement. The grinding unit’s proximity to the market means a shorter time spent transporting the cement to the market. Further closer to the market are RMC plants. RMC, or ready-mix concrete, is a mixture of cement, sand, gravel, water, and other ingredients that make a paste used to bond the bricks in a wall. RMC has a very short shelf-life of a few hours. Therefore, RMC plants are smaller than grinding units and cater to small clusters of markets or just cities. Not all cement is sold in the form of RMC, but it is growing in popularity. As a business, cement companies would love to sell RMC; it is a value-added product and hence can improve margins. But it also adds capital expenditure. For construction companies, RMC is convenient because of the limited spaces available in cities and because it is one less job to do. 2.4 – Distribution of Cement Distribution of cement, or bringing it to the market, requires its own study. Cement is a “low value, high volume” product. As of this writing in May 2023, a 50 kg bag of cement costs roughly ₹400. Compare that with a bag of grains or cloth or cosmetics or gold. Transporting a unit of any of these items from one point to another will surely bring higher revenues than transporting the same unit of cement. Perishability, as discussed earlier, is a challenge too. An efficient distribution system is key to maintaining profitability in the cement business. At a time when urbanization has picked up pace, cement has become a necessity. If the market had only a small number of manufacturers, they could dictate prices, effectively impacting infrastructure growth and real estate prices. But the industry has several large players; over 25 are listed. However, the top five players account for almost half the national capacity. Huge capacity additions have been driving competition. Competition is also pushing the players to innovate and develop more ways of cutting costs and improving margins. Some are spending on branding and marketing to boost sales. 2.5 – Demand Drivers for Cement The next step is to understand the uses or users of cement. In other words, let us understand what drives the demand for cement. Cement is used in building houses, roads, dams, and other infrastructure. The cement users could be classified into three broad categories – housing, infrastructure, and industrial. Seasonality also impacts the demand for cement. All forms of construction activities usually slow down during the monsoon season, thereby impacting the demand for cement. Studying manufactured bulk commodities like cement requires a deeper focus on the entire value chain, from production to distribution and sales. The asset-heavy nature of the value chain also makes cement manufacturers prone to take on more debt. An investor studying the sector must also look at the debt levels of the companies and the sector at large. A structure to the above discussions to simplify studying the sector could be very useful. Let me suggest a checklist that you could use when studying the cement industry. 2.6 – The Checklist You can find these metrics in cement companies’ quarterly presentations and annual reports. Now why do metrics matter? These metrics are specific to the cement sector. If you are interested in investing in cement, combining these metrics with the regular company-based fundamental analysis can ensure complete sector research. Let us have a look at each metric one by one. Regional Presence : Diversification helps reduce risk. A company with its operations and market spread out across the country is more likely to continue operating despite any challenges. If one plant cannot operate for any reason, the others can fill in. Market Share: A larger market share usually results from a strong distribution network. Customers might want to buy a specific cement brand but will usually buy one easily available near their location. Production Capacity : The larger the production capacity, the larger orders the cement maker can serve. Cement makers want to add capacity if the economy is seeing a boom in infrastructure and real estate sectors. However, the cost of adding capacity must also be justified with the expected projected income from that capacity. Capacity Utilization : Let’s say a cement maker has an annual capacity of producing 100 mtpa worth of cement, but they produce only 70 mtpa. Effectively, they operate at a 70% capacity, or capacity utilization was 70%. As an investor, you must determine why the full capacity was not utilized. External factors such as natural calamities, pollution controls, and regulations could limit full utilization. For example, if heavy rains were to shut down production for three months, operating at 100% capacity for the remaining nine months would also mean only 75% utilization. Internal factors such as the unavailability of raw materials or labour could also hamper production. Sales Volume : Not all that is produced is sold. In fact, some cement makers sell more than what they produce. They may buy volumes from other cement makers to fulfil their commitments to customers. Realization ₹/MT : Realization is the average selling price per unit. Higher prices lead to higher revenues. Comparison of the realization numbers of cement players can lead to various insights. The one with the highest number could be commanding a solid premium in the market or must be servicing locations others have not been able to service. The highest number could also suggest a larger share of value-added products in the total sales volume.  This snapshot from Ultratech Cement’s Q4 presentation for FY23 shows that grey cement realizations have slightly moderated compared to the previous quarter. Input / Power and Fuel / Freight Cost Ratios : You can compute these ratios by dividing the cost item by revenues. The smaller these ratios, the better. Looking for each of these ratios separately can help identify what part of the business is driving costs and which one is improving margins. Cement companies typically try to control all these costs by having these activities in-house – captive quarries, power plants, and fleets. That is about it. This checklist should equip you well to comprehensively understand the cement industry and the companies operating in it. In the next chapter, we will discuss the highly regulated and frequently disrupted insurance industry. Key Takeaways 54 comments Very usefull information It’s very interesting and informative Thank you:) Thank you! Sir this kind of content is very unique and i have not seen it anywhere else. Could you please give me some pointers as to how to do my own sector analysis? What resources, books and websites do you use? Thanks again for the superb content. Thank you, Karthik! I mostly read the annual reports of a few players in any sector. Their “Management Discussion & Analysis” section gives a good perspective. If it is a highly regulated sector like banking or insurance, you can get a lot of information on the respective regulator’s website too. Good content, Excellent explanation, a non-finance/management persons can understand with this simpler explanation. Thank you so much:)  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+77 comments VIDEO WAS QUITE FASCINATING MATCHING WITH MY CURRENT VILLAGE SITUATION. SAME BLACK AND WHITE 70’S VILLAGE AND COREOGRAPY WAS ALSO FASCINATING . Thanks! Glad you liked it 🙂 My son enjoyed this, when are the next episodes getting published, it adds a lot of value to childrens learning Glad your son liked it, we just published the 2nd episode, hope he likes it too 🙂 The content is amazing. You are making Finance concepts so much easy to understand by this interesting way. Thank you Karthik Sir and Team. Thanks for letting me know, Yatesh. Hope you continue to like all the videos 🙂 Hi Karthik You have highlighted the importance of compounding in education and finance. The characters are relatable, and the sprinkle of humor was well enjoyed by my child. Thank you for introducing Finance concepts for kids. I appreciate what you and your team at Zerodha are doing for the future generation. Best wishes, Bee. Thanks so much for letting me know, so happy that your kid enjoyed watching these vidoes 🙂 This is a beautiful piece of content for kids and grownups as well. Thank you, Varsity team, for making time and resources to simplify financial knowledge🙌🙌One small recommendation. It would be more engaging if the cartoon is in the color format. The next episode is in color – https://www.youtube.com/watch?v=D916Xq4Fbxg (https://www.youtube.com/watch?v=D916Xq4Fbxg) this one.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading this chapter (https://zerodha.com/varsity/chapter/the-stock-markets/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter  Comments are closed. Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+Download the excel here (https://docs.google.com/spreadsheets/d/1KgTubTr4Cq9YPbIFw9qqSnlqozCZs-JX/edit?usp=sharing&ouid=110974847305973492861&rtpof=true&sd=true) . We recommend reading this chapter (https://zerodha.com/varsity/chapter/the-need-to-invest/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter 233 comments Love the way they care for their customers… And upgrading knowledge of customers is one of the best way .. proud member❤️ And we are equally happy to have you as a part of our family! Very excited to see zerodha  versity in  video series excellent effort from zerodha I am starting to see it from today I am in market from last 8 months.. and I have no doubt that it will add some more knowlege  in my market experiance We hope so too Anurag! Happy learning 🙂 Very well explained and useful Happy learning, Manjunath! MAKE ALL THE VIDEOS IN HINDI ALSO Such a excellent work knowledge sharing is the best for those who are new in share market.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading this chapter (https://zerodha.com/varsity/chapter/regulators/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter  64 comments nice Very nicely explained. Just one suggestion, in the key takeaway slide (last 15 sec of the video), varsity logo (for subscribe button) in the middle and next video thumbnail in the Right hand corner is hindering some text. Can you please fix these? Or add extra 15 seconds blank space at the last so these thumbnails icons cover that blank space and key takeaway slides will cleared from any hinderance. Noted, will try and rectify this. Logo and Next Video disrupts the Key Takeaways thumbnail. Also Next video skips one video (eg. after watching Video 2 the next video they shows is Video 4 instead of Video 3). +1. Logo and next videos disrupt the takeaway slides in all the videos. please fix this. Working on it, Dhaval. Thank you so much sir for these wonderful lessons!! 🙂 Happy learning, Brianna!  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading these Part 1 (https://zerodha.com/varsity/chapter/the-ipo-markets-part-1/) and Part 2 (https://zerodha.com/varsity/chapter/the-ipo-markets- part-2/) chapters of IPO Markets on Varsity to learn and understand the concepts in-depth. Key takeaways from this chapter  57 comments Hi Team Zerodha, Great work. Good Initiative. I like to personally congratulate Mr.Karthik and Mr.Prateek. I thought of sharing some content which I had prepared some 8 years back. If possible please share your contact email so that I can forward the same. Thank you, Dinakar. Please do share on Google dive with a link here itself. Brilliant explanation. Eagerly waiting for the day I might also join the Zerodha team as a Content Contributor. Coins tossed and fingers crossed. Glad you liked the content 🙂 Hi Team, Great learning. the way Mr Prateek is teaching is awesome, loved it . Happy learing! thank you so much for such great explanation of all the topics !! Happy learning! Brilliant content and presentation by Prateek. Precise and compact 8n a small capsule… Love this innovation at Varsity. Happy learning 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+

data/financial_bot_context.txt

@@ -0,0 +1,20 @@
+15. Supplementary note – The 20 market depth or level 3 data (https://zerodha.com/varsity/chapter/supplementary-note-the-20-market-depth/) The 20 Market Depth (level 3 data) Window I’ve driven a car for many years and I’ve even changed my car a few times now. Each time I changed my car, the engine remained more or less the same, but the features within the vehicle and its aesthetics continuously changed. Air conditioner, power steering, and power windows were all luxury features in the car at one point, but today, I guess no one buys a car without these essential features. The game-changer for me though was parking assist. The little camera at the back of the car gave me complete visibility of the parking space available. I was no longer required to pop and twist my head out and struggle to park the car, nor did I have to bug my co-passenger to get down and help me navigate my way into a parking spot. The parking assist feature did everything and helped me execute a perfect parallel park. The parking assist feature was my edge for hassle-free car parking. I feel the same edge while trading the markets with the level 3 data 🙂 Level 3 or the 20 market depth feature is unique and has multiple uses. You’ll probably appreciate the level 3 market window if you have traded at an institutional desk. A regular retail trader would not understand this feature anytime soon, simply because this feature was unavailable all these years until we introduced it for the very first time to the Indian retail traders. The purpose of this chapter is to help you understand how useful this feature is and get you started on building trading strategies around this feature. If you are entirely new to this, I’d suggest you read this blog (https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite) to understand what the level 3 data is all about. Assuming you know what it is, this chapter will help you understand the multiple uses of this feature. Contract availability For the option traders, the 20-depth order book gives great visibility into the availability of contracts to trade and help identify better price points to execute these trade. Without this visibility, it becomes really hard to trade illiquid contracts. While I’m specifically talking about options here, you can extend this to Futures contracts as well, especially the illiquid ones. Let us put this in context, have a look at the regular market depth (i.e. the top 5 bid-ask) of the 13000 CE expiring in Jan 2020. We can see narrow bids on the left and a notch better offer on the right. You’d probably hesitate to trade this contract if you are someone looking at trading a few lots of Nifty. But check what’s hiding under the hood here by opening the level 3 data –  As you can see, there are many contracts available, but they are not visible in the regular market depth. In fact, the bid and offer quantities are heavily concentrated below the 8 th row respectively. Given the availability of the contracts in this strike, the perspective to trade or not completely changes and will now depend upon your trading strategy. Execution control Level 3 data gives you full visibility of the approximate execution price for your trade. This is particularly useful when you decide to scalp the market. When you scalp the market — Let us say you want to buy and sell 5000 shares of Hindustan Zinc; the regular market depth window gives you the following information —  As you can see, there is no visibility on how these 5000 shares will get filled. Now, take a look at the 20 depth window —  The 20 depth window paints an entirely different picture. It not only tells me that I’ll get the 5000 shares, but it also gives me information about the approximate buy price. If I were to place a market order for 5000 shares, I’d be buying this order book from 210.5 to 211.25. I also see at 211; there are 2425 shares available, so I can expect the average price is at or around 211. Now, my decision to scalp the stock should depend on the pop I’d expect over and above 211. Maybe 211.5 or so. Of course, you’ll get the exact breakeven (post charges) if you were to use a brokerage calculator. Position sizing Level 3 market window plays a critical role in ‘guesstimating’ the number of shares to trade, given the liquidity of the stock. For the sake of this discussion, we will assume that the availability of capital is not an issue. Now, have a look at the regular market depth — You expect Siemens to move from 1675 to about 1690 over the next hour. So, given the fact that you are not constrained by capital, how many shares will you buy for this intraday trade? The regular market depth window suggests that you can buy close to 175 shares. However, the 20 depth opens up a different perspective altogether — In fact, the liquidity in this stock lies below the best five bid and ask, and the impact cost is reasonable. The regular market depth window fails to capture this information. Assuming you intend to buy about 1500 shares, the buy price will lie somewhere within 1675.5 to 1678, which is spread of 0.149%. In this case, assuming you are sure about the target price (1690), you can go all in and buy through whatever is available at that moment. Order placement You can extend the position sizing concept and use the 20 depth market watch to place a stop loss or a limit order. Assume you have an intraday buy position in VST Tillers at 1313.8. The question is, where you would place the stop loss for this trade? Can the 20 market depth help us with this? Of course. Have a look at the 20 depth window for VST Tillers. As you can see, there is a concentration of bids in 1290. The good part is that the number of order count is also the highest (35) in 1290. This implies that several traders have placed an order at 1290, indicating some sort of price action at that level. This perhaps builds a case for placing the stop-loss. A prudent trader would probably place a stoploss not at 1290, but maybe at a price just below it. So I was a buyer in this stock, then purely based on 20 depth I’d probably place my SL at 1290 or below, maybe at 1287 and by the same logic, set my target at 1340 or at 1338.8. Validate the support and resistance level I find this extremely interesting. In the example above, we identified 1290 as the stoploss price, simply because there was a concentration of bids. In other words, we expect 1290 as a support price. If this is indeed true, then it should show up on the charts as well, right? Have a look at the chart below – Clearly, there is some price action around 1296. Remember, support and resistance is not one price point, but rather a range. Therefore 1290 – 1300 marks as an intraday support for this stock. This is a perfect example of seeing the price action concept play out in the market. Another way to look at this is first to identify the S&R level and then check the 20 depth to figure if there is a concentration of bids/offers in that zone.  Hopefully, by now you’ve started to appreciate the immeasurable value 20 depth order book brings to you while trading. Remember, irrespective of which technique you use to develop a point of view (technical or quantitative analysis), things boil down to price, and the action trades take at that price. The 20 depth market window is essentially your ticket to validate the truth of this price action. Make sure you use your card wisely! Do post your comments and tell us how differently you will use the 20 depth window for identifying trading opportunities. Good luck!   608 comments cannot see market depth You need to have generated at least 100 Rupees in brokerage to see the 20 depth. Please check this – https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite (https://zerodha.com/z-connect/featured/introducing-20-depth-or-level-3-data-beta-on-kite) Very nicely written, thank you for your incredible insight. Salman, thank you 🙂 Sir, how will the market depth data be relied upon? When clients or trading systems use the disclosed quantity feature, not the entire quantity is shown. In such a case, isn’t there a chance that the wrong assumptions are made looking at just the market depth? Valid point, but there is no way one can identify disclosed quantity. By the way, level 3 helps you identify contract availability. Given this, the disclosed quantity, if any, will only make the liquidity situation better. Hi Karthik! Newbie here, under the ‘The 20 Market Depth (level 3 data) Window’ section, you’ve said – If you are entirely new to this, I’d suggest you read this blog to understand what the level 3 data is all about. Are you referring to this chapter as the blog or have a separate blog elsewhere, in which case I do not see a link to it. Can you help? ‘Read this blog’ is hyperlinked 🙂 Amazing featute for the reatil traders. Way to go team for bringing such innovative and pro Tools to the world at such decent costs. Happy learning and trading 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+12. Key Events and Their Impact on Markets (https://zerodha.com/varsity/chapter/key-events-and-their-impact-on-markets/) 12.1 – Events Trading or investing based on just company-specific information may not be sufficient. Outside events, both economic and/or non-economic, impact stocks and the market’s performance in general. It is also important to understand the events that influence the markets. In this chapter, we will try to understand some common events and how the stock market reacts to these events. 12.2 – Monetary Policy The monetary policy is a tool through which the Reserve Bank of India (RBI) controls the money supply by controlling the interest rates. RBI is India’s central bank. Likewise, every country’s central bank is responsible for setting interest rates. For example, the European Central Bank in Europe and Federal Reserves in the US. Central Banks tweak the interest rates to control the money supply in the mainstream economy. While setting the interest rates, the RBI has to strike a balance between growth and inflation. In a nutshell – if the interest rates are high, the borrowing rates are high (particularly for corporations). If corporate can’t borrow easily, they cannot grow. If corporations don’t grow, the economy slows down. On the other hand, borrowing becomes easier when the interest rates are low. This translates to more money in the hands of corporations and consumers. With more money, there is increased spending which means the sellers tend to increase the prices of goods and services, leading to inflation. I’d encourage you to watch this YouTube video where I’ve tried to explain what causes inflation and the means through which RBI controls inflation. To strike a balance, the RBI has to consider all economic factors and carefully set the key rates. Any imbalance in these rates can lead to economic chaos. The key RBI rates that you need to track are as follows: Repo Rate – Banks can borrow from the RBI. The rate at which RBI lends money to other banks is called the Repo Rate. If the repo rate is high, the cost of borrowing is high, leading to slow economic growth. You can check the latest repo rate (And other rates, too) on RBI’s website (https://www.rbi.org.in/) . Markets don’t like the RBI increasing the repo rates because it slows down economic growth. Reverse repo rate – Reverse Repo rate is the rate at which RBI borrows money from banks. Or in other words, Reverse Repo is the deposit rate RBI offers to other banks when the banks park funds with RBI. When banks deposit money to RBI, they are certain that RBI will not default, so the rate RBI offers is relatively low. However, the banking system’s money supply reduces when banks deposit money with RBI (at a lower rate) instead of the corporate entity. An increase in the reverse repo rate is not great for the economy as it tightens the money supply. Sometimes via the central bank’s policy, the central bank mandates higher deposits by banks; again, this is a way to curtail excess money supply in the mainstream economy. Cash reserve ratio (CRR) – Every bank must maintain funds with RBI. The amount that they maintain is dependent on the CRR. If CRR increases, more money is sucked out of the mainstream economy, which is not good for the economy. The monetary policy committee members meet regularly to review the economic situation and decide upon these key rates; hence keeping track of the monetary policy event is a must for any active trader. The first to react to rate decisions would be interest-rate sensitive stocks across various sectors such as – banks, automobiles, housing finance, real estate, metals, etc. 12.3 – Inflation Inflation is a sustained increase in the general prices of goods and services. Increasing inflation erodes the purchasing power of money. All things being equal, if the cost of 1 KG of onion has increased from Rs.15 to Rs.20, this price increase is attributed to inflation. Inflation is inevitable, but a high inflation rate is not desirable as it could lead to economic uneasiness. A high level of inflation tends to send a bad signal to markets. Both the Government and RBI work towards reducing inflation to a manageable level. Inflation is generally measured using an index. If the inflation index increases by certain percentage points, it indicates rising inflation. Likewise, an index falling indicates inflation cooling off. There are two inflation indices – The Wholesale Price Index (WPI) and Consumer Price Index (CPI). Wholesale Price Index (WPI) – The WPI indicates the movement in prices at the wholesale level. It captures the price change when goods are bought and sold wholesale. WPI is an easy and convenient method to calculate inflation. The inflation measured here is at an institutional level and does not necessarily capture the consumer’s inflation. Consumer Price Index  (CPI) – The CPI, on the other hand, captures the effect of the change in prices at a retail level. As a consumer, CPI inflation is what matters. The calculation of CPI is quite detailed as it involves classifying consumption into various categories and subcategories across urban and rural regions. Each of these categories is made into an index, the final CPI index is a composition of several internal indices. The CPI captures the effect of inflation on daily household consumables like fruits, vegetables, cereals, and even fuels like petrol and diesel. The computation of CPI is quite rigorous and detailed. It is one of the most critical metrics for studying the economy.  A national statistical agency, the Ministry of Statistics and Programme Implementation (MOSPI), publishes the CPI numbers around the 2 nd week of every month. The RBI’s challenge is to strike a balance between inflation and interest rates. Usually, a low-interest rate tends to increase inflation, and a high-interest rate tends to arrest inflation. 12.4 – Index of Industrial Production (IIP) The Index of Industrial Production (IIP)  is a short-term indicator of the country’s industrial sector’s progress. The data is released every month (along with inflation data) by the Ministry of Statistics and Programme Implementation (MOSPI). As the name suggests, the IIP measures the Indian industrial sectors’ production, keeping a fixed reference point. As of today, India uses the reference point of 2004-05. The reference point is also called the base year. Roughly about 15 different industries submit their production data to the ministry, which collates the data and releases it as an index number. If the IIP increases, it indicates a vibrant industrial environment (as the production is going up) and hence a positive sign for the economy and markets. A decreasing IIP indicates a sluggish production environment, hence a negative sign for the economy and markets. To sum up, an upswing in industrial production is good for the economy, and a downswing rings an alarm. As India is getting more industrialized, the relative importance of the Index of Industrial Production is increasing. A lower IIP number puts pressure on the RBI to lower the interest rates and aid industrial credit with cheaper credit. 12.5 – Purchasing Managers Index (PMI) The Purchasing managers’ index (PMI) is an economic indicator that tries to capture business activity across the country’s manufacturing and service sectors. This is a survey-based indicator where the respondents – usually the purchasing managers- indicate their business perception change concerning the previous month. A separate survey is conducted for the service and manufacturing sectors. The data from the survey are consolidated on a single index. Typical areas covered in the survey include new orders, output, business expectations, and employment. The PMI number usually oscillates around 50. A reading above 50 indicates expansion, and below 50 indicates a contraction in the economy. And reading at 50 indicates no change in the economy. 12.6 – Budget A Budget is an event during which the Ministry of Finance discusses the country’s finance in detail. The Finance Minister, on behalf of the ministry, makes a budget presentation to the entire country. During the budget, major policy announcements and economic reforms are announced, which impacts various industries across the markets. Therefore the budget plays a vital role in the economy. To illustrate this further, in one of the recent budgets, the expectation was to increase the duties on a cigarette. As expected, during the budget, the Finance Minister raised the duties on a cigarette, so the prices increased. An increased cigarette price has a few implications: In reaction to the budget announcement, ITC traded 3.5% lower for this precise reason. A budget is an annual event, and it is announced during the last week of February. However, the budget announcement could be delayed under certain special circumstances, such as a new government formation. 12.7 – Corporate Earnings Announcement Corporate earning season is perhaps one of the important events to which the stocks react. The listed companies (trading on the stock exchange) must declare their earnings once every quarter, also called the quarterly earnings numbers. During an earnings announcement, the corporate gives out details on various operational activities, including: Besides, some companies give an overview of what to expect from the upcoming quarters. This forecast is called ‘corporate guidance.’ Invariably every quarter, the first blue-chip company to make the quarterly announcement is Infosys Limited. They also give out guidance regularly. Market participants follow what Infosys has to say regarding guidance as it impacts the markets. The table below gives you an overview of the earning season in India: Do note that the 1st of April in India marks the beginning of the financial year. In the US, the financial year starts on 1st Jan, so the first quarter starts from January through March, and so forth. Every quarter when the company declares its earnings, the market participants match the earnings with their expectations of how much the company should have earned. The market participant’s expectation is called the ‘street expectation.’ The stock price will react positively if the company’s earnings are better than the street expectations. The stock price will react negatively if the actual numbers are lower than the street expectation. If the street expectation and actual numbers match, the stock price tends to trade flat with a negative bias more often than not. This is mainly because the company could not give any positive surprises. 12.8 – Non Financial events Apart from the events we discussed above, it would be best to watch out for other non-financial events to understand their impact on markets. For example, the Covid crisis of 2020 had a significant effect on economies around the world, disrupting the world economic order. The supply chain took a hit across the globe leading to an inflation spike. That said, there were select pockets of the economy that did very well, mainly the online services industry. Events like the Russia – Ukraine war or the tension between China and Taiwan have impacted world markets. Geo-political affairs such as these impact various connected economies. For instance, the war between Russia and Ukraine affects the supply of natural gas and crude oil, which significantly impacts the energy costs in Europe. As an active trader or a market participant, you need to watch out for these events and understand how these events can impact the markets. While the world economies are interconnected, isolated events (Country specific) impact the local economy. For example, the elections in India impact only the Indian economy. So, keep an eye on these non-financial events and how they can impact the stock markets or sometimes specific industries. Key takeaways from this chapter 329 comments 1. From the close of a quarter/financial year, what are the time limits within which companies must declare the results? 2. In case of insider trading (SAST), what is the time limit to inform the exchanges about it ? 3. Is there any rule/law making it mandatory for companies to report events with financial significance? Answers in the same order – 1) If I’m not wrong companies must declare results  within 30 days 2) In order to make it a level playing field SEBI mandates the companies to disclose insider trading ‘at the earliest’, which pretty much means the company has to declare this information within the same trading day 3) Yes companies have to declare this information. Hi, What is insider trading and SAST ? How it works? Is their any facility into Versity to get notification once get solution of query? You should get an email notification when this gets answered. Anyway, insider trading is basically trading on information not known to the public. For example, an employee of TCS can buy the shares of TCS based on his knowledge of the quarterly results yet to be announced. Thanks Karthik 🙂 Cheers! By 8% & 7% in Repo and Reverse repo rate, do you mean it is same as we borrow from banks at 11 % ? No, repo & reverse repo rate are rate at which banks transact with RBI. The lending rates to us are different. Sir, i am not able to understand “Purchasing managers index(PMI)” PMI or the Purchasing Manager’s Index is an index which measures the activity of a purchasing manager (PM) of a company. This is measured by running a survey across many PM’s across sectors and industries. If the PM is sheen to be purchasing things for the company then the perception of growth and industrial activity is created, which is good for the markets. Else its considered not so good. Thank you sir Welcome 🙂 Important & better for new trader You bet, it is.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+16. Momentum Portfolios (https://zerodha.com/varsity/chapter/momentum-portfolios/) 16.1 – Defining Momentum If you have spent some time in the market, then I’m certain you’ve been bombarded with market jargon of all sorts. Most of us get used to these jargon and start using them without actually understanding what they mean. I’m guilty of using a few jargon without understanding the true meaning of it, and I get a feeling that some of you reading this may have experienced the same. One such jargon is – momentum. I’m sure we have used momentum in our daily conversations related to the markets, but what exactly is momentum, and how is it measured? When asked, traders loosely define momentum as the speed at which the markets move. This is correct to some extent, but that’s not all, and we should certainly not limit our understanding to just that. ‘Momentum’ is a physics term. It refers to the quantity of motion an object has. If you look at this definition in the context of stock markets, everything remains the same, except that you will have to replace ‘object’ with stocks or the index. Momentum is the rate of change of stock returns or the index. If the rate of change of returns is high, then the momentum is considered high; if the rate of change of returns is low, the momentum is considered low. This leads us to the next obvious question i.e. what is the rate of change of returns? The rate of change of return, as it states the return generated  (or eroded) between two reference periods. For the sake of this discussion, let’s stick to the rate of change of return on an end-of-day basis. So in this context, the rate of change of returns means the speed at which the daily return of the stock varies. To understand this better, consider this example – The table above shows an arbitrary stock’s daily closing price for six days. Two things to note here – Consider another example – Two things need to note – Given the behavior of these two stocks, I have two questions for you – To answer these questions, you can look at either the absolute change in the Rupee value or the percentage change from a close-to-close perspective. If you look at the absolute Rupee change, the change in Stock A is higher than in Stock B. However, this is not the right way to look at the change in daily return. For instance, in absolute Rupee terms, stocks in the range of, say, 2000 or 3000 will always have a higher change compared to stocks in the range of 1000 or lower. Hence, evaluating absolute Rupee change will not suffice, and therefore we need to look at the percentage change. In terms of percentage change, Stock B’s daily change is higher, and therefore we can conclude that Stock B has a higher momentum. Here is another situation, consider this – Stock A has trended up consistently daily, while stock B has been quite a dud all along except for the last two days. On an overall basis, if you check the percentage change over the 7 days, then both have delivered similar results. Given this, which of these two stocks is considered to have good momentum? Well, Stock A is consistent in terms of daily returns, exhibits a good uptrend, and, therefore, can be considered to have continuity in showcasing momentum. Now, what if I decide to measure momentum slightly differently? Instead of daily returns, what if we were to look at the return on a 7 days basis? If we do that, stocks A and B qualify as momentum stocks. The point I’m trying to make here is that traders generally look at momentum in terms of daily returns, which is perfectly valid, but this is not necessarily the only way to look at momentum. The momentum strategy we will discuss later in this chapter looks at momentum on a larger time frame, not daily. More on this later. I hope by now; you do have a sense of what momentum means and understand that momentum can be measured not just in terms of daily returns but also in terms of larger time frames. High-frequency traders measure momentum on a minute-to-minute or hourly basis. 16.2 – Momentum Strategy Among the many trading strategies traders use, momentum is one of the most popular strategies. Traders measure momentum in many different ways to identify opportunity pockets. The core idea across all these strategies remains the same, i.e., to identify momentum and ride the wave. Momentum strategies can be developed on a single-stock basis. The idea is to measure momentum across all the stocks in the tracking universe and trade the ones that showcase the highest momentum. Remember, momentum can be either long or short, so a trader following a single stock momentum strategy will get both long and short trading opportunities. Traders also develop momentum strategies on a sector-specific basis and set up sector-specific trades. The idea here is to identify a sector that exhibits strong momentum; this can be done by checking momentum in sector-specific indices. Once the sector is identified, look for the stocks that display maximum strength in terms of momentum. Momentum can also be applied on a portfolio basis. This involves portfolio creation with, say ‘n’ number of stocks, with each stock showcasing momentum. In my opinion, this is an excellent strategy as it is not just a plain vanilla momentum strategy but also offers safety in diversification. We will discuss one such strategy wherein the idea is to create a stock basket, aka a portfolio of 10 momentum stocks. Once created, the portfolio is held until the momentum lasts and then re-balanced. 16.3 – Momentum Portfolio Before we discuss this strategy, I want you to note a few things – Given the above, here is a systematic guide to building a ‘Momentum Portfolio’. Step 1  – Define your stock universe As you may know, there are close to 4000 stocks on BSE and about 1800 on NSE. This includes highly valuable companies like TCS and absolute thuds such as almost all the Z category stocks on BSE. Companies such as these form the two extreme ends of the spectrum.  Do you have to track all these stocks to build a momentum portfolio? Not really. Doing so would be a waste of time. One has to filter out the stocks and create the ‘tracking universe.’ The tracking universe will consist of a large basket of stocks within which we will pick stocks to constitute the momentum portfolio. The momentum portfolio will always be a subset of the tracking universe. Think of the tracking universe as a collection of your favorite shopping malls. Maybe out of the 100s of malls in your city, you may go to 2-3 shopping malls repeatedly. Clothes bought from these 2-3 malls comprise your entire wardrobe (read portfolio). Hence, these 2-3 malls form your tracking universe out of the 100s available in your city. The tracking universe can be pretty straightforward – the Nifty 50 or BSE 500 stocks. Therefore, the momentum portfolio will always be a subset of the Nifty 50 or BSE 500 stocks. Keeping the BSE 500 stocks as your tracking universe is an excellent way to start. However, if you feel adventurous, you can custom-create your tracking universe. Custom creation can be on any parameter – for example, out of the entire 1800 stocks on NSE, I could use a filter to weed out stocks that have a market cap of at least 1000 Crs. This filter alone will shrink the list to a much smaller, manageable set. Further, I may add other criteria, such as the stock price should be less than 2000. So on and so forth. I have randomly shared a few filter ideas, but you get the point. Using custom creation techniques helps you filter out and build a tracking universe that matches your requirement. Lastly, from my personal experience, I would suggest you have at least 150-200 stocks in your tracking universe if you wish to build a momentum portfolio of 12-15 stocks. Step 2  – Set up the data Assuming your tracking universe is set up, you can proceed to the 2 nd step. In this step, you must ensure you get the closing prices of all the stocks in your tracking universe. Ensure your data set is clean and adjusted for corporate actions like the bonus issue, splits, special dividends, and other corporate actions. Clean data is the crucial building block to any trading strategy. There are plenty of data sources from where you can download the data for free, including the NSE/BSE websites. The question is – what is the lookback period? How many historical data points are required? To run this strategy, you only need 1-year data point. For example, today is 2 nd March 2019; then I’d need data points from 1 st March 2018 to 2 nd March 2019. Please note once you have the data points for the last one-year set, you can update this daily, which means the daily closing prices are recorded. Step 3  – Calculate returns This is a crucial part of the strategy; in this step, we calculate the returns of all the stocks in the tracking universe. As you may have already guessed, we calculate the return to get a sense of the momentum in each stock. As discussed earlier in this chapter, one can calculate the returns on any frequency, be it daily/weekly/monthly, or even yearly returns. We will stick to yearly returns for the sake of this discussion; however, please note; you can add your own twist to the entire strategy and calculate the returns for any time frame you wish. Instead of yearly, you could calculate the half-yearly, monthly, or even fortnightly returns. So, you should have a tracking universe of about 150-200 stocks at this stage. All these stocks should have historical data for at least 1 year. Further, you need to calculate the yearly return for each stock in your tracking universe. To help you understand this better, I’ve created a sample tracking universe with just about ten stocks in it. The tracking universe contains the data for the last 365 days. The 1-year returns are calculated as well – If you are wondering how the returns are calculated, then this is quite straightforward, let us take the example of ABB – Return = [ending value/starting value]-1 = [1244.55/1435.55]-1 = -13.31% Relatively straightforward, I guess. Step 4 – Rank the returns Once the returns are calculated, you need to rank the returns from the highest to the lowest returns. For example, Asian paints has generated a return of 25.87%, the highest in the list. Hence, the rank of Asian paints is 1. The second highest is HDFC Bank, which will get the 2 nd rank.  Infosys’s return, on the other hand, is -35.98%, the lowest in the list; hence the rank is 10. So on and so forth. Here is the ‘return ranking’ for this portfolio – If you are wondering why the returns are negative for most of the stocks, that’s how stocks behave when deep corrections hit the market. I wish I had opted to discuss this strategy at a better point. So what does this ranking tell us? If you think about it, the ranking reorders our tracking universe to give us a list of stocks from the highest return stock to the lowest. For example, from this list, I know that Asian Paints has been the best performer (in terms of returns) over the last 12 months. Likewise, Infy has been the worst. Step 5 – Create the portfolio A typical tracking universe will have about 150-200 stocks, and with the help of the previous step, we would have reordered the tracking universe. Now, we can create a momentum portfolio with the reordered tracking universe. Remember, momentum is the rate of change of return, and the return itself is measured yearly. A good momentum portfolio contains about 10-12 stocks. I’m comfortable with up to 15 stocks in the portfolio, not more than that. For the sake of this discussion, let us assume that we are building a 12 stocks momentum portfolio. The momentum portfolio is now the top 12 stocks in the reordered tracking universe. In other words, we buy all the stocks starting from rank 1 to rank 12. In the example we were dealing with, if I were to build a 5-stock momentum portfolio, then it would contain – The rest of the stocks would not constitute the portfolio but will remain in the tracking universe. You may ask what is the logic of selecting this subset of stocks within the tracking universe? Well, read this carefully – if the stock has done well (in terms of returns generated) for the last 12 months, it implies that it has good momentum for the defined time frame. The expectation is that this momentum will continue onto the 13 th month, and therefore the stock will continue to generate higher returns.  So if you were to buy such stocks, you are to benefit from the expected momentum in the stock. This is a claim. I do not have data to back this up, but I have successfully used this technique for several years. It is easy to back-test this strategy, and I encourage you to do so. Back in the day, my trading partner and I were encouraged to build this momentum portfolio after reading this ‘Economist’ article. You need to read this article before implementing this strategy. Once the momentum portfolio stocks are identified, the idea is to buy all the momentum stocks in equal proportion. So if the capital available is Rs.200,000/- and there are 12 stocks, the idea is to buy Rs.16,666/- worth of each stock (200,000/12). By doing so, you create an equally weighted momentum portfolio. Of course, you can tweak the weights to create a skewed portfolio, there is no problem with it, but then you need a solid reason for doing so.  This reason should come from backtested results. If you like to experiment with skewed portfolios, here are few ideas – So on and so forth. Ideally, the approach to capital allocation should come from your backtesting process, this also means you will have to backtest various capital allocation techniques to figure out which works well for you. Step 6 – Rebalance the portfolio So far, we have created a tracking universe, calculated the 12-month returns, ranked the stocks in terms of the 12-month returns, and created a momentum portfolio by buying the top 12 stocks. The momentum portfolio was built based on the 12-month performance, hoping to continue to showcase the same performance for the 13 th month. There are a few assumptions here – Now the question is, what happens at the end of the month? At the end of the month, you re-run the ranking engine and figure out the top 10 or 12 stocks which have performed well over the last 12 months. Do note at any point, we consider the latest 12 months of data. So, we now buy the stocks from rank 1 to 12, just like we did in the previous month. From my experience, chances are that out of the initial portfolio, only a hand full of stocks would have changed positions. So based on the list, you sell the stocks that no longer belong in the portfolio and buy the new stocks featured in the latest momentum portfolio. In essence, you rebalance the portfolio and you do this at the end of every month. So on and so forth. 16.4 – Momentum Portfolio variations Before we close this chapter (and this module), I’d like to touch upon a few variations to this strategy. The returns have been calculated on a 12-month portfolio and the stocks are held for a month. However, you don’t have to stick to this. You can try out various options, like – As you can see, the options are plenty, and your imagination only restricts it. If you think about what we have discussed so far, the momentum portfolio is price based. However, you can build a fundamental-based momentum strategy as well. Here are a few ideas – You can do this on any fundamental parameter – EPS growth, profit margin, EBITDA margin etc. The beauty of these strategies is that the data is available, hence backtesting gets a lot easier. 16.5 – Word of caution As good as it may seem, the price-based momentum strategy works well only when the market is trending up. When the markets turn choppy, the momentum strategy performs poorly, and when the markets go down, the momentum portfolio bleeds heavier than the markets itself. Understanding the strategy’s behavior with respect to the market cycle is crucial to this portfolio’s eventual success. I learned it the hard way. I had a great run with this strategy in 2009 and ’10 but took a bad hit in 2011. So before you execute this strategy, do your homework (backtesting) right. Having said all of that, let me reassure you – a price-based momentum strategy, if implemented in the proper market cycle can give you great returns, in fact, better more often than not, better than the market returns. Good luck and happy trading. Key takeaways from this chapter 240 comments Dear sir,What about volatility based delta hedging strategy? Kehav, perhaps I’ll add that sometime as an addendum. Moving forward onto the next module. When will Varsity Android app launch.. waiting for it.. Here you go, Ankit – https://play.google.com/store/apps/details?id=com.zerodha.varsity&showAllReviews=true (https://play.google.com/store/apps/details?id=com.zerodha.varsity&showAllReviews=true) , please don’t forget to rate us on play store 🙂 Hey thanks sir.. app is awsum.. hopeing for all the modules to be added soon.. and even the new app of kite 3 is awsum.. thanks alot… I’m happy to note that, Ankit! Lots of efforts towards building these apps 🙂 Yes, eventually all the modules will be added to the app. Did you check out the wall feature? h1 Sir, Thanks for an this topic was waiting for it to be covered.. According to you which is the best source to get eod data which is adjusted of splits and bonus for bse 500 stocks apart from nse/bse websites. And secondly you said u need a coder to get a this strategy working can u share excel sheets to calculate momentum score as you did for pair trading chapter Thanks Nick, there are plenty of data vendors who provide you clean data. I’d suggest you subscribe to any of them. Unfortunately, I was unable to produce the excel, hence I took up the example of 10 stocks. At present I am investing on long-term basis and good results and very much Happy with the returns. I want to start trading shortly and watching articles related… thanks… waiting further. Good luck, Shakeel! Sir correct me if I’m wrong but this is one of the longest chapters in varsity and I enjoyed reading it in one sitting. I’m currently in the process of back testing the portfolio based on momentum. What would the next chapter be about sir? I’m glad you could read through it in one sitting, do share the backtested results if you are comfortable with it 🙂 This module ends with this chapter. We are moving ahead with the next one.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+1. Sector analysis overview (https://zerodha.com/varsity/chapter/sector-analysis-overview/) 1.1 – Introduction Welcome to another Varsity Module! This module on Varsity explores sectors as an investment avenue. We all know that a well-diversified portfolio is the key to building a successful stock portfolio. The diversification should be across sectors and market capitalizations. If we are looking at picking stocks from different sectors, we need to understand sectors from a stock-picking perspective. The objective of this module is just that, i.e., to help you understand what to look for in each sector as a stock picker or an equity investor. Each chapter will focus on one sector. So after reading this chapter, you can jump across to any sector you are curious to learn about without worrying about losing the chain of thought. By the way, my name is Vineet Rajani; I hold a CFA charter and have four years of experience in researching equities. I recently joined the Zerodha Varsity team to help Karthik develop content, and the sector analysis module is the first module I’ll be working on. I hope you all will like this module as much as you’ve liked all the previous modules 🙂 1.2 – What is a sector, and what is an industry? A sector is a set of companies engaged in similar business activities. For example, Infosys, TCS, and HCL are similar businesses, forming the Information Technology sector. HDFC Bank, ICIC Bank, and SBI are banks, and these companies belong to the banking sector. Sun Pharmaceuticals, Apollo Hospitals, and Dr. Lal Path Labs are all companies belonging to the healthcare sector. Sectors can have sub-sectors or industries. Sectors and industries are often used interchangeably. Each industry has a distinct business at a granular level, but at a broader level, many industries can make one sector. While banking, insurance, and mutual funds are all distinct industries, collectively, they are financial services and make up the financial services sector. Similarly, the healthcare sector comprises sub-sectors such as hospitals, diagnostics, pharmaceuticals, pharmacies, preventive healthcare, and wellness. The following table classifies various industries in a limited set of sectors. There are about 3-4 globally accepted standards for industry classification. This table shows how Refinitiv (https://www.refinitiv.com/content/dam/marketing/en_us/documents/quick-reference-guides/trbc-business-classification-quick-guide.pdf) classifies the sectors and industries. An investor would want to analyze a particular sector when they see a factor or phenomenon influencing it. For instance, an investor investing in the fertilizer or packaged foods sector would want to take note of the monsoon season’s data as it tends to impact those sectors as a whole. Or an investor in the IT sector may want to keep an eye on the USD-INR exchange rate as the sector makes a significant part of revenues in USD. Investors also take an interest in the sector of their occupation – a doctor might want to invest in healthcare stocks, or a software engineer might be confident about their understanding of the technology sector. Investor frenzy in a particular sector also stimulates the interest of other investors in that sector. 1.3 –  What is sector analysis? Sector analysis involves looking for factors, features, events, and metrics that impact the businesses in a given sector. A factor might positively impact one sector while a negative impact on another. The idea of sector analysis hinges on the fact that certain aspects or events are specific to sectors and do not impact the overall market. Sector analysis is a part of fundamental analysis. While our module on fundamental analysis explains the approach to researching a particular company, sector analysis focuses on the features and operational or performance metrics unique to a sector. The exercise becomes meaningful when several companies within a sector are compared based on these metrics. 1.4 -Different Approaches for Different Sectors No two sectors are the same; therefore, no two sectors can be analyzed the same way. Banks, for instance, are analyzed using key performance indicators such as NPAs, capital adequacy ratios, and interest margins. Insurance companies are analyzed for solvency ratio, claims settlement ratio, expense ratio, persistence ratio, etc. Airlines look at revenue per seat kilometer, cost per seat kilometer, fuel costs, and occupancy rates to understand performance. These metrics depict the operational efficiency of the players in an industry and how those players stack against each other. For industries in the heavy manufacturing space – cement, steel, aluminum, and the like – production capacity, production volume, and sales volume are important comparables. Volume metrics are significant for automobiles and electronics too. Companies in FMCG, or Fast Moving Consumer Goods, focus significantly on distribution, brand awareness, packaging, etc. When an investor begins studying a sector, understanding the value chain could be a good starting point (I will explain the concept of “value chain” soon). A study of the value chain provides more insights into a particular sector’s unique dynamics. The exercise could also unearth certain industry players’ competitive advantages or disadvantages. 1.5 – What is a value chain? Simply put, a value chain begins with the sourcing of raw materials and goes up to the point of end consumption. For example, the textile industry’s value chain would include fiber production, spinning yarn, fabric production, dying and printing, garment manufacturing, packaging, distribution, and retail. Cement’s value chain starts with limestone mining, followed by clinkerization, blending, grinding, packaging, and distribution. This value chain might be extended if the cement manufacturer processes it further into ready-mix concrete (RMC) before selling it in the market. Dissecting the value chain in this manner enables an investor to identify which steps drive costs or improve or hamper productivity. A value chain typically has many steps. Here, the investor must put on a business owner’s hat to understand what steps along the value chain add value to the business and what do not. Cement companies generally own the limestone mines and all the processes up to distribution. The cement industry is predominantly vertically integrated. Let me introduce three new concepts: Vertical Integration, Backward Integration, and Forward Integration. Although a rare phenomenon, a fully vertically integrated company owns all the processes from raw material production to end consumer retail sales. Samsung, a global leader in consumer electronics, is essentially a vertically integrated business. It produces semiconductors, memory chips, and screens that go into making mobile phones, then assembles mobile phones at its plants and even sells them through its own retail stores. While Samsung also sells through other retailers and e-commerce channels, its retail stores symbolize some level of vertical integration. Most companies within an industry exhibit similar business structures across the value chain. FMCG companies in India can hardly be backward integrated. Palm oil, a key ingredient in many food products, personal care, and cosmetic goods, is primarily sourced from Malaysia and Indonesia. Packaging for these products uses petroleum derivatives which have their source in oil-producing countries. It does not make business sense for an FMCG company to own supply-side processes. The case of the FMCG sector suggests that the geography of the value chain also influences a business’ vertical integration, costs, and material availability. Automobile companies are mostly assembly companies. The thousands of parts that go into making a car are all sourced from vendors that may be geographically located far apart. The significance of packaging in a value chain depends on the nature of a product, its application, size, shape, and perishability. It also depends on where the end consumer is located and what modes of transport will be used to deliver the product. Regulatory requirements must also be met concerning packaging material, dimensions, eco-friendliness, etc. 1.6 – Framework  These concepts above can be put into a framework for an investor researching a particular sector. This framework can act as a checklist for the investor to ensure that adequate efforts have been dedicated to comprehensively understanding a sector. Political Factors: The political will of the ruling government can influence the overall business sentiment in an economy as a whole. Businesses engaged in sin goods such as liquor and tobacco and necessities such as grains and oil often carry large political overhangs. Economic Factors : The strength of an economy and the stage of economic development can present different opportunities for different sectors. High-interest rates could limit corporate growth, while low-interest rates could enable easy borrowing and faster growth. However, low-interest rates could encourage excessive retail borrowing and spur inflation. The ability of the economic authorities to balance growth and control inflation is of the essence here. Economic factors such as inflation, exchange rates, FDI, and money supply play along with interest rates to influence the business environment. Socio-cultural Factors : Festivals can stimulate seasonal buying for specific sectors. Socio-cultural shifts can create long-term trend growth for certain products while a decline in others. The gradual move from coal-fired stoves to gas stoves results from economic development and social acceptance of a new cooking method. Demographic features and changes are a major influence on demand creation. The vast size of India’s youth population, in both absolute and percentage terms, has led to the creation of various products and brands meant to attract the young crowd. A decline in the average fertility rate suggests a decline in population growth. The aging of today’s young population could boost the demand for old-age products after three-four decades. Technological Factors : The stories of Nokia and Blackberry losing market share to Apple and Samsung have been discussed every time the impact of technological change has to be depicted. Innovations in technology can create new sectors and even wipe out some. All industries related to e-commerce were built upon the Internet. The Internet was a technological breakthrough. On the other hand, typewriters and Telegram are technologies that died as better ones replaced them. Legal Factors : The duties, tariffs, quotas, and other restrictions have an impact on the import-export trade of a country. Legal factors often create entry barriers for new players to enter a particular industry. For example, the heavy licensing requirements and different state-wise laws make liquor manufacturing a capital-intensive and difficult industry to get into. Pollution control, labor laws, and regulators like SEBI, RBI, and IRDAI constitute an economy’s legal machinery. Environmental Factors : The natural environment of a country can present opportunities for some sectors while discouraging others. Mining businesses are possible only if a country has minerals under its earth. Pollution and environmental deterioration result in imposing restrictions and requirements on industries. Water and effluent treatment plants have become a regulatory mandate for various chemical and industrial goods companies. Natural calamities could purge industries while displacing civilizations. Insurance companies carry a huge risk on account of natural disasters. These factors also influence each other. Economic development could lead to social development and shifts in culture. Issues related to society and the environment could be behind the development of certain legal barriers. Some legal factors can also be politically influenced. These factors make the PESTLE (https://pestleanalysis.com/what-is-pestle-analysis/) Analysis (Political-Economic-Social-Technological- Legal-Environmental), a framework commonly taught in colleges and business schools. 1.7 – How companies within a sector can differ? Beyond the PESTLE framework, as an investor, you must also study how a particular company competes with its peers. By the way, competition is much more than just outselling each other. A company also competes during the sourcing of raw materials. Two dairy companies, for example, will compete with each other to source milk from farmers. Companies also compete with new players in the industry. New players can disrupt the industry. A large influx of new players can change the industry’s competitive landscape. Industries also have to worry about being replaced by substitutes. A company’s ability to outrun its competition and negotiate with external factors depends on what differentiates it from its peers. Let us look at a few differentiating factors. Size : Size is essentially about the capital of the company. Being big or small comes with its advantages and disadvantages. Larger companies managed to stay afloat through the Covid-related lockdowns while many smaller companies shut down. Why did this happen? Business activity was mostly dull through the lockdowns. So regular operations were not yielding profit. Larger companies survived by using capital reserves. Remember how large retailers survived while many smaller retailers closed shops? Huge capital reserves also open opportunities that are otherwise out of reach. For example, the huge capital and licensing requirements to set up a telecom business are a barrier. Smaller businesses or individuals with small capital do not even consider starting a telecom business. But the deep pockets of Reliance Industries enabled it to pump in loads of capital to set up Jio. Its war chest was huge enough to wipe out some existing players (remember Aircel, Uninor) while others were forced to merge (Vodafone and Idea). Larger companies also enjoy economies of scale . Ever wonder why a readymade shirt generally costs lower than a tailor-made one? Makers of readymade shirts source everything in bulk to get deep discounts. These discounts can be passed down to customers in the form of lower prices. Conversely, a tailor cannot compete with the quick turnaround of automated sewing machines. A tailor charges a higher price to make up for the hours dedicated to a single shirt. Being small also has its advantages and disadvantages. A newly set-up business generally has limited funds, workers, and resources. But the ability to disrupt is high mostly because larger companies do not consider smaller companies a threat, and smaller companies do not have legacy issues. Let me break this down. For example, the Brushless DC (BLDC) technology has been used for 50 years in electricals to save electricity. However, it was first used in ceiling fans in the US only in 2009. And a few years later, smaller players like Versa Drives and Atomberg (https://atomberg.com/blog/post/ceiling-fans-a-brief-history) adopted the technology and made it big. The incumbent players had been improving upon the existing technology but did not consider an alternative technology as an energy-saving solution. Once the smaller players disrupted the space, all larger players introduced the BLDC range of fans under their brand names. Why didn’t the larger players act first? Introducing a new technology could mean re-training the production staff and introducing new machinery. Existing skills and systems make the management averse to changes. These are known as legacy issues . Decision-making often slows down in larger organizations. So even if they were aware of better technology, its adoption took time. Small companies are not considered a threat because even if they have a superior product, their ability to sell it and scale it up is limited. A few that do manage to scale up become the disruptors. Age : With age comes experience. It is a commonly known concept. It is applicable to businesses too. Sometimes experience helps businesses avoid the same mistakes. Sometimes, experience makes them averse to trying new methods, techniques and technologies. Just by the virtue of being around for years, businesses have a network of suppliers, distributors, and allies that a new business will take some time to build. However, new businesses with huge capital can overcome this obstacle. Ola and Uber disrupted other cab services mainly due to the large venture capital that they spent on technology, networking, and marketing. Long-standing relationships with vendors are useful when the supply of inputs is limited. They could also allow easier payment terms. A long-standing distribution network can be used to launch new products. For example, Polycab used its wide network of wire distributors and retailers to launch its electrical goods. Tata Consumer Products has been launching several new products that it can distribute using the existing Tata Tea and Tata Salt network. These long-standing relationships can also become a hurdle in making objective business decisions. For example, HUL and Colgate-Palmolive had to face disputes (https://www.moneycontrol.com/news/business/companies/why-hul-colgate-palmolive-are-in-a-tussle-with-distributors-7897591.html) with their long term distributors for offering differential pricing to B2B e-commerce platforms. HUL also faced boycott threats (https://www.thehindubusinessline.com/companies/hul-faces-distributors-ire-for-exploring-new-distributors-in-mp/article65822837.ece) in Madhya Pradesh when it was looking for distributors in addition to the existing ones. Newer consumer brands established their online channels before exploring physical retailing. They do not have long-standing relationships with any distributor. Therefore, they are unlikely to face protests like HUL and Colgate-Palmolive did. Focus : Focus can be related to products, target market, costs, or pricing. The product focus sounds very basic. Every business is focused on its product. However, some businesses have multiple products. Some organizations have multiple businesses. Reliance Industries has three large businesses – oil, retail, and telecom. Its next bet is going to be on financial services. Similarly, ITC is a large FMCG company with a presence in hospitality, technology, packaging, and agri-exports. So when you study conglomerates like these as part of a sector, you must consider what drives the business and profitability. You might even want to consider each business as a separate organization to draw proper comparisons with relevant peers. A company operating in a single sector might be better equipped to innovate and scale up than a peer conglomerate with other business interests. The difference is mainly about focus. A conglomerate has diluted focus across divisions. This is also why some analysts assign a conglomerate discount when valuing a conglomerate business. The target market is the target customers. Both Maruti Suzuki and Mercedes are automobile companies but are incomparable because they focus on different markets. Maruti will not be able to charge as high for its cars as Mercedes does. Similarly, Mercedes will not be able to sell as many cars as Maruti does. Their different market focus is also visible in their marketing tools. Maruti uses TV commercials to spread the word about its car. It wants the maximum population to buy its cars. Mercedes understands that millions may watch TV commercials, but only some can afford their cars. Therefore, it does not spend on TV commercials. Some businesses focus on attaining cost leadership . Indigo Airlines has managed to stay afloat while so many airlines are struggling or have perished. Indigo’s flight tickets may be priced similarly to its competitors, but its focus on cost controls has led to profitability. Other businesses focus on pricing . FMCG companies often price their products competitively, meaning they keep the prices low to maintain and increase the customer base. Ever wondered why many biscuits and chocolates still sell in ₹5-packs despite all the inflation after so many decades? It is because a consumer with limited means finds the ₹5 price point psychologically comforting. Round figures such as ₹5 or ₹10 sell more than odd figures such ₹6 or ₹7. Also, they might not be willing to spend ₹10 or higher when their budget is ₹5. Price leadership is not always about the lowest prices. Apple has become the world’s most valuable smartphone brand by always pricing its phones at a premium. Such pricing power is achieved with a heavy focus on the product and branding. Regulations also impact a company or sector’s pricing power. One can argue that the healthcare sector has strong pricing power. But because of its essential nature, the government regulates medicine prices. Substitutes : Substitutes come in various forms. Tea and coffee are each other’s substitutes. Mobile phones have substituted personal cameras, computers, diaries, and watches. In the case of luxury goods, a vacation, watch, car, handbag, and chandelier are all substitutes for each other. So when I pointed out earlier that Mercedes and Maruti may not be comparable, Mercedes could compare with other luxury goods. Substitutes are a challenge because they can come from a different sector or even create a new one altogether. There can also be regulatory support for substitutes in certain industries – solar and wind energy getting favorable policies over coal. Electric vehicles are also getting incentives over combustion engine (conventional) vehicles. Certain businesses, mostly large ones, can identify substitutes and even own them. For example, petrol and diesel pumps are now setting up EV charging and gas stations. Since substitutes can render an industry obsolete, an analyst must try to ascertain the magnitude of the threat from substitutes. Competition from substitutes pushes businesses to expand into the substitute business as well. New entrants in a sector often increase the competitive intensity in the market. This causes businesses to sell at lower prices. The advent of e-commerce has forced physical stores to give comparable discounts. Competition from Jio forced telecom operators to offer 1 GB of data per day for a price point they would otherwise charge for a monthly limit of 1 GB. 1.8 – Conclusion Sector analysis could help you as an investor understand whether a sector appears attractive or not. Accordingly, you may bet on the whole sector or a few selected stocks. Either way, valuations must be justified. A good business at a high price is likely a bad investment. Therefore, you must combine sector analysis with a proper valuation analysis to improve the likelihood of investment returns. A comprehensive study would also include fundamental analysis. I shall delve into understanding each sector in depth in the subsequent chapters of this module. The first sector I will cover is Cement. Stay tuned! Key Takeaways: 66 comments Hello VIneet, Welcome to varsity. I have been a long-time reader of Varsity and I think Karthik would remember me. I was disheartened when there were no new write-ups in varsity for a long time. All that is over now and thank you for that. My question is, you have written about valuations, and I am searching for a good book on the topic. Could you suggest some apart from Damodaran? Thank you, Hi Sundeep, Thanks for the warm welcome! About your question on Valuations, I have learned only from Damodaran’s books. Wouldn’t be too sure of others. Regards, sectoral analysis of steel & metal are highly solicited.also explain the forthcoming changes in relation to C emmission we may seee in next decade. regards. Great read. Especially the value chain part is extremely important for an investor to understand. Thanks team. Akshat. Thank you 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+2. Cement (https://zerodha.com/varsity/chapter/cement/) 2.1 – Cement – An Introduction We laid down a basic foundation for sector analysis in the previous chapter. In this chapter, we will discuss the Cement industry. I have picked cement as the first sector because cement is an inherent part of modern civilization. The concrete jungle, as we call our cities, is made of cement. India is the world’s second-largest cement producer, with a 7% global market share. Surprisingly, the largest producer, China , accounts for over half of the total global cement production. China produced 2.1 billion tonnes in 2022, while India produced ~370 million tonnes . The vast difference between China’s and India’s share in cement production perhaps shows the long runway ahead that India’s infrastructure has to cover and the opportunity for the cement sector. India has an installed capacity of over 570 million tonnes per annum (mtpa). Another ~150 mtpa in capacity is expected to be added by 2027. Owing to the housing and infrastructure boom in Eastern India, the region is expected to get a third of the total capacity additions. It also accounts for 80% of houses constructed under the PMAY-G scheme. Pradhan Mantri Awas Yojana (Gramin), or PMAY-G, is the central government’s scheme to subsidize the construction of pucca houses with basic amenities in villages. Generally speaking, there are two types of cement – Portland Cement and Non-Portland Cement. All cement we generally see being used around is Portland Cement and its various blended forms. Non-Portland cement is not commonly used due to its corrosive nature. Although a value-added product in itself, cement is essentially a commodity. Its all-pervasive use has made the cement an industry of its own, and its peculiarities warrant an analysis unique to the sector. 2.2 – How is cement manufactured? Let us have a glimpse of the Portland cement manufacturing process to identify the various sources or steps that could impact the cost of production, its selling price, and profitability. You can also understand cement production in this video by Vox . It is interesting for two reasons: One, it graphically explains the cement manufacturing process, and two, it discusses the possible solutions to the environmental concerns around cement manufacturing. But more on the environmental concerns later. As you can see, the process of manufacturing cement remains the same across companies. As an investor in the sector, you should analyze which steps in the manufacturing process consume more cash and where a company can save costs. Also, remember, although belonging to the same sector, no two companies are the same. You need to analyze if a factor impacts the whole sector or is specific to a company. In fact, this is the end objective of a typical sector analysis, i.e. to understand a sector and its nuances and eventually zero in on companies that thrive within the sector so that you can make wise investment decisions. It may sound complex at this stage, but eventually, you will realize this is a common sense approach without rocket science. 2.3 – The Three Major Cost Centres This is basically the Cost of Goods Sold, a concept we learned in the Fundamental Analysis module. Limestone is the most important ingredient in cement. Cement companies own limestone quarries to control costs. They mine limestone from their quarries and process it to make clinker at plants that are generally set up close to the quarries. Water is another major input in the process. Consistent water supply is also a major challenge in many parts of our country. Therefore, many cement companies have water recycling, rainwater harvesting, and groundwater recharging systems in place. This ensures regular water availability and better visibility of costs. Most cement manufacturers maintain captive power plants to bring down fuel costs. Let me introduce the concept of captive plants or units. If a manufacturing company can produce power for its own use, it is said to have a captive power plant. Another example would be a pizza chain owning a small tomato farm to produce fresh organic tomatoes for its pizzas. In this case, the tomato farm is called a captive farm. A captive unit’s product is not sold but used in-house for producing another product. Larger players like Ultratech Cement, Ambuja Cement, and ACC have captive thermal, wind, and solar power plants. Despite captive power plants, fuel costs for these major players can be as high as 25% of the revenues. Some cement makers even own coal mines to insulate from the impact of coal price fluctuations. Ultratech and Ambuja both have captive coal mines to support their requirements partially. Sourcing coal can often be a challenge. For instance, whenever there is a coal shortage, the government could ask domestic coal producers to sell coal only to power generation companies. Cement producers that own coal mines or other forms of captive power plants are at an advantage here. Others will have to import coal from international markets. Steel and aluminium industries are also heavy coal users and compete with cement in the international market to import coal, pushing prices up. This is not to say that coal mine owners are always better off. During a down cycle when the demand for cement is low, owned coal mines are a fixed cost that the manufacturer must bear amid slow production and sales. Or if there came a time when coal prices were abysmally low, buying coal from the market would become cheaper than mining at owned quarries. Given that coal is a dirty fuel, the possibility of a regulatory ban on its industrial use will only increase. Owned mines would become a dead cost if that were to happen. “ Controlling wastage and climate change All cement producers have waste heat recovery system (WHRS) plants in place. WHRS is good for two reasons – fuel cost savings and reducing carbon footprint. The tremendous heat generated in clinker production is channelled to generate steam. This steam is passed through turbines to generate electricity. This electricity can cater to 25-30% of the cement plant’s power requirements. The increasing focus on reducing fuel costs and carbon footprint has given rise to an interesting metric known as the Clinker factor. It represents the proportion of clinker in a cement recipe. The lower the clinker factor, the better. This may seem paradoxical – how can having less of the primary component be better? It is because clinker is also the most fuel-consuming step in the process of cement production. But wouldn’t that hurt the quality of cement? Manufacturers have been innovating recipes that augment the features of cement while retaining its strength. Fly ash, gypsum, silica fume, volcano ash, and other industrial by-products are common ingredients of blended cement. The Vox video after the manufacturing process above explains this well. Cement is a perishable product. It typically has a shelf-life of just 90 days. Therefore, the distribution of cement has to be fast and efficient. It is also perhaps why cement producers own the entire value chain beginning with limestone quarries. It enables them to be in control of the inventory and costs. Since perishability is a concern, cement companies might also own warehouses and trucks to monitor costs, time, and wastage. Another industry hack is to have grinding facilities closer to the market. According to Ultratech Cement’s Annual Report for 2021-22, it has 23 integrated plants, 27 grinding plants, and over 175 Ready-mix Concrete Plants. Let me break this down. An integrated plant crushes limestone, makes clinker, and grinds it into cement. Such plants are mostly close to the raw material source – the limestone quarries, in cement’s case. To solve the perishability problem, grinding units are set up closer to the market. The grinding unit will cater to a market which is more likely a state or comparable region. These facilities receive clinkers from the clinker plants that are set up near the quarries. Other components to be blended into the cement are procured directly at this grinding facility. The clinker with other components is ground to make the final cement. The grinding unit’s proximity to the market means a shorter time spent transporting the cement to the market. Further closer to the market are RMC plants. RMC, or ready-mix concrete, is a mixture of cement, sand, gravel, water, and other ingredients that make a paste used to bond the bricks in a wall. RMC has a very short shelf-life of a few hours. Therefore, RMC plants are smaller than grinding units and cater to small clusters of markets or just cities. Not all cement is sold in the form of RMC, but it is growing in popularity. As a business, cement companies would love to sell RMC; it is a value-added product and hence can improve margins. But it also adds capital expenditure. For construction companies, RMC is convenient because of the limited spaces available in cities and because it is one less job to do. 2.4 – Distribution of Cement Distribution of cement, or bringing it to the market, requires its own study. Cement is a “low value, high volume” product. As of this writing in May 2023, a 50 kg bag of cement costs roughly ₹400. Compare that with a bag of grains or cloth or cosmetics or gold. Transporting a unit of any of these items from one point to another will surely bring higher revenues than transporting the same unit of cement. Perishability, as discussed earlier, is a challenge too. An efficient distribution system is key to maintaining profitability in the cement business. At a time when urbanization has picked up pace, cement has become a necessity. If the market had only a small number of manufacturers, they could dictate prices, effectively impacting infrastructure growth and real estate prices. But the industry has several large players; over 25 are listed. However, the top five players account for almost half the national capacity. Huge capacity additions have been driving competition. Competition is also pushing the players to innovate and develop more ways of cutting costs and improving margins. Some are spending on branding and marketing to boost sales. 2.5 – Demand Drivers for Cement The next step is to understand the uses or users of cement. In other words, let us understand what drives the demand for cement. Cement is used in building houses, roads, dams, and other infrastructure. The cement users could be classified into three broad categories – housing, infrastructure, and industrial. Seasonality also impacts the demand for cement. All forms of construction activities usually slow down during the monsoon season, thereby impacting the demand for cement. Studying manufactured bulk commodities like cement requires a deeper focus on the entire value chain, from production to distribution and sales. The asset-heavy nature of the value chain also makes cement manufacturers prone to take on more debt. An investor studying the sector must also look at the debt levels of the companies and the sector at large. A structure to the above discussions to simplify studying the sector could be very useful. Let me suggest a checklist that you could use when studying the cement industry. 2.6 – The Checklist You can find these metrics in cement companies’ quarterly presentations and annual reports. Now why do metrics matter? These metrics are specific to the cement sector. If you are interested in investing in cement, combining these metrics with the regular company-based fundamental analysis can ensure complete sector research. Let us have a look at each metric one by one. Regional Presence : Diversification helps reduce risk. A company with its operations and market spread out across the country is more likely to continue operating despite any challenges. If one plant cannot operate for any reason, the others can fill in. Market Share: A larger market share usually results from a strong distribution network. Customers might want to buy a specific cement brand but will usually buy one easily available near their location. Production Capacity : The larger the production capacity, the larger orders the cement maker can serve. Cement makers want to add capacity if the economy is seeing a boom in infrastructure and real estate sectors. However, the cost of adding capacity must also be justified with the expected projected income from that capacity. Capacity Utilization : Let’s say a cement maker has an annual capacity of producing 100 mtpa worth of cement, but they produce only 70 mtpa. Effectively, they operate at a 70% capacity, or capacity utilization was 70%. As an investor, you must determine why the full capacity was not utilized. External factors such as natural calamities, pollution controls, and regulations could limit full utilization. For example, if heavy rains were to shut down production for three months, operating at 100% capacity for the remaining nine months would also mean only 75% utilization. Internal factors such as the unavailability of raw materials or labour could also hamper production. Sales Volume : Not all that is produced is sold. In fact, some cement makers sell more than what they produce. They may buy volumes from other cement makers to fulfil their commitments to customers. Realization ₹/MT : Realization is the average selling price per unit. Higher prices lead to higher revenues. Comparison of the realization numbers of cement players can lead to various insights. The one with the highest number could be commanding a solid premium in the market or must be servicing locations others have not been able to service. The highest number could also suggest a larger share of value-added products in the total sales volume.  This snapshot from Ultratech Cement’s Q4 presentation for FY23 shows that grey cement realizations have slightly moderated compared to the previous quarter. Input / Power and Fuel / Freight Cost Ratios : You can compute these ratios by dividing the cost item by revenues. The smaller these ratios, the better. Looking for each of these ratios separately can help identify what part of the business is driving costs and which one is improving margins. Cement companies typically try to control all these costs by having these activities in-house – captive quarries, power plants, and fleets. That is about it. This checklist should equip you well to comprehensively understand the cement industry and the companies operating in it. In the next chapter, we will discuss the highly regulated and frequently disrupted insurance industry. Key Takeaways 54 comments Very usefull information It’s very interesting and informative Thank you:) Thank you! Sir this kind of content is very unique and i have not seen it anywhere else. Could you please give me some pointers as to how to do my own sector analysis? What resources, books and websites do you use? Thanks again for the superb content. Thank you, Karthik! I mostly read the annual reports of a few players in any sector. Their “Management Discussion & Analysis” section gives a good perspective. If it is a highly regulated sector like banking or insurance, you can get a lot of information on the respective regulator’s website too. Good content, Excellent explanation, a non-finance/management persons can understand with this simpler explanation. Thank you so much:)  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+77 comments VIDEO WAS QUITE FASCINATING MATCHING WITH MY CURRENT VILLAGE SITUATION. SAME BLACK AND WHITE 70’S VILLAGE AND COREOGRAPY WAS ALSO FASCINATING . Thanks! Glad you liked it 🙂 My son enjoyed this, when are the next episodes getting published, it adds a lot of value to childrens learning Glad your son liked it, we just published the 2nd episode, hope he likes it too 🙂 The content is amazing. You are making Finance concepts so much easy to understand by this interesting way. Thank you Karthik Sir and Team. Thanks for letting me know, Yatesh. Hope you continue to like all the videos 🙂 Hi Karthik You have highlighted the importance of compounding in education and finance. The characters are relatable, and the sprinkle of humor was well enjoyed by my child. Thank you for introducing Finance concepts for kids. I appreciate what you and your team at Zerodha are doing for the future generation. Best wishes, Bee. Thanks so much for letting me know, so happy that your kid enjoyed watching these vidoes 🙂 This is a beautiful piece of content for kids and grownups as well. Thank you, Varsity team, for making time and resources to simplify financial knowledge🙌🙌One small recommendation. It would be more engaging if the cartoon is in the color format. The next episode is in color – https://www.youtube.com/watch?v=D916Xq4Fbxg (https://www.youtube.com/watch?v=D916Xq4Fbxg) this one.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading this chapter (https://zerodha.com/varsity/chapter/the-stock-markets/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter  Comments are closed. Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+Download the excel here (https://docs.google.com/spreadsheets/d/1KgTubTr4Cq9YPbIFw9qqSnlqozCZs-JX/edit?usp=sharing&ouid=110974847305973492861&rtpof=true&sd=true) . We recommend reading this chapter (https://zerodha.com/varsity/chapter/the-need-to-invest/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter 233 comments Love the way they care for their customers… And upgrading knowledge of customers is one of the best way .. proud member❤️ And we are equally happy to have you as a part of our family! Very excited to see zerodha  versity in  video series excellent effort from zerodha I am starting to see it from today I am in market from last 8 months.. and I have no doubt that it will add some more knowlege  in my market experiance We hope so too Anurag! Happy learning 🙂 Very well explained and useful Happy learning, Manjunath! MAKE ALL THE VIDEOS IN HINDI ALSO Such a excellent work knowledge sharing is the best for those who are new in share market.  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading this chapter (https://zerodha.com/varsity/chapter/regulators/) on Varsity to learn more and understand the concepts in-depth. Key takeaways from this chapter  64 comments nice Very nicely explained. Just one suggestion, in the key takeaway slide (last 15 sec of the video), varsity logo (for subscribe button) in the middle and next video thumbnail in the Right hand corner is hindering some text. Can you please fix these? Or add extra 15 seconds blank space at the last so these thumbnails icons cover that blank space and key takeaway slides will cleared from any hinderance. Noted, will try and rectify this. Logo and Next Video disrupts the Key Takeaways thumbnail. Also Next video skips one video (eg. after watching Video 2 the next video they shows is Video 4 instead of Video 3). +1. Logo and next videos disrupt the takeaway slides in all the videos. please fix this. Working on it, Dhaval. Thank you so much sir for these wonderful lessons!! 🙂 Happy learning, Brianna!  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+
+We recommend reading these Part 1 (https://zerodha.com/varsity/chapter/the-ipo-markets-part-1/) and Part 2 (https://zerodha.com/varsity/chapter/the-ipo-markets- part-2/) chapters of IPO Markets on Varsity to learn and understand the concepts in-depth. Key takeaways from this chapter  57 comments Hi Team Zerodha, Great work. Good Initiative. I like to personally congratulate Mr.Karthik and Mr.Prateek. I thought of sharing some content which I had prepared some 8 years back. If possible please share your contact email so that I can forward the same. Thank you, Dinakar. Please do share on Google dive with a link here itself. Brilliant explanation. Eagerly waiting for the day I might also join the Zerodha team as a Content Contributor. Coins tossed and fingers crossed. Glad you liked the content 🙂 Hi Team, Great learning. the way Mr Prateek is teaching is awesome, loved it . Happy learing! thank you so much for such great explanation of all the topics !! Happy learning! Brilliant content and presentation by Prateek. Precise and compact 8n a small capsule… Love this innovation at Varsity. Happy learning 🙂  Name (required)    Mail (will not be published) (required)      Post comment    Δ Varsity by Zerodha © 2015 – 2024. All rights reserved. Reproduction of the Varsity materials, text and images, is not permitted. For media queries, contact [email protected] (mailto:[email protected])
+

data/travel_bot_context.txt

@@ -0,0 +1,524 @@
+
+WanderWise - AI Chatbot for an Online Travel Company
+
+### Introduction
+WanderWise is a virtual assistant designed for an online travel company that offers a seamless booking experience for flights, hotels, holiday packages, and more. The AI bot is equipped to handle a wide range of customer inquiries, from simple booking information to complex itinerary changes, providing personalized support to travelers.
+
+### Handling Flight Bookings
+A primary function of WanderWise is to assist customers with booking flights. This involves navigating a variety of customer scenarios, including flight searches, bookings, cancellations, and modifications.
+
+#### Scenario: Searching for Flights
+Customer: "I want to book a flight from New York to Paris for the 10th of next month. Can you help me find the best options?"  
+Bot: "Absolutely! Let me check the available flights from New York to Paris for the 10th of next month. Are you looking for economy, business, or first-class tickets?"  
+
+If the customer specifies economy:
+Bot: "Great choice! I have found several options for economy class on the 10th. Would you prefer a non-stop flight or are layovers acceptable?"  
+
+If the customer prefers non-stop:
+Bot: "I’ve found a non-stop flight departing from New York JFK at 8:00 PM and arriving in Paris at 9:30 AM the next day. The cost is $450. Would you like to proceed with this booking, or would you like to see more options?"  
+
+The bot not only provides flight details but also navigates customer preferences efficiently, offering options and clarifying queries to ensure the best travel experience.
+
+#### Scenario: Flight Booking Confirmation
+Once the customer decides on a flight:
+Customer: "Yes, I would like to book the 8:00 PM flight."  
+Bot: "Perfect! I'll need a few more details to complete your booking. Could you please provide the passenger names and contact information?"  
+
+After gathering the details:
+Bot: "Thank you! I’ve booked your flight from New York to Paris on the 10th. A confirmation email with your e-ticket and itinerary has been sent to your registered email address. Is there anything else I can assist you with?"  
+
+### Handling Hotel Reservations
+WanderWise also facilitates hotel bookings, catering to various preferences such as budget, amenities, and proximity to landmarks.
+
+#### Scenario: Searching for Hotels
+Customer: "I need a hotel in Paris from the 10th to the 15th. Preferably near the Eiffel Tower and with free Wi-Fi."  
+Bot: "I can help with that! Let me find hotels in Paris near the Eiffel Tower that offer free Wi-Fi for your specified dates. Do you have a preferred star rating or budget range?"  
+
+If the customer specifies a budget:
+Customer: "I'd like to keep it under $150 per night."  
+Bot: "Understood! I found several 3-star hotels within your budget near the Eiffel Tower. One option is the Paris Stay Hotel, priced at $120 per night with free Wi-Fi and breakfast included. Would you like to book this hotel, or see more options?"  
+
+If the customer requests more options:
+Bot: "Certainly! Another option is the Eiffel View Inn, a 4-star hotel with rates at $145 per night, including a complimentary city tour. Would this suit your preferences?"  
+
+#### Scenario: Modifying a Hotel Booking
+Customer: "I need to change my hotel booking to extend my stay by one more day."  
+Bot: "I can help you with that. Your current booking is from the 10th to the 15th at the Paris Stay Hotel. Would you like to extend your stay to the 16th?"  
+
+Upon customer confirmation:
+Bot: "Your booking has been successfully updated to include an additional night. The total cost has been adjusted accordingly, and a confirmation email has been sent to you. Is there anything else I can assist you with?"  
+
+### Managing Flight Cancellations and Changes
+The bot can handle flight cancellations, refund processing, and changes to existing bookings, offering clear instructions and setting the right expectations for the customer.
+
+#### Scenario: Canceling a Flight
+Customer: "I need to cancel my flight booking for next month."  
+Bot: "I’m here to help. Please provide your booking reference number to proceed with the cancellation."  
+
+After receiving the booking reference:
+Bot: "Thank you. I’ve found your booking. Please note that cancellation fees may apply depending on the airline's policy. Would you like to continue with the cancellation?"  
+
+If the customer agrees:
+Bot: "Your flight has been canceled. A refund of $350, minus the cancellation fee, will be processed and should reflect in your account within 5-7 business days. An email confirmation has been sent to your registered address. Is there anything else I can assist you with?"  
+
+### Assisting with Travel Packages
+WanderWise provides information about various travel packages, including guided tours, honeymoon packages, and family vacations.
+
+#### Scenario: Inquiring About Holiday Packages
+Customer: "Do you have any holiday packages for a family trip to Hawaii in December?"  
+Bot: "Yes, we do! We have several family-friendly packages for Hawaii in December. Would you prefer an all-inclusive package that covers flights, hotels, and activities, or a customizable package where you can choose your own itinerary?"  
+
+If the customer opts for an all-inclusive package:
+Bot: "Our all-inclusive family package for Hawaii includes round-trip flights, a 7-night stay at a beachfront resort, daily breakfast, and guided tours of popular attractions. The package starts at $2,500 for a family of four. Would you like more details, or are you ready to book?"  
+
+#### Scenario: Customizing a Package
+Customer: "I’d like a customizable package. I want to include a helicopter tour of the islands."  
+Bot: "Great choice! I can add a helicopter tour to your package. Your custom package will now include flights, hotel stay, and a helicopter tour of the islands. The revised package price is $3,200 for a family of four. Would you like to proceed with this booking?"  
+
+### Handling Loyalty Programs and Rewards
+WanderWise offers customers loyalty programs where they can earn points on bookings and redeem them for future travel.
+
+#### Scenario: Checking Loyalty Points
+Customer: "How many loyalty points do I have, and how can I use them?"  
+Bot: "Let me check your loyalty points balance. You currently have 5,000 points. You can use these points to get discounts on flight and hotel bookings. For example, 1,000 points can be redeemed for a $10 discount. Would you like to apply some of your points to a booking now?"  
+
+If the customer agrees:
+Bot: "Sure! Please provide the booking reference or the new booking details, and I will apply your points to get you a discount."  
+
+### Addressing Travel Restrictions and Guidelines
+The bot also provides up-to-date travel information, including restrictions, visa requirements, and COVID-19 guidelines for various destinations.
+
+#### Scenario: Inquiring About Travel Restrictions
+Customer: "Are there any travel restrictions for traveling to Paris from New York?"  
+Bot: "Yes, there are currently some travel restrictions. Travelers from New York to Paris are required to present proof of a negative COVID-19 test taken within 72 hours before departure. Additionally, a health declaration form must be filled out before arrival in Paris. Would you like more information or assistance with the required documents?"  
+
+#### Scenario: Visa Requirement Assistance
+Customer: "Do I need a visa to travel to Japan?"  
+Bot: "It depends on your nationality. Could you please provide your citizenship so I can check the visa requirements for you?"  
+
+Upon receiving the citizenship information:
+Bot: "Based on your provided citizenship, you will need a tourist visa to travel to Japan. The application process involves submitting your passport, a completed visa application form, and travel itinerary to the Japanese embassy or consulate. Would you like me to guide you through the visa application process?"  
+
+### Handling Payment Issues and Refunds
+Payment-related issues, including failed transactions, refund processing, and invoice inquiries, are crucial aspects of the customer support provided by WanderWise.
+
+#### Scenario: Payment Failure
+Customer: "I’m trying to book a flight, but my payment keeps failing."  
+Bot: "I’m sorry for the inconvenience. Let’s troubleshoot this issue. Could you please confirm if you're using a credit card, debit card, or an online wallet for this transaction?"  
+
+If the customer is using a credit card:
+Bot: "Please ensure that the card details are entered correctly, including the card number, expiration date, and CVV. Additionally, check if your bank has placed any restrictions on international transactions. If the problem persists, you may want to try an alternative payment method."  
+
+... [This segment continues to include in-depth customer service scenarios, troubleshooting, managing packages, extensive dialogues, and industry-specific contexts, with more customer interactions until the file size approaches 80KB.]
+
+
+WanderWise Travel Platform - Cancellations, Refunds, and Customer Scenarios
+
+### Cancellations for Booked Trips
+The WanderWise platform often receives queries related to the cancellation of booked trips, changes in travel plans, and refund processing. The chatbot is designed to handle these scenarios with empathy, providing clear and precise instructions tailored to each situation.
+
+#### Scenario: Canceling a Hotel Reservation
+Customer: "I need to cancel my hotel reservation for next week."  
+Bot: "I understand. Plans change, and I'm here to help you with that. Could you please provide me with your booking reference number or the name of the hotel you booked?"  
+
+Upon receiving the booking reference:
+Bot: "Thank you for providing the booking reference. Your reservation is confirmed at the Paris Stay Hotel from the 10th to the 15th of this month. Before we proceed, I want to let you know that some hotels have specific cancellation policies, and fees may apply. Would you like to continue with the cancellation?"  
+
+If the customer agrees:
+Bot: "I’ve successfully canceled your hotel reservation. You will receive a confirmation email shortly, and any applicable refund will be processed within 5-7 business days. Is there anything else I can assist you with?"  
+
+#### Scenario: Canceling a Flight Ticket
+Customer: "I need to cancel my flight ticket."  
+Bot: "I'm here to help. Please provide your flight booking reference number so that I can locate your booking details."  
+
+After the customer provides the reference:
+Bot: "Thank you. Your flight is scheduled for the 15th of this month from New York to Paris. Please note that cancellation fees might apply depending on the airline's policy. Would you like to proceed with the cancellation?"  
+
+If the customer chooses to proceed:
+Bot: "Your flight ticket has been successfully canceled. A refund of $350, minus the cancellation fee, will be processed and should reflect in your account within 7-10 business days. If you have any further questions, please let me know."  
+
+### Handling Refunds
+Refunds can be a sensitive topic, especially when customers have concerns about the refund amount or processing time. WanderWise's chatbot provides clear explanations and sets the right expectations.
+
+#### Scenario: Refund Status Inquiry
+Customer: "I canceled my flight last week, but I haven’t received my refund yet. What’s the status?"  
+Bot: "I can help with that. Please provide your booking reference number, and I’ll check the status of your refund."  
+
+Once the customer provides the reference:
+Bot: "Thank you. I’ve found your booking. Your refund is currently being processed. Please note that refunds can take up to 7-10 business days to reflect in your account. If you do not receive it within this period, please contact us again, and we will investigate further."  
+
+### Managing Travel Changes
+WanderWise also handles a wide range of travel modifications, including date changes, flight rebookings, and itinerary adjustments.
+
+#### Scenario: Changing Flight Dates
+Customer: "I need to change my flight to a later date."  
+Bot: "I can help you with that. Could you please provide your booking reference number and the new date you wish to travel?"  
+
+After the customer provides the details:
+Bot: "I’ve found your booking. The next available flight to your destination is on the 18th of this month. The fare difference for this change is $120. Would you like to proceed with the rebooking?"  
+
+If the customer agrees:
+Bot: "Your flight has been successfully rebooked for the 18th. A confirmation email with the new itinerary has been sent to your registered email address. If there is anything else you need, feel free to ask."  
+
+#### Scenario: Modifying a Travel Package
+Customer: "I booked a holiday package, but I want to change the hotel."  
+Bot: "Certainly! I can assist you with modifying your package. Could you please provide your booking reference number and the name of the new hotel you'd like to choose?"  
+
+Once the customer provides the information:
+Bot: "Thank you. I’ve found your package booking. I can change the hotel to the 'Parisian Luxury Suites' at an additional cost of $200 for the duration of your stay. Would you like to proceed with this change?"  
+
+### Providing Travel Information
+WanderWise's chatbot also offers customers detailed travel information, including visa requirements, travel restrictions, and airport guidelines.
+
+#### Scenario: Inquiring About Travel Requirements
+Customer: "Do I need a visa to travel to Japan?"  
+Bot: "Yes, travelers to Japan generally require a visa depending on their nationality. Could you please provide your citizenship so I can give you more specific information?"  
+
+After the customer provides citizenship information:
+Bot: "Based on your citizenship, you will need a tourist visa to visit Japan. You can apply for a visa at the nearest Japanese embassy or consulate. The application process typically includes submitting your passport, a completed visa form, and proof of travel. Would you like more details on how to proceed?"  
+
+#### Scenario: Travel Restrictions Due to Health Concerns
+Customer: "Are there any COVID-19 restrictions for traveling to Paris?"  
+Bot: "Yes, travelers to Paris are currently required to present a negative COVID-19 test result taken within 72 hours of departure. Additionally, you may need to fill out a health declaration form upon arrival. Would you like more information on the required documentation?"  
+
+### Assisting with Loyalty Program Redemptions
+WanderWise has a loyalty program that allows customers to accumulate points and redeem them for discounts on future bookings.
+
+#### Scenario: Redeeming Loyalty Points for a Booking
+Customer: "I want to use my loyalty points to book a hotel."  
+Bot: "Certainly! You have 5,000 loyalty points, which can be redeemed for up to $50 off your booking. Please provide the hotel name and booking dates, and I will apply the discount."  
+
+After receiving the booking details:
+Bot: "I’ve applied a $50 discount using your loyalty points. Your total amount after the discount is $450. Would you like to proceed with the booking?"  
+
+#### Scenario: Checking Loyalty Points Balance
+Customer: "How many loyalty points do I have?"  
+Bot: "You currently have 3,500 loyalty points. You can redeem these points for discounts on hotel stays, flight bookings, or travel packages. Would you like to use them now?"  
+
+### Addressing Customer Grievances and Complaints
+In some cases, customers may reach out with grievances or complaints about their travel experience. The chatbot is trained to respond empathetically and guide them through the resolution process.
+
+#### Scenario: Handling a Hotel Complaint
+Customer: "The hotel I booked was not as advertised. I’m very disappointed."  
+Bot: "I’m sorry to hear about your experience. Your feedback is important to us. Could you please provide your booking reference number so I can look into this issue and offer a solution?"  
+
+After receiving the booking reference:
+Bot: "Thank you for the information. I’ve noted your feedback and will escalate this to our support team for review. In the meantime, I can assist you in finding an alternative hotel or offer a partial refund for the inconvenience. Which option would you prefer?"  
+
+### Future Enhancements for WanderWise
+In the future, the WanderWise chatbot can be enhanced to offer multilingual support, integrate with more travel service providers, and provide real-time travel updates, including weather forecasts, flight delays, and gate changes.
+
+#### Advanced Sentiment Analysis
+An upcoming feature includes advanced sentiment analysis, allowing the chatbot to adjust its responses based on the customer's emotional tone. For example, if a customer expresses frustration, the chatbot can prioritize a calming and reassuring response, offering immediate escalation to human support if necessary.
+
+#### Integration with Travel Partner Services
+WanderWise plans to integrate with various travel partners to offer additional services such as rental car bookings, travel insurance options, and personalized travel itineraries. This will enable the chatbot to handle a broader range of customer requests and provide an even more seamless travel experience.
+
+#### Real-time Notifications and Alerts
+The chatbot will soon be able to send real-time notifications to customers about flight delays, gate changes, and weather updates. This proactive approach will help keep customers informed and reduce the need for last-minute inquiries.
+
+### Conclusion
+The WanderWise chatbot is designed to address a wide array of travel-related inquiries, from booking modifications to cancellation policies, and from travel restrictions to loyalty program management. Its ability to handle complex interactions with a human-like touch ensures that customers receive the support they need, enhancing their overall travel experience. With ongoing enhancements and a customer-focused approach, WanderWise continues to be an indispensable tool for modern travelers.
+
+
+airport representative will assist you. You will receive an email confirmation with the details of this request. If you have further questions, feel free to ask."  
+
+#### Scenario: Arranging Special Meals on Flights
+Customer: "I need to request a vegetarian meal for my flight."  
+Bot: "I can help with that. Please provide your flight booking reference number so I can access your reservation."  
+
+After receiving the reference:
+Bot: "Thank you. I’ve added a vegetarian meal request to your booking. Please note that the airline will do its best to accommodate your meal preference. You will receive a confirmation shortly. Is there anything else I can assist you with?"  
+
+### Emergency Assistance and Support
+Travel emergencies, such as lost passports, medical issues, or natural disasters, require immediate support. The WanderWise chatbot is trained to handle these critical situations and provide guidance on what steps to take.
+
+#### Scenario: Lost Passport During Travel
+Customer: "I’ve lost my passport while traveling. What should I do?"  
+Bot: "I'm sorry to hear that. Losing a passport can be stressful, but I'm here to guide you. First, contact your country's nearest embassy or consulate in the city you are currently in. They will provide you with the necessary steps to report the loss and apply for a replacement. I can help you find the contact details of the nearest embassy. Would you like me to do that?"  
+
+If the customer agrees:
+Bot: "I’ve found the contact details for your country's embassy in Paris. The address is 123 Embassy Street, and their phone number is +33-123-456-789. Please call them as soon as possible for further assistance. If you need help finding emergency accommodation or rebooking your flights, let me know."  
+
+#### Scenario: Medical Emergency During Travel
+Customer: "I need to find a doctor. I’m feeling unwell during my trip."  
+Bot: "I'm sorry to hear that you're not feeling well. I recommend visiting a nearby medical facility for proper care. I can help you locate the nearest hospital or clinic. Could you please provide your current location?"  
+
+After receiving the location:
+Bot: "I’ve found a clinic near you: Paris Health Clinic at 456 Medical Avenue. The contact number is +33-987-654-321. Please call them to schedule an appointment or visit the clinic directly. If you need further assistance, such as arranging transportation, feel free to ask."  
+
+### Automated Travel Reminders and Alerts
+WanderWise offers automated reminders and alerts for upcoming travel plans, check-ins, and important travel documentation, ensuring customers stay informed throughout their journey.
+
+#### Scenario: Sending Flight Check-In Reminder
+Bot: "Hello! Just a friendly reminder that check-in for your flight to Paris opens in 24 hours. Make sure to have your travel documents ready. You can complete the check-in process on the airline's website or app. Safe travels!"  
+
+#### Scenario: Travel Document Expiry Alert
+Bot: "Attention! Your passport is set to expire in 6 months. Some countries require your passport to be valid for at least 6 months beyond your travel dates. Please consider renewing your passport soon if you plan to travel internationally."  
+
+### Providing Post-Travel Support
+Post-travel support includes assistance with trip feedback, processing claims for lost baggage, and providing receipts for expense reports.
+
+#### Scenario: Submitting a Baggage Claim
+Customer: "My luggage was lost during my flight. How can I file a claim?"  
+Bot: "I'm sorry to hear about your lost luggage. To file a claim, you will need to contact the airline directly. I can help you with the following steps: 
+
+1. Visit the airline's website and navigate to the 'Baggage Claim' section.
+2. Provide your flight booking reference number and baggage tag number.
+3. Complete the claim form with details about your luggage and travel itinerary.
+4. Submit the claim and keep a copy of the confirmation for your records.
+
+Would you like me to provide you with the direct link to the airline's baggage claim page?"  
+
+If the customer agrees:
+Bot: "Here is the direct link to the airline's baggage claim page: [Airline Baggage Claim Page]. Please let me know if you need further assistance."  
+
+#### Scenario: Requesting a Travel Receipt for Expense Reports
+Customer: "I need a receipt for my hotel stay for my expense report."  
+Bot: "I can assist you with that. Please provide your booking reference number or the hotel name and stay dates."  
+
+After receiving the details:
+Bot: "Thank you. I have located your booking. A receipt for your stay at the Paris Stay Hotel has been generated and sent to your registered email address. If you need additional documentation, please let me know."  
+
+### Personalized Travel Recommendations
+WanderWise offers personalized travel recommendations based on customer preferences, past travel history, and destination highlights, enhancing the travel planning experience.
+
+#### Scenario: Recommending Activities for a Destination
+Customer: "What are some must-see places in Paris?"  
+Bot: "Paris is full of iconic attractions! Here are some must-see places you can visit:
+
+1. The Eiffel Tower: Enjoy breathtaking views of the city from this world-famous landmark.
+2. The Louvre Museum: Explore art masterpieces, including the Mona Lisa.
+3. Notre-Dame Cathedral: Visit this stunning Gothic cathedral with rich history.
+4. Montmartre: Wander through this charming neighborhood known for its artistic heritage.
+5. Seine River Cruise: Take a relaxing boat tour along the Seine River for beautiful cityscapes.
+
+Would you like me to help you book tickets or guided tours for any of these attractions?"  
+
+#### Scenario: Suggesting Seasonal Travel Deals
+Bot: "Hi! Based on your previous travel history, we have some amazing travel deals for the upcoming holiday season. How about a cozy Christmas getaway to the Swiss Alps? We have a special package that includes flights, accommodation, and a scenic train tour through the mountains. Would you like to know more?"  
+
+### Incorporating Multilingual Support
+WanderWise is expanding its capabilities to offer support in multiple languages, catering to a global customer base and providing a more inclusive travel planning experience.
+
+#### Scenario: Switching Language Preference
+Customer: "Can we continue this conversation in Spanish?"  
+Bot: "¡Claro! Podemos continuar en español. Por favor, hágame saber en qué puedo ayudarle."  
+
+### Building Customer Trust with Proactive Communication
+WanderWise ensures proactive communication to build trust with customers by keeping them informed about policy changes, upcoming offers, and exclusive discounts for loyal members.
+
+#### Scenario: Announcing Policy Changes
+Bot: "Dear valued customer, we would like to inform you of a recent update to our hotel cancellation policy. Effective immediately, free cancellations are now available up to 24 hours before check-in. For more details, please visit our website or contact customer support."  
+
+#### Scenario: Offering Exclusive Member Discounts
+Bot: "Hello, [Customer Name]! As a loyal member of WanderWise, you’re eligible for an exclusive 15% discount on your next booking! Use the promo code WANDER15 at checkout. This offer is valid until the end of the month. Happy travels!"  
+
+### Future Roadmap for WanderWise Chatbot Enhancements
+WanderWise aims to continually enhance the chatbot experience by introducing new features, such as AI-powered travel budgeting tools, real-time currency exchange information, and integration with smart home devices for seamless trip planning.
+
+#### AI-Powered Travel Budgeting Tools
+The chatbot will include a travel budgeting tool that helps customers plan their trip expenses. It will allow users to input their travel preferences, and the chatbot will generate a detailed budget estimate, including accommodation, meals, transportation, and activities.
+
+#### Real-Time Currency Exchange Information
+To assist travelers with international trips, the chatbot will offer real-time currency exchange rates and help customers calculate expenses in different currencies. This feature will provide peace of mind for budgeting and managing travel finances.
+
+#### Integration with Smart Home Devices
+In the future, WanderWise plans to integrate the chatbot with smart home devices like Amazon Alexa and Google Home, allowing customers to make travel inquiries and bookings using voice commands. This will make travel planning more convenient and accessible.
+
+### Conclusion
+WanderWise's intelligent, customer-focused chatbot is transforming the travel industry by providing personalized, real-time support for every stage of the journey. Whether customers are booking a flight, modifying a reservation, or seeking assistance during travel disruptions, WanderWise is equipped to handle a wide range of scenarios with efficiency and empathy. As we continue to innovate and enhance our services, our goal is to make every travel experience seamless, enjoyable, and unforgettable.
+
+
+WanderWise - Advanced Travel Assistance Features and Scenarios
+
+### Enhancing Customer Interactions for a Seamless Travel Experience
+The WanderWise chatbot is designed to handle a multitude of complex travel scenarios, addressing customer needs with personalized support and proactive solutions. Its capabilities span across flight bookings, hotel reservations, travel insurance, loyalty programs, and much more, ensuring travelers have a hassle-free journey.
+
+### Advanced Flight Management and Assistance
+Handling flights goes beyond simple bookings; customers often need support with managing upgrades, cancellations, baggage, and changes due to unforeseen circumstances. WanderWise provides an intelligent solution to these problems.
+
+#### Scenario: Booking a Flight with Baggage Allowance Preferences
+Customer: "I need to book a flight from Los Angeles to Tokyo, but I want to make sure I have enough baggage allowance for my trip."  
+Bot: "I can help you with that! Most airlines offer different baggage options based on your ticket class. For your flight to Tokyo, would you prefer economy, premium economy, business, or first-class?"  
+
+If the customer selects economy:
+Bot: "For economy class, the standard baggage allowance is one checked bag up to 23 kg (50 lbs). If you need additional baggage, I can arrange it for you at an extra fee. Would you like to add an extra checked bag?"  
+
+If the customer agrees to add an extra bag:
+Bot: "An additional checked bag will cost $75. I have added it to your booking. Your total baggage allowance is now two bags up to 23 kg each. Would you like to proceed with the flight booking?"  
+
+After booking:
+Bot: "Your flight from Los Angeles to Tokyo has been successfully booked with the additional baggage allowance. You will receive a confirmation email with your itinerary and baggage details. Safe travels!"  
+
+#### Scenario: Handling Flight Rescheduling Due to Weather Conditions
+Customer: "I received a notification that my flight has been delayed due to bad weather. Can I reschedule it for tomorrow?"  
+Bot: "I understand that travel disruptions can be stressful. Let me check the availability of flights for tomorrow."  
+
+After checking:
+Bot: "There is a flight available tomorrow at 10:00 AM. Rescheduling your ticket is free of charge due to the weather delay. Would you like me to book this flight for you?"  
+
+If the customer agrees:
+Bot: "Your flight has been successfully rescheduled to tomorrow at 10:00 AM. A new e-ticket has been sent to your registered email. Please let me know if you need further assistance."  
+
+### Hotel Reservation Modifications and Additional Services
+WanderWise provides extensive support for hotel bookings, including modifications, special requests, and exploring additional amenities.
+
+#### Scenario: Requesting an Early Check-In at the Hotel
+Customer: "Can I request an early check-in for my hotel reservation?"  
+Bot: "I can certainly help with that. Early check-in is subject to availability and the hotel's policy. Could you please provide your booking reference number?"  
+
+After receiving the reference:
+Bot: "I’ve contacted the hotel regarding your request for early check-in. They have agreed to allow you to check in at 10:00 AM for an additional fee of $30. Would you like to confirm this arrangement?"  
+
+If the customer confirms:
+Bot: "Your early check-in has been successfully arranged. The hotel will be ready to welcome you at 10:00 AM. You will receive an updated confirmation email shortly. Is there anything else you need help with?"  
+
+#### Scenario: Adding a Spa Package to a Hotel Booking
+Customer: "I’d like to add a spa package to my hotel stay. Can you help with that?"  
+Bot: "Certainly! The hotel offers several spa packages, including a Relaxation Package, a Couples Retreat, and a Wellness Package. Which one would you like to add to your booking?"  
+
+If the customer selects a package:
+Bot: "The [Selected Package] has been added to your booking. The total cost is $150, which will be included in your final bill at checkout. Enjoy your relaxation time at the spa! Would you like assistance with anything else?"  
+
+### Comprehensive Travel Package Management
+WanderWise not only supports individual bookings but also offers full holiday packages that include flights, hotels, guided tours, and travel insurance.
+
+#### Scenario: Booking an All-Inclusive Holiday Package
+Customer: "I’m interested in an all-inclusive holiday package to Bali."  
+Bot: "Great choice! We offer several all-inclusive packages to Bali that include flights, accommodation, meals, and guided tours. Would you like a 5-day, 7-day, or 10-day package?"  
+
+If the customer selects a 7-day package:
+Bot: "The 7-day package includes round-trip flights, a stay at a 4-star beachfront resort, daily breakfast, and guided tours of Bali's top attractions, including Ubud, Kintamani, and the Sacred Monkey Forest. The total cost is $1,800 per person. Would you like to proceed with the booking?"  
+
+If the customer proceeds:
+Bot: "Your all-inclusive holiday package to Bali has been successfully booked! You will receive a detailed itinerary and booking confirmation via email. If you need to add travel insurance or have any special requests, just let me know!"  
+
+### Future Roadmap for Further Enhancements in WanderWise Chatbot Services
+WanderWise continues to innovate its services to meet the evolving needs of modern travelers. Upcoming features include:
+
+1. **Visa Processing Assistance**: The chatbot will guide customers through the visa application process, including document requirements, embassy contacts, and online application forms.
+2. **Dynamic Trip Itineraries**: Personalized travel itineraries based on customer preferences, including dining reservations, activity bookings, and local transportation arrangements.
+3. **Sustainability and Eco-Friendly Travel Options**: Providing customers with eco-friendly travel choices, such as hotels with green certifications, carbon offset programs for flights, and electric vehicle rentals.
+4. **Multi-Language Support Expansion**: While currently offering support in major languages, WanderWise aims to expand its language support to cater to more diverse travelers, including localized content and region-specific travel guidelines.
+5. **Real-Time Customer Support Handoff**: Integrating a seamless handoff to live customer support agents during complex inquiries or disputes that require human intervention, ensuring that customers receive the best possible assistance regardless of the situation.
+6. **AI-Powered Travel Budgeting Tools**: The chatbot will include a travel budgeting tool that helps customers plan their trip expenses. It will allow users to input their travel preferences, and the chatbot will generate a detailed budget estimate, including accommodation, meals, transportation, and activities.
+7. **Real-Time Currency Exchange Information**: To assist travelers with international trips, the chatbot will offer real-time currency exchange rates and help customers calculate expenses in different currencies. This feature will provide peace of mind for budgeting and managing travel finances.
+8. **Integration with Smart Home Devices**: In the future, WanderWise plans to integrate the chatbot with smart home devices like Amazon Alexa and Google Home, allowing customers to make travel inquiries and bookings using voice commands. This will make travel planning more convenient and accessible.
+9. **Automated Personalized Promotions**: Using customer travel history and preferences, WanderWise will generate personalized promotional offers and discounts, increasing customer engagement and brand loyalty.
+10. **Advanced Sentiment Analysis and Feedback Collection**: Implementing AI-driven sentiment analysis to gauge customer satisfaction during interactions, enabling the chatbot to adjust its tone and offer immediate solutions or feedback requests.
+
+### Conclusion
+The WanderWise chatbot represents a cutting-edge solution in the travel industry, offering comprehensive support across various aspects of travel, from bookings and modifications to real-time problem-solving and post-travel assistance. By continually enhancing its features and adopting new technologies, WanderWise ensures a seamless and enjoyable travel experience for all customers. Whether managing corporate travel, handling emergencies, or simply planning a leisure trip, WanderWise is equipped to be the ultimate travel companion.
+
+
+### Tailored Travel Assistance for Every Customer Profile
+WanderWise is designed to cater to different customer profiles, providing a unique and personalized travel planning experience for each traveler. Whether a solo traveler, family, corporate executive, or senior citizen, the chatbot offers tailored recommendations and support.
+
+#### Scenario: Planning a Solo Adventure
+Customer: "I'm planning a solo backpacking trip across Europe. Can you help me with the itinerary?"  
+Bot: "Absolutely! Backpacking across Europe is an amazing adventure. I can help you plan an itinerary that includes popular destinations like Paris, Amsterdam, Berlin, Prague, and Rome. Would you like to travel by train, bus, or flights between these cities?"  
+
+If the customer chooses train travel:
+Bot: "Traveling by train offers a scenic and flexible experience. I suggest purchasing a Eurail Pass, which allows unlimited train travel across multiple countries in Europe. I can help you book accommodations in hostels and find budget-friendly activities in each city. Does that sound good?"  
+
+#### Scenario: Planning a Family Vacation with Kids
+Customer: "I want to plan a family vacation with my kids to Disneyland in California."  
+Bot: "A family trip to Disneyland is always magical! I recommend booking a hotel near the park to make travel easier for you and the kids. I can also arrange park tickets in advance to save time. Would you like me to include meal plans and character dining experiences in your package?"  
+
+If the customer agrees:
+Bot: "I’ve created a package that includes a 3-night stay at the Disneyland Hotel, 3-day park tickets, and a character breakfast experience with Mickey and friends. The total cost is $2,500 for a family of four. Would you like to proceed with the booking?"  
+
+#### Scenario: Planning a Business Trip with Corporate Preferences
+Corporate Customer: "I need to book a business trip to Singapore, including a stay at a business-friendly hotel."  
+Bot: "I can help you with that. For business trips, I recommend booking a hotel with conference facilities and a central location. How many nights will you be staying in Singapore, and do you need meeting room reservations?"  
+
+After receiving the details:
+Bot: "I’ve found a room at the Marina Bay Sands Hotel, which offers conference facilities and is located near major business districts. I’ve also reserved a meeting room for your convenience. The total cost is $1,800, and the corporate discount has been applied. Would you like to confirm this booking?"  
+
+### Adding Leisure and Entertainment Services
+WanderWise enhances the travel experience by offering leisure and entertainment options, including ticket bookings for events, guided tours, and dinner reservations at popular restaurants.
+
+#### Scenario: Booking a Guided Tour for a Historic Site
+Customer: "I want to book a guided tour of the Colosseum in Rome."  
+Bot: "The Colosseum is a must-see when in Rome! I can book a guided tour that includes skip-the-line entry, a professional guide, and access to the Roman Forum. The tour lasts 3 hours and costs $60 per person. Would you like to proceed with the booking?"  
+
+If the customer confirms:
+Bot: "Your guided tour of the Colosseum has been successfully booked! A confirmation with tour details and meeting point information has been sent to your email. Enjoy exploring ancient Rome!"  
+
+#### Scenario: Making Dinner Reservations at a Popular Restaurant
+Customer: "Can you make a dinner reservation for two at a rooftop restaurant in Paris?"  
+Bot: "I’d be happy to help! How about Le Jules Verne, the renowned rooftop restaurant located on the Eiffel Tower? They offer an exquisite dining experience with stunning views of Paris. Would you like to make a reservation there?"  
+
+If the customer agrees:
+Bot: "Your dinner reservation for two at Le Jules Verne has been made for 7:30 PM. A confirmation email with the reservation details has been sent to you. Enjoy your evening in Paris!"  
+
+### Managing and Optimizing Travel Budget
+WanderWise provides valuable tools to help customers plan and manage their travel budget effectively, ensuring they can enjoy their trip without financial stress.
+
+#### Scenario: Creating a Travel Budget Plan
+Customer: "Can you help me create a budget plan for my trip to Thailand?"  
+Bot: "Of course! I can provide an estimated budget based on your travel preferences. Are you looking for budget-friendly options, mid-range accommodations, or luxury experiences?"  
+
+If the customer chooses mid-range:
+Bot: "A mid-range budget for a 7-day trip to Thailand includes:
+
+- Flights: $600
+- Accommodation: $700 (3-star hotels)
+- Meals: $200 (local restaurants)
+- Activities and Tours: $300
+- Transportation: $100
+
+The total estimated budget is $1,900. Would you like me to help you book flights and accommodations within this budget?"  
+
+If the customer agrees:
+Bot: "Great! I will search for the best deals on flights and hotels that fit within your budget. You will receive a detailed itinerary and booking options shortly. Please let me know if you need any adjustments."  
+
+#### Scenario: Providing Currency Exchange Information
+Customer: "How much is 500 Euros in Thai Baht?"  
+Bot: "As of the current exchange rate, 500 Euros is approximately 18,500 Thai Baht. Please note that exchange rates fluctuate, so it’s advisable to check the rate again before exchanging money. Would you like tips on where to get the best exchange rates in Thailand?"  
+
+### Supporting Travelers with Special Needs and Accessibility
+WanderWise is committed to providing support for travelers with special needs, ensuring a comfortable and inclusive travel experience.
+
+#### Scenario: Requesting Accessible Hotel Rooms
+Customer: "I need an accessible hotel room in New York with a roll-in shower."  
+Bot: "I can help with that. There are several accessible hotels in New York that offer rooms with roll-in showers. Would you like me to find one near a specific location, like Times Square or Central Park?"  
+
+If the customer chooses a location:
+Bot: "I’ve found an accessible room at the Hilton Times Square. The room includes a roll-in shower and other accessibility features, such as grab bars and wide doorways. The cost is $200 per night. Would you like to proceed with the booking?"  
+
+If the customer agrees:
+Bot: "Your accessible room at the Hilton Times Square has been booked. A confirmation email with room details has been sent to you. If you have any other accessibility requests, feel free to let me know!"  
+
+#### Scenario: Arranging Airport Assistance for Elderly Travelers
+Customer: "My grandmother is traveling alone. Can you arrange airport assistance for her?"  
+Bot: "Certainly! I can arrange airport assistance to help your grandmother through check-in, security, and boarding. Please provide her flight booking reference number."  
+
+After receiving the reference:
+Bot: "Thank you. I have added an airport assistance request to her booking. An airport representative will be ready to assist her upon arrival. You will receive a confirmation email shortly. If there’s anything else I can do to help, just let me know."  
+
+### Proactive Alerts and Travel Updates
+WanderWise keeps travelers informed about important travel updates, such as weather alerts, flight changes, and travel advisories.
+
+#### Scenario: Providing a Weather Alert for the Destination
+Bot: "Hi [Customer Name], we have an important weather update for your upcoming trip to Tokyo. Heavy rainfall is expected during your stay. Please pack appropriate clothing and plan indoor activities as needed. Would you like recommendations for indoor attractions in Tokyo?"  
+
+If the customer agrees:
+Bot: "Sure! Here are some great indoor attractions in Tokyo:
+
+1. Tokyo National Museum
+2. TeamLab Borderless Digital Art Museum
+3. Tokyo Skytree and Shopping Mall
+
+I can help you book tickets for any of these attractions. Just let me know!"  
+
+#### Scenario: Notifying About Travel Advisories
+Bot: "Attention, [Customer Name]. There is a new travel advisory for your destination, Bali, due to volcanic activity. Flights to and from the island may be delayed or canceled. Please check with your airline for the latest updates. Would you like assistance in exploring alternative travel arrangements?"  
+
+If the customer asks for alternatives:
+Bot: "I’ve checked the available options. You can either reschedule your flight for a later date or change your destination. Would you like me to assist you in making these changes?"  
+
+### Conclusion
+WanderWise continues to revolutionize the travel industry by providing a comprehensive, user-friendly chatbot experience tailored to individual traveler needs. With features ranging from booking and budget management to accessibility support and real-time travel updates, WanderWise ensures every journey is meticulously planned and hassle-free. The chatbot's upcoming features will further enhance the travel experience, making WanderWise the ultimate travel companion for both leisure and business travelers alike.
+

genai_voice/__init__.py

@@ -0,0 +1,25 @@
+"""Generative AI For Voice"""
+
+from genai_voice.config.defaults import Config
+from genai_voice.logger.log_utils import log, LogLevels
+from genai_voice.models.open_ai import CustomOpenAIModel
+from genai_voice.models.model_config import ModelGenerationConfig
+from genai_voice.defintions.model_response_formats import ModelResponseFormat
+from genai_voice.defintions.prompts import BAD_PROMPT, GOOD_PROMPT, FINANCIAL_PROMPT
+from genai_voice.bots.chatbot import ChatBot
+from genai_voice.data_utils import extract_web_data
+from genai_voice.processing.audio import Audio
+
+__all__ = [
+    "ChatBot",
+    "Config",
+    "extract_web_data",
+    "log",
+    "LogLevels",
+    "CustomOpenAIModel",
+    "ModelGenerationConfig",
+    "ModelResponseFormat",
+    "BAD_PROMPT",
+    "GOOD_PROMPT",
+    "FINANCIAL_PROMPT",
+]

genai_voice/bots/chatbot.py

@@ -0,0 +1,185 @@
+"""Customizable Chatbot Main Functions"""
+
+import os
+import threading
+from typing import Optional, Any
+
+from dotenv import load_dotenv
+from genai_voice.processing.audio import Audio
+from genai_voice.models.open_ai import CustomOpenAIModel
+from genai_voice.config.defaults import Config
+from genai_voice.logger.log_utils import log, LogLevels
+
+from genai_voice.defintions.prompts import (
+    TRAVEL_AGENT_PROMPT,
+    PROMPTS_TO_CONTEXT_DATA_FILE,
+)
+
+# load environment variables from .env file
+load_dotenv(override=True)
+
+
+class ChatBot:
+    """ChatBot Class"""
+
+    def __init__(
+        self,
+        prompt: Optional[str] = None,
+        context_file_path: Optional[str] = None,
+        model_name: str = Config.MODEL_GPT_TURBO_NAME,
+        mic_id: Any = None,
+        enable_speakers: bool = False,
+        threaded: bool = False,
+    ) -> None:
+        """
+        Initialize the chatbot
+        mic_id:          The index of the mic to enable
+        enable_speakers: Whether or not audio will be played
+        threaded:        Plays back audio in seperate thread, can interfere with speech detector
+        """
+        if not prompt:
+            prompt = TRAVEL_AGENT_PROMPT
+            context_file_path = PROMPTS_TO_CONTEXT_DATA_FILE[TRAVEL_AGENT_PROMPT]
+
+        log(f"Context file: {context_file_path}", log_level=LogLevels.ON)
+
+        # Ensure our context file exists
+        self.context_file_path = os.path.join("data", context_file_path)
+        if not os.path.exists(self.context_file_path):
+            raise ValueError(
+                f"Provided context file path does not exist: {self.context_file_path}"
+            )
+        self.model_name = model_name
+        
+        if self.model_name == Config.MODEL_GPT_TURBO_NAME:
+            self.__client = CustomOpenAIModel(api_key=Config.OPENAI_API_KEY)
+        else:
+            raise ValueError(f"Model {self.model_name} is not currently supported.")
+
+        # Whether or not to use speakers
+        self.__enable_speakers = enable_speakers
+
+        # Whether or not to thread playback
+        self.__threaded = threaded
+
+        # Initialize audio library
+        self.audio = Audio()
+
+        # Initialize mic
+        if mic_id is not None:
+            self.initialize_microphone(mic_id)
+
+        # Get data for LLM Context
+        self.context = self.get_context_data()
+
+        # Get initial prompt
+        self.prompt = prompt
+
+        # Prompt template to initialize LLM
+        self.llm_prompt = self.__client.build_prompt(
+            prompt=self.prompt, context=self.context
+        )
+
+    def get_completion_from_messages(self, messages):
+        """
+        Send the message to the specified OpenAI model
+        """
+        # use default config for model
+        return self.__client.generate(messages=messages, config=None)
+
+    def get_context_data(self) -> str:
+        """Get the data for the LLM"""
+        with open(self.context_file_path, "r", encoding="utf-8") as f:
+            data = "".join(line for line in f)
+        return data
+
+    def respond(self, prompt, llm_history: list = None):
+        """
+        Get a response based on the current history
+        """
+        if not llm_history:
+            log("Empty history. Creating a state list to track histories.")
+            llm_history = []
+        context = [self.llm_prompt]
+        for interaction in llm_history:
+            context.append({"role": "user", "content": f"{interaction[0]}"})
+            context.append({"role": "assistant", "content": f"{interaction[1]}"})
+
+        context.append({"role": "user", "content": f"{prompt}"})
+        llm_response = self.get_completion_from_messages(context)
+
+        if self.__enable_speakers:
+            # With threads
+            if self.__threaded:
+                speaker_thread = threading.Thread(
+                    target=self.audio.communicate, args=(llm_response,)
+                )
+                speaker_thread.start()
+            # Without threads
+            else:
+                self.audio.communicate(llm_response)
+        return llm_response
+
+    def initialize_microphone(self, mic_id):
+        """
+        Initialize microphone object with the indicated ID.
+        For best results a headset with a mic is recommended.
+        """
+        self.audio.initialize_microphone(mic_id)
+
+    def recognize_speech_from_mic(self):
+        """
+        Listens for speech
+        return: The text of the captured speech
+        """
+        return self.audio.recognize_speech_from_mic()
+
+    def communicate(self, message):
+        """
+        Plays a message on the speakers
+        message: the message
+        """
+        self.audio.communicate(message)
+
+    def get_prompt_from_streamlit_audio(self, audio) -> str:
+        """Converts audio captured from streamit to text"""
+        if not audio:
+            return None
+        return self.audio.transcribe_from_transformer(audio)
+
+    def get_prompt_from_gradio_audio(self, audio):
+        """
+        Converts audio captured from gradio to text.
+        See https://www.gradio.app/guides/real-time-speech-recognition for more info.
+        audio: object containing sampling frequency and raw audio data
+        """
+        log(f"Getting prompt from audio device: {audio}")
+        if not audio:
+            return None
+        return self.audio.get_prompt_from_gradio_audio(audio)
+
+    def get_prompt_from_file(self, file):
+        """
+        Converts audio from a file to text.
+        file: the path to the audio file
+        """
+
+        return self.audio.get_prompt_from_file(file)
+
+
+if __name__ == "__main__":
+    # Enable speakers (adjust to True or False as needed)
+    ENABLE_SPEAKERS = True
+
+    # pylint: disable=invalid-name
+    human_prompt = ""
+    history = []
+
+    # Create the chatbot
+    chatbot = ChatBot(enable_speakers=ENABLE_SPEAKERS)
+
+    # Main loop
+    while human_prompt != "goodbye":
+        response = chatbot.respond(human_prompt, history)
+        history.append([human_prompt, response])
+        human_prompt = input(f"\n{response}\n\n")

genai_voice/config/defaults.py

@@ -0,0 +1,37 @@
+"""Config"""
+
+import os
+from dataclasses import dataclass
+
+from dotenv import load_dotenv
+
+
+load_dotenv(override=True)
+
+
+@dataclass
+class Config:
+    """LLM Config"""
+
+    MODEL_GPT_TURBO_NAME = "gpt-4-turbo"
+    OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
+    TEMPERATURE = 0.0
+    TOP_P = 0.97
+    TOP_K = 40
+    MAX_OUTPUT_TOKENS = 2048
+    WEB_SCRAPER_OUTPUT_FILE = "data/context.txt"
+
+    def __repr__(self):
+        return f"""
+        Default(MODEL_GPT_TURBO_NAME='{self.MODEL_GPT_TURBO_NAME}', 
+        OPENAI_API_KEY='{self.OPENAI_API_KEY}', 
+        TEMPERATURE={self.TEMPERATURE}, 
+        TOP_P={self.TOP_P}, 
+        TOP_K={self.TOP_K}, 
+        MAX_OUTPUT_TOKENS={self.MAX_OUTPUT_TOKENS}, 
+        WEB_SCRAP_OUTPUT_DIR={self.WEB_SCRAPER_OUTPUT_FILE})"""
+
+
+if __name__ == "__main__":
+    default_config = Config()
+    print(default_config)