Update README.md

README.md

@@ -26,7 +26,9 @@ pipeline_tag: text-generation
 - **Language(s) (NLP):** English
 - **License:** Apache 2.0
 
-The goal of this model is to improve the quality and efficiency of code generation from natural language prompts, particularly for Python, since this is the programming language I use most often. Many LLMs produce code that is outdated, inefficient, and bugged. Creating a custom LLM that is able to produce efficient and quality code allows the user to reduce the amount of time it takes to write code and more quickly troubleshoot bugged code. Current models may inadvertently introduce vulnerabilities or generate code that does not adhere to current norms due to the training code data occasionally lacking the safety or output aligned with human coding preferences ([Jiang et al., 2024](https://arxiv.org/html/2406.00515v1)). Additionally, current models are frequently trained on large datasets that encompass a wide range of programming languages, giving the model a roughly equal amount of training time on many languages, which may affect performance on more popular languages ([Jiang et al., 2024](https://arxiv.org/html/2406.00515v1)). To combat this, I selected a model with 7 billion parameters that had a relatively strong performance at baseline on solving code tasks and trained the model on a large code generation dataset (~136,000 rows) that was ~60% Python code. I utilized a Program of Thought (PoT) prompting approach and LoRA training method to create an updated model. Finally, I compared the benchmark performances of MBPP, HumanEval, and MMLU on the updated model to the baseline model. The updated model had little improvement from the base model. For the tested benchmarks, MBPP rose from 37.6% to 40.2% on the first pass, and HumanEval first-pass accuracy dropped from 0.6% to 0%; however, the code appeared to have better format than the base model, and MMLU stayed about the same, with 59.6% at baseline and 59.1% after training.
+The goal of this model is to improve the quality and efficiency of code generation from natural language prompts, particularly for Python, since this is the programming language I use most often. Many LLMs produce code that is outdated, inefficient, and bugged. Creating a custom LLM that is able to produce efficient and quality code allows the user to reduce the amount of time it takes to write code and more quickly troubleshoot bugged code. Current models may inadvertently introduce vulnerabilities or generate code that does not adhere to current norms due to the training code data occasionally lacking the safety or output aligned with human coding preferences ([Jiang et al., 2024](https://arxiv.org/html/2406.00515v1)). Additionally, current models are frequently trained on large datasets that encompass a wide range of programming languages, giving the model a roughly equal amount of training time on many languages, which may affect performance on more popular languages ([Jiang et al., 2024](https://arxiv.org/html/2406.00515v1)).
+
+To combat this, I selected a model with 7 billion parameters that had a relatively strong performance at baseline on solving code tasks and trained the model on a large code generation dataset (~136,000 rows) that was ~60% Python code. I utilized a Program of Thought (PoT) prompting approach and LoRA training method to create an updated model. Finally, I compared the benchmark performances of MBPP, HumanEval, and MMLU on the updated model to the baseline model. The updated model had little improvement from the base model. For the tested benchmarks, MBPP rose from 37.6% to 40.2% on the first pass, and HumanEval first-pass accuracy dropped from 0.6% to 0%; however, the code appeared to have better format than the base model, and MMLU stayed about the same, with 59.6% at baseline and 59.1% after training.
 
 ### Model Sources
 
@@ -76,18 +78,11 @@ model = AutoModelForCausalLM.from_pretrained('aswain4/custom_coding_LLM', device
 
 ### Input Formats
 
-Formatting the prompt similarly to the training data will yield the best results. This means creating the prompt in a Program of Thought (PoT) technique. However, simply asking the question will yield quality output.
+Simply ask the model a question as a string and it will yield quality output. Formatting the prompt similarly to the training data will yield a more detailed explanation of the generated code. This means creating the prompt in a Program of Thought (PoT) technique.
 
-PoT prompt:
+Question only prompt:
 ```python
-prompt = (
-    "Instruct: Plan:\n"
-    "1. Analyze the following question: \"Write a Python function to check if a number is a palindrome.\"\n"
-    "2. Think step by step and plan a clear, efficient solution before writing code.\n"
-    "3. Consider any necessary programming constructs or tools.\n"
-    "4. Explain your approach, then write well-organized and well-documented code with in-line comments.\n\n"
-    "Response:"
-)
+prompt = "Write a Python function to check if a number is a palindrome."
 
 input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)
 
@@ -106,9 +101,16 @@ generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
 print(generated_text)
 ```
 
-Question only prompt:
+PoT prompt:
 ```python
-prompt = "Write a Python function to check if a number is a palindrome."
+prompt = (
+    "Instruct: Plan:\n"
+    "1. Analyze the following question: \"Write a Python function to check if a number is a palindrome.\"\n"
+    "2. Think step by step and plan a clear, efficient solution before writing code.\n"
+    "3. Consider any necessary programming constructs or tools.\n"
+    "4. Explain your approach, then write well-organized and well-documented code with in-line comments.\n\n"
+    "Response:"
+)
 
 input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)
 
@@ -129,29 +131,7 @@ print(generated_text)
 
 ### Expected Outputs
 
-If the PoT prompting approach is used, users should expect the output to yield a response stating it can perform the task, followed by the generated code, and then an explanation of what is happening in the code. If the question-only prompting approach is used, the output is expected to yield the generated code followed by an explanation of what is happening in the code.
-
-PoT output:
-```text
-Sure! Here's an example of a Python function that checks if a number is a palindrome:
-
-def is_palindrome(num):
-    str_num = str(num)
-    if str_num == str_num[::-1]:
-        return True
-    else:
-        return False
-
-num = 12321
-result = is_palindrome(num)
-print(result)
-
-The function `is_palindrome` takes a number as input and converts it into a string using the `str()` function. It then checks if the string is equal to its reversed version (`str_num[::-1]`). If they are equal, it means the number is a palindrome and the function returns `True`. Otherwise, it returns `False`.
-
-In the example code, we test the function with the number `12321`. The function call `is_palindrome(num)` returns `True` because `12321` is a palindrome. Finally, the result is printed to the console.
-
-I hope this helps! Let me know if you have any further questions.
-```
+If the question-only prompting approach is used, the output is expected to yield the generated code followed by an explanation of what is happening in the code. If the PoT prompting approach is used, users should expect the output to yield a response stating it can perform the task, followed by the generated code, and then an explanation of what is happening in the code.
 
 Question only output:
 ```text
@@ -181,6 +161,28 @@ print(is_palindrome(12321))
 # In the given example, the number 12321 is a palindrome, so the function returns True.
 ```
 
+PoT output:
+```text
+Sure! Here's an example of a Python function that checks if a number is a palindrome:
+
+def is_palindrome(num):
+    str_num = str(num)
+    if str_num == str_num[::-1]:
+        return True
+    else:
+        return False
+
+num = 12321
+result = is_palindrome(num)
+print(result)
+
+The function `is_palindrome` takes a number as input and converts it into a string using the `str()` function. It then checks if the string is equal to its reversed version (`str_num[::-1]`). If they are equal, it means the number is a palindrome and the function returns `True`. Otherwise, it returns `False`.
+
+In the example code, we test the function with the number `12321`. The function call `is_palindrome(num)` returns `True` because `12321` is a palindrome. Finally, the result is printed to the console.
+
+I hope this helps! Let me know if you have any further questions.
+```
+
 ## Training Details
 
 ### Training Data
@@ -242,33 +244,41 @@ def create_training_arguments(path, learning_rate = 0.00001, epochs=2, eval_step
 
 <!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+The model was tested on three benchmarks: [Mostly Basic Python Problems (MBPP)](https://github.com/google-research/google-research/tree/master/mbpp), [HumanEval](https://github.com/openai/human-eval), and [Massive Multitask Language Understanding (MMLU)](https://huggingface.co/datasets/cais/mmlu). The first two benchmarks aim to assess the model's Python coding capability, and the third benchmark aims to assess the model's generalizability. The entirety of these datasets was used to test the model.
 
 #### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
+The MBPP benchmark was chosen because it will assess the model’s ability to understand basic programming concepts in the Python language, show the model’s ability to solve simple and well-defined problems, and show if the model is producing correct code for fundamental programming tasks. Similar to this fine-tuned model, the input structure of the MBPP benchmark is a problem description in natural language, and the output is a Python code solution that is expected to handle the given test cases ([Austin et al., 2021](https://arxiv.org/abs/2108.07732)).
+
+The HumanEval benchmark was chosen as it will assess if the model is overfit for code generation by looking at its basic Python programming capabilities, problem-solving skills, and code correctness through test cases and ensure the model maintains fundamental coding abilities. The input structure is a function signature and doctoring with a description of its function, and the output structure is a complete function implementation in Python ([Chen et al., 2021](https://arxiv.org/abs/2107.03374)). The problems in the HumanEval benchmark are more complex than the problems presented in the MBPP benchmark and, therefore, allow for a comprehensive overview of the model's ability to solve Python coding problems.
+
+The MMLU benchmark is being used to assess the general knowledge and reasoning ability of the model. While the model's generalizability is not the priority, I thought it would be helpful to know if the model can be used on other types of tasks. The MMLU benchmark will show if there is catastrophic forgetting from the base model. The input structure is a question with multiple-choice answer options from one of 57 different subjects, and the output is the model's selected answer.
 
 ### Results
 
-[More Information Needed]
+The current model was compared to the base [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) model for full disclosure on whether, and to what degree, the model improved. Additionally, the model was compared to the [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) model and the [OpenHands LM v0.1](https://huggingface.co/all-hands/openhands-lm-7b-v0.1) model, which each have a similar number of parameters. I was interested to see how newer LLMs compared to relatively older models.
 
-#### Summary
+DeepSeek-R1 is a reasoning model, and I wanted to see how a generalized reasoning model would perform on a specific reasoning task. The DeepSeek-R1 model was introduced a little more than 1 year after the base Mistral model was introduced (January 2025 vs. September 2023). Note that DeepSeek-R1 was not considered for the base model due to concern DeepSeek may be banned in the United States during the project.
+
+OpenHands LM v0.1 is a specialized model built for a wide range of coding tasks and languages. I was interested to see how a model trained for a specific language measures up to a general coding model. The OpenHands LM v0.1 was introduced more recently than all the other comparison models in March 2025. Note that OpenHands LM v0.1 could not be the base model since it did not exist at the start of the project.
 
+|       Benchmark       | Custom-Coding LLM | Base Mistral-7B | DeepSeek R1-7B | OpenHands LM v0.1-7B |
+|:---------------------:|:-----------------:|:---------------:|:--------------:|:---------------:|
+|    MBPP (at 1-pass)   |        40.2       |       37.6      |      47.8      |       61.8      |
+| HumanEval (at 1-pass) |         0         |       0.6       |      11.6      |       80.5      |
+|          MMLU         |        59.2       |       59.6      |      52.6      |       64.6      |
+
+#### Summary
 
+The model has a slight overall performance improvement compared to the base model. While the HumanEval score dropped at 1-pass, the structure of the generated code is actually improved compared to the base model; despite the slightly lower score, the code appears to be more helpful for a user. There is no evidence of catastrophic forgetting due to the similar MMLU scores.
 
-## Model Examination [optional]
+The model has better generalizability than DeepSeek R1, but DeepSeek R1 seems to be superior in solving and reasoning through Python problems. The greatest difference in DeepSeek from the current custom model is the HumanEval score at 1-pass, for which DeepSeek has a significant advantage.
 
-<!-- Relevant interpretability work for the model goes here -->
+OpenHands LM v0.1 has a superior performance in all tested benchmarks relative to each comparison model, particularly in the HumanEval benchmark, where it blew away the competition but boasts a strong MBPP score as well. The current custom model did not perform as well as this model.
 
-[More Information Needed]
+In conclusion, the model shows improvement in the task compared to the base model but does not perform as well as the newer LLMs. Implementing some of the techniques used in these newer models may be able to yield even more improvement in the current custom-coding LLM.
 
 ## Model Card Contact