EC2 Default User commited on
Commit
7e6a400
Β·
1 Parent(s): 2a9402e

First commit: Add STM32 Model Zoo

Browse files
Dockerfile ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM python:3.10-slim
2
+
3
+ RUN apt-get update && apt-get install -y \
4
+ build-essential \
5
+ libgl1 \
6
+ libglib2.0-0 \
7
+ wget \
8
+ git \
9
+ unzip \
10
+ && rm -rf /var/lib/apt/lists/*
11
+
12
+ RUN useradd -ms /bin/bash appuser
13
+ USER appuser
14
+
15
+ ENV HOME=/home/appuser \
16
+ PATH=/home/appuser/.local/bin:$PATH \
17
+ STATS_TYPE='HuggingFace_devcloud' \
18
+ PYTHONDONTWRITEBYTECODE=1 \
19
+ PYTHONUNBUFFERED=1
20
+
21
+ WORKDIR $HOME/app
22
+
23
+ RUN pip install --no-cache-dir --upgrade pip
24
+
25
+ COPY --chown=appuser . $HOME/app
26
+
27
+ #To use modelzoo training scripts clone modelzoo-services
28
+ RUN git clone https://github.com/STMicroelectronics/stm32ai-modelzoo-services.git
29
+
30
+ #To benchmark pre-trained models from stm32ai-modelzoo clone this repo
31
+ #RUN git clone https://github.com/STMicroelectronics/stm32ai-modelzoo.git
32
+
33
+ COPY --chown=appuser download_datasets.py $HOME/app/download_datasets.py
34
+
35
+ RUN pip install --no-cache-dir -r requirements_dash.txt
36
+ RUN pip install --no-cache-dir -r stm32ai-modelzoo-services/requirements.txt
37
+
38
+ #To upload your dataset
39
+ #COPY datasets/yourdataset.zip datasets/
40
+ #RUN unzip datasets/yourdataset.zip -d datasets/
41
+
42
+
43
+ EXPOSE 7860
44
+
45
+ CMD ["sh", "-c", "python download_datasets.py && python dash_app.py"]
LICENCE.md ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ # Copyright (c) 2017 STMicroelectronics
2
+
3
+ This software component is licensed by STMicroelectronics under the **BSD 3-Clause** license. You may not use this file except in compliance with this license. You may obtain a copy of the license [here](https://opensource.org/licenses/BSD-3-Clause).
README.md CHANGED
@@ -1,11 +1,10 @@
1
  ---
2
- title: Stm32 Modelzoo App
3
- emoji: 😻
4
- colorFrom: red
5
  colorTo: gray
6
  sdk: docker
7
  pinned: false
8
- license: bsd-3-clause
9
  ---
10
 
11
  Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
1
  ---
2
+ title: stm32 model zoo app
3
+ emoji: πŸš€
4
+ colorFrom: blue
5
  colorTo: gray
6
  sdk: docker
7
  pinned: false
 
8
  ---
9
 
10
  Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
assets/ST_logo_2024_white.png ADDED
assets/github-mark-white.png ADDED
assets/logs.jpg ADDED
assets/style.css ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ html, body {
2
+ font-family: 'Arial', sans-serif;
3
+ background-color: #fcfcfc;
4
+ color: #333;
5
+ margin: 0;
6
+ padding: 0;
7
+ }
8
+
9
+ .top-bar {
10
+ display: flex;
11
+ justify-content: space-between;
12
+ align-items: center;
13
+ width: 100%;
14
+ background-color: #03234b;
15
+ color: white;
16
+ padding: 7px 5px;
17
+ font-size: 18px;
18
+ font-weight: bold;
19
+ border-bottom: 2px solid #03234b;
20
+ box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1);
21
+ }
22
+
23
+ .credentials-section {
24
+ display: none;
25
+ background-color: #f8f9fa;
26
+ border: 1px solid #e0e0e0;
27
+ padding: 20px;
28
+ border-radius: 8px;
29
+ box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1);
30
+ }
31
+
32
+
33
+ .credentials-col {
34
+ padding: 10px;
35
+ }
36
+
37
+
38
+ .credentials-text {
39
+ font-size: 15px;
40
+ font-weight: bold;
41
+ color: #03234b;
42
+ }
43
+
44
+ .input-field {
45
+ display: block;
46
+ width: 100%;
47
+ padding: 8px;
48
+ margin-bottom: 10px;
49
+ font-size: 14px;
50
+ border: 1px solid #ced4da;
51
+ border-radius: 4px;
52
+ box-shadow: inset 0 1px 2px rgba(0, 0, 0, 0.1);
53
+ }
54
+
55
+
56
+ .input-field:focus {
57
+ border-color: #357ab7;
58
+ outline: none;
59
+ box-shadow: 0 0 5px rgba(53, 122, 183, 0.5);
60
+ }
61
+
62
+
63
+ .start-button {
64
+ display: block;
65
+ width: 100%;
66
+ background-color: #03234b;
67
+ color: #ffffff;
68
+ font-size: 14px;
69
+ padding: 10px;
70
+ border-radius: 4px;
71
+ border: none;
72
+ cursor: pointer;
73
+ transition: background-color 0.3s ease;
74
+ }
75
+
76
+ .start-button:hover {
77
+ background-color: #3cb4e6;
78
+ }
79
+
80
+ @keyframes blink{
81
+ 0% { opacity: 1; }
82
+ 50% { opacity: 0; }
83
+ 100% { opacity: 1; }
84
+ }
85
+
86
+ .blinking {
87
+ animation: blink 1s infinite;
88
+ }
dash_app.py ADDED
@@ -0,0 +1,1438 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /*---------------------------------------------------------------------------------------------
2
+ # * Copyright (c) 2024 STMicroelectronics.
3
+ # * All rights reserved.
4
+ # *
5
+ # * This software is licensed under terms that can be found in the LICENSE file in
6
+ # * the root directory of this software component.
7
+ # * If no LICENSE file comes with this software, it is provided AS-IS.
8
+ # *--------------------------------------------------------------------------------------------*/
9
+ import os
10
+ import re
11
+ import uuid
12
+ import time
13
+ import shutil
14
+ import zipfile
15
+ import threading
16
+ import subprocess
17
+ import select
18
+ from datetime import datetime
19
+ from concurrent.futures import ThreadPoolExecutor
20
+
21
+ import dash
22
+ from dash import dcc, html
23
+ import dash_daq as daq
24
+ from dash.dependencies import Input, Output, State, ALL
25
+ import dash_bootstrap_components as dbc
26
+ from dash.exceptions import PreventUpdate
27
+ import dash_daq as daq
28
+ from flask import Flask, render_template, request, send_file, jsonify, abort
29
+ import plotly.graph_objects as go
30
+ import plotly.colors as pc
31
+
32
+ import yaml
33
+ import ruamel.yaml
34
+ import pandas as pd
35
+
36
+ import logging
37
+ import base64
38
+
39
+ logging.basicConfig(level=logging.DEBUG)
40
+ logger = logging.getLogger(__name__)
41
+
42
+ server = Flask(__name__)
43
+ server.secret_key = os.urandom(24)
44
+
45
+
46
+ @server.route('/')
47
+ def welcome_page():
48
+ """
49
+ Handles the welcome page route.
50
+ This function extracts the username from the request host,
51
+ determines if the duplicate mode should be enabled, and renders
52
+ the welcome page template with the duplicate mode state.
53
+ Returns:
54
+ str: The rendered 'index.html' template with the duplicate_mode parameter.
55
+ """
56
+ host = request.host
57
+ print("host:", host)
58
+ usr_match = re.match(r'^(.*?)\-stm32', host)
59
+ print("usr_match:", usr_match)
60
+
61
+ if usr_match:
62
+ hf_user = usr_match.group(1)
63
+ else:
64
+ hf_user = "modelzoo_user"
65
+
66
+ if hf_user == "stmicroelectronics":
67
+ duplicate_mode = True
68
+ else:
69
+ duplicate_mode = False
70
+
71
+ print("hf_user:", hf_user)
72
+ print("duplicate_mode:", duplicate_mode)
73
+
74
+ return render_template('index.html', duplicate_mode=duplicate_mode)
75
+
76
+
77
+ external_stylesheets = [dbc.themes.LITERA]
78
+ app = dash.Dash(__name__, server=server,external_stylesheets=external_stylesheets, url_base_pathname='/dash_app/', suppress_callback_exceptions=True)
79
+
80
+ local_yamls = {
81
+ 'image_classification': 'stm32ai-modelzoo-services/image_classification/src/user_config.yaml',
82
+ 'human_activity_recognition': 'stm32ai-modelzoo-services/human_activity_recognition/src/user_config.yaml',
83
+ 'hand_posture': 'stm32ai-modelzoo-services/hand_posture/src/user_config.yaml',
84
+ 'object_detection': 'stm32ai-modelzoo-services/object_detection/src/user_config.yaml',
85
+ 'audio_event_detection': 'stm32ai-modelzoo-services/audio_event_detection/src/user_config.yaml',
86
+ 'pose_estimation': 'stm32ai-modelzoo-services/pose_estimation/src/user_config.yaml',
87
+ 'semantic_segmentation': 'stm32ai-modelzoo-services/semantic_segmentation/src/user_config.yaml'
88
+ }
89
+
90
+ def banner():
91
+ return html.Div(
92
+ id="banner",
93
+ className="top-bar",
94
+ style={
95
+ "display": "flex",
96
+ "align-items": "center",
97
+ "justify-content": "space-between",
98
+ "position": "fixed",
99
+ "top": "0",
100
+ "left": "0",
101
+ "width": "100%",
102
+ "z-index": "1000",
103
+ "background": "linear-gradient(to right, #03234b, #054080)",
104
+ "box-shadow": "0px 4px 8px rgba(0, 0, 0, 0.2)",
105
+ "border-radius": "0 0 10px 10px"
106
+ },
107
+ children=[
108
+ html.A(
109
+ id="learn-more-button",
110
+ children=[
111
+ html.Img(
112
+ src=app.get_asset_url("github-mark-white.png"),
113
+ style={"width": "22px", "height": "22px", "margin-right": "8px"}
114
+ ),
115
+ html.Span("stm32ai-modelzoo", style={"font-weight": "bold"})
116
+ ],
117
+ href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services",
118
+ target="_blank",
119
+ style={
120
+ "display": "flex",
121
+ "align-items": "center",
122
+ "color": "#ffffff",
123
+ "text-decoration": "none",
124
+ "font-size": "16px",
125
+ "font-family": "Arial, sans-serif",
126
+ "transition": "color 0.3s ease"
127
+ }
128
+ ),
129
+ html.Div(
130
+ [
131
+ dbc.Button(
132
+ html.Img(
133
+ src=app.get_asset_url("logs.jpg"),
134
+ style={"width": "22px", "height": "22px","margin-right":"10px"}
135
+ ),
136
+ id="toggle-log",
137
+ n_clicks=0,
138
+ className="",
139
+ style={
140
+ "background": "none",
141
+ "border": "none",
142
+ "padding": "0",
143
+ "margin-right": "12px",
144
+ "cursor": "pointer"
145
+ }
146
+ ),
147
+ html.A(
148
+ html.H5(
149
+ "ST Edge AI Developer Cloud",
150
+ style={
151
+ "margin": "0",
152
+ "color": "#ffffff",
153
+ "font-size": "16px",
154
+ "font-weight": "bold",
155
+ "font-family": "Arial, sans-serif",
156
+ "transition": "color 0.3s ease"
157
+ }
158
+ ),
159
+ href="https://stm32ai-cs.st.com/home",
160
+ target="_blank",
161
+ style={
162
+ "display": "flex",
163
+ "align-items": "center",
164
+ "text-decoration": "none"
165
+ }
166
+ )
167
+ ],
168
+ style={"display": "flex", "align-items": "center"}
169
+ )
170
+ ]
171
+ )
172
+
173
+ def create_dashboard_layout():
174
+ """
175
+ Creates the layout for the application: STM32ModelZoo dashboard.
176
+
177
+ This function defines the structure and components of the dashboard,
178
+ including the banner, model selection dropdown, YAML update options,
179
+ credentials input, output display, training metrics graphs, and download button.
180
+
181
+ Returns:
182
+ dbc.Container: A Dash Bootstrap Component container with the dashboard layout.
183
+ """
184
+ return html.Div([
185
+ banner(),
186
+ dbc.Container([
187
+ dcc.Location(id='url', refresh=False),
188
+ dbc.Row(dbc.Col(html.H3("STM32 Model zoo Dashboard", style={'color': '#03234b', 'text-align': 'center', "margin-top": "80px", "font-family": "Arial, sans-serif"}), className="mb-4")),
189
+ dbc.Row([
190
+ dbc.Col(
191
+ html.H5("Use case selection", style={'color': '#03234b', 'margin-bottom': '10px'}),
192
+ width=12
193
+ )
194
+ ], id="use-case-section", style={"display": "none"}),
195
+ dbc.Row(dbc.Col(dcc.Dropdown(
196
+ id='selected-model',
197
+ options=[
198
+ {'label': 'Image Classification (IC)', 'value': 'image_classification'},
199
+ {'label': 'Human Activity Recognition (HAR)', 'value': 'human_activity_recognition'},
200
+ {'label': 'Hand Posture', 'value': 'hand_posture'},
201
+ {'label': 'Audio Event Detection(AED)', 'value': 'audio_event_detection'},
202
+ {'label': 'Object Detection', 'value': 'object_detection'},
203
+ {'label': 'Pose estimation', 'value': 'pose_estimation'},
204
+ {'label': 'Semantic Segmentation', 'value': 'semantic_segmentation'},
205
+ ],
206
+ placeholder="Please select your use case",
207
+ className="mb-4"
208
+ ))),
209
+ dbc.Row(
210
+ dbc.Col(
211
+ html.Div(
212
+ id='toggle-yaml',
213
+ children=[
214
+ dbc.Button("How to update User Config ", id="open-offcanvas", n_clicks=0),
215
+ dbc.Offcanvas(
216
+ html.Div(
217
+ [
218
+ html.P([
219
+ html.Strong("Configure Dataset section:"),
220
+ html.Br(),
221
+ "- Dataset path: ../datasets/your_use_case/name_of_dataset or datasets/your_prepared_dataset.",
222
+ html.Br(),
223
+ html.Br(),
224
+ "- For more details, refer to the ",
225
+ html.A("README", href="https://huggingface.co/spaces/STMicroelectronics/stm32-modelzoo-app/blob/main/datasets/README.md", target="_blank", style={'color': '#007bff', 'text-decoration': 'underline'}),
226
+ ".",
227
+ html.Br(),
228
+ html.Br(),
229
+ "- If you need to upload your model for evaluation, benchmarking or quantizig:",
230
+ html.Br(),
231
+ "- Update model path under General section: models/your_model_name",
232
+ html.Br(),
233
+ "- For more details, refer to the ",
234
+ html.A("README", href="https://huggingface.co/spaces/STMicroelectronics/stm32-modelzoo-app/blob/main/models/README.md", target="_blank", style={'color': '#007bff', 'text-decoration': 'underline'}),
235
+ "."
236
+ ], style={'font-family': 'Arial, sans-serif', 'color': '#03234b', 'fontSize': '18px'})
237
+ ]
238
+ ),
239
+ id="offcanvas",
240
+ is_open=False,
241
+ title="πŸ“š Help",
242
+ placement="end",
243
+ ),
244
+ dcc.RadioItems(
245
+ id='modify-yaml-choice',
246
+ labelStyle={'display': 'inline-block', 'margin-right': '10px'},
247
+ className="mb-4",
248
+ ),
249
+ dcc.Upload(
250
+ id='load-yaml-file',
251
+ children=html.Button('Upload YAML File'),
252
+ style={'display': 'none'}
253
+ ),
254
+ html.Div(id='load-state', style={'margin-top': '10px'}),
255
+ html.Div(id='yaml-layout', style={'display': 'none'})
256
+ ],
257
+ style={'font-family': 'Arial, sans-serif'}
258
+ )
259
+ )
260
+ ),
261
+ dbc.Row([
262
+ dbc.Col([
263
+ html.P("Enter your ST Edge AI Developer Cloud credentials:", style={'color': '03234b', 'fontSize': '15px', 'fontWeight': 'bold'}, className="credentials-text"),
264
+ dcc.Input(id='devcloud-username-input', type='text', placeholder='Enter username', className="input-field mb-2"),
265
+ dcc.Input(id='devcloud-password-input', type='password', placeholder='Enter password', className="input-field mb-4")
266
+ ], width=6),
267
+ dbc.Col([
268
+ dbc.Button('Launch training', id='process-button', color="#ceecf9", className="start-button mb-4", style={'display': 'none', 'box-shadow': '0px 4px 6px rgba(0, 0, 0, 0.1)'})
269
+ ], className="credentials-col")
270
+ ], id='credentials-section', style={
271
+ 'display': 'none',
272
+ 'justify-content': 'center',
273
+ 'align-items': 'center',
274
+ 'height': '100vh',
275
+ }, className="credentials-section mb-4"),
276
+
277
+ dbc.Row([
278
+ dbc.Col(
279
+ html.H5("Results visualization", style={'color': '#03234b', 'margin-bottom': '10px'}),
280
+ width=12
281
+ )
282
+ ], id="results-section", style={"display": "none"}),
283
+ dbc.Row([
284
+ dbc.Col(dbc.Card([
285
+ dbc.CardHeader("Metrics", style={'background-color': '#03234b', 'color': 'white'}),
286
+ dbc.CardBody(
287
+ dcc.Graph(id='acc-visualization', style={'height': '100%', 'width': '100%'}),
288
+ style={'height': '400px', 'display': 'flex', 'justify-content': 'center', 'align-items': 'center'}
289
+ )
290
+ ]), width=6, style={'padding': '10px'}),
291
+ dbc.Col(dbc.Card([
292
+ dbc.CardHeader("Metrics", style={'background-color': '#03234b', 'color': 'white'}),
293
+ dbc.CardBody(
294
+ dcc.Graph(id='loss-visualization', style={'height': '100%', 'width': '100%'}),
295
+ style={'height': '400px', 'display': 'flex', 'justify-content': 'center', 'align-items': 'center'}
296
+ )
297
+ ]), width=6, style={'padding': '10px'})
298
+ ], style={'margin-bottom': '30px'}),
299
+ dbc.Row([
300
+ dbc.Col(dbc.Card([
301
+ dbc.CardHeader("Memory Usage", style={'background-color': '#8191a5', 'color': 'white', 'font-size': '20px'}),
302
+ dbc.CardBody(dcc.Graph(id='memory-bar'))
303
+ ]), width=4),
304
+ dbc.Col(dbc.Card([
305
+ dbc.CardHeader("Inference Time", style={'background-color': '#8191a5', 'color': 'white', 'font-size': '20px'}),
306
+ dbc.CardBody(dcc.Graph(id='inference-time'))
307
+ ]), width=4),
308
+ ],justify="center"),
309
+
310
+ dbc.Row([
311
+ html.Div(id='metric-graphs-container', style={
312
+ 'margin-bottom': '30px'
313
+ })
314
+ ]),
315
+ dcc.Interval(id='interval-widget', interval=1000, n_intervals=0),
316
+ dcc.Download(id="download-resource"),
317
+ dbc.Row(
318
+ dbc.Col(
319
+ dbc.Button('Download outputs', id='download-action', className="mb-4", style={
320
+ 'background-color': '#ffd200',
321
+ 'color': '#ffffff',
322
+ 'font-size': '14px',
323
+ 'padding': '10px 10px',
324
+ 'border-radius': '5px',
325
+ 'box-shadow': '0px 4px 6px rgba(0, 0, 0, 0.1)',
326
+ 'margin-top': '20px'
327
+ }),
328
+ style={
329
+ 'display': 'flex',
330
+ 'justify-content': 'center',
331
+ 'alignItems': 'center',
332
+ }
333
+ )
334
+ ),
335
+ dbc.Row([
336
+ dbc.Col(dbc.Card([
337
+ dbc.CardHeader("Confusion Matrix", style={'background-color': '#8191a5', 'color': 'white', 'font-size': '20px'}),
338
+ dbc.CardBody(
339
+ html.Div(
340
+ html.Img(id='confusion-matrix-img', style={'max-width': '100%', 'height': 'auto'}),
341
+ style={'display': 'flex', 'justify-content': 'center', 'align-items': 'center', 'height': '100%'}
342
+ )
343
+ )
344
+ ]), width=12)
345
+ ], justify="center")
346
+ ], fluid=True),
347
+ dbc.Offcanvas(
348
+ html.Div(id='log-reader', style={'whiteSpace': 'pre-wrap', 'padding': '15px', 'height': '200px', 'overflow': 'auto'}),
349
+ id="log-offcanvas",
350
+ is_open=False,
351
+ placement="bottom",
352
+ style={'height': '200px', 'background-color': '#343a40', 'color': 'white', 'resize': 'vertical', 'overflow': 'auto'}
353
+ ),
354
+ ])
355
+
356
+ def read_configs(selected_model):
357
+ """
358
+ Loads a YAML file based on the selected model by the user.
359
+ Args:
360
+ selected_model (str): The key to select the appropriate YAML file path.
361
+ Returns:
362
+ dict: The loaded YAML data.
363
+ """
364
+ if not selected_model:
365
+ raise ValueError("No model selected. Please select a valid model.")
366
+ if selected_model not in local_yamls:
367
+ raise ValueError(f"Model '{selected_model}' not found in local_yamls")
368
+
369
+ yaml_path = local_yamls[selected_model]
370
+ try:
371
+ with open(yaml_path, 'r') as file:
372
+ return yaml.safe_load(file)
373
+ except Exception as e:
374
+ raise ValueError(f"Error reading YAML file at {yaml_path}: {e}")
375
+
376
+
377
+ def build_yaml_form(yaml_content, parent_key=''):
378
+ """
379
+ Recursively builds a form based on the provided YAML content.
380
+
381
+ Parameters:
382
+ - yaml_content (dict): The YAML content to build the form from.
383
+ - parent_key (str): The parent key to maintain the hierarchy of nested keys. Default is an empty string.
384
+
385
+ Returns:
386
+ - list: A list of Dash Bootstrap Components (dbc) AccordionItems representing the form fields.
387
+ """
388
+
389
+ hidden_sections = {'tools', 'deployment', 'mlflow', 'hydra'}
390
+ accordion_items = []
391
+
392
+ for key, value in yaml_content.items():
393
+ if key in hidden_sections and parent_key == '':
394
+ continue
395
+
396
+ full_key = f"{parent_key}.{key}" if parent_key else key
397
+
398
+ if isinstance(value, dict):
399
+ if full_key == "dataset":
400
+ section_title = html.Span([
401
+ "Dataset ",
402
+ html.Span("*", style={"color": "red", "fontWeight": "bold"}),
403
+ html.Span(" (Set dataset path)", style={"fontSize": "0.85rem", "color": "#dc3545", "marginLeft": "5px"})
404
+ ])
405
+ else:
406
+ section_title = key.capitalize()
407
+
408
+ nested_accordion = build_yaml_form(value, full_key)
409
+ accordion_items.append(
410
+ dbc.AccordionItem(
411
+ nested_accordion,
412
+ title=section_title
413
+ )
414
+ )
415
+ else:
416
+ field = [html.Label(key, style={"font-weight": "bold", "margin-bottom": "5px"})]
417
+
418
+ if isinstance(value, bool):
419
+ field.append(
420
+ dcc.Checklist(
421
+ id={'type': 'yaml-setting', 'index': full_key},
422
+ options=[{'label': '', 'value': True}],
423
+ value=[True] if value else [],
424
+ style={"padding": "10px", "border": "1px solid #ddd", "margin-bottom": "10px"}
425
+ )
426
+ )
427
+
428
+
429
+ elif isinstance(value, list):
430
+ field.append(
431
+ dcc.Dropdown(
432
+ id={'type': 'yaml-setting', 'index': full_key},
433
+ options=[{'label': str(v), 'value': v} for v in value],
434
+ value=value,
435
+ multi=True,
436
+ style={"padding": "10px", "border": "1px solid #ddd", "margin-bottom": "10px"}
437
+ )
438
+ )
439
+
440
+ else:
441
+ input_style = {
442
+ "padding": "10px",
443
+ "border": "1px solid #ddd",
444
+ "margin-bottom": "10px",
445
+ "width": "100%"
446
+ }
447
+ helper = None
448
+
449
+ if full_key == "dataset.training_path":
450
+ input_style.update({
451
+ "border": "2px solid #ffc107",
452
+ "backgroundColor": "#fff8e1"
453
+ })
454
+ helper = html.Div(
455
+ "⚠️ Please update dataset path.",
456
+ style={
457
+ "color": "#856404",
458
+ "fontSize": "0.85rem",
459
+ "marginTop": "-8px",
460
+ "marginBottom": "10px"
461
+ }
462
+ )
463
+
464
+ field.append(
465
+ dcc.Input(
466
+ id={'type': 'yaml-setting', 'index': full_key},
467
+ value=value,
468
+ type='text',
469
+ style=input_style
470
+ )
471
+ )
472
+
473
+ if helper:
474
+ field.append(helper)
475
+
476
+ accordion_items.append(
477
+ dbc.AccordionItem(
478
+ field,
479
+ title=key.capitalize()
480
+ )
481
+ )
482
+
483
+ return accordion_items
484
+
485
+ def create_yaml(yaml_content):
486
+ """
487
+ Creates a YAML form using Dash Bootstrap Components (dbc) and Dash HTML Components (html).
488
+
489
+ Parameters:
490
+ yaml_content (dict): The content of the YAML file to be used for building the form.
491
+
492
+ Returns:
493
+ dbc.Form: A Dash form component containing an accordion with the YAML content and a submit button.
494
+ """
495
+ accordion_items = build_yaml_form(yaml_content)
496
+ accordion = dbc.Accordion(
497
+ accordion_items,
498
+ start_collapsed=True
499
+ )
500
+
501
+ return dbc.Form([
502
+ accordion,
503
+ html.Div(
504
+ dbc.Button(
505
+ 'Submit',
506
+ id='apply-button',
507
+ style={
508
+ 'background-color': '#FFD200',
509
+ 'color': '#03234b',
510
+ 'font-size': '14px',
511
+ 'padding': '10px 10px 10px 10px',
512
+ 'border-radius': '5px',
513
+ 'margin-top': '15px',
514
+ 'border': '2px solid #FFD200',
515
+ 'box-shadow': '0px 4px 6px rgba(0, 0, 0, 0.1)',
516
+ }
517
+ ),
518
+ style={
519
+ 'display': 'flex',
520
+ 'justify-content': 'center',
521
+ 'margin-top': '15px',
522
+ }
523
+ ),
524
+ html.Div(
525
+ id='submission-outcome',
526
+ style={
527
+ 'marginTop': '10px',
528
+ 'textAlign': 'center',
529
+ 'fontStyle': 'italic',
530
+ 'color': '#03234b',
531
+ 'font-size': '14px'
532
+ }
533
+ )
534
+ ])
535
+
536
+
537
+ def process_form_configs(form_configs):
538
+ """
539
+ Extracts and processes form data to update YAML content.
540
+
541
+ This function processes the form data, converting values to appropriate types
542
+ and updating the YAML content accordingly.
543
+
544
+ Args:
545
+ form_configs (dict): The form data to be processed.
546
+
547
+ Returns:
548
+ dict: The updated YAML content with processed form data.
549
+ """
550
+ updated_yaml = {}
551
+ for key, value in form_configs.items():
552
+ if value is not None:
553
+ if isinstance(value, list) and len(value) == 1:
554
+ value = value[0]
555
+
556
+ if isinstance(value, str):
557
+ try:
558
+ if '.' in value:
559
+ value = float(value)
560
+ else:
561
+ value = int(value)
562
+ except ValueError:
563
+ pass
564
+
565
+ updated_yaml[key] = value
566
+
567
+ return updated_yaml
568
+
569
+
570
+ def create_archive(archive_path, directory_to_compress):
571
+ """
572
+ Creates a ZIP archive of a specified directory.
573
+
574
+ Parameters:
575
+ archive_path (str): The path where the ZIP archive will be created.
576
+ directory_to_compress (str): The directory whose contents will be compressed into the ZIP archive.
577
+
578
+ Returns:
579
+ None
580
+ """
581
+ def add_file_to_zip(zipf, file_path, arcname):
582
+ """
583
+ Adds a file to the ZIP archive.
584
+
585
+ Parameters:
586
+ zipf (zipfile.ZipFile): The ZIP file object.
587
+ file_path (str): The path of the file to add to the ZIP archive.
588
+ arcname (str): The archive name for the file within the ZIP archive.
589
+
590
+ Returns:
591
+ None
592
+ """
593
+ zipf.write(file_path, arcname=arcname)
594
+
595
+ with zipfile.ZipFile(archive_path, 'w', compression=zipfile.ZIP_DEFLATED) as zipf:
596
+ with ThreadPoolExecutor() as executor:
597
+ for root_dir, sub_dirs, files in os.walk(directory_to_compress):
598
+ for file_name in files:
599
+ file_path = os.path.join(root_dir, file_name)
600
+ if os.path.abspath(file_path) != os.path.abspath(archive_path):
601
+ arcname = os.path.relpath(file_path, directory_to_compress)
602
+ executor.submit(add_file_to_zip, zipf, file_path, arcname)
603
+
604
+
605
+ app.layout = create_dashboard_layout
606
+
607
+ logs = []
608
+ lock = threading.Lock()
609
+ new_training = False
610
+
611
+ def fill_logs(message):
612
+ """
613
+ Appends a message to the logs list in a thread-safe manner and returns the formatted logs.
614
+
615
+ Parameters:
616
+ message (str): The message to be appended to the logs.
617
+
618
+ Returns:
619
+ html.Pre: The formatted logs as HTML content.
620
+ """
621
+ with lock:
622
+ logs.append(message)
623
+ filtered_logs = filter_logs("\n".join(logs))
624
+ formatted_logs = format_logs(filtered_logs)
625
+ return html.Pre(formatted_logs, style={'whiteSpace': 'pre-wrap', 'wordBreak': 'break-all'})
626
+
627
+ def filter_logs(logs):
628
+ important_lines = []
629
+ for line in logs.split('\n'):
630
+ if '[INFO]' in line or 'Epoch' in line or 'Total params' in line or 'Trainable params' in line or 'Non-trainable params' in line:
631
+ important_lines.append(line)
632
+ elif 'Segments built' in line:
633
+ important_lines.append(line)
634
+ return '\n'.join(important_lines)
635
+
636
+ def format_logs(logs):
637
+ formatted_logs = logs.replace('[INFO]', '\n[INFO]').replace('Epoch', '\nEpoch').replace('Segments built', '\nSegments built')
638
+ return formatted_logs
639
+
640
+ def extract_metrics(logs):
641
+ metrics = {
642
+ 'float': {},
643
+ 'quantized': {},
644
+ 'oks': {},
645
+ }
646
+
647
+ float_match = re.search(r"Accuracy of float model(?: on validation_set)?\s*=\s*([\d.]+)\s*%", logs)
648
+ if float_match:
649
+ metrics['float']['accuracy'] = float(float_match.group(1))
650
+
651
+ quant_match = re.search(r"Accuracy of quantized model(?: on validation_set)?\s*=\s*([\d.]+)\s*%", logs)
652
+ if quant_match:
653
+ metrics['quantized']['accuracy'] = float(quant_match.group(1))
654
+
655
+ precision_match = re.search(r"Mean precision:\s*([\d.]+)", logs)
656
+ if precision_match:
657
+ metrics['oks']['precision'] = float(precision_match.group(1))
658
+
659
+ recall_match = re.search(r"Mean recall:\s*([\d.]+)", logs)
660
+ if recall_match:
661
+ metrics['oks']['recall'] = float(recall_match.group(1))
662
+
663
+ ap_match = re.search(r"Mean AP $mAP$:\s*([\d.]+)", logs)
664
+ if ap_match:
665
+ metrics['oks']['mean_ap'] = float(ap_match.group(1))
666
+
667
+ oks_match = re.search(r"The mean OKS is :\s*([\d.]+)", logs)
668
+ if oks_match:
669
+ metrics['oks']['mean_oks'] = float(oks_match.group(1))
670
+
671
+ iou_match = re.search(r"Average IoU of float model \(all classes\) on validation_set\s*=\s*([\d.]+)\s*%", logs)
672
+ if iou_match:
673
+ metrics['float']['average_iou'] = float(iou_match.group(1))
674
+
675
+ return metrics
676
+
677
+
678
+ def _parse_inference_memory(logs):
679
+ metrics = {}
680
+ """
681
+ patterns = {
682
+ "ram": r"Total RAM\s*:\s*([\d.]+)\s*\(KiB\)",
683
+ "flash": r"Total Flash\s*:\s*([\d.]+)\s*\(KiB\)",
684
+ "inference_time": r"Inference Time\s*:\s*([\d.]+)\s*\(ms\)"
685
+ }
686
+ """
687
+ patterns = {
688
+ "ram": r"Total RAM\s*:\s*([\d.]+)\s*\(KiB\)",
689
+ "flash": r"Total Flash\s*:\s*([\d.]+)\s*\(KiB\)",
690
+ "inference_time": r"Inference Time\s*:\s*([\d.]+)\s*\(ms\)"
691
+ }
692
+
693
+ for key, pattern in patterns.items():
694
+ matches = re.findall(pattern, logs)
695
+ if matches:
696
+ metrics[key] = float(matches[-1])
697
+
698
+ return metrics
699
+
700
+ def create_accuracy_gauge(accuracy):
701
+ return go.Figure(go.Indicator(
702
+ mode="gauge+number",
703
+ value=accuracy,
704
+ title={'text': "Accuracy (%)"},
705
+ gauge={'axis':{'range': [0, 100]}, 'bar':{'color':"#49B170"}},
706
+ ))
707
+
708
+ def create_iou_gauge(iou_value):
709
+ fig = go.Figure(go.Indicator(
710
+ mode="gauge+number",
711
+ value=iou_value,
712
+ title={'text': "IoU (%)"},
713
+ gauge={'axis': {'range': [0, 100]}, 'bar': {'color': "#49B170"}}
714
+ ))
715
+ return fig
716
+
717
+ def latest_confusion_matrix(outputs_folder, recent_directory):
718
+ cf_path = os.path.join(outputs_folder, recent_directory)
719
+
720
+ for filename in os.listdir(cf_path):
721
+ if "confusion_matrix" in filename and filename.endswith(".png"):
722
+ image_path = os.path.join(cf_path, filename)
723
+ with open(image_path, "rb") as f:
724
+ encoded = base64.b64encode(f.read()).decode()
725
+ return f"data:image/png;base64,{encoded}"
726
+ return None
727
+
728
+ def run_script(script, devcloud_username, devcloud_password):
729
+ """
730
+ Executes a given script with the provided ST Developer Cloud credentials and logs the output.
731
+
732
+ Parameters:
733
+ - script (str): The path to the script to be executed.
734
+ - devcloud_username (str): Username for ST Developer Cloud.
735
+ - devcloud_password (str): Password for ST Developer Cloud.
736
+
737
+ Returns:
738
+ - None
739
+ """
740
+ global logs
741
+
742
+ with lock:
743
+ logs = []
744
+
745
+ isolated_env = os.environ.copy()
746
+ isolated_env['stmai_username'] = devcloud_username
747
+ isolated_env['stmai_password'] = devcloud_password
748
+ isolated_env['STATS_TYPE'] = 'HuggingFace_devcloud'
749
+
750
+ execution = subprocess.Popen(['python3', script], env=isolated_env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
751
+ while True:
752
+ file_descriptors = [execution.stdout.fileno(), execution.stderr.fileno()]
753
+ selected_descriptors = select.select(file_descriptors, [], [])
754
+
755
+ for descriptor in selected_descriptors[0]:
756
+ if descriptor == execution.stdout.fileno():
757
+ out = execution.stdout.readline()
758
+ if out:
759
+ fill_logs(out)
760
+ if out == '' and execution.poll() is not None:
761
+ return
762
+ if descriptor == execution.stderr.fileno():
763
+ error = execution.stderr.readline()
764
+ if error:
765
+ fill_logs(error)
766
+
767
+ def execute_async(script, devcloud_username, devcloud_password):
768
+ """
769
+ Executes a Python script asynchronously in a separate thread.
770
+
771
+ Parameters:
772
+ script (str): The path to the Python script to be executed.
773
+ devcloud_username (str): The username for the DevCloud environment.
774
+ devcloud_password (str): The password for the DevCloud environment.
775
+
776
+ Returns:
777
+ None
778
+ """
779
+ thread = threading.Thread(target=run_script, args=(script, devcloud_username, devcloud_password))
780
+ thread.start()
781
+
782
+
783
+ @app.callback(
784
+ Output("config-section", "style"),
785
+ Input('selected-model', 'value')
786
+ )
787
+ def toggle_config_section(selected_model):
788
+ """
789
+ Toggles the visibility of the configuration section based on the selected model.
790
+
791
+ Parameters:
792
+ selected_model (str): The value of the selected model from the dropdown.
793
+
794
+ Returns:
795
+ dict: A dictionary containing the CSS style for the configuration section.
796
+ """
797
+ if selected_model:
798
+ return {"display": "block"}
799
+ else:
800
+ return {"display": "none"}
801
+
802
+
803
+ @app.callback(
804
+ Output('toggle-yaml', 'style'),
805
+ Input('selected-model', 'value')
806
+ )
807
+ def dipslay_yaml_container(selected_model):
808
+ """
809
+ Toggles the display of the YAML update container based on the selected model.
810
+ This function updates the CSS style of the YAML update container to either
811
+ show or hide it based on whether a model is selected from the dropdown.
812
+ Args:
813
+ selected_model (str): The selected model from the dropdown.
814
+ Returns:
815
+ dict: A dictionary containing the CSS style to either display or hide the container.
816
+ """
817
+ if selected_model:
818
+ return {'display': 'block'}
819
+ return {'display': 'none'}
820
+
821
+
822
+ @app.callback(
823
+ Output("offcanvas", "is_open"),
824
+ [Input("open-offcanvas", "n_clicks")],
825
+ [dash.dependencies.State("offcanvas", "is_open")],
826
+ )
827
+ def toggle_offcanvas(n1, is_open):
828
+ if n1:
829
+ return not is_open
830
+ return is_open
831
+
832
+ @app.callback(
833
+ Output("log-offcanvas", "is_open"),
834
+ [Input("toggle-log", "n_clicks")],
835
+ [State("log-offcanvas", "is_open")],
836
+ )
837
+ def toggle_log(n, is_open):
838
+ if n:
839
+ return not is_open
840
+ return is_open
841
+
842
+ @app.callback(
843
+ [Output('yaml-layout', 'style'),
844
+ Output('yaml-layout', 'children')],
845
+ [Input('modify-yaml-choice', 'value'),
846
+ Input('selected-model', 'value')]
847
+ )
848
+
849
+ def display_yaml_form(selection_update, selected_model):
850
+ """
851
+ Toggles the display of the YAML form and updates its content based on user input.
852
+ This function updates the CSS style and content of the YAML form based on whether
853
+ the user chooses to update the YAML file and a model is selected from the dropdown.
854
+ Args:
855
+ selection_update (str): The user's choice to update the YAML file ('yes' or 'no').
856
+ selected_model (str): The selected model from the dropdown.
857
+ Returns:
858
+ tuple: A tuple containing the CSS style to either display or hide the form,
859
+ and the form content generated from the YAML data.
860
+ """
861
+
862
+ if not selected_model:
863
+ return {'display': 'none'}, "Please select a model to display its configuration."
864
+
865
+ try:
866
+ yaml_conf = read_configs(selected_model)
867
+ form_conf = create_yaml(yaml_conf)
868
+ return {'display': 'block'}, form_conf
869
+ except ValueError as e:
870
+ return {'display': 'none'}, f"Error: {str(e)}"
871
+ except Exception as e:
872
+ return {'display': 'none'}, f"Unexpected Error: {str(e)}"
873
+
874
+
875
+ @app.callback(
876
+ Output("log-reader", "style"),
877
+ Input('apply-button', 'n_clicks')
878
+ )
879
+ def toggle_output_section(n_clicks):
880
+ """
881
+ Toggles the visibility of the output logs section based on the number of clicks.
882
+
883
+ Parameters:
884
+ selected_model (str): The value of the selected model from the dropdown.
885
+
886
+ Returns:
887
+ dict: A dictionary containing the CSS style for the configuration section.
888
+ """
889
+ if n_clicks is None or n_clicks == 0:
890
+ return {'display': 'none'}
891
+ return {'display': 'block'}
892
+
893
+
894
+ @app.callback(
895
+ Output('credentials-section', 'style'),
896
+ [Input('modify-yaml-choice', 'value'),
897
+ Input('selected-model', 'value'),
898
+ Input('apply-button', 'n_clicks')]
899
+
900
+ )
901
+ def display_credentials(selection_update, selected_model, n_clicks):
902
+ """
903
+ Toggles the display of the credentials input fields based on user input.
904
+
905
+ This function updates the CSS style of the credentials input fields to either
906
+ show or hide them based on the user's choice to update the YAML file and the
907
+ selection of a model from the dropdown.
908
+
909
+ Args:
910
+ selection_update (str): The user's choice to update the YAML file ('yes' or 'no').
911
+ selected_model (str): The selected model from the dropdown.
912
+
913
+ Returns:
914
+ dict: A dictionary containing the CSS style to either display or hide the credentials input fields.
915
+ """
916
+ if n_clicks is None or n_clicks == 0:
917
+ return {'display': 'none'}
918
+ return {'display': 'block'}
919
+
920
+ @app.callback(
921
+ Output('process-button', 'style'),
922
+ [Input('apply-button', 'n_clicks')]
923
+
924
+ )
925
+ def display_launch_training(n_clicks):
926
+ """
927
+ Displays the process button based on the number of clicks on the apply button.
928
+ Parameters:
929
+ n_clicks (int): The number of times the apply button has been clicked.
930
+ Returns:
931
+ dict: A dictionary containing the CSS style for the process button.
932
+ """
933
+ if n_clicks and n_clicks > 0:
934
+ return {'display': 'inline-block'}
935
+ return {'display': 'none'}
936
+
937
+ @app.callback(
938
+ Output("results-section", "style"),
939
+ Output("toggle-log", "className"),
940
+ Input("process-button", "n_clicks")
941
+ )
942
+ def display_results_section(n_clicks):
943
+ """
944
+ Affiche la section des rΓ©sultats et dΓ©clenche le clignotement du logo.
945
+ """
946
+ if n_clicks and n_clicks > 0:
947
+ return {"display": "block"}, "blinking"
948
+ else:
949
+ return {"display": "none"}, ""
950
+
951
+ @app.callback(
952
+ [Output('log-reader', 'children'),
953
+ Output('acc-visualization', 'figure'),
954
+ Output('acc-visualization', 'style'),
955
+ Output('loss-visualization', 'figure'),
956
+ Output('loss-visualization', 'style'),
957
+ Output('confusion-matrix-img', 'src')],
958
+ [Input('interval-widget', 'n_intervals'),
959
+ Input('process-button', 'n_clicks')],
960
+ [State('selected-model', 'value'),
961
+ State('devcloud-username-input', 'value'),
962
+ State('devcloud-password-input', 'value')]
963
+ )
964
+ def refresh_metrics(n_intervals, nb_clicks, selected_model, devcloud_username, devcloud_password):
965
+ """
966
+ Updates the log display and training metrics based on user actions and intervals.
967
+
968
+ This function handles the following:
969
+ - Executes the training script when the run button is clicked and updates the logs.
970
+ - Periodically checks for new training metrics and updates the accuracy and loss graphs.
971
+ - Manages the display of the log and metrics components based on the training status.
972
+
973
+ Args:
974
+ n_intervals (int): The number of intervals that have passed for the interval component.
975
+ nb_clicks (int): The number of times the run button has been clicked.
976
+ selected_model (str): The selected model from the dropdown.
977
+ devcloud_username (str): The username for authentication.
978
+ devcloud_password (str): The password for authentication.
979
+
980
+ Returns:
981
+ tuple: A tuple containing:
982
+ - str: The updated log messages.
983
+ - dict: The figure data for the accuracy graph.
984
+ - dict: The CSS style to display or hide the accuracy graph.
985
+ - dict: The figure data for the loss graph.
986
+ - dict: The CSS style to display or hide the loss graph.
987
+ - str: The base64 encoded image source for the confusion matrix.
988
+
989
+ Raises:
990
+ PreventUpdate: If the callback context is not triggered by a relevant input.
991
+ """
992
+
993
+ global logs, new_training
994
+
995
+ callback_context = dash.callback_context
996
+ if not callback_context.triggered:
997
+ raise PreventUpdate
998
+
999
+ button = callback_context.triggered[0]['prop_id'].split('.')[0]
1000
+
1001
+ if button == 'process-button' and nb_clicks:
1002
+ if devcloud_username and devcloud_password:
1003
+ st_script = f"stm32ai-modelzoo-services/{selected_model}/src/stm32ai_main.py"
1004
+ execute_async(st_script, devcloud_username, devcloud_password)
1005
+ new_training = True
1006
+ logs.append("Starting application ...")
1007
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1008
+ else:
1009
+ logs.append("Please enter both ST Developer Cloud username and password:")
1010
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1011
+
1012
+ elif button == 'interval-widget':
1013
+ if not new_training:
1014
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1015
+
1016
+ outputs_folder = "experiments_outputs"
1017
+
1018
+ if not os.path.exists(outputs_folder):
1019
+ os.makedirs(outputs_folder)
1020
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1021
+
1022
+ dated_directories = [d for d in os.listdir(outputs_folder) if os.path.isdir(os.path.join(outputs_folder, d)) and d.startswith('20')]
1023
+ if dated_directories:
1024
+ recent_directory = max(dated_directories, key=lambda d: datetime.strptime(d, '%Y_%m_%d_%H_%M_%S'))
1025
+ train_metrics_file = os.path.join(outputs_folder, recent_directory, 'logs', 'metrics', 'train_metrics.csv')
1026
+ print(f"Metrics file : {train_metrics_file}")
1027
+ if os.path.exists(train_metrics_file) and new_training:
1028
+ metrics_dataframe = pd.read_csv(train_metrics_file)
1029
+ if not metrics_dataframe.empty:
1030
+ figures = []
1031
+ metrics_pairs = [
1032
+ ('accuracy', 'val_accuracy'),
1033
+ ('loss', 'val_loss'),
1034
+ ('oks', 'val_oks'),
1035
+ ('val_map',)
1036
+ ]
1037
+ for pair in metrics_pairs:
1038
+ if len(pair) == 2:
1039
+ train_metric, val_metric = pair
1040
+ if train_metric in metrics_dataframe.columns and val_metric in metrics_dataframe.columns:
1041
+ fig = {
1042
+ 'data': [
1043
+ {
1044
+ 'x': metrics_dataframe['epoch'],
1045
+ 'y': metrics_dataframe[train_metric],
1046
+ 'type': 'line',
1047
+ 'name': train_metric.capitalize(),
1048
+ 'line': {'color': '#FFD200', 'width': 2, 'dash': 'solid'},
1049
+ 'hoverinfo': 'x+y+name',
1050
+ 'hoverlabel': {'bgcolor': '#EEEFF1', 'font': {'color': '#525A63'}}
1051
+ },
1052
+ {
1053
+ 'x': metrics_dataframe['epoch'],
1054
+ 'y': metrics_dataframe[val_metric],
1055
+ 'type': 'line',
1056
+ 'name': val_metric.capitalize(),
1057
+ 'line': {'color': '#3CB4E6', 'width': 2, 'dash': 'solid'},
1058
+ 'hoverinfo': 'x+y+name',
1059
+ 'hoverlabel': {'bgcolor': '#EEEFF1', 'font': {'color': '#525A63'}}
1060
+ }
1061
+ ],
1062
+ 'layout': {
1063
+ 'title': {
1064
+ 'text': f'{train_metric.capitalize()} vs {val_metric.capitalize()}',
1065
+ 'x': 0.5,
1066
+ 'xanchor': 'center'
1067
+ },
1068
+ 'xaxis': {
1069
+ 'title': 'Epochs',
1070
+ 'showgrid': True,
1071
+ 'gridcolor': '#EEEFF1',
1072
+ 'tickangle': 45
1073
+ },
1074
+ 'yaxis': {
1075
+ 'title': train_metric.capitalize(),
1076
+ 'showgrid': True,
1077
+ 'gridcolor': '#EEEFF1'
1078
+ },
1079
+ 'showlegend': True,
1080
+ 'legend': {
1081
+ 'x': 1,
1082
+ 'y': 1,
1083
+ 'traceorder': 'normal',
1084
+ 'font': {'size': 10},
1085
+ 'bgcolor': '#EEEFF1',
1086
+ 'bordercolor': '#A6ADB5',
1087
+ 'borderwidth': 1
1088
+ },
1089
+ 'hovermode': 'closest',
1090
+ 'plot_bgcolor': '#ffffff'
1091
+ }
1092
+ }
1093
+ figures.append(fig)
1094
+ elif len(pair) == 1:
1095
+ val_metric = pair[0]
1096
+ if val_metric in metrics_dataframe.columns:
1097
+ fig = {
1098
+ 'data': [
1099
+ {
1100
+ 'x': metrics_dataframe['epoch'],
1101
+ 'y': metrics_dataframe[val_metric],
1102
+ 'type': 'line',
1103
+ 'name': val_metric.capitalize(),
1104
+ 'line': {'color': '#3CB4E6', 'width': 2, 'dash': 'solid'},
1105
+ 'hoverinfo': 'x+y+name',
1106
+ 'hoverlabel': {'bgcolor': '#EEEFF1', 'font': {'color': '#525A63'}}
1107
+ }
1108
+ ],
1109
+ 'layout': {
1110
+ 'title': {
1111
+ 'text': f'{val_metric.capitalize()} over Epochs',
1112
+ 'x': 0.5,
1113
+ 'xanchor': 'center'
1114
+ },
1115
+ 'xaxis': {
1116
+ 'title': 'Epochs',
1117
+ 'showgrid': True,
1118
+ 'gridcolor': '#EEEFF1',
1119
+ 'tickangle': 45
1120
+ },
1121
+ 'yaxis': {
1122
+ 'title': val_metric.capitalize(),
1123
+ 'showgrid': True,
1124
+ 'gridcolor': '#EEEFF1'
1125
+ },
1126
+ 'showlegend': True,
1127
+ 'legend': {
1128
+ 'x': 1,
1129
+ 'y': 1,
1130
+ 'traceorder': 'normal',
1131
+ 'font': {'size': 10},
1132
+ 'bgcolor': '#EEEFF1',
1133
+ 'bordercolor': '#A6ADB5',
1134
+ 'borderwidth': 1
1135
+ },
1136
+ 'hovermode': 'closest',
1137
+ 'plot_bgcolor': '#ffffff'
1138
+ }
1139
+ }
1140
+ figures.append(fig)
1141
+
1142
+ confusion_matrix_src = latest_confusion_matrix(outputs_folder, recent_directory)
1143
+
1144
+ if figures:
1145
+ return "\n".join(logs), figures[0], {'display': 'block'}, figures[1] if len(figures) > 1 else {}, {'display': 'block'}, confusion_matrix_src
1146
+ else:
1147
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, confusion_matrix_src
1148
+ else:
1149
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1150
+ else:
1151
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1152
+ else:
1153
+ return "\n".join(logs), {}, {'display': 'none'}, {}, {'display': 'none'}, None
1154
+
1155
+ raise PreventUpdate
1156
+
1157
+ @app.callback(
1158
+ Output('metric-graphs-container', 'children'),
1159
+ Input('log-reader', 'children')
1160
+ )
1161
+ def update_metrics_dashboard(logs):
1162
+ metrics = extract_metrics(logs)
1163
+ graphs = []
1164
+
1165
+ def get_metric_card(title, figure):
1166
+ return dbc.Col(
1167
+ dbc.Card([
1168
+ dbc.CardHeader(title, style={'background-color': '#8191a5', 'color': 'white', 'font-size': '18px'}),
1169
+ dbc.CardBody(dcc.Graph(figure=figure, config={'displayModeBar': False}))
1170
+ ]),
1171
+ width=4
1172
+ )
1173
+
1174
+ average_iou = metrics.get("float", {}).get("average_iou")
1175
+ if average_iou is not None:
1176
+ graphs.append(get_metric_card("Average IoU - Float Model", create_iou_gauge(average_iou)))
1177
+
1178
+ if "float" in metrics:
1179
+ acc = metrics["float"].get("accuracy")
1180
+ if acc is not None:
1181
+ graphs.append(get_metric_card("Accuracy - Float Model", create_accuracy_gauge(acc)))
1182
+
1183
+ if "quantized" in metrics:
1184
+ acc = metrics["quantized"].get("accuracy")
1185
+ if acc is not None:
1186
+ graphs.append(get_metric_card("Accuracy - Quantized Model", create_accuracy_gauge(acc)))
1187
+
1188
+ if "oks" in metrics:
1189
+ mean_oks = metrics["oks"].get("mean_oks")
1190
+ if mean_oks is not None:
1191
+ graphs.append(get_metric_card("Mean OKS", create_accuracy_gauge(mean_oks)))
1192
+
1193
+ precision = metrics["oks"].get("precision")
1194
+ if precision is not None:
1195
+ graphs.append(get_metric_card("Mean Precision", create_accuracy_gauge(precision)))
1196
+
1197
+ recall = metrics["oks"].get("recall")
1198
+ if recall is not None:
1199
+ graphs.append(get_metric_card("Mean Recall", create_accuracy_gauge(recall)))
1200
+
1201
+ mean_ap = metrics["oks"].get("mean_ap")
1202
+ if mean_ap is not None:
1203
+ graphs.append(get_metric_card("Mean AP (mAP)", create_accuracy_gauge(mean_ap)))
1204
+
1205
+ return dbc.Row(graphs, justify="center")
1206
+
1207
+
1208
+ @app.callback(
1209
+ Output('memory-bar', 'figure'),
1210
+ Input('log-reader', 'children')
1211
+ )
1212
+ def update_memory_bar(logs):
1213
+ metrics = _parse_inference_memory(logs)
1214
+ ram = metrics.get('ram', 0)
1215
+ flash = metrics.get('flash', 0)
1216
+
1217
+ fig = go.Figure()
1218
+ fig.add_trace(go.Bar(
1219
+ y=["Total RAM ", "Total Flash"],
1220
+ x=[ram, flash],
1221
+ orientation='h',
1222
+ marker_color=["#E6007E", "#3cb4e6"]
1223
+ ))
1224
+
1225
+ fig.update_layout(title="Memory Usage (KiB)", xaxis_title="Size (KiB)")
1226
+ return fig
1227
+
1228
+ @app.callback(
1229
+ Output('inference-time', 'figure'),
1230
+ Input('log-reader', 'children')
1231
+ )
1232
+ def update_inference_time(logs):
1233
+ metrics = _parse_inference_memory(logs)
1234
+ inference_time = metrics.get('inference_time', 0)
1235
+
1236
+ fig = go.Figure(go.Indicator(
1237
+ mode="number",
1238
+ value=inference_time,
1239
+ title={'text': "Inference Time (ms)"},
1240
+ ))
1241
+ return fig
1242
+
1243
+ @app.callback(
1244
+ Output('submission-outcome', 'children'),
1245
+ [Input('apply-button', 'n_clicks'),
1246
+ Input('process-button', 'n_clicks')],
1247
+ [State({'type': 'yaml-setting', 'index': ALL}, 'id'),
1248
+ State({'type': 'yaml-setting', 'index': ALL}, 'value'),
1249
+ State('selected-model', 'value'),
1250
+ State('devcloud-username-input', 'value'),
1251
+ State('devcloud-password-input', 'value')]
1252
+ )
1253
+ def process_button_actions(submit_clicks, exec_nb_clicks, form_input_ids, form_input_values, selected_model, devcloud_username, devcloud_password):
1254
+ """
1255
+ Handles the actions triggered by the submit and run buttons.
1256
+
1257
+ This function processes the form data when the submit button is clicked,
1258
+ updates the corresponding YAML file, and executes the training script when
1259
+ the run button is clicked.
1260
+
1261
+ Args:
1262
+ submit_clicks (int): The number of times the submit button has been clicked.
1263
+ exec_nb_clicks (int): The number of times the execution/run button has been clicked.
1264
+ form_input_ids (list): A list of dictionaries containing the IDs of the form inputs.
1265
+ form_input_values (list): A list of values from the form inputs.
1266
+ selected_model (str): The selected model from the dropdown.
1267
+ devcloud_username (str): The username for DevCloud authentication.
1268
+ devcloud_password (str): The password for DevCloud authentication.
1269
+
1270
+ Returns:
1271
+ str: A message indicating the result of the action, such as successful YAML update or script execution status.
1272
+
1273
+ Raises:
1274
+ PreventUpdate: If the callback context is not triggered by a relevant input or if no action is taken.
1275
+ """
1276
+ new_fields = []
1277
+
1278
+ callback_context = dash.callback_context
1279
+ if not callback_context.triggered:
1280
+ raise PreventUpdate
1281
+
1282
+ triggered_button = callback_context.triggered[0]['prop_id'].split('.')[0]
1283
+
1284
+ if triggered_button == 'apply-button':
1285
+ if submit_clicks:
1286
+ try:
1287
+ form_fields_data = {}
1288
+ for i in range(len(form_input_ids)):
1289
+ input_id = form_input_ids[i]['index']
1290
+ input_value = form_input_values[i]
1291
+ form_fields_data[input_id] = input_value
1292
+
1293
+ yaml_file_path = local_yamls.get(selected_model)
1294
+ if yaml_file_path :
1295
+ yaml_parser = ruamel.yaml.YAML()
1296
+ with open(yaml_file_path , 'r') as file:
1297
+ current_yaml_data = yaml_parser.load(file)
1298
+
1299
+ updated_yaml_data = process_form_configs(form_fields_data)
1300
+ for key, value in updated_yaml_data.items():
1301
+ keys = key.split('.')
1302
+ nested_dict = current_yaml_data
1303
+ for k in keys[:-1]:
1304
+ nested_dict = nested_dict.setdefault(k, {})
1305
+ if nested_dict[keys[-1]] != value:
1306
+ nested_dict[keys[-1]] = value
1307
+ new_fields.append(key)
1308
+
1309
+ with open(yaml_file_path , 'w') as file:
1310
+ yaml_parser.dump(current_yaml_data, file)
1311
+
1312
+ return f"User config yaml file has been updated successfully ! Updated fields are: {', '.join(new_fields)}"
1313
+ else:
1314
+ return f"ERROR: No user config yaml found for '{selected_model}'."
1315
+ except Exception as e:
1316
+ return f"ERROR: UPDATING USER CONFIG YAML file: {e}"
1317
+ else:
1318
+ raise PreventUpdate
1319
+ elif triggered_button == 'process-button':
1320
+ if exec_nb_clicks:
1321
+ st_script = f"stm32ai-modelzoo-services/{selected_model}/src/stm32ai_main.py"
1322
+ execute_async(st_script, devcloud_username, devcloud_password)
1323
+ return "Application is running ..."
1324
+ else:
1325
+ raise PreventUpdate
1326
+
1327
+
1328
+
1329
+ @app.callback(
1330
+ Output('download-action', 'style'),
1331
+ [Input('interval-widget', 'n_intervals')],
1332
+ [State('selected-model', 'value')]
1333
+ )
1334
+ def toggle_download_button(n_intervals, selected_model):
1335
+ """
1336
+ Toggles the display of the download button based on the existence of output directories.
1337
+
1338
+ This function checks if the output directories for the selected model exist and
1339
+ toggles the display of the download button accordingly.
1340
+
1341
+ Args:
1342
+ n_intervals (int): The number of intervals that have passed for the interval component.
1343
+ model_choice (str): The selected model from the dropdown.
1344
+
1345
+ Returns:
1346
+ dict: A dictionary containing the CSS style to either display or hide the download button.
1347
+ """
1348
+ out_directory = os.path.join(os.getcwd(), "experiments_outputs")
1349
+
1350
+ if not os.path.exists(out_directory ):
1351
+ return {'display': 'none'}
1352
+
1353
+ output_subdirectories = [d for d in os.listdir(out_directory ) if os.path.isdir(os.path.join(out_directory , d)) and d.startswith('20')]
1354
+
1355
+ if output_subdirectories:
1356
+ return {'display': 'block'}
1357
+ return {'display': 'none'}
1358
+
1359
+
1360
+ @app.callback(
1361
+ Output('download-resource', 'data'),
1362
+ [Input('download-action', 'n_clicks')],
1363
+ [State('selected-model', 'value')]
1364
+ )
1365
+ def generate_download_link(n_clicks, selected_model):
1366
+ """
1367
+ Generates a download link based on the selected model and operation mode.
1368
+
1369
+ This function reads the YAML configuration for the selected model, determines the operation mode,
1370
+ and generates a download link for the appropriate file (ZIP or ELF/BIN) based on the operation mode.
1371
+
1372
+ Args:
1373
+ click_count (int): The number of times the download button has been clicked.
1374
+ selected_model (str): The selected model from the dropdown.
1375
+
1376
+ Returns:
1377
+ dcc.send_file: A Dash component to send the file for download.
1378
+
1379
+ Raises:
1380
+ PreventUpdate: If no relevant action is taken or the required files do not exist.
1381
+ """
1382
+
1383
+ if n_clicks is None:
1384
+ raise PreventUpdate
1385
+
1386
+
1387
+ output_directory = os.path.join(os.getcwd(), "./experiments_outputs")
1388
+
1389
+ if not os.path.exists(output_directory ):
1390
+ raise PreventUpdate
1391
+
1392
+
1393
+ timestamped_directories = [d for d in os.listdir(output_directory ) if os.path.isdir(os.path.join(output_directory , d)) and d.startswith('20')]
1394
+
1395
+ timestamped_directories = [
1396
+ d for d in os.listdir(output_directory)
1397
+ if os.path.isdir(os.path.join(output_directory, d)) and d.startswith("20")
1398
+ ]
1399
+
1400
+ if timestamped_directories:
1401
+ recent_directory = max(
1402
+ timestamped_directories,
1403
+ key=lambda d: datetime.strptime(d, "%Y_%m_%d_%H_%M_%S")
1404
+ )
1405
+ recent_directory_path = os.path.join(output_directory, recent_directory)
1406
+ zip_file_path = os.path.join(recent_directory_path, f"{recent_directory}.zip")
1407
+
1408
+
1409
+ if not os.path.exists(zip_file_path):
1410
+ create_archive(zip_file_path, recent_directory_path)
1411
+
1412
+
1413
+ if os.path.exists(zip_file_path):
1414
+ return dcc.send_file(zip_file_path)
1415
+
1416
+ raise PreventUpdate
1417
+
1418
+ @server.route('/download/<path:subpath>')
1419
+ def download_file(subpath):
1420
+ """
1421
+ Route to download a file from the server.
1422
+
1423
+ Parameters:
1424
+ - subpath (str): The subpath of the file to be downloaded, relative to the './experiments_outputs' directory.
1425
+
1426
+ Returns:
1427
+ - Response: A Flask response object to send the file as an attachment if it exists.
1428
+ - tuple: A tuple containing an error message and a 404 status code if the file is not found.
1429
+ """
1430
+ file_path = os.path.join(os.getcwd(), './experiments_outputs', subpath)
1431
+ if os.path.exists(file_path):
1432
+ return send_file(file_path, as_attachment=True)
1433
+ else:
1434
+ return "File not found", 404
1435
+
1436
+
1437
+ if __name__ == '__main__':
1438
+ app.run_server(host='0.0.0.0',port=7860, dev_tools_ui=True, dev_tools_hot_reload=True, threaded=True)
datasets/README.md ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # STM32 Model Zoo App
2
+
3
+
4
+ ## Directory components:
5
+
6
+ The template provides a structured layout for your project, ensuring that your prepared datasets are organized in the appropriate folders for training purposes.
7
+
8
+ ```
9
+ |-- assets
10
+ |-- models
11
+ |-- datasets
12
+ | |-- human_activity_recognition
13
+ | |-- dataset_name
14
+ | |-- image_classification
15
+ | |-- dataset_name
16
+ | |-- USE_CASE
17
+ | |-- dataset_name
18
+ |-- README.md
19
+ |-- static
20
+ |-- templates
21
+ |-- Dockerfile
22
+ |-- dash_app.py
23
+ |-- download_datasets.py
24
+ |-- requirements_dash.txt
25
+ |-- LICENCE.md
26
+ ```
download_datasets.py ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /*---------------------------------------------------------------------------------------------
2
+ # * Copyright (c) 2024 STMicroelectronics.
3
+ # * All rights reserved.
4
+ # *
5
+ # * This software is licensed under terms that can be found in the LICENSE file in
6
+ # * the root directory of this software component.
7
+ # * If no LICENSE file comes with this software, it is provided AS-IS.
8
+ # *--------------------------------------------------------------------------------------------*/
9
+ import os
10
+ import requests
11
+ import tarfile
12
+ import zipfile
13
+
14
+ def download_and_extract(url, dest_path, rename_to=None):
15
+ if not os.path.exists(dest_path):
16
+ os.makedirs(dest_path)
17
+ filename = url.split('/')[-1]
18
+ filepath = os.path.join(dest_path, filename)
19
+
20
+ try:
21
+ with requests.get(url, stream=True) as r:
22
+ r.raise_for_status()
23
+ with open(filepath, 'wb') as f:
24
+ for chunk in r.iter_content(chunk_size=8192):
25
+ f.write(chunk)
26
+ print(f"Downloaded {filename} successfully.")
27
+
28
+ extracted_folder = None
29
+ if filepath.endswith('.tar.gz') or filepath.endswith('.tgz'):
30
+ with tarfile.open(filepath, 'r:gz') as tar:
31
+ tar.extractall(path=dest_path)
32
+ extracted_folder = tar.getnames()[0].split('/')[0]
33
+ print(f"Extracted {filename} successfully.")
34
+ elif filepath.endswith('.zip'):
35
+ with zipfile.ZipFile(filepath, 'r') as zip_ref:
36
+ zip_ref.extractall(dest_path)
37
+ extracted_folder = zip_ref.namelist()[0].split('/')[0]
38
+ print(f"Extracted {filename} successfully.")
39
+
40
+ os.remove(filepath)
41
+ print(f"Removed {filename} successfully.")
42
+
43
+ if rename_to and extracted_folder:
44
+ extracted_folder_path = os.path.join(dest_path, extracted_folder)
45
+ new_folder = os.path.join(dest_path, rename_to)
46
+ if os.path.exists(extracted_folder_path):
47
+ os.rename(extracted_folder_path, new_folder)
48
+ print(f"Renamed {extracted_folder_path} to {new_folder} successfully.")
49
+ else:
50
+ print(f"Extracted folder {extracted_folder_path} does not exist.")
51
+ except Exception as e:
52
+ print(f"An error occurred: {e}")
53
+
54
+ datasets = {
55
+ 'image_classification': 'http://download.tensorflow.org/example_images/flower_photos.tgz',
56
+ 'human_activity_recognition': 'https://www.cis.fordham.edu/wisdm/includes/datasets/latest/WISDM_ar_latest.tar.gz',
57
+ 'hand_posture': 'https://raw.githubusercontent.com/STMicroelectronics/stm32ai-modelzoo-services/main/hand_posture/datasets/ST_VL53L8CX_handposture_dataset.zip', # Updated URL
58
+ 'audio_event_detection': 'https://github.com/karolpiczak/ESC-50/archive/master.zip'
59
+ }
60
+
61
+ for dataset, url in datasets.items():
62
+ print(f"Processing {dataset}...")
63
+ if dataset == 'audio_event_detection':
64
+ download_and_extract(url, f'/home/appuser/datasets/{dataset}', rename_to='ESC-50')
65
+ else:
66
+ download_and_extract(url, f'/home/appuser/datasets/{dataset}')
67
+ print(f"Finished processing {dataset}.")
models/README.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # STM32 Model Zoo App
2
+
3
+
4
+ ## Directory components:
5
+
6
+
7
+ The template provides a structured layout for your project, ensuring that your pretrained models are stored appropriately for evaluation and benchmarking.
8
+
9
+ ```
10
+ |-- assets
11
+ |-- models
12
+ | |-- model_1
13
+ | |-- ...
14
+ | |-- model_n
15
+ |-- datasets
16
+ |-- README.md
17
+ |-- static
18
+ |-- templates
19
+ |-- Dockerfile
20
+ |-- dash_app.py
21
+ |-- download_datasets.py
22
+ |-- requirements_dash.txt
23
+ |-- LICENCE.md
24
+ ```
requirements_dash.txt ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ dash==2.17.0
2
+ dash-bootstrap-components==1.6.0
3
+ dash-html-components==2.0.0
4
+ dash-table==5.0.0
5
+ ruamel.yaml==0.18.6
6
+ Flask
7
+ flask-session
8
+ gunicorn==20.1.0
9
+ dash-chartjs
10
+ plotly
11
+ dash-daq
12
+ pybase64
static/ST_logo_dark_blue.jpg ADDED
templates/index.html ADDED
@@ -0,0 +1,378 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>STM32AI Model Zoo</title>
7
+ <style>
8
+ * {
9
+ box-sizing: border-box;
10
+ margin: 0;
11
+ padding: 0;
12
+ }
13
+
14
+ body {
15
+ font-family: 'Arial', sans-serif;
16
+ background-color: #f8f9fa;
17
+ color: #03234b;
18
+ display: flex;
19
+ }
20
+
21
+ .sidebar {
22
+ width: 200px;
23
+ height: 100vh;
24
+ position: fixed;
25
+ left: 0;
26
+ top: 0;
27
+ background-color: white;
28
+ padding: 20px;
29
+ box-shadow: 2px 0 10px rgba(0, 0, 0, 0.1);
30
+ overflow-y: auto;
31
+ transition: all 0.3s ease-in-out;
32
+ }
33
+
34
+ .sidebar h2 {
35
+ font-size: 18px;
36
+ margin-bottom: 10px;
37
+ font-weight: bold;
38
+ }
39
+
40
+ .sidebar ul {
41
+ list-style-type: none;
42
+ padding: 0;
43
+ }
44
+
45
+ .sidebar ul li {
46
+ margin: 10px 0;
47
+ }
48
+
49
+ .sidebar ul li a {
50
+ text-decoration: none;
51
+ color: #1a73e8;
52
+ font-size: 16px;
53
+ display: block;
54
+ padding: 6px 10px;
55
+ border-radius: 5px;
56
+ }
57
+
58
+ .sidebar ul li a:hover {
59
+ background-color: #e3f2fd;
60
+ }
61
+
62
+ .content {
63
+ margin-left: 220px;
64
+ padding: 40px;
65
+ max-width: 1200px;
66
+ width: 100%;
67
+ line-height: 1.6;
68
+ word-wrap: break-word;
69
+ overflow-wrap: break-word;
70
+ transition: margin-left 0.3s ease-in-out;
71
+ }
72
+
73
+ h1, h2, h3 {
74
+ color: #03234b;
75
+ }
76
+
77
+ p {
78
+ margin-bottom: 15px;
79
+ }
80
+
81
+ hr {
82
+ border: 0;
83
+ height: 1px;
84
+ background: #ddd;
85
+ margin: 20px 0;
86
+ }
87
+
88
+ .content-wrapper {
89
+ display: flex;
90
+ flex-direction: column;
91
+ gap: 20px;
92
+ }
93
+
94
+ @media (max-width: 768px) {
95
+ .sidebar {
96
+ width: 180px;
97
+ }
98
+ .content {
99
+ margin-left: 200px;
100
+ }
101
+ }
102
+
103
+ @media (max-width: 576px) {
104
+ .sidebar {
105
+ width: 100%;
106
+ height: auto;
107
+ position: relative;
108
+ padding: 15px;
109
+ text-align: center;
110
+ }
111
+ .content {
112
+ margin-left: 0;
113
+ width: 100%;
114
+ padding: 20px;
115
+ }
116
+ .sidebar ul {
117
+ display: flex;
118
+ justify-content: center;
119
+ gap: 15px;
120
+ }
121
+ .sidebar ul li {
122
+ display: inline;
123
+ }
124
+ }
125
+
126
+ .image-small {
127
+ width: 150px;
128
+ height: auto;
129
+ }
130
+
131
+ .title-container {
132
+ display: flex;
133
+ align-items: center;
134
+ justify-content: space-between;
135
+ margin-bottom: 20px;
136
+ }
137
+
138
+ .title-container h1 {
139
+ margin: 0;
140
+ }
141
+
142
+ .title-container img {
143
+ margin-left: 20px;
144
+ }
145
+
146
+ .button {
147
+ display: inline-block;
148
+ padding: 10px 20px;
149
+ font-size: 16px;
150
+ color: #fff;
151
+ background-color: #007bff;
152
+ border: none;
153
+ border-radius: 5px;
154
+ text-align: center;
155
+ text-decoration: none;
156
+ margin-top: 20px;
157
+ }
158
+
159
+ .button:hover {
160
+ background-color: #0056b3;
161
+ }
162
+
163
+ .badge {
164
+ display: inline-block;
165
+ padding: 5px 10px;
166
+ font-size: 14px;
167
+ color: #fff;
168
+ background-color: #28a745;
169
+ border-radius: 5px;
170
+ margin-right: 5px;
171
+ }
172
+
173
+ /* Table Styles */
174
+ .centered-table {
175
+ width: 100%;
176
+ border-collapse: collapse;
177
+ margin: 20px 0;
178
+ font-size: 16px;
179
+ text-align: left;
180
+ }
181
+
182
+ .centered-table th, .centered-table td {
183
+ padding: 12px 15px;
184
+ border: 1px solid #ddd;
185
+ }
186
+
187
+ .centered-table th {
188
+ background-color: #f2f2f2;
189
+ color: #03234b;
190
+ }
191
+
192
+ .centered-table tr:nth-child(even) {
193
+ background-color: #f9f9f9;
194
+ }
195
+
196
+ .centered-table tr:hover {
197
+ background-color: #f1f1f1;
198
+ }
199
+
200
+ .centered-table a {
201
+ color: #1a73e8;
202
+ text-decoration: none;
203
+ }
204
+
205
+ .centered-table a:hover {
206
+ text-decoration: underline;
207
+ }
208
+ </style>
209
+ </head>
210
+ <body>
211
+ <div class="sidebar">
212
+ <h2>Table of Contents</h2>
213
+ <ul>
214
+ <li><a href="#Introduction">Introduction</a></li>
215
+ <li><a href="#UserGuide">User Guide</a></li>
216
+ <li><a href="#UsageScenarios">Usage Scenarios and Examples</a></li>
217
+ <li><a href="#StartApplication">How to Start the Application</a></li>
218
+ </ul>
219
+ </div>
220
+
221
+ <div class="content">
222
+ <div class="content-wrapper">
223
+ <div class="title-container">
224
+ <h1> πŸš€ STMicroelectronics - STM32AI model zoo</h1>
225
+ <img src="{{url_for('static', filename='ST_logo_dark_blue.jpg')}}" alt="Logo" class="image-small">
226
+ </div>
227
+ <hr>
228
+ <p class="lead">
229
+ Welcome to the STM32 Model zoo dockerized Dashboard πŸ“ˆ <br>
230
+ This space is dedicated to run STM32 model zoo from the dashboard using Dash Plotly. This zoo is a collection of reference machine learning models that are optimized to run on STM32 microcontrollers. Available on GitHub, this is a valuable resource for anyone looking to add AI capabilities to their STM32-based projects.
231
+ </p>
232
+
233
+ <section id="Summary">
234
+ <h2>Quick Start</h2>
235
+ <p>
236
+ Get started quickly with the STM32 Model Zoo Dashboard. Click the button below to start the application.
237
+ </p>
238
+ <a href="#StartApplication" class="button" style="background-color: #FFD200; color: #03234b; padding: 10px 20px; text-decoration: none; border-radius: 5px; border: 2px solid #FFD200; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);"> Start Application</a>
239
+ </section>
240
+
241
+ <section id="Introduction">
242
+ <h2>Introduction</h2>
243
+ <p>
244
+ The application provides a comprehensive dashboard interface for interacting with STM32 models and exploring their capabilities. It is developed using Flask and Dash, and is hosted on Hugging Face Spaces.<br>
245
+ This dashboard simplifies the use of the STM32 AI Model Zoo, enabling you to start training these models on Hugging Face with ease.
246
+ </p>
247
+ <div align="center" style="margin-top: 80px; padding: 20px 0;">
248
+ <p align="center">
249
+ <a href="https://www.python.org/downloads/" target="_blank"><img src="https://img.shields.io/badge/python-3.10-blue" /></a>
250
+ <a href="https://dash.plotly.com/installation" target="_blank"><img src="https://img.shields.io/badge/Dash%20Plotly-2.0.0-3CAEA3?style=flat&logo=plotly&logoColor=white&link=https://dash.plotly.com/installation"/></a>
251
+ <a href="https://stm32ai-cs.st.com/home" target="_blank"><img src="https://img.shields.io/badge/STM32Cube.AI-Developer%20Cloud-FFD700?style=flat&logo=stmicroelectronics&logoColor=white"/></a>
252
+ </p>
253
+ </section>
254
+
255
+ <section id="UserGuide">
256
+ <h2>User Guide</h2>
257
+ <p>
258
+ This application provides the necessary code to build a docker container that runs on Hugging Face Docker Space. This space hosts :</p>
259
+ <li style="color: #03234b" ><b>Dash application script:</b> This application provides a dashboard developed with Dash Plotly to run use cases from the STM32-Model Zoo. You can <b>Train</b>, <b>Evaluate</b> and <b>Benchmark</b> models seamlessly.</li>
260
+ <li style="color: #03234b" ><b>Dockerfile:</b> Sets up the desired environment and clones the repository from GitHub to ensure alignment with the latest updates. Model zoo is hosted <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services" target="_blank">here</a>. You can find the licences information <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services/blob/main/LICENSE.md" target="_blank">here</a>.</li>
261
+ <li style="color: #03234b" ><b>Datasets:</b> Downloads the dataset based on the use case. The user must provide the dataset link before building the docker image and launching the application. The table below summarizes the datasets handled by the script: <br>
262
+ <br>
263
+ <table class="centered-table">
264
+ <thead>
265
+ <tr>
266
+ <th>Dataset</th>
267
+ <th>License</th>
268
+ </tr>
269
+ </thead>
270
+ <tbody>
271
+ <tr>
272
+ <td><a href="http://download.tensorflow.org/example_images/flower_photos.tgz">Flower Photos</a></td>
273
+ <td><a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></td>
274
+ </tr>
275
+ <tr>
276
+ <td><a href="https://data.mendeley.com/datasets/tywbtsjrjv/1">Plant Village</a></td>
277
+ <td><a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0</a></td>
278
+ </tr>
279
+ <tr>
280
+ <td><a href="https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/">Food-101</a></td>
281
+ <td><a href="https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/image_classification/pretrained_models/mobilenetv2">-</a></td>
282
+ </tr>
283
+ <tr>
284
+ <td><a href="https://www.cis.fordham.edu/wisdm/includes/datasets/latest/WISDM_ar_latest.tar.gz">WISDM Activity Recognition</a></td>
285
+ <td><a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></td>
286
+ </tr>
287
+ <tr>
288
+ <td><a href="https://github.com/STMicroelectronics/stm32ai-modelzoo/raw/main/hand_posture/datasets/ST_VL53L8CX_handposture_dataset.zip">Hand Posture</a></td>
289
+ <td><a href="https://github.com/STMicroelectronics/stm32ai-modelzoo/blob/main/hand_posture/datasets/LICENSE.md">SLA008</a></td>
290
+ </tr>
291
+ <tr>
292
+ <td><a href="https://github.com/karolpiczak/ESC-50/archive/master.zip">ESC-50</a></td>
293
+ <td><a href="https://github.com/karolpiczak/ESC-50/blob/master/LICENSE">-</a></td>
294
+ </tr>
295
+ </tbody>
296
+ </table>
297
+ </p>
298
+ </section>
299
+
300
+ <section id="UsageScenarios">
301
+ <h2>Usage Scenarios</h2>
302
+ <p>
303
+ To start using the features of this application follow these steps:
304
+ <div class="indented">
305
+ <li><strong>Go to the <em>Start Application</em> section</strong>: Duplicate this space to launch it on your Hugging Face space. This will automatically build the image and launch the container on your space.</li>
306
+ <p style= "margin-left: 2em">
307
+ <li> If you'd like to use one of these models, please follow these instructions:
308
+ <ul>
309
+ <li><em>Object Detection:</em> Please follow this <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services/tree/main/object_detection/src#1" target="_blank">link</a> to perform dataset preparation.</li>
310
+ <br>
311
+ <li><em>Semantic Segmentation:</em> Please follow this <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services/tree/main/semantic_segmentation/src#2-3" target="_blank">link</a> to get the dataset.</li>
312
+ <br>
313
+ <li><em>Pose Estimation:</em> Please follow this <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services/tree/main/pose_estimation/src#1" target="_blank">link</a> to get the dataset.</li>
314
+ </ul>
315
+ </li>
316
+
317
+ <li><strong>Setup and launch your application:</strong> To launch one of the other use cases, first check the download_datasets.py script and make sure you have the link corresponding to your dataset to download. We provided links the script for datasets mentioned in the previous table.<br>
318
+ These datasets are downloaded automatically and do not require a data preparation phase as mentioned for Object Detection, Pose Segmentation, and Semantic Segmentation use cases.</li>
319
+ <br>
320
+ <strong>How to use the dashboard ? </strong><br>
321
+ <ol>
322
+ <li>Select the use case from the drop-down list.</li>
323
+ <br>
324
+ <li>A corresponding YAML configuration file will open, and you'll need to configure it to launch a training session.<br>
325
+ If you're using a use case with one of the datasets handled by the download_datasets.py script, go to the Dataset section in the YAML file and configure the path to your data accordingly : <em>../datasets/your_use_case/name_of_dataset</em><br>
326
+ <br>
327
+ For example, if you selected Image classification use case with flowers dataset the path should be this way: <em>../datasets/image_classification/flowers_photo</em> <br>
328
+ <br>
329
+ For the other use cases that need data preparation, we recommend you place your dataset in the datasets folder and set your path accordingly: <em>datasets/name_of_your_dataset</em>
330
+ </li>
331
+ <br>
332
+ <li>Submit your modifications and insert your ST Edge AI Developer cloud to benefit from the whole experience.<br>
333
+ Your training session will start, allowing you to visualize output logs and metrics plots. Finally, you can download your experiment outputs.<br>
334
+ </li>
335
+ </ol>
336
+
337
+ <p> If you need to explore, evaluate, and benchmark pre-trained models, you can use <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo" target="_blank">stm32ai-modelzoo</a></p>
338
+ <br>
339
+
340
+ <em> Note: </em>
341
+ <li>Deployment on edge is not supported.</li>
342
+ <li>If you need to reproduce the same experience provided in the model zoo, please refer to <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo" target="_blank">stm32ai-modelzoo</a> </li>
343
+ <li>If you want to discover different services offered by the model zoo, please refer to <a href="https://github.com/STMicroelectronics/stm32ai-modelzoo-services" target="_blank">stm32ai-modelzoo-services</a> </li></li>
344
+ <li> If you need to use your own model, please add it to the models folder, rebuild the Docker image, and then update the model path in the user configuration YAML file.</li>
345
+ </p>
346
+ </p>
347
+ </section>
348
+
349
+ <section id="StartApplication">
350
+ <h2>How to Start the Application</h2>
351
+ <p>
352
+ Once the space has been duplicated from the home page, you need to update the script <em>download_datasets.py</em> script that will allow you to download and unarchive the dataset corresponding to your use case.
353
+ </p>
354
+ <br>
355
+ {% if duplicate_mode %}
356
+ <h3>Duplicate this Space</h3>
357
+ <p style="color: #03234b;">
358
+ To start using this application, duplicate this space to your own Hugging Face account.
359
+ </p>
360
+ <a href="https://huggingface.co/spaces/STMicroelectronics/stm32-modelzoo-app?duplicate=true" target="_blank" style="background-color: #03234b; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px; border: 2px solid #03234b; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
361
+ Duplicate this Space
362
+ </a>
363
+ {% else %}
364
+ <h3>Launch the Dashboard</h3>
365
+ <p>
366
+ You have duplicated the space. Now you can launch the interactive dashboard below.
367
+ </p>
368
+ <a href="/dash_app/" target="_blank" style="background-color: #03234b; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px; border: 2px solid #03234b; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
369
+ Launch Dashboard
370
+ </a>
371
+ {% endif %}
372
+ </section>
373
+
374
+ </div>
375
+ </div>
376
+
377
+ </body>
378
+ </html>