Project

General

Profile

ML beam test PID Meetings » History » Version 52

Richard Trotta, 06/03/2024 11:16 AM

1 1 Richard Trotta
{{>toc}}
2
3
h1. ML beam test PID Meetings
4
5 3 Richard Trotta
---
6
7 1 Richard Trotta
h2. Summer 2024
8 4 Richard Trotta
9
---
10
11 25 Richard Trotta
h3. May 28th, 2024
12 1 Richard Trotta
13
* Darren ran with Python3.9
14
* Docker (containerization) Definition
15
** Containerization is a technology that allows developers to package and run applications along with all their dependencies in isolated environments called containers. This ensures that the application runs consistently across different computing environments, from a developer's laptop to testing, staging, and production.
16
** Docker is a popular platform that simplifies containerization. It provides tools to create, deploy, and manage containers. With Docker, developers can write code locally, share their work with colleagues, and deploy to production in a seamless and efficient manner. Docker containers are lightweight, fast, and portable, making them ideal for modern software development and deployment.
17 9 Richard Trotta
18 39 Richard Trotta
* "A Crash Course for Summer Research":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing
19 17 Richard Trotta
* "GitHub and Python Introduction":https://github.com/trottar/UVA_summer_students
20 18 Richard Trotta
** Navigate to "python_tutorials":https://github.com/trottar/UVA_summer_students/tree/master/python_tutorials and read through the two html files ("Introduction":https://github.com/trottar/UVA_summer_students/blob/master/python_tutorials/Introduction.ipynb and "python_tutorial_2":https://github.com/trottar/UVA_summer_students/blob/master/python_tutorials/python_tutorial_2.ipynb)
21
*** Note, these are html files so you'll need 
22 8 Richard Trotta
23
h4. Near-term Goals
24
25
# Setup Linux Subsystem for Windows
26
# Fork the ML Beam Testing GitHub repository
27
# Containerize ML Beam Testing GitHub repository with Docker
28 5 Richard Trotta
29 14 Richard Trotta
h4. Homework
30
31 16 Richard Trotta
* "Read chapters 0-3 and finish exercises":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing
32 14 Richard Trotta
33 3 Richard Trotta
---
34 19 Richard Trotta
35 25 Richard Trotta
h3. May 29th, 2024
36 19 Richard Trotta
37
h4. Setting Up Jupyter Notebooks in Docker
38
39
# Create a directory to store, build, and run Docker container
40
	<pre><code class="bash">
41
cd /path/to/directory
42
mkdir beamtest_dir
43 20 Richard Trotta
cd beamtest_dir
44
</code></pre>
45 21 Richard Trotta
# Once in the new directory, create a Dockerfile
46 20 Richard Trotta
	<pre><code class="bash">
47
touch Dockerfile
48
</code></pre>
49 21 Richard Trotta
# Using the text editor (e.g., vim, gedit, or emacs) of your choice, edit the Dockerfile with the following information
50 20 Richard Trotta
	<pre><code class="bash">
51
# Use the official Python 3.9 image as the base image
52
FROM python:3.9
53
54
# Install Jupyter Notebook and other dependencies
55
RUN pip install --no-cache-dir jupyter
56
57
# Create a working directory
58
WORKDIR /workspace
59
60
# Expose the Jupyter Notebook port
61
EXPOSE 8888
62
63
# Set the default command to start Jupyter Notebook
64 1 Richard Trotta
CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]
65
</code></pre>
66 21 Richard Trotta
# Build the Docker image (e.g., helloworld)
67
	<pre><code class="bash">
68
docker build -t helloworld .
69
</code></pre>
70
# Run the Docker container
71
</code></pre>
72
	<pre><code class="bash">
73
docker run -p 8888:8888 helloworld
74
</code></pre>
75
# Jupyter Notebook is now running. Navigate to a browser and type in either
76 23 Richard Trotta
77 21 Richard Trotta
* The custom token authorization screen</code></pre>
78
	<pre><code class="bash">
79
http://localhost:8888
80
</code></pre>
81 22 Richard Trotta
82 21 Richard Trotta
* To bypass the token screen, copy/paste the URL that splashes in the terminal into your browser
83 24 Richard Trotta
84
h4. Near-term Goals
85
86
# Try and get the "python tutorial":https://github.com/trottar/UVA_summer_students Jupyter notebooks working in a container.
87
88
h4. Homework
89
90
* "Read chapters 0-3 and finish exercises":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing
91 27 Richard Trotta
92
---
93
94
h3. May 30th, 2024
95
96
h4. Setting Up Bash Script for Running Docker
97
98 28 Richard Trotta
# Create a bash script
99
	<pre><code class="bash">
100
touch run_docker.sh
101
</code></pre>
102 29 Richard Trotta
# Using the text editor (e.g., vim, gedit, or emacs) of your choice, edit run_docker.sh
103
	<pre><code class="bash">
104
#!/bin/bash
105 1 Richard Trotta
106 29 Richard Trotta
# Build docker image ($1 is the first bash script argument)
107 37 Richard Trotta
docker build -t $1 .
108 29 Richard Trotta
109
# Run the docker container (renaming it with the '_container' string)
110
docker run -p 8888:8888 --name "${1}_container" $1
111
</code></pre>
112
# Adjust permissions with "chmod":https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/how-permissions-chmod-with-numbers-command-explained-777-rwx-unix to allow script execution
113
	<pre><code class="bash">
114
chmod 755 run_docker.sh
115
</code></pre>
116 31 Richard Trotta
# Execute run_docker.sh with container name (e.g., helloworld)
117
	<pre><code class="bash">
118
./run_docker.sh helloworld
119
</code></pre>
120 1 Richard Trotta
121 31 Richard Trotta
The docker image should be properly built and the container should be running. Follow instructions from "yesterday":https://redmine.jlab.org/projects/uva-phys-zheng/wiki/ML_beam_test_PID_Meetings#May-29th-2024 to continue.
122
123
h4. Setting Up Bash Script for committing Docker container
124
125 32 Richard Trotta
This should be done while a docker container is running and performed in another terminal.
126
127 31 Richard Trotta
# Create a bash script
128
	<pre><code class="bash">
129
touch commit_docker.sh
130
</code></pre>
131
# Using the text editor (e.g., vim, gedit, or emacs) of your choice, edit commit_docker.sh
132
	<pre><code class="bash">
133
#!/bin/bash
134
135
# Create a formatted string with date to organize committed containers
136
# - `date` is a command that prints or sets the system date and time.
137
# - `+%H` extracts the hour in 24-hour format.
138
# - `+%M` extracts the minute.
139
# - `+%S` extracts the second.
140
# - `+%Y` extracts the year.
141
# - `+%m` extracts the month.
142
# - `%d` extracts the day.
143
f_date=$(date +%Y-%m-%d_h%Hm%Ms%S)
144
145
# While a docker container is running (with root name given as first argument), 
146
# commit to a dated container so that changes in the container are saved
147 38 Richard Trotta
docker commit "${1}_container" "${1}_${f_date}"
148 32 Richard Trotta
</code></pre>
149
# Adjust permissions with "chmod":https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/how-permissions-chmod-with-numbers-command-explained-777-rwx-unix to allow script execution
150
	<pre><code class="bash">
151
chmod 755 commit_docker.sh
152
</code></pre>
153
# Execute run_docker.sh with container name (e.g., helloworld)
154
	<pre><code class="bash">
155
./commit_docker.sh helloworld
156 31 Richard Trotta
</code></pre>
157 28 Richard Trotta
158
h4. Useful Docker Commands
159 27 Richard Trotta
160
* To see all docker images
161
	<pre><code class="bash">
162
docker images
163
</code></pre>
164
* To see all docker containers running
165
	<pre><code class="bash">
166
docker ps
167
</code></pre>
168
* To end specific docker container
169
	<pre><code class="bash">
170 34 Richard Trotta
docker stop <container_id>
171 27 Richard Trotta
</code></pre>
172
* To end all docker container
173
	<pre><code class="bash">
174 36 Richard Trotta
docker stop $(docker ps -q)
175 27 Richard Trotta
</code></pre>
176 33 Richard Trotta
177
h4. Near-term Goals
178
179
# Try and get the "python tutorial":https://github.com/trottar/UVA_summer_students Jupyter notebooks working in a container.
180
# Take a look at the "GitHub ML Beam Test Repository":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/tree/main/aiml
181
* In particular, look at the script "solid_beamtest_hallc_2022/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/blob/main/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb
182
** Try to get this notebook working in a container by creating a requirements.txt file (use this "site as a general guide":https://www.docker.com/blog/containerized-python-development-part-1/)
183
** We will cover this in detail tomorrow
184
185
h4. Homework
186
187
* "Read chapters 0-3 and finish exercises":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing
188 40 Richard Trotta
189
---
190
191
h3. May 31st, 2024
192
193
h4. Clone a git repository into the Docker container by adding a line to the Dockerfile
194
195 41 Richard Trotta
* Using the text editor (e.g., vim, gedit, or emacs) of your choice, edit the Dockerfile with the following information (the python tutorial GitHub repo is used as an example)
196 40 Richard Trotta
	<pre><code class="bash">
197
# Use the official Python 3.9 image as the base image
198
FROM python:3.9
199
200
# Install Jupyter Notebook and other dependencies
201
RUN pip install --no-cache-dir jupyter
202
203
# Create a working directory
204
WORKDIR /workspace
205
206
# Clone GitHub repo
207 1 Richard Trotta
RUN git clone https://github.com/trottar/UVA_summer_students.git
208 41 Richard Trotta
209
# Expose the Jupyter Notebook port
210
EXPOSE 8888
211
212
# Set the default command to start Jupyter Notebook
213
CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]
214
</code></pre>
215
216
h4. Quick requirements.txt introduction
217
218
* requirements.txt is a way to easily install many python packages by recursively reading through a text file
219 43 Richard Trotta
* This text file can either be generated...
220
** Manually, by writing the required python packages into the text file.
221
*** Note: it is good practice to specify the version of the package. Such as...
222
	<pre><code class="bash">
223
matplotlib==3.3.4
224
</code></pre>
225
** Automatically, by running the following command
226
	<pre><code class="bash">
227
pip freeze > requirements.txt
228
</code></pre>
229 47 Richard Trotta
*** *Note:* This command will include all packages installed.
230 43 Richard Trotta
231 41 Richard Trotta
* A Docker container can install all these packages upon building its image by adding a few lines to the Dockerfile
232
* Using the text editor (e.g., vim, gedit, or emacs) of your choice, edit the Dockerfile with the following information
233 42 Richard Trotta
	<pre><code class="bash">
234 41 Richard Trotta
# Use the official Python 3.9 image as the base image
235
FROM python:3.9
236
237 46 Richard Trotta
# Use the most up-to-date version of pip3.9
238
RUN pip install --upgrade pip
239 1 Richard Trotta
240 46 Richard Trotta
# Install git and ssh
241
RUN apt-get update && apt-get install -y git openssh-client
242
243 1 Richard Trotta
# Create a working directory
244 41 Richard Trotta
WORKDIR /workspace
245
246 46 Richard Trotta
# Clone GitHub
247
RUN git clone https://github.com/trottar/solid_beamtest_hallc_2022.git
248 41 Richard Trotta
249
# copy the dependencies file to the working directory
250
COPY requirements.txt .
251
252
# install dependencies
253
RUN pip install -r requirements.txt
254 40 Richard Trotta
255
# Expose the Jupyter Notebook port
256
EXPOSE 8888
257 1 Richard Trotta
258
# Set the default command to start Jupyter Notebook
259
CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]
260
</code></pre>
261 46 Richard Trotta
262
* *Note:* I also made some other adjustments such as...
263
** Moved the pip installation of jupyter directly into the requirements.txt
264
** Added an installation for git and ssh to avoid completely clean image creations. 
265
** Added a pip install --upgrade pip to assure pip is the latest version for python3.9
266 40 Richard Trotta
267
h4. Useful Docker Commands
268
269
* Remove all stopped containers (use with caution)
270
	<pre><code class="bash">
271 45 Richard Trotta
docker system prune -a
272 40 Richard Trotta
</code></pre>
273
274
h4. Near-term Goals
275
276
# Take a look at the "GitHub ML Beam Test Repository":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/tree/main/aiml
277
* In particular, look at the script "solid_beamtest_hallc_2022/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/blob/main/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb
278
** Try to get this notebook working in a container by creating a requirements.txt file (use this "site as a general guide":https://www.docker.com/blog/containerized-python-development-part-1/)
279
280
h4. Homework
281
282
* "Read chapters 0-3 and finish exercises":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing
283 48 Richard Trotta
284
---
285
286
h3. June 3rd, 2024
287
288
h4. First Time General Docker Procedure
289
290
# Clear out previous images and containers
291
	<pre><code class="bash">
292
docker system prune -a
293
</code></pre>
294
# Build the Docker image and run the container with bash script
295
	<pre><code class="bash">
296
./run_docker.sh beamtest
297
</code></pre>
298
# While the container is running, create snapshots of changes by committing
299
	<pre><code class="bash">
300
./commit_docker.sh beamtest
301
</code></pre>
302 49 Richard Trotta
# *IMPORTANT* Properly stop a container by checking its ID with _docker ps_ then 
303 48 Richard Trotta
	<pre><code class="bash">
304
docker stop <contatiner_ID>
305
</code></pre>
306
307
h4. Running a snapshot
308
309
* Snapshots are built images so no rebuilding required
310
* To run a container of a snapshot...
311
# Check out all built images/snapshots
312
	<pre><code class="bash">
313 50 Richard Trotta
docker images
314 48 Richard Trotta
</code></pre>
315
# Run the specific snapshot
316
	<pre><code class="bash">
317
docker run -p 8888:8888 <snapshot_name>
318
</code></pre>
319
# While the container of snapshot is running, create further snapshots of changes by committing
320
	<pre><code class="bash">
321
./commit_docker.sh beamtest
322
</code></pre>
323 51 Richard Trotta
# *IMPORTANT* Properly stop a container by checking its ID with _docker ps_ then 
324 48 Richard Trotta
	<pre><code class="bash">
325
docker stop <contatiner_ID>
326
</code></pre>
327
328
h4. Near-term Goals
329
330
# Take a look at the "GitHub ML Beam Test Repository":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/tree/main/aiml
331
* In particular, look at the script "solid_beamtest_hallc_2022/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb":https://github.com/JeffersonLab/solid_beamtest_hallc_2022/blob/main/aiml/Pencil_Beam/Pencil_Cher_ML.ipynb
332
** Try to get this notebook working in a container by creating a requirements.txt file (use this "site as a general guide":https://www.docker.com/blog/containerized-python-development-part-1/)
333
* Compare the plots in this notebook with those in "Darren's report":https://wordpress.its.virginia.edu/zhenggroup/files/2023/10/SoLID_beamtest_ML_PID_Upton.pdf to try and get an understanding.
334
335
h4. Homework
336
337 52 Richard Trotta
* "Read chapters 0-4 and finish exercises":https://drive.google.com/file/d/1-TPLo5VGSwCUDjqj_j4g--p4FbBgYHbw/view?usp=sharing