Terminal Tricks For Machine Learning Researchers
NVIDIA GPU monitoring
Machine Learning algorithms need plenty of parallel computation and the most common way of doing it lightning-fast is using NVIDIA GPUs. While using NVIDIA GPUs we need to monitor the usage of GPU on the terminal. The easiest way to monitor it is by the following command:
In case we want to check GPU usage regularly one can use the following command to refresh the GPU usage information every second:
watch -n 1 nvidia-smi
When you add the following bash script to the bottom of ~/.bashrc you set watch_gpu as an alias for compact GPU monitoring:
alias watch_gpu="watch -t -n 1 'nvidia-smi | grep -oP \" *[0-9]+MiB \/ *[0-9]+MiB \| *[0-9]+\%\" '"
after adding this command to the ~/.bashrc, use source ~/.bashrc to load the recently updated bashrc. Then when you run watch_gpu in your terminal you will be able to see the following compact GPU usage information on your terminal.
In our case, we have 4 GPUs and as you can see from the figure we have 4 rows. Out of 11178 MiB, we use 0 MiB which means we don't use any of the GPUs yet and all 4 GPUs are free to use. In the last column, the percentage shows how much of the GPU is running now. Observe that GPU memory usage and GPU usage are not the same concepts.
Setting a specific GPU to use
In case we use a multiple GPU machine and we want to utilize specific GPUs for a given task, we need to specify it before running our script with the following command:
Here we tell our machine to use only GPUs indexed by 0 and 1.
We connect to the server with ssh commands. SSH is known as Secure Shell or Secure Socket Shell, is a network protocol that gives users, particularly system administrators, a secure way to access a computer over an unsecured network. Connecting to a server while mapping a specific port to a local machine port is as simple as adding a couple of words to your command. In the following command, we map port 8888 of the server machine to port 1234 of our local machine.
This way we can monitor and play with Jupyter notebook, tensorboard, etc useful tools on our local machine while the actual job is done over the server machine.
Copy a file to server:
scp -r ~/some/folder user@server:~/to/where
Copy a file from server to local machine:
scp -r user@server:~/to/where ~/some/folder
You can also download the stuff from an online source utilizing wget command:
You should use the corresponding terminal of the machine. If you want to download online resource into the server then use the command in the server terminal.
Mount a remote folder to a local folder:
In order to mount a folder from the remote server to a local folder, first, create a free folder. Then run following command on your local machine's terminal:
Tmux is a terminal multiplier where you can create multiple panes. The main advantage of the "tmux" is that it allows users to run their codes in the background and even if they lose connection to the server, the code will be running. The next time you connect you can check out the results of your code by just attaching the correct tmux session. Another advantage is since the session is an independent terminal session it will keep the settings and environment variables as they are, therefore, you do not need to do manual work over and over again.
First of all, install tmux:
Ubuntu and Debian:
CentOS and Fedora:
I have a configuration that I find very useful, which basically converts tmux configuration similar to gnu screen configuration. In order to use the settings;
Create a .tmux.conf file and then copy-paste the following code to the file:
Create a tmux session:
tmux new -s session_name
Attach to an already created session:
tmux a -t session_name
Kill a session:
tmux kill-session -t session_name
Cheatsheet for shortcuts:
(ctrl + a) + | :Divide pane
(ctrl + a) + - :Divide pane
(ctrl + a) + d :Detach session
(ctrl + a) + z :Full-screen the current pane / minimize from full screen
for further: https://tmuxcheatsheet.com
Change the appearance of the terminal line:
export PS1="\u@MacBook 🤓 \W $ "
export PS1="\[\033[1;33m\]\u@MacBook 🤓 \W $ \[\033[00m\] "
Add new paths to the PATH: