Terminal Tricks For Machine Learning Researchers

NVIDIA GPU monitoring

Machine Learning algorithms need plenty of parallel computation and the most common way of doing it lightning-fast is using NVIDIA GPUs. While using NVIDIA GPUs we need to monitor the usage of GPU on the terminal. The easiest way to monitor it is by the following command:

nvidia-smi

In case we want to check GPU usage regularly one can use the following command to refresh the GPU usage information every second:

watch -n 1 nvidia-smi

Advanced Monitoring

When you add the following bash script to the bottom of ~/.bashrc you set watch_gpu as an alias for compact GPU monitoring:

alias watch_gpu="watch -t -n 1 'nvidia-smi | grep -oP \" *[0-9]+MiB \/ *[0-9]+MiB \| *[0-9]+\%\" '"

after adding this command to the ~/.bashrc, use source ~/.bashrc to load the recently updated bashrc. Then when you run watch_gpu in your terminal you will be able to see the following compact GPU usage information on your terminal.

In our case, we have 4 GPUs and as you can see from the figure we have 4 rows. Out of 11178 MiB, we use 0 MiB which means we don't use any of the GPUs yet and all 4 GPUs are free to use. In the last column, the percentage shows how much of the GPU is running now. Observe that GPU memory usage and GPU usage are not the same concepts.

Setting a specific GPU to use

In case we use a multiple GPU machine and we want to utilize specific GPUs for a given task, we need to specify it before running our script with the following command:

export CUDA_VISIBLE_DEVICES=0,1

Here we tell our machine to use only GPUs indexed by 0 and 1.

Server Side

We connect to the server with ssh commands. SSH is known as Secure Shell or Secure Socket Shell, is a network protocol that gives users, particularly system administrators, a secure way to access a computer over an unsecured network. Connecting to a server while mapping a specific port to a local machine port is as simple as adding a couple of words to your command. In the following command, we map port 8888 of the server machine to port 1234 of our local machine.

ssh -L 1234:127.0.0.1:8888 user_name@server_address

This way we can monitor and play with Jupyter notebook, tensorboard, etc useful tools on our local machine while the actual job is done over the server machine.

Copy a file to server:

scp -r ~/some/folder user@server:~/to/where

Copy a file from server to local machine:

scp -r user@server:~/to/where ~/some/folder

You can also download the stuff from an online source utilizing wget command:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

You should use the corresponding terminal of the machine. If you want to download online resource into the server then use the command in the server terminal.

Mount a remote folder to a local folder:

In order to mount a folder from the remote server to a local folder, first, create a free folder. Then run following command on your local machine's terminal:

sshfs -o volname=any_name -o follow_symlinks username@server:/remote/folder /local/machine/free_folder

Tmux

Tmux is a terminal multiplier where you can create multiple panes. The main advantage of the "tmux" is that it allows users to run their codes in the background and even if they lose connection to the server, the code will be running. The next time you connect you can check out the results of your code by just attaching the correct tmux session. Another advantage is since the session is an independent terminal session it will keep the settings and environment variables as they are, therefore, you do not need to do manual work over and over again.

First of all, install tmux:

Ubuntu and Debian:

sudo apt install tmux

CentOS and Fedora:

sudo yum install tmux

macOS:

brew install tmux

I have a configuration that I find very useful, which basically converts tmux configuration similar to gnu screen configuration. In order to use the settings;

Create a .tmux.conf file and then copy-paste the following code to the file:

# remap prefix from 'C-b' to 'C-a'

unbind C-b

set-option -g prefix C-a

bind-key C-a send-prefix

# split panes using | and -

bind | split-window -h

bind - split-window -v

unbind '"'

unbind %

# switch panes using Alt-arrow without prefix

bind -n M-Left select-pane -L

bind -n M-Right select-pane -R

bind -n M-Up select-pane -U

bind -n M-Down select-pane -D

# Enable mouse control (clickable windows, panes, resize

set-option -g mouse on

# make scrolling with wheels work

bind -n WheelUpPane if-shell -F -t = "#{mouse_any_flag}" "send-keys -M" "if -Ft= '#{pane_in_mode}' 'send-keys -M' 'select-pane -t=; copy-mode -e; send-keys -M'"

bind -n WheelDownPane select-pane -t= \; send-keys -M

# reload config file (change file location to your the tmux.conf you want to use)

bind r source-file ~/.tmux.conf

# don't rename windows automatically

set-option -g allow-rename off

set-option -g history-limit 10000

Create a tmux session:

tmux new -s session_name

Attach to an already created session:

tmux a -t session_name

Kill a session:

tmux kill-session -t session_name

Cheatsheet for shortcuts:

• (ctrl + a) + | :Divide pane

• (ctrl + a) + - :Divide pane

• (ctrl + a) + d :Detach session

• (ctrl + a) + z :Full-screen the current pane / minimize from full screen

• for further: https://tmuxcheatsheet.com

Bash Profile

Change the appearance of the terminal line:

export PS1="\u@MacBook 🤓 \W $ "

export PS1="\[\033[1;33m\]\u@MacBook 🤓 \W $ \[\033[00m\] "

Add new paths to the PATH:

export PATH="/some/path/:$PATH"