2019년 11월 29일 금요일

Monitoring Jetson Series

I used Jetson Nano, Jetson TX2, Xavier NX, Ubuntu 18.04 Official image with root account.

Traditional Monitoring

How do you monitor Jetson? Do you use top or htop?

top

Top is a traditional command-line tool for monitoring real-time processes in a Unix/Linux systems, it’s comes preinstalled on most if not all Linux distributions and shows a useful summary of system information including uptime, total number of processes (and number of: running, sleeping, stopped and zombie processes), CPU and RAM usage, and a list of processes or threads currently being managed by the kernel.


htop

htop is an interactive, ncurses-based processes viewer for Linux systems. It is practically a top-like tool, but it displays colorful text, and uses ncurses to implement a text-graphical interface, and allows for output scrolling. It doesn’t come preinstalled on most mainstream Linux distributions.


  • It has a nicer text-graphics interface, with colored output.
  • It is easy to use and highly configurable.
  • Allows for scrolling process list vertically and horizontally to see all processes and complete command lines.
  • It also displays a process tree and comes with mouse support.
  • Allows you to easily perform certain functions related to processes (killing, renicing etc) which can be done without entering their PIDs.
  • htop is also much faster than top.


Jetson monitoring

If you're monitoring Jetson, you're most interested in CPU, memory, and GPU usage. Yes, GPU... Top or htop can't monitor Jetson's GPU information. Therefore, new tools are needed to effectively monitor Jetson series.

tegrastats

The tools to effectively monitor the GPUs of the Nvidia Jetson series are tegrastats.

spypiggy@spypiggy-desktop:~$ tegrastats --help
Usage: tegrastats [-option]
Options:
    --help                  : print this help screen
    --interval <millisec>   : sample the information in <milliseconds>
    --logfile  <filename>   : dump the output of tegrastats to <filename>
    --load_cfg <filename>   : load the information from <filename>
    --save_cfg <filename>   : save the information to <filename>
    --start                 : run tegrastats as a daemon process in the background
    --stop                  : stop any running instances of tegrastats
    --verbose               : print verbose message


Let's run the tegrastats command.

spypiggy@spypiggy-desktop:~$ tegrastats
RAM 385/3964MB (lfb 721x4MB) SWAP 0/8126MB (cached 0MB) CPU [2%@102,1%@102,0%@102,0%@102] EMC_FREQ 0% GR3D_FREQ 0% PLL@20.5C CPU@22.5C PMIC@100C GPU@22.5C AO@26C thermal@22.5C POM_5V_IN 825/825 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 385/3964MB (lfb 721x4MB) SWAP 0/8126MB (cached 0MB) CPU [2%@102,6%@102,1%@102,2%@102] EMC_FREQ 0% GR3D_FREQ 0% PLL@20.5C CPU@22.5C PMIC@100C GPU@22.5C AO@26C thermal@22.5C POM_5V_IN 865/845 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 385/3964MB (lfb 721x4MB) SWAP 0/8126MB (cached 0MB) CPU [3%@102,2%@102,1%@102,0%@102] EMC_FREQ 0% GR3D_FREQ 0% PLL@20.5C CPU@22.5C PMIC@100C GPU@22.5C AO@26.5C thermal@22.75C POM_5V_IN 825/838 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 385/3964MB (lfb 721x4MB) SWAP 0/8126MB (cached 0MB) CPU [3%@102,1%@102,1%@102,0%@102] EMC_FREQ 0% GR3D_FREQ 0% PLL@20.5C CPU@22.5C PMIC@100C GPU@22.5C AO@26.5C thermal@22.5C POM_5V_IN 825/835 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 385/3964MB (lfb 721x4MB) SWAP 0/8126MB (cached 0MB) CPU [3%@102,1%@102,0%@102,0%@102] EMC_FREQ 0% GR3D_FREQ 0% PLL@20C CPU@22.5C PMIC@100C GPU@22C AO@26.5C thermal@22.5C POM_5V_IN 825/833 POM_5V_GPU 0/0 POM_5V_CPU 123/123
^C
spypiggy@spypiggy-desktop:~$


Information does not come at a glance. An important piece of information about the GPU is GR3D_FREQ. In the above output this value is zero. Therefore, it can be estimated that there is little GPU usage.

Jetson stats - ( TUI version of tegrastats ? )

There is a tool jtop that makes tegrastat similar to htop. When jetson-stats is installed, the jetson_stats service is registered. After starting this service, the jtop command is available. Therefore, after installing jetson-stats, execute the service first or execute the jto command after rebooting.

root@JetsonNano:~# pip3 install -U jetson-stats
root@JetsonNano:~# systemctl restart jetson_stats.service
root@JetsonNano:~# systemctl list-units --type service --all
  UNIT                                     LOAD      ACTIVE   SUB     DESCRIPTION
  accounts-daemon.service                  loaded    active   running Accounts Service
  alsa-restore.service                     loaded    active   exited  Save/Restore Sound Card State
  alsa-state.service                       loaded    inactive dead    Manage Sound Card State (restore and store)
  anacron.service                          loaded    inactive dead    Run anacron jobs
  apparmor.service                         loaded    inactive dead    AppArmor initialization
  apport-autoreport.service                loaded    inactive dead    Process error reports when automatic reporting is enabled
  apport.service                           loaded    active   exited  LSB: automatic crash report generation
  apt-daily-upgrade.service                loaded    inactive dead    Daily apt upgrade and clean activities
  apt-daily.service                        loaded    inactive dead    Daily apt download activities
● auditd.service                           not-found inactive dead    auditd.service
  avahi-daemon.service                     loaded    active   running Avahi mDNS/DNS-SD Stack
  bluetooth.service                        loaded    active   running Bluetooth service
  bolt.service                             loaded    active   running Thunderbolt system service
● chronyd.service                          not-found inactive dead    chronyd.service
  colord.service                           loaded    active   running Manage, Install and Generate Color Profiles
● connman.service                          not-found inactive dead    connman.service
● console-screen.service                   not-found inactive dead    console-screen.service
  console-setup.service                    loaded    active   exited  Set console font and keymap
  containerd.service                       loaded    active   running containerd container runtime
  cron.service                             loaded    active   running Regular background program processing daemon
  dbus.service                             loaded    active   running D-Bus System Message Bus
  dns-clean.service                        loaded    inactive dead    Clean up any mess left by 0dns-up
  docker.service                           loaded    inactive dead    Docker Application Container Engine
  emergency.service                        loaded    inactive dead    Emergency Shell
● festival.service                         not-found inactive dead    festival.service
● firewalld.service                        not-found inactive dead    firewalld.service
  fstrim.service                           loaded    inactive dead    Discard unused blocks
  gdm.service                              loaded    active   running GNOME Display Manager
  getty-static.service                     loaded    inactive dead    getty on tty2-tty6 if dbus and logind are not available
  getty@tty1.service                       loaded    inactive dead    Getty on tty1
  getty@tty7.service                       loaded    inactive dead    Getty on tty7
  gpsd.service                             loaded    inactive dead    GPS (Global Positioning System) Daemon
  grub-common.service                      loaded    active   exited  LSB: Record successful boot for GRUB
  haveged.service                          loaded    active   running Entropy daemon using the HAVEGE algorithm
  jetson_stats.service                loaded   active  running jetson_stats service
● kbd.service                              not-found inactive dead    kbd.service
  kerneloops.service                       loaded    active   running Tool to automatically collect and submit kernel crash signatures
  keyboard-setup.service                   loaded    active   exited  Set the console keyboard layout

root@JetsonNano:~# jtop 



<jtop>













댓글 없음:

댓글 쓰기