We describe how to use pre built and pre configured packages of PCP, flame graph plugin and customized perf agent to generate flame graphs of Java and system calls. We will be making use of Vector which is optional.

A sample flame graph obtained by following this post:

javaflames

Integrating flame graph extension with Performance Co-pilot(PCP):

In-order to generate flame graphs we need to integrate flame graph extension with PCP.

To profile java method calls using flame graphs we need additional component – an agent that will attach to JVM and download mappings of memory address to method names, we use pref-map-agent.

This is modular and needs manual work to configure and make them work together, so to save time and effort, we have made the necessary changes and created packages installable from PPA on Ubuntu 14.04 and 16.04.

Prerequisites:

The PCP and plugins we configured need the dependencies listed below

  1. Java 8 update 60 (or higher)
  2. perf
  3. cmake
  4. pkg-config
  5. libmicrohttpd10

Java needs to be running with the -XX:+PreserveFramePointer.

You can set the environment variable command to run java with -XX:+PreserveFramePointer

export JAVA_OPTIONS=”-XX:+StartAttachListener -XX:+PreserveFramePointer”

Use the below command to install the dependencies, we are using OpenJDK, however,  relevant version of Oracle Java could be used.

sudo apt-get install openjdk-8-jdk pkg-config cmake  linux-tools-$(uname -r) libmicrohttpd-dev

Install PCP

sudo add-apt-repository ppa:anil400-yadav/pcp-3.11.4-1

sudo apt-get update

In order to make the flame graph extension work with PCP we will install pcp development headers files along with pcp and pcp-webapi.

sudo apt-get install pcp pcp-webapi libpcp-pmda3-dev

Note: The libpcp-pmda-dev package contains the Performance Co-Pilot (PCP) library and header files needed to develop Performance Metric Domain Agents (PMDAs). Each PMDA exports performance data for one domain; for example the operating system kernel, Cisco routers, a database, an application, etc.

After installing PCP make sure the services pmcd and pmwebd are running by using below commands

service pmcd status

service pmwebd status

Install perf-map-agent to translate java symbols

How Perf-map-agent works:

To know how does the perf-map-agent work please go through the below link.

https://github.com/jrudolph/perf-map-agent

Before installing perf-map-agent make sure JAVA_HOME is configured to point to a JDK.

You need to add ppa:ayv/perf-map-agent-1.0.1 to your system

sudo add-apt-repository ppa:ayv/perf-map-agent-1.0.1

sudo apt-get update
sudo apt-get install perf-map-agent

How to determine whether your setup is working?

After running the systack.sh script which is under your generic directory you can check the files which has been generated under /tmp directory like /tmp/perf-<pid>.map should contain the java symbols. If the /tmp/perf-<pid>.map file is empty then think that something went wrong with your setup.

Visualising flame graphs:

You can access the svg file on a browser without additional software, to access the svg file on a browser:

Step 1: move to your generic directory which will be under /usr/lib/pcp/pmdas/generic and execute the systack.sh script as root user

sudo sh systack.sh

Step 2: go to the directory /usr/share/pcp/webapps/systack, a file called systack.svg will be created

Access the the svg file using your hostname or ip address and verify the java calls are being monitored or not.

http://hostname/systack/systack.svg

Optional: Using vector to visualize and analyze the metrics

Vector is an open source host level performance monitoring framework which exposes hand picked, high resolution system and application metrics to the browser. Vector provides a simple way for users to visualize system level metrics in real time.

Install Vector in the directory /var/www/html. INstall vector on any web-server like apache or nginx

– cd /var/www/html

– sudo wget https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz

– sudo tar xvf vector.tar.gz

Now just open your browser and point to your web-server. you will be able to access Vector.

Once you are done with Vector configuration you have to enable flame graphs in file /var/www/html/maps/scripts/app-cc8970137d.js.map

enableCpuFlameGraph : true

Now restart web-server and try to access vector on your browser, now you will be able to see the option CPU Flame Graph

On Vector dashboard click on widget and hover on CPU now options will be displayed along with CPU Flame Graph.