We describe how to use pre built and pre configured packages of PCP, flame graph plugin and customized perf agent to generate flame graphs of Java and system calls. We will be making use of Vector which is optional.
A sample flame graph obtained by following this post:
Integrating flame graph extension with Performance Co-pilot(PCP):
In-order to generate flame graphs we need to integrate flame graph extension with PCP.
To profile java method calls using flame graphs we need additional component – an agent that will attach to JVM and download mappings of memory address to method names, we use pref-map-agent.
This is modular and needs manual work to configure and make them work together, so to save time and effort, we have made the necessary changes and created packages installable from PPA on Ubuntu 14.04 and 16.04.
The PCP and plugins we configured need the dependencies listed below
- Java 8 update 60 (or higher)
Java needs to be running with the -XX:+PreserveFramePointer.
You can set the environment variable command to run java with -XX:+PreserveFramePointer
export JAVA_OPTIONS=”-XX:+StartAttachListener -XX:+PreserveFramePointer”
Use the below command to install the dependencies, we are using OpenJDK, however, relevant version of Oracle Java could be used.
sudo apt-get install openjdk-8-jdk pkg-config cmake linux-tools-$(uname -r) libmicrohttpd-dev
sudo add-apt-repository ppa:anil400-yadav/pcp-3.11.4-1
sudo apt-get update
In order to make the flame graph extension work with PCP we will install pcp development headers files along with pcp and pcp-webapi.
sudo apt-get install pcp pcp-webapi libpcp-pmda3-dev
Note: The libpcp-pmda-dev package contains the Performance Co-Pilot (PCP) library and header files needed to develop Performance Metric Domain Agents (PMDAs). Each PMDA exports performance data for one domain; for example the operating system kernel, Cisco routers, a database, an application, etc.
After installing PCP make sure the services pmcd and pmwebd are running by using below commands
service pmcd status
service pmwebd status
Install perf-map-agent to translate java symbols
How Perf-map-agent works:
To know how does the perf-map-agent work please go through the below link.
Before installing perf-map-agent make sure JAVA_HOME is configured to point to a JDK.
You need to add ppa:ayv/perf-map-agent-1.0.1 to your system
sudo add-apt-repository ppa:ayv/perf-map-agent-1.0.1
sudo apt-get update
sudo apt-get install perf-map-agent
How to determine whether your setup is working?
After running the systack.sh script which is under your generic directory you can check the files which has been generated under /tmp directory like /tmp/perf-<pid>.map should contain the java symbols. If the /tmp/perf-<pid>.map file is empty then think that something went wrong with your setup.
Visualising flame graphs:
You can access the svg file on a browser without additional software, to access the svg file on a browser:
Step 1: move to your generic directory which will be under /usr/lib/pcp/pmdas/generic and execute the systack.sh script as root user
sudo sh systack.sh
Step 2: go to the directory /usr/share/pcp/webapps/systack, a file called systack.svg will be created
Access the the svg file using your hostname or ip address and verify the java calls are being monitored or not.
Optional: Using vector to visualize and analyze the metrics
Vector is an open source host level performance monitoring framework which exposes hand picked, high resolution system and application metrics to the browser. Vector provides a simple way for users to visualize system level metrics in real time.
Install Vector in the directory /var/www/html. INstall vector on any web-server like apache or nginx
– cd /var/www/html
– sudo wget https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz
– sudo tar xvf vector.tar.gz
Now just open your browser and point to your web-server. you will be able to access Vector.
Once you are done with Vector configuration you have to enable flame graphs in file /var/www/html/maps/scripts/app-cc8970137d.js.map
enableCpuFlameGraph : true
Now restart web-server and try to access vector on your browser, now you will be able to see the option CPU Flame Graph
On Vector dashboard click on widget and hover on CPU now options will be displayed along with CPU Flame Graph.