Uber JVM Profiler实战

原创
2017/06/08 13:17
阅读数 334

背景

        日常工作经常需要监控一些java进程的内存、cpu资源使用情况,也就是jmx中的信息。一般的java应用都有暴露jmx的机制,然后由客户端连接进程信息查看。此种方式有个限制,就是你必须知道jmx的地址,然后才能连接,如果该应用是动态漂移的,比如docker、spark等分布式程序,每次运行的节点是变化的。这时不妨换个思路,由应用程序自主的push jmx信息出来,然后我们去一个集中的地方进行信息收集。

        uber的jvm profiler就是这样一个工程,这个项目之初是为了收集spark的jvm信息的,不过也可以在其他java进程上使用。ps:和jmxtrans有点类似,都是在服务端引用相关jar包,然后向外push数据,不过uber的jmx profile可以把数据push到redis、kafka、datadog、influxdb等系统中。

         jvm profile依赖jdk8,详细信息查看github

        由于实际工作中使用场景关系,本次实战tomcat和jar的使用方式。

下载

git clone https://github.com/uber-common/jvm-profiler.git

编译

mvn clean package -Dmaven.test.skip=true
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building uber-jvm-profiler 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ jvm-profiler ---
[INFO] Deleting /mnt/jvm-profiler/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ jvm-profiler ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /mnt/jvm-profiler/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ jvm-profiler ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 53 source files to /mnt/jvm-profiler/target/classes
[INFO] /mnt/jvm-profiler/src/main/java/com/uber/profiling/util/StringUtils.java: /mnt/jvm-profiler/src/main/java/com/uber/profiling/util/StringUtils.java uses or overrides a deprecated API.
[INFO] /mnt/jvm-profiler/src/main/java/com/uber/profiling/util/StringUtils.java: Recompile with -Xlint:deprecation for details.
[INFO] /mnt/jvm-profiler/src/main/java/com/uber/profiling/YamlConfigProvider.java: Some input files use unchecked or unsafe operations.
[INFO] /mnt/jvm-profiler/src/main/java/com/uber/profiling/YamlConfigProvider.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ jvm-profiler ---
[INFO] Not copying test resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ jvm-profiler ---
[INFO] Not compiling test sources
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ jvm-profiler ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ jvm-profiler ---
[INFO] Building jar: /mnt/jvm-profiler/target/jvm-profiler-1.0.0.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.1.0:shade (default) @ jvm-profiler ---
[INFO] Including org.apache.kafka:kafka-clients:jar:0.11.0.2 in the shaded jar.
[INFO] Including net.jpountz.lz4:lz4:jar:1.3.0 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.6 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.25 in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.5 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.8.11 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.8.11 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.8.0 in the shaded jar.
[INFO] Including org.javassist:javassist:jar:3.21.0-GA in the shaded jar.
[INFO] Including org.yaml:snakeyaml:jar:1.18 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.3.6 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.3.3 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.1.3 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.6 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /mnt/jvm-profiler/target/jvm-profiler-1.0.0.jar with /mnt/jvm-profiler/target/jvm-profiler-1.0.0-shaded.jar
[INFO] Dependency-reduced POM written at: /mnt/jvm-profiler/dependency-reduced-pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.168 s
[INFO] Finished at: 2019-05-07T15:46:03+08:00
[INFO] Final Memory: 29M/639M
[INFO] ------------------------------------------------------------------------

配置

    tomcat类型配置

-javaagent:/mnt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.FileOutputReporter,metricInterval=6000,ioProfiling=true,tag=dalu,outputDir=/tmp

        ps:在java_opts中添加上面的配置

    jar类型配置

-javaagent:/mnt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,metricInterval=6000,ioProfiling=true,tag=dalu,brokerList=192.168.1.6:9092,topicPrefix=profiler_

        ps:在java_opts中添加上面的配置

启动

    tomcat类型

Java Agent 1.0.0 premain args: reporter=com.uber.profiling.reporters.FileOutputReporter,metricInterval=6000,ioProfiling=true,tag=dalu,outputDir=/
tmp
1557212575527 com.uber.profiling.Arguments: Got argument value for reporter: com.uber.profiling.reporters.FileOutputReporter
1557212575527 com.uber.profiling.Arguments: Got argument value for metricInterval: 6000
1557212575527 com.uber.profiling.Arguments: Got argument value for tag: dalu
1557212575527 com.uber.profiling.Arguments: Got argument value for ioProfiling: true
1557212575533 com.uber.profiling.reporters.FileOutputReporter: Got argument value for outputDir: /tmp
1557212576119 com.uber.profiling.AgentImpl: Finished one time profiler: com.uber.profiling.profilers.ProcessInfoProfiler@6f1fba17
1557212576133 com.uber.profiling.AgentImpl: Ran periodic profiler (first run): com.uber.profiling.profilers.CpuAndMemoryProfiler@6325a3ee
1557212576139 com.uber.profiling.AgentImpl: Ran periodic profiler (first run): com.uber.profiling.profilers.IOProfiler@1d16f93d
1557212576146 com.uber.profiling.AgentImpl: Scheduled profiler com.uber.profiling.profilers.CpuAndMemoryProfiler@6325a3ee with interval 6000 mill
is
1557212576146 com.uber.profiling.AgentImpl: Scheduled profiler com.uber.profiling.profilers.IOProfiler@1d16f93d with interval 6000 millis

        ps:看到上面的tomcat日志,表示jvm profile加载成功

查看

    tomcat类型

        会在/tmp下面生成三个文件:CpuAndMemory.json、 IO.json、 ProcessInfo.json。注意这三个文件的内容是追加的操作,也就是文件会越来越大。

        CpuAndMemory.json

{"nonHeapMemoryTotalUsed":1.2519632E7,"bufferPools":[{"totalCapacity":16384,"name":"direct","count":2,"memoryUsed":16384},{"totalCapacity":0,"nam
e":"mapped","count":0,"memoryUsed":0}],"heapMemoryTotalUsed":2.9360128E7,"vmRSS":97513472,"epochMillis":1557209593342,"nonHeapMemoryCommitted":1.
4876672E7,"heapMemoryCommitted":4.294967296E9,"memoryPools":[{"peakUsageMax":251658240,"usageMax":251658240,"peakUsageUsed":567808,"name":"Code C
ache","peakUsageCommitted":2555904,"usageUsed":567808,"type":"Non-heap memory","usageCommitted":2555904},{"peakUsageMax":-1,"usageMax":-1,"peakUs
ageUsed":10728624,"name":"Metaspace","peakUsageCommitted":11010048,"usageUsed":10728624,"type":"Non-heap memory","usageCommitted":11010048},{"pea
kUsageMax":1073741824,"usageMax":1073741824,"peakUsageUsed":1223200,"name":"Compressed Class Space","peakUsageCommitted":1310720,"usageUsed":1223
200,"type":"Non-heap memory","usageCommitted":1310720},{"peakUsageMax":-1,"usageMax":-1,"peakUsageUsed":29360128,"name":"G1 Eden Space","peakUsag
eCommitted":450887680,"usageUsed":29360128,"type":"Heap memory","usageCommitted":450887680},{"peakUsageMax":-1,"usageMax":-1,"peakUsageUsed":0,"n
ame":"G1 Survivor Space","peakUsageCommitted":0,"usageUsed":0,"type":"Heap memory","usageCommitted":0},{"peakUsageMax":4294967296,"usageMax":4294
967296,"peakUsageUsed":0,"name":"G1 Old Gen","peakUsageCommitted":3844079616,"usageUsed":0,"type":"Heap memory","usageCommitted":3844079616}],"pr
ocessCpuLoad":0.3,"systemCpuLoad":0.5,"processCpuTime":700000000,"vmHWM":97513472,"appId":null,"name":"9440@192.168.1.9","host":"192.168.1.9","processUuid":"d1db056d-857f-467f-9f82-9a5d68193781","gc":[{"collectionTime":0,"name":"G1 Young Generation","collectionCount":0},{"col
lectionTime":0,"name":"G1 Old Generation","collectionCount":0}]}

        IO.json

{"system":7506813,"idle":2448896944,"cpu":"cpu2","iowait":22125,"user":13151461,"nice":612246},{"system":6973702,"idle":2450832959,"cpu":
"cpu3","iowait":18469,"user":12478415,"nice":553689}],"appId":null,"name":"9440@192.168.1.9","host":"192.168.1.9","processUuid":"d1
db056d-857f-467f-9f82-9a5d68193781","self":{"io":{"wchar":237804,"write_bytes":237568,"rchar":98387611,"read_bytes":0}},"epochMillis":15572096173
41}

        ProcessInfo.json

{"jvmInputArguments":"","jvmClassPath":"","epochMillis":1557212126677,"cmdline":"test -server -XX:+UseCompressedOops -Xmx4G -Xms4G -se
rver -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -XX:G1HeapRegionS
ize=2M -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX
:G1MaxNewSizePercent=75  -javaagent:/mnt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.u
ber.profiling.reporters.FileOutputReporter,metricInterval=6000,ioProfiling=true,tag=dalu,outputDir=/tmp -Xloggc:/opt/app/applications/test/logs/test.gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/app/applications/test/logs -Dcom.sun.management
.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -classpath /opt/programs/jdk1.8.0_111/lib/t
ools.jar:/opt/programs/tomcat_8.5.31/bin/bootstrap.jar:/opt/programs/tomcat_8.5.31/bin/tomcat-juli.jar:/opt/app/applications/test/conf
ig:/opt/app/applications/test/lib: -Dapp.name=test -Djava.io.tmpdir=/opt/app/applications/test/temp -Djava.endor
sed.dirs=/opt/programs/tomcat_8.5.31/endorsed -Dcatalina.base=/opt/app/applications/test -Dcatalina.home=/opt/programs/tomcat_8.5.31 o
rg.apache.catalina.startup.Bootstrap start ","appId":null,"name":"10161@192.168.1.9","host":"192.168.1.9","processUuid":"5228e88d-e
838-4607-8a1e-5e6f1c40bcd0","agentVersion":"1.0.0","appClass":null,"tag":"dalu","xmxBytes":4294967296,"appJar":null}

 

 

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部