在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
Uber JVM ProfilerUber JVM Profiler provides a Java Agent to collect various metrics and stacktraces for Hadoop/Spark JVM processesin a distributed way, for example, CPU/Memory/IO metrics. Uber JVM Profiler also provides advanced profiling capabilities to trace arbitrary Java methods and arguments onthe user code without user code change requirement. This feature could be used to trace HDFS name node call latencyfor each Spark application and identify bottleneck of name node. It could also trace the HDFS file paths each Sparkapplication reads or writes and identify hot files for further optimization. This profiler is initially created to profile Spark applications which usually have dozens of or hundreds ofprocesses/machines for a single application, so people could easily correlate metrics of these differentprocesses/machines. It is also a generic Java Agent and could be used for any JVM process as well. How to Build
This command creates jvm-profiler.jar file with the default reporters like ConsoleOutputReporter, FileOutputReporter and KafkaOutputReporter bundled in it. If you want to bundle the custom reporters like RedisOutputReporter or InfluxDBOutputReporter in the jar file then provide the maven profile id for that reporter in the build command. For example to build a jar file with RedisOutputReporter, you can execute Example to Run with Spark ApplicationYou could upload jvm-profiler jar file to HDFS so the Spark application executors could access it. Then add configuration like following when launching Spark application: --conf spark.jars=hdfs://hdfs_url/lib/jvm-profiler-1.0.0.jar--conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar Example to Run with Java ApplicationFollowing command will start the example application with the profiler agent attached, which will report metrics to the console output: java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=mytag,metricInterval=5000,durationProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod,argumentProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1,sampleInterval=100 -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication Example to Run with Executable JarUse following command to run jvm profiler with executable jar application. java -javaagent:/opt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,metricInterval=5000,durationProfiling=foo.bar.FooAppication.barMethod,sampleInterval=5000 -jar foo-application.jar Example to Run with TomcatSet the jvm profiler in CATALINA_OPTS before starting the tomcat server. Check logs/catalina.out file for metrics. export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/opt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,metricInterval=5000,durationProfiling=foo.bar.FooController.barMethod,sampleInterval=5000" Example to Run with Spring Boot Maven PluginUse following command to use jvm profiler with Spring Boot 2.x. For Spring Boot 1.x use mvn spring-boot:run -Dspring-boot.run.jvmArguments="-javaagent:/opt/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,metricInterval=5000,durationProfiling=foo.bar.FooController.barMethod,sampleInterval=5000" Send Metrics to KafkaUber JVM Profiler supports sending metrics to Kafka. For example, java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.KafkaOutputReporter,metricInterval=5000,brokerList=localhost:9092,topicPrefix=profiler_ -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication It will send metrics to Kafka topic profiler_CpuAndMemory. See bottom of this document for an example of the metrics. More DetailsFeature ListUber JVM Profiler supports following features:
Parameter ListThe java agent supports following parameters, which could be used in Java command line like "-javaagent:agent_jar_file.jar=param1=value1,param2=value2":
YAML Config FileThe parameters could be provided as arguments in java command, or in a YAML config file if you use configProvider=com.uber.profiling.YamlConfigProvider. Following is an example of the YAML config file: reporter: com.uber.profiling.reporters.ConsoleOutputReportermetricInterval: 5000 Metrics ExampleFollowing is an example of CPU and Memory metrics when using ConsoleOutputReporter or KafkaOutputReporter: { "nonHeapMemoryTotalUsed": 11890584.0, "bufferPools": [ { "totalCapacity": 0, "name": "direct", "count": 0, "memoryUsed": 0 }, { "totalCapacity": 0, "name": "mapped", "count": 0, "memoryUsed": 0 } ], "heapMemoryTotalUsed": 24330736.0, "epochMillis": 1515627003374, "nonHeapMemoryCommitted": 13565952.0, "heapMemoryCommitted": 257425408.0, "memoryPools": [ { "peakUsageMax": 251658240, "usageMax": 251658240, "peakUsageUsed": 1194496, "name": "Code Cache", "peakUsageCommitted": 2555904, "usageUsed": 1173504, "type": "Non-heap memory", "usageCommitted": 2555904 }, { "peakUsageMax": -1, "usageMax": -1, "peakUsageUsed": 9622920, "name": "Metaspace", "peakUsageCommitted": 9830400, "usageUsed": 9622920, "type": "Non-heap memory", "usageCommitted": 9830400 }, { "peakUsageMax": 1073741824, "usageMax": 1073741824, "peakUsageUsed": 1094160, "name": "Compressed Class Space", "peakUsageCommitted": 1179648, "usageUsed": 1094160, "type": "Non-heap memory", "usageCommitted": 1179648 }, { "peakUsageMax": 1409286144, "usageMax": 1409286144, "peakUsageUsed": 24330736, "name": "PS Eden Space", "peakUsageCommitted": 67108864, "usageUsed": 24330736, "type": "Heap memory", "usageCommitted": 67108864 }, { "peakUsageMax": 11010048, "usageMax": 11010048, "peakUsageUsed": 0, "name": "PS Survivor Space", "peakUsageCommitted": 11010048, "usageUsed": 0, "type": "Heap memory", "usageCommitted": 11010048 }, { "peakUsageMax": 2863661056, "usageMax": 2863661056, "peakUsageUsed": 0, "name": "PS Old Gen", "peakUsageCommitted": 179306496, "usageUsed": 0, "type": "Heap memory", "usageCommitted": 179306496 } ], "processCpuLoad": 0.0008024004394748531, "systemCpuLoad": 0.23138430784607697, "processCpuTime": 496918000, "appId": null, "name": "24103@machine01", "host": "machine01", "processUuid": "3c2ec835-749d-45ea-a7ec-e4b9fe17c23a", "tag": "mytag", "gc": [ { "collectionTime": 0, "name": "PS Scavenge", "collectionCount": 0 }, { "collectionTime": 0, "name": "PS MarkSweep", "collectionCount": 0 } ]} Metric DetailsA list of all metrics and information corresponding to them can be found here. Generate flamegraph of Stacktrack Profiling resultWe can take the output of Stacktrack Profiling to generate flamegraph to visualize CPU time. Using the Python script python stackcollapse.py -i Stacktrace.json > Stacktrace.foldedflamegraph.pl Stacktrace.folded > Stacktrace.svg Note that it is required to enable stacktrace sampling, in order to generate flamegraph. To enable it, please set |
请发表评论