Profiling a black swan using Intel Processor Trace, or what to do when your code takes 20 ms to run instead of 2 ms

In the beginning we'll see why performance of a really small part of code can directly affect an application's success. Then we'll try to profile this small part using the perf profiler and get the profile that we wanted, but with great additional charges.

Then we'll try to improvise a small profiler based on interface of the Linux kernel. The profiler will be using Intel Processor Trace technology, which allows us to reconstruct the application running trace. We'll walk through all the steps of creating this profiler: trace collecting and decoding, adding information about methods and profile aggregation. At the end, we'll get a small profiler. We'll use it to profile JSON review and study the results.

Then we'll see that average metrics often aren't enough to describe the code performance, and we need metrics that describe how slow our code works. It appears that our users have to face these rare delays much more often than we thought and wish.

We'll try to learn why sometimes our code works slower than average and find out that all the profilers we used before don't really help us to find reasons for these delays. So we'll make a tool based on the profiler we just built, and this tool will allow us to see into these rare delays with unprecedented precision and understand what causes them. And in the very end we'll finally learn the connection between black swans and profiling Java applications.