Examining File System Latency in Production

Brendan Gregg, Lead Performance Engineer, Joyent

This paper introduces file system latency as a metric for understanding application

performance. With the increased functionality and caching of file systems, the

traditional approach of studying disk-based metrics can be confusing and incomplete.

The different reasons for this will be explained in detail, including new behavior that has

been caused by I/O throttling in cloud computing environments. Solutions for

measuring file system latency are demonstrated, including the use of DTrace to create

custom analysis tools. We also show different ways this metric can be presented,

including the use of heat maps to visualize the full distribution of file system latency,

from Joyent’s Cloud Analytics.