Lab 8: Monitoring I/O Performance on Linux – 5pts*10 = 50 pts

Introduction

The iostat command is used to monitor the load on server input/output (I/O) devices by observing the time the devices are active compared to the average transfer rate of the device. iostat generates several report lines that can be used to monitor and subsequently change the system configuration to better balance the I/O workload between physical disk devices.

Description

In this lab, we’ll use the iostat utility to monitor I/O on Linux.

iostat is part of the sysstat utilities which is maintained by Sebastien Godard.

sysstat is open source software written in C, and is available under the GNU General Public License, version 2.

You can download the latest version here:

http://sebastien.godard.pagesperso-orange.fr/download.html

We’ll use the ASU Unix box for this lab.

To log into the system, you’ll need a terminal emulator, such as PuTTY.

After installing and launching PuTTY, connect to the ASU Linux machine using the credentials below:

Hostname: general.asu.edu

User name: YOUR ASURITE username

Password: YOUR ASURITE password  

On Mac, you don’t need PuTTY. Simply bring up the Terminal and type:

ssh general.asu.edu    OR  ssh@<YOUR USERNAME> general.asu.edu

The iostat utility is already installed on the ASU General system. We’ll just use it.

By default iostat produces two reports; the CPU Utilization report and the Device Utilization report.

The default invocation shows the statistics since system start up, then exits.

-bash-3.2$ iostat

Linux 2.6.18-416.el5xen (general1)      11/14/2016

avg-cpu:           %user   %nice   %system          %iowait             %steal   %idle

                        3.80      0.00      0.25                  0.06                  0.01      95.88

Device:            tps       Blk_read/s   Blk_wrtn/s   Blk_read          Blk_wrtn

xvda                 9.50         3.15               171.97              2581377            140838290

xvda1               0.00         0.04         0.00                       30698               1186

xvda2               9.50         3.11               171.96              2550479            140837104

xvdb                 1.08         0.55               15.10                449235              12362776

xvdb1               1.08         0.55               15.10                449035              12362776

dm-0                 23.58       3.66               187.06              2996194            153199880

dm-1                 0.00         0.00               0.00                  2696                                         0

-bash-3.2$

Inclusion of the CPU and Device Utilization reports can be controlled with the -c and -d options respectively.

CPU

-bash-3.2$ iostat -c

Linux 2.6.18-416.el5xen (general1)      11/14/2016

avg-cpu:           %user   %nice   %system          %iowait             %steal              %idle

                                    3.80      0.00      0.25                  0.06                  0.01                  95.88

-bash-3.2$

Devices

-bash-3.2$ iostat -d

Linux 2.6.18-416.el5xen (general1)      11/14/2016

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read               Blk_wrtn

xvda              9.50         3.15       171.90             2581385            141023426

xvda1             0.00         0.04         0.00             30698               1186

xvda2             9.49         3.11       171.90            2550487            141022240

xvdb              1.08         0.55        15.07             449235              12364960

xvdb1             1.08         0.55        15.07            449035              12364960

dm-0             23.57         3.65       186.98            2996202            153387200

dm-1              0.00         0.00         0.00              2696                 0

-bash-3.2$

The default Device or CPU Utilization report can be replaced with extended

Statistics using the -x option. Below is an example for the extended CPU statistics since system startup  

-bash-3.2$ iostat -cx

Linux 2.6.18-416.el5xen (general1)      11/14/2016

avg-cpu:  %user   %nice            %system          %iowait             %steal              %idle

                        3.79    0.00        0.25                  0.06                  0.01                  95.89

-bash-3.2$

Whilst reviewing the stats since system start up can be useful, more often you will want to review current activity.

Current activity can be displayed by specifying an interval measured in seconds.

Output will continue until interrupted or a specified count has been reached.

The syntax is as followed:

iostat -dx <interval> <count>

<interval> is the number of seconds iostat waits between each report. Specifying the number of seconds causes iostat to print periodic reports where IO statistics are averaged for the time period since previous report. Note that the first report will be the device statistics since system start up.

Example: $ iostat -dx 3

will display the extended device statistics since system start up in the first report, and every 3 seconds in subsequent reports until interrupted.

<count> specifies the number of reports.

Example: $ iostat -dx 3 4

will display extended device statistics since system start up in the first report, and the deltas for the last 3 seconds in three further reports.

-bash-3.2$ iostat -dx 3 3

Linux 2.6.18-416.el5xen (general1)      11/14/2016

Device:    rrqm/s   wrqm/s   r/s           w/s          rsec/s   wsec/s      avgrq-sz avgqu-sz               await      svctm    %util

xvda        0.03       12.14      0.14        9.35        3.14        171.86   18.44     0.08                        8.62        0.31        0.29

xvda1      0.00       0.00        0.00        0.00        0.04        0.00        34.92     0.00                         10.09      2.15        0.00

xvda2      0.02       12.14      0.14        9.35        3.10        171.85   18.44     0.08                        8.62        0.31        0.29

xvdb        0.00       0.83        0.03        1.05        0.55        15.05    14.49       0.01                        13.45      0.40        0.04

xvdb1      0.00       0.83        0.03        1.05        0.55        15.05    14.49       0.01                        13.45      0.40        0.04

dm-0        0.00      0.00        0.20        23.36      3.65        186.91   8.09        0.14                        6.00        0.14        0.32

dm-1        0.00      0.00        0.00         0.00       0.00        0.00      8.00          0.00                        9.73        0.52        0.00

Device:    rrqm/s   wrqm/s   r/s           w/s          rsec/s     wsec/s    avgrq-sz avgqu-sz               await      svctm    %util

xvda        0.00       20.00      0.00        3.67        0.00        189.33    51.64     0.00                        0.00        0.00        0.00

xvda1      0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvda2      0.00       20.00      0.00        3.67        0.00        189.33    51.64     0.00                        0.00        0.00        0.00

xvdb        0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvdb1      0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

dm-0       0.00       0.00        0.00        23.67     0.00         189.33    8.00       0.00                        0.00        0.00        0.00

dm-1       0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

Device:   rrqm/s   wrqm/s   r/s                            w/s          rsec/s     wsec/s    avgrq-sz avgqu-sz               await      svctm    %util

xvda       0.00        0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvda1     0.00        0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvda2     0.00        0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvdb       0.00        0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

xvdb1     0.00        0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

dm-0       0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

dm-1       0.00       0.00        0.00        0.00        0.00        0.00        0.00        0.00                        0.00        0.00        0.00

-bash-3.2$

If you have many devices and you want to watch for only some of them, you can also specify device names on command line: $ iostat -x -d sda 5

-bash-3.2$ cat /proc/partitions

major minor  #blocks  name

 202     0   20971520 xvda

 202     1     104391 xvda1

 202     2   20860402 xvda2

 202    16   20971520 xvdb

-bash-3.2$ iostat -dx xvda

Linux 2.6.18-416.el5xen (general1)      11/14/2016

Device:     rrqm/s   wrqm/s   r/s          w/s          rsec/s     wsec/s    avgrq-sz   avgqu-sz             await      svctm    %util

xvda         0.03      12.13      0.14        9.34        3.14        171.80    18.44       0.08                      8.61        0.31        0.29

-bash-3.2$

Now, what are those cryptic extended statistics?

§  r/s and w/s— respectively, the number of read and write requests issued by processes to the OS for a device.

§  rsec/s and wsec/s — sectors read/written (each sector 512 bytes).

§  avgrq-sz — average sectors per request (for both reads and writes). Do the math — (rsec + wsec) / (r + w) = (3.14+171.80)/(0.14+9.34)=18.45

§  avgqu-sz — average queue length for this device.

§  await — average response time (ms) of IO requests to a device. The name is a bit confusing as this is the total response time including wait time in the requests queue (let call it qutim), and service time that device was working servicing the requests (see next column — svctim). So the formula is await = qutim + svctim.

§  svctim — average time (ms) a device was servicing requests. This is a component of total response time of IO requests.

§  %util — Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

Activities

You will estimate the performance of xvda and dm-0 devices by (1) looking at each device utilization, and (2) looking at how loaded each device (partition) is.

A.   Device utilization

1.    Generate the extended device statistics report since system start up. Copy your output report and paste here. Be sure to format it so it is readable, as done in the description above.

2.    Provide the number of read requests issued by processes to the OS for each device:

xvda:  ______________

dm-0: ______________

3.    Provide the number of write requests issued by processes to the OS for each device

xvda:  ______________

dm-0: ______________

4.    Provide the average time (ms) each device was servicing requests ( aka service time)

xvda:  _________________

dm-0: _________________

5.    Compute the percentage of time each device was servicing requests as followed:

( (read requests + write requests) * service time in ms / 1000 ms ) * 100%

6.    Compare to the %util and draw your conclusion. What can you say about device saturation?

B.   We’ll now try to estimate how loaded the devices are. To do so, we look at the non-existent column mentioned above — qutim — the average time a request is spending in the queue. If it’s insignificant (compared to svctim), then the IO device is not saturated. When it becomes comparable to svctim and goes above it, then requests are queued longer and a major part of response time is actually time spent waiting in the queue.

7.    Provide the average response time (ms) of IO requests to each device.

xvda:  _________________

dm-0: _________________

8.    For each device, compute qutim

9.    Compare qutim to svctim and draw your conclusion

C.   Conclusion

10. Write a short paragraph essay on what you learned from this lab

Deliverable

Prepare your deliverable using this lab document as followed:

1.    Insert your answers to all the questions in “Activities” section

2.    Save your file as lab8_<YOUR USERNAME>.docx. Replace <YOUR USERNAME> with your ASURITE username. If you’re not using MS Word (for example, if you’re using Mac), please submit a PDF document.

3.    Be sure to include your full name on every page. You can insert it in the header, above the “IFT 201” banner.

4.    Upload your file to Blackboard.