My compatriot Chris Gibson highlighted a great performance tuning attribute for Fibre Channel adapters in this blog post. I was so excited that I dusted off my blog-writing skills (much neglected) and got fired up enough to put Chris' advice into practice via a script.
First, the advice - which came originally from IBM's Dan Braden - is to display the number of command elements -num_cmd_elems - on a Fibre Channel adapter.
num_cmd_elems? Huh?
On AIX (and on the Virtual I/O Server) there is an attribute on Fibre Channel adapters called num_cmd_elems. This is the maximum number of in-flight IOs for a fibre channel adapter. Physical Volumes (PVs –also known as hdisks) have a corresponding attribute called queue_depth, which I’ve written about here. On the FC adapter, the num_cmd_elems is the total for the adapter, not just the one disk.
For more of Dan Braden's advice, check out his document on AIX Disk Queue Depth Tuning for Performance. It's known as Technote TD105745.
So, onto applying the advice in your environment.
Three Key Values
There are three values that you're interested in knowing here:
-
the default value for num_cmd_elems
-
the actual (or effective) value, and
-
the high water mark.
On a Fibre Channel adapter, you can get these settings using the following commands;
Default value for num_cmd_elems
You get the default value with the lsattr command.
lsattr -Dl fcs0 -a num_cmd_elems -F value
Here is an explanation of those flags:
-
-D is to show the default
-
-l points to the name of the adapter (in this case, fcs0)
-
-a is the attribute
-
-F displays the format. In this case, we're after the value only.
Now on to the actual value - the value which is in effect.
Actual value for num_cmd_elems
Now, if you've changed the value for your adapter (most likely by increasing it), then you need to see what it is currently set to. Same command as above, except you use the -E to show the Effective value:
lsattr -El fcs0 -a num_cmd_elems -F value
High Water Mark
Now it's time to find out exactly what the High Water Mark is for the adapter. You can do that using the fcstat command with the new (and currently undocumented) flag -D.
Look at the FC SCSI Adapter Driver statistics, where you will find the High water mark of active commands. This example stolen borrowed from Chris Gibson's blog.
# fcstat –D fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
…
FC SCSI Adapter Driver Queue Statistics
Number of active commands: 0
High water mark of active commands: 17
Number of pending commands: 0
High water mark of pending commands: 1
Number of commands in the Adapter Driver Held off queue: 0
High water mark of number of commands in the Adapter Driver Held off queue: 0
…
Alright. We now have three values: default, effective and high-water-mark.
Now, Chris has provided this formula which he got from Dan Braden:
Set num_cmd_elems to >= high water mark of active commands or the default, whichever is greater.
# chdev –l fcs0 –a num_cmd_elems=4096 –a max_xfer_size=0x200000 –P
; requires a reboot
Now as you may have a number of fibre channel adapters, and you may have to do this for a number of AIX environments, it probably calls for a script.
The following script displays the three values we're after: the current (effective) num_cmd_elems, the default setting for the adapter, and the high water mark you've actually reached. It then displays the command to run (chdev) to set the num_cmd_elems to the High Water Mark or the default setting for the adapter, if the current setting is less than both of those.
The script does not run the chdev command, which requires a reboot anyway.
The usual disclaimers apply: run this script at your own risk. it should be run in a non-critical environment etc.
#!/bin/ksh
# set_num_cmd_elems.sh
print "Adapter Current Default High Water Mark"
for i in $(lsdev -cadapter -F name | grep "^fcs")
do
# Find actual setting for num_cmd_elems
actual_num_cmd_elems=$(lsattr -El $i -a num_cmd_elems -F value)
# Find default setting for the FC adapter
default_num_cmd_elems=$(lsattr -Dl $i -a num_cmd_elems -F value)
# High Water Mark
# Three greps required here. The first is to extract the correct stanza from the fcstat -D output
# the second and third are to extract the correct line, which in some environments shows the output
# with an extra space after the "mark"
high_water_mark=$(fcstat -D $i | \
grep -p 'FC SCSI Adapter Driver Queue Statistics' | \
grep 'High water mark' | \
grep 'active commands' | awk '{print $NF}')
if (( "$high_water_mark" > "$default_num_cmd_elems" ))
then
desired_num_cmd_elems=$high_water_mark
else
desired_num_cmd_elems=$default_num_cmd_elems
fi
if (( "$actual_num_cmd_elems" < "$desired_num_cmd_elems" ))
then
echo "chdev -l $i -a num_cmd_elems=$desired_num_cmd_elems-a max_xfer_size=0x200000 -P"
fi
done
A tip of ye olde hat to Chris Gibson and to Dan Braden for providing this information. And my thanks for David Tansley for assistance with the script.
Tags: fc dan chris gibson aix fibre channel chdev queue adapter num_cmd_elems technote depth performance disk braden
