Tuning Oracle disk I/O: The SCSI bus, part 2

The following is part of a series on the different aspects of disk I/O performance and optimization for Oracle databases.

The following is part of a series on the different aspects of disk I/O performance and optimization for Oracle databases. Each tip is excerpted from the not-yet-released Rampant TechPress book, "Oracle disk I/O tuning," by Mike Ault. Check back to the main series page for upcoming installments.

Mike Ault

Mike Ault is one of SearchOracle.com's Oracle Internals experts. Mike is senior Oracle consultant with Burleson Consulting, and one of the leading names in Oracle technology.

To view Mike's expert responses or to ask him a question, click here.

This tip follows from Oracle disk I/O tuning: The SCSI bus, part 1.

SCSI bus

In later Solaris releases, you can specify scsi_options per target or per SCSI bus. See esp(7d), isp(7d), from which this example /kernel/drv/esp.conf file is derived:

name="esp" parent="/[email protected],e0000000/[email protected],e0001000/[email protected],400000"  reg=0xf,0x800000,0x40
It has also been reported that some hardware RAIDs support a number of different LUNs (logical disks), but these LUNs share a common set of I/O buffers between them. This can cause SCSI QFULL conditions on those devices that do not have commands queued. Since the usual algorithm is to retry the command when a previous command is completed, Solaris doesn't handle this situation very well.

The workaround is to decrease sd_max_throttle such that there's always at least 1 slot available for each LUN, e.g., if you have 3 LUNs and your RAID supports a maximum of 64 outstanding commands, sd_max_throttle must be at most 31. Any two LUNs can get 31 requests and you still have two slots left over for number 3.

For some hardware RAIDs, it has been found that decreasing sd_max_throttle improved performance due to better load balancing among LUNs. It might be worth a shot if your hardware based RAID system seems sluggish.

There are several additional host configuration variables that can be specified in addition to sd_max_throttle and in relation to the SCSI interface on SUN Solaris. These are usually specified in the /etc/system file as was shown above.

The following variables in the /etc/systems file should be set to maximize system performance. When any of these variables are changed, the system must be rebooted for changes to take effect.

sd_max_throttle -- The sd_max_throttle variable sets the maximum number of commands that the SCSI sd driver will attempt to queue to a single HBA driver. The default value is 256. This variable must be set to a value less than or equal to the maximum queue depth of each LUN connected to each instance of the sd driver. If this is not done, then commands may be rejected because of a full queue condition and the sd driver instance that receives the queue full message will throttle down sd_max_throttle to 1. This obviously will result in degraded performance. The variable is set in the /etc/system file as follows:

set sd:sd_max_throttle=20

sd_io_time -- The sd_io_time variable is command time out value. Setting this time out value to 120 seconds will prevent the host from issuing warning messages while non-disruptive operations are performed on the disk array. For example, in a EMC Symmettrix power path environment this variable can be set to 0x3C (60 seconds). As with the sd_max_throttle value, this is set in the /etc/system file. Using the EMC value in an example:

set sd:sd_io_time = 0x3C

scsi_options -- The value scsi_options is a setting for SCSI options included as outlined in the bit meaning table above. Solaris 7 default SCSI settings are for the maximum Ultra Wide Speed. Since the /etc/system file can also affect other disks in the system, consider the effects of implementing the scsi_options=0x7F8 mask on the other disks. You might consider the bus level or device level setpoints as we discussed above rather than a system wide setting. An example entry in the /etc/system file would be:

set scsi_options=0x7F8
In addition to the /etc/system file, the file /kernel/drv/sd.conf can also be used to specify SCSI driver specific settings. However, these usually deal with specifying addresses for hardware RAID. For example, you must modify the host configuration file /kernel/drv/sd.conf to insure smooth integration from the host and hardware based RAID or disk systems such as EMC's Symmetrix.

Using Symmetrix as an example, Symmetrix devices are addressed on the Fibre Channel bus using the Symmetrix Hard Loop ID as a Target ID. There is only one Target ID per Symmetrix port. A Target ID can have up to 128 luns. By default, Solaris searches only for Target IDs; if LUN addressing is required, you must modify the file /kernel/drv/sd.conf. However, this isn't a SCSI configuration book, so we aren't going to cover that aspect.

Sun conclusions for SCSI interface tuning

In order to have a properly tuned interface, you must make sure the proper SCSI options are set at the system (/etc/system) and bus as well as device level (scsi_options). Make sure that the proper command queued depth (sd_max_throttle) and proper command delay times (sd_io_time) are set. In a mixed environment (multiple SCSI devices with different queue depths and IO considerations) set these at the device level or the entire SCSI system or bus will be drug down to the performance of the least powerful device. Finally, try to isolate tape or other low speed devices on their own SCSI bus as their device numbers are usually set to 6, which causes them to usurp service from other devices in the chain.

Click to buy the book, "Oracle disk I/O tuning," by Mike Ault.

About the author

Mike Ault is a SearchOracle.com expert and a senior Oracle consultant with Burleson Consulting, and one of the leading names in Oracle technology. The author of more than 20 Oracle books and hundreds of articles in national publications, Mike Ault has five Oracle Masters Certificates and was the first popular Oracle author with his landmark book "Oracle7 administration and management." Mike also wrote several of the "Exam Cram" books, and enjoys a reputation as a leading author and Oracle consultant. Ask Mike a question today!

Dig Deeper on Oracle database performance problems and tuning