Multiple partitions
are necessary to read the data in parallel. The option PARTSIZE= forces
the software to partition SPD Engine data files at the specified size.
The actual size of the partition is computed to accommodate the maximum
number of observations that fit in the specified size of
n megabytes, gigabytes, or terabytes. If you
have a table with an observation length greater than 65K, you might
find that the PARTSIZE= that you specify and the actual partition
size do not match. To get these numbers to match, specify a PARTSIZE=
that is a multiple of 32 and the observation length.
By splitting (partitioning)
the data portion of an SPD Engine data set into fixed-sized files,
the software can introduce a high degree of scalability for some operations.
The SPD Engine can spawn threads in parallel (for example, up to one
thread per partition for WHERE evaluations). Separate data partitions
also enable the SPD Engine to process the data without the overhead
of file access contention between the threads. Because each partition
is one file, the trade-off for a small partition size is that an increased
number of files (for example, UNIX i-nodes) are required to store
the observations.
Scalability limitations
using PARTSIZE= depend on how you configure and spread the file systems
specified in the DATAPATH= option across striped volumes. (You should
spread each individual volume's striping configuration across multiple
disk controllers or SCSI channels in the disk storage array.) The
goal for the configuration, at the hardware level, is to maximize
parallelism during data retrieval. For information about disk striping,
see “I/O Setup and Validation” under “SPD Engine”
in Scalability and Performance at
http://support.sas.com/rnd/scalability.
The PARTSIZE= specification
is limited by the SPD Engine system option MINPARTSIZE=, which is
usually maintained by the system administrator. MINPARTSIZE= ensures
that an inexperienced user does not arbitrarily create small partitions,
thereby generating a large number of data files.
The partition size determines
a unit of work for many of the parallel operations that require full
data set scans. But, more partitions does not always mean faster processing.
The trade-offs involve balancing the increased number of physical
files (partitions) required to store the data set against the amount
of work that can be done in parallel by having more partitions. More
partitions means more open files to process the data set, but a smaller
number of observations in each partition. A general rule is to have
10 or fewer partitions per data path, and 3 to 4 partitions per CPU.
(Some operating systems have a limit on the number of open files that
you can use.)
To determine an adequate
partition size for a new SPD Engine data set, you should be aware
of the following:
-
the types of applications that
run against the data
-
-
how many CPUs are available to
the applications
-
which disks are available for storing
the partitions
-
the relationships of these disks
to the CPUs
For example, if each
CPU controls only one disk, then an appropriate partition size would
be one in which each disk contains approximately the same amount of
data. If each CPU controls two disks, then an appropriate partition
size would be one in which the load is balanced. Each CPU does approximately
the same amount of work.
Note: The PARTSIZE= value for a
data set cannot be changed after a data set is created. To change
PARTSIZE=, you must re-create the data set and specify a different
PARTSIZE= value in the LIBNAME statement or on the new (output) data
set.