Best Oracle configuration for HP DL585, six 6402 controllers and42 disks?

[Database Administrator]

Case: A new HP DL585 server dedicated to Oracle database use. Equipped
with two AMD dual core processors (upgradeable to four CPUs if CPU
becomes the bottleneck). Initially 16 gigabytes of RAM, more RAM will be
added if required. Red Hat Linux 4 AS. Four internal 15k 146 GB U320
disks for OS and database software (two mirrored pairs) on integrated
RAID controller. Six HP 6402 RAID controllers for database disks, each
controller has 256 MB of battery backed cache. The controllers have two
U320 channels but only one channel per controller is used to maximize
controller throughput. Each controller drives seven 15k 146 GB U320
disks. The disks are on MSA30 DB (dual bus) arrays, 14 disks per array
so the total number of disks on arrays is 42. A separate storage system
is used for RMAN disk-to-disk backups and it is connected via dual 2 Gb FC.

Question: how to configure the controllers and Oracle software to make
the best out of the disk subsystem. The server will host several
databases, one large and several small ones. The databases are weighted
on reads and data warehouse use (multi-user sequential). The system
should be up and running most of the time but has no real 24/7
requirements so some downtime can be tolerated.

Assumably we will configure each 7 disk set as RAID 1+0 with hot spare
and present each set as one raw device on Red Hat. The Oracle ASM will
be used so that it sees six logical raw devices and they will then form
a single ASM disk group. All data on all databases (control files,
tablespaces, redos, archived redos etc.) will be put on this single ASM
disk group so the data and the IO load will be spread equally over all
six logical raw devices. Assumably this will give us a maximal IO
performance. There is dual striping (stripe on stripe): ASM does
striping over the six logical raw devices and under each device there is
internal striping done by 6402 controller over the active disks. The
controllers would be set to use the cache only for writes and not for
reads since Oracle has its own read cache = the buffer cache.

Does this configuration sound reasonable? If ASM does the striping on 1
MB slices what should be the stripe size on 6402 controllers? What is
the reliability and failure rate of a 6402 controller - in our
configuration all the databases will go down if one controller malfunctions.

[Database Administrator]

Heikki Siltala wrote:

>What is
>the reliability and failure rate of a 6402 controller - in our
>configuration all the databases will go down if one >controller malfunctions.

I don't know, but I do know that I've seen controller problems on every
type of hardware I've seen over the last 25 years, as the software guy
breathing a sigh of relief because the problem wasn't something I've
done. There seem to be peaks in failure when they are new and when
they are a few years old. An MSA1000 I'm working on had a controller
failure when it was new, as well as more than one disk at a time. Now
it is giving frame time out errors when under heavy load (it seems to
consider exports and RMAN heavy load, and sometimes high levels of
redo), so far they appear innocuous, I'm thinking the fibre channel
driver just complains too much. While this may sound bad out of
context, actually I'm quite impressed with it and would recommend it
for a small site. But I'm a software guy, what do I know?

I wouldn't be dependent on single controller paths if I had a choice.
It's never up to me.

jg
--
@home.com is bogus.
Hey, someone is using a nickname I used to use!
http://app.businessweek.com/UserComm...productId=3305

[Database Administrator]

Joel Garry schreef:

>
> I wouldn't be dependent on single controller paths if I had a choice.
> It's never up to me.
>
> jg

Neither would I; is it possible to have every disk connected to every
controller?

[Database Administrator]

frank.van.bortel@gmail.com wrote:
> Joel Garry schreef:
>
> >
> > I wouldn't be dependent on single controller paths if I had a choice.
> > It's never up to me.
> >
> > jg
>
> Neither would I; is it possible to have every disk connected to every
> controller?

Perhaps reading the question asked by the OP would help you get the
answer to your question.

[Database Administrator]

frank.van.bortel@gmail.com kirjoitti:
>
> Neither would I; is it possible to have every disk connected to every
> controller?
>

With this hardware: no. Each disk is driven by one HP 6402 RAID card.

--
Heikki

[Database Administrator]

Narrowing down the disk configuration options. Assuming that we will
spread all disk usage over all the disk and over all HP 6402 controllers
(S.A.M.E) so no separate disks for redos etc.

Setup 1: Each controller driving 7 disks will be configured RAID 1+0
with hot spare. Oracle ASM is used to tie together the six logical raw
devices (one logical raw device per controller). ASM is set to use no
redundancy, just striping the data over the six raw devices. Capacity:
6*6*146 GB = 5256 GB. Failure tolerance: disk failures are toleared well
since there is mirroring and hot spare in every logical raw device and
the controller will start using the spare if a drive fails. Controller
failures are not tolerated since ASM stripes over the six controllers
and if any controller goes down, all data goes down.

Setup 2: Each controller driving 7 disks will be configured RAID 0
without hot spare. Oracle ASM is used to form two failgroups each of
which consists of three logical raw devices (one logical raw device per
controller). ASM does striping and mirroring so that each block is
written on both controller sets. Capacity: 6*7*146 GB = 6132 GB. Failure
tolerance: one disk failure on one RAID 0 set will be tolerated since
all blocks are written to two RAID 0 sets. There could be also multiple
disk failures on one RAID 0 set or a couple of disk failures on
different RAID 0 sets as long as one ASM failgroup gets no failures.
Controller failures are tolerated: any one controller could fail without
data loss and also two or three controllers could fail if they are in
same ASM failgroup.

Setup 3: Each controller driving 7 disks will be configured RAID 1+0
with hot spare. Oracle ASM is used to form two failgroups each of which
consists of three logical raw devices (one logical raw device per
controller). ASM does striping and mirroring so that each block is
written on both controller sets. Capacity: 0.5*6*6*146 GB = 2628 GB.
Failure tolerance: disk failures are toleared well since there are
mirroring and hot spare in every logical raw device and the controller
will start using the spare if a drive fails. Controller failures are
tolerated: any one controller could fail without data loss and also two
or three controllers could fail if they are in same ASM failgroup.

Setup 4: this would be all 6402 controllers set to JBOD mode and ASM
configured to stripe and mirror over the disks and controllers (two
failgroups, each of which having 21 disks on 3 controllers). This could
be the optimal solution but since 6402 has no JBOD mode we can forget
about it.

Comparison: Setup 1 tolerates disk failures but no controller failures.
Setup 2 has rather a limited disk failure toleration but it tolerates
controller failures. Setup 3 tolerates both disk failures and controller
failures but the disk capacity is cut to half.

Performance comparison: Setup 2 is assumed to be the fastest since seven
disk RAID 0 sets are used. Setup 3 is the slowest at least in write
operations since every write goes to two controllers and four disks.

[Database Administrator]

Oops! And now the correct capacity calcuations. :-) Theoretical maximum
capacity is 42*146 GB = 6132 GB.

Setup 1: 6*3*146 GB = 2628 GB (43 percent of the theoretical capacity)

Setup 2: 0.5*6*7*146 GB = 3066 GB (50 percent of the theoretical capacity)

Setup 3: 0.5*6*3*146 GB = 1314 GB (21 percent of the theoretical capacity)

Setup 4, if possible, would give same capacity as Setup 2.

[dbasrusty]

Which config did you go for and how's it doing?
thx