Cannot mirrorvg after replacing defective hdisk

[Database Administrator]

The scenario:

1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
2) hdisk1 died with messages in errpt - LVs listed as open/stale
3) Ran rmlvcopy and removed all copies on the failed hdisk1
4) Was able to reducevg and remove the affected disk.
5) rmdev -l hdisk1 -d
6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP

IBM sent a hot-swappable replacement, and we took out the old disk and
inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
we go ahead and try to add it back into the rootvg.

We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
tells us that it could not synchronize the data. Checking the system
shows the mirroring in place but most of the LVs remained stale.

Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
then syncvg -l hd1, and goes back to open/stale. Here is the error:

0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.

oslevel -s
5300-07-01-0748

A reboot of the server had no effect.

Also to be noted: My first thought was the new disk was DOA, but it
was certified, and if I add it to another VG and do mklvcopy/syncvg on
those LVs, it works fine.



We expect to call IBM AIX Support as our next step. Just wondered if
anyone has seen this before and wanted to be sure it's not something
silly before we call them.

[Database Administrator]

On Nov 6, 10:05 pm, "steven_nospam at Yahoo! Canada"
wrote:
> The scenario:
>
> 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
> 2) hdisk1 died with messages in errpt - LVs listed as open/stale
> 3) Ran rmlvcopy and removed all copies on the failed hdisk1
> 4) Was able to reducevg and remove the affected disk.
> 5) rmdev -l hdisk1 -d
> 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP
>
> IBM sent a hot-swappable replacement, and we took out the old disk and
> inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
> we go ahead and try to add it back into the rootvg.
>
> We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
> tells us that it could not synchronize the data. Checking the system
> shows the mirroring in place but most of the LVs remained stale.
>
> Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
> then syncvg -l hd1, and goes back to open/stale. Here is the error:
>
> 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
>
> oslevel -s
> 5300-07-01-0748
>
> A reboot of the server had no effect.
>
> Also to be noted: My first thought was the new disk was DOA, but it
> was certified, and if I add it to another VG and do mklvcopy/syncvg on
> those LVs, it works fine.
>
> We expect to call IBM AIX Support as our next step. Just wondered if
> anyone has seen this before
So far no.
but
- Is the errpt saying anythink ?
- do mirroring one LV at a time - beginning with hd5

Search the IBM support side seaching for ' mwc , syncvg ' ;-(

cheers
Hajo

[Database Administrator]

On 6 Nov, 21:05, "steven_nospam at Yahoo! Canada"
wrote:
> The scenario:
>
> 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
> 2) hdisk1 died with messages in errpt - LVs listed as open/stale
> 3) Ran rmlvcopy and removed all copies on the failed hdisk1
> 4) Was able to reducevg and remove the affected disk.
> 5) rmdev -l hdisk1 -d
> 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP
>
> IBM sent a hot-swappable replacement, and we took out the old disk and
> inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
> we go ahead and try to add it back into the rootvg.
>
> We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
> tells us that it could not synchronize the data. Checking the system
> shows the mirroring in place but most of the LVs remained stale.
>
> Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
> then syncvg -l hd1, and goes back to open/stale. *Here is the error:
>
> 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
>
> oslevel -s
> 5300-07-01-0748
>
> A reboot of the server had no effect.
>
> Also to be noted: *My first thought was the new disk was DOA, but it
> was certified, and if I add it to another VG and do mklvcopy/syncvg on
> those LVs, it works fine.
>
> We expect to call IBM AIX Support as our next step. Just wondered if
> anyone has seen this before and wanted to be sure it's not something
> silly before we call them.

Does the logical volume has bad-block relocation policy turned off?
This may have caused the command to fail.

HTH
Duncan

[Database Administrator]

On Nov 6, 4:57*pm, Hajo Ehlers wrote:
> On Nov 6, 10:05 pm, "steven_nospam at Yahoo! Canada"
>
>
>
> wrote:
> > The scenario:
>
> > 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
> > 2) hdisk1 died with messages in errpt - LVs listed as open/stale
> > 3) Ran rmlvcopy and removed all copies on the failed hdisk1
> > 4) Was able to reducevg and remove the affected disk.
> > 5) rmdev -l hdisk1 -d
> > 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP
>
> > IBM sent a hot-swappable replacement, and we took out the old disk and
> > inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
> > we go ahead and try to add it back into the rootvg.
>
> > We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
> > tells us that it could not synchronize the data. Checking the system
> > shows the mirroring in place but most of the LVs remained stale.
>
> > Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
> > then syncvg -l hd1, and goes back to open/stale. *Here is the error:
>
> > 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
>
> > oslevel -s
> > 5300-07-01-0748
>
> > A reboot of the server had no effect.
>
> > Also to be noted: *My first thought was the new disk was DOA, but it
> > was certified, and if I add it to another VG and do mklvcopy/syncvg on
> > those LVs, it works fine.
>
> > We expect to call IBM AIX Support as our next step. Just wondered if
> > anyone has seen this before
>
> So far no.
> but
> *- Is the errpt saying anythink ?
> *- do mirroring one LV at a time - beginning with hd5
>
> Search the IBM support side seaching for ' mwc , syncvg ' ;-(
>
> cheers
> Hajo- Hide quoted text -
>
> - Show quoted text -

Hi Hajo,

When we were first testing the new disk, we had no errors showing up,
but I just worked with an IBM rep this morning to run a series of
tests, and now we are getting these errors when we try the syncvg:

LVM_IO_FAIL
DISK_ERR2
DISK_ERR4

Both the AIX support person and I agreed that it looks like the new
disk installed is operational, but the LVM_IO_FAIL is pointing to a
possible SCSI cable or port issue, so we are calling IBM back in to
take another look at the hardware.

More news next week when the IBM SE shows up to review the situation.

Another thing I noticed was that the failing disk (hdisk1) was a 72GB
and the working one is 144GB (hdisk3). There are sufficient FREE PPs
on the small disk to handle the USED PPs on the working disk so it is
not a space issue, and bad block relocation is on and there are no
signs the issue was a space problem, but I pointed it out to the IBM
engineer just in case. They said we normally would get a different
error if insufficient PPs were available on the other disk for
mirroring.

Anyway, they will be checking it out and I will update this post when
I find out more.