check-circle-line exclamation-circle-line close-line

Three sets of VMs were configured for this solution:

  • Set of 2 VMs for production 2-node Oracle RAC prdrac with RAC VMs prdrac01 and prdrac02
  • 1 VM for production Oracle far sync prdfs with VM prdfs
  • Set of 2 VMs for DR 2-node Oracle RAC prddg with RAC VMs prddg01 and prddg02

 

Below show the on-prem production RAC , Far Sync and DR RAC setup.

 

All VMs were created with the following components and settings:

  • 8 vCPUs and 32GB memory
  • Oracle Enterprise Linux 7.4 OS
  • Oracle 12.2.0.1.0 Grid Infrastructure and RDBMS binaries installed on all VMs
  • Oracle SGA set to 16GB and PGA set to 4GB for all database instances
  • Oracle ASM and Oracle ASMLib were used
  • All RAC shared VMDKs are Eager Zero Thick for non-vSAN storage (thin-provisioned starting SDDC 1.8v2 / vSAN 6.7 P01) . Remember, Independent persistent mode is NOT required for enabling multiwriter attribute.
  • All ASM disk groups disks presented on different PVSCSI controllers for purposes of performance and queue depth
  • All database-related VMDKs partitioned using Linux utilities, with proper alignment offset and labeled using Oracle ASMLib
  • Oracle ASM disk groups DATA_DG, FRA_DG, GIMR_DG, REDO_DG, and CRS_DG created with external redundancy
  • Oracle ASM disk group VOTE_DG created with high redundancy

Find steps to add shared VMDKs with the multiwriter attribute for vSphere attached traditional storage here and for vSAN here.

See “Appendix A” for the complete list of Oracle initialization parameters for all three sets of VMs.

All best practices for Oracle workloads on a VMware SDDC were followed in accordance with the Oracle Databases on VMware Best Practices Guide here.

Table below provides Oracle solution architecture on-premises networking information.

 

network

prdrac01

prdrac02

prdfs

PRDDG01

 

PRDDG02

Public FDQN

prdrac01.corp.localdomain

prdrac02.corp.localdomain

prdfs.corp.localdomain

prddg01.corp.localdomain

prddg02.corp.localdomain

Public IP

10.128.138.200

10.128.138.201

10.128.138.207

10.128.138.210

10.128.138.211

 

 

 

 

 

 

Private FDQN

prdrac01-priv.corp.localdomain

prdrac02-priv.corp.localdomain

 

prddg01-priv.corp.localdomain

prddg02-priv.corp.localdomain

Private IP

192.168.138.200

192.168.138.201

 

192.168.138.210

192.168.138.211

 

 

 

 

 

 

VIP FDQN

prdrac01-vip.corp.localdomain

prdrac02-vip.corp.localdomain

 

prddg01-vip.corp.localdomain

prddg02-vip.corp.localdomain

VIP IP

10.128.138.202

10.128.138.203

 

10.128.138.215

10.128.138.216

 

 

 

 

 

 

SCAN

prdrac-scan.corp.localdomain

prdrac-scan.corp.localdomain

 

prddg-scan.corp.localdomain

prddg-scan.corp.localdomain

 

10.128.138.204

10.128.138.204

 

10.128.138.219

10.128.138.219

 

10.128.138.205

10.128.138.205

 

10.128.138.220

10.128.138.220

 

10.128.138.206

10.128.138.206

 

10.128.138.221

10.128.138.221

 

Below shows the Oracle RAC VMs prdrac01 and prdrac02 public and private interconnect networking details

 

 

Below table provides Oracle RAC VM prdrac01 disk layout and ASM disk group configuration.

!!!T9!!! 'VM prdrac02' 2 Rows [/dev/sdb1]

Name

SCSI TYPE

SCSI ID (Controller, LUN)

Size

Type

Hard Disk

Disk Name

Operating System /

Paravirtual

SCSI (0:0)

50GB

ext4 file system

1

/dev/sda1

Oracle binary disk /u01

Paravirtual

SCSI (0:1)

50GB

ext4 file system

2

/dev/sdb1

 

Table : RAC VM prdrac02 disk layout

 

In addition to the two VMDKs shown in table above , attach the previously created prdrac01 shared VMDKs to prdrac02 on the same SCSI X:Y slot with the multiwriter attribute set.

 

Below is a list of Oracle RAC prdracOracle ASM disk groups.

 

 

Below command output displays the Oracle RAC prdrac cluster services:


[root@prdrac01 ~]# /u01/app/12.2.0/grid/bin/crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.CRS_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.DATA_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.FRA_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.GIMR_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.LISTENER.lsnr
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.REDO_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.VOTE_DG.dg
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.chad
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.net1.network
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.ons
     ONLINE ONLINE prdrac01 STABLE
     ONLINE ONLINE prdrac02 STABLE
ora.proxy_advm
     OFFLINE OFFLINE prdrac01 STABLE
     OFFLINE OFFLINE prdrac02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
   1 ONLINE ONLINE prdrac02 STABLE
ora.LISTENER_SCAN2.lsnr
   1 ONLINE ONLINE prdrac01 STABLE
ora.LISTENER_SCAN3.lsnr
   1 ONLINE ONLINE prdrac01 STABLE
ora.MGMTLSNR
   1 ONLINE ONLINE prdrac01 169.254.208.150 192.168.138.200, STABLE
ora.asm
   1 ONLINE ONLINE prdrac01 Started,STABLE
   2 ONLINE ONLINE prdrac02 Started,STABLE
   3 OFFLINE OFFLINE STABLE
ora.cvu
   1 ONLINE ONLINE prdrac01 STABLE
ora.mgmtdb
   1 ONLINE ONLINE prdrac01 Open,STABLE
ora.prdrac.db
   1 ONLINE ONLINE prdrac01 Open,HOME=/u01/app/oracle/product/12.2.0/dbhome_1, STABLE
   2 ONLINE ONLINE prdrac02 Open,HOME=/u01/app/oracle/product/12.2.0/dbhome_1,STABLE
ora.prdrac01.vip
   1 ONLINE ONLINE prdrac01 STABLE
ora.prdrac02.vip
   1 ONLINE ONLINE prdrac02 STABLE
ora.qosmserver
   1 ONLINE ONLINE prdrac01 STABLE
ora.scan1.vip
   1 ONLINE ONLINE prdrac02 STABLE
ora.scan2.vip
   1 ONLINE ONLINE prdrac01 STABLE
ora.scan3.vip
   1 ONLINE ONLINE prdrac01 STABLE
--------------------------------------------------------------------------------
[root@prdrac01 ~]#

Below shows the Oracle Far Sync VMs prdfs network details:

 

 

Below table provides Oracle Far Sync VM prdfs disk layout and ASM disk group configuration.

!!!T10!!! 'Oracle Far Sync' and 'FRA_DISK01' 4Rows [REDO_DISK01]

Name

SCSI TYPE

SCSI ID (Controller, LUN)

Size

Type

Hard Disk

Disk Name

Operating System (OS) /

Paravirtual

SCSI (0:0)

50GB

ext4 file system

1

/dev/sda1

Oracle binary disk /u01

Paravirtual

SCSI (0:1)

50GB

ext4 file system

2

/dev/sdb1

FRA Disk 1

Paravirtual

SCSI (1:0)

200GB

FRA_DG

3

FRA_DISK01

REDO Disk 1

Paravirtual

SCSI (1:1)

30GB

REDO_DG

4

REDO_DISK01

 

Below table provides Oracle RAC VM prddg02 disk layout:

Name

SCSI TYPE

SCSI ID (Controller, LUN)

Size

Type

Hard Disk

Disk Name

Operating System /

Paravirtual

SCSI (0:0)

50GB

ext4 file system

1

/dev/sda1

Oracle binary disk /u01

Paravirtual

SCSI (0:1)

50GB

ext4 file system

2

/dev/sdb1

 

In addition to the two VMDKs shown in the table above , attach the previously created prddg01 shared VMDKs to prddg02 on the same SCSI X:Y slot with the multiwriter attribute set.

 

The following is a list of Oracle RAC prddg Oracle ASM disk groups:

 

 

Below shows the Oracle RAC VMs prddg01 and prddg02 public and private interconnect networking details:

 

 

Below table provides Oracle RAC VM prddg01 disk layout and ASM disk group configuration.

Name

SCSI TYPE

SCSI ID (Controller, LUN)

Size

Type

Hard Disk

Disk Name

Operating System (OS) /

Paravirtual

SCSI (0:0)

50GB

ext4 file system

1

/dev/sda1

Oracle binary disk /u01

Paravirtual

SCSI (0:1)

50GB

ext4 file system

2

/dev/sdb1

DATA Disk 1

Paravirtual

SCSI (1:0)

200GB

DATA_DG

3

DATA_DISK01

FRA Disk 1

Paravirtual

SCSI (2:0)

100GB

FRA_DG

4

FRA_DISK01

GIMR Disk 1

Paravirtual

SCSI (2:1)

50GB

GIMR_DG

5

GIMR_DISK01

REDO Disk 1

Paravirtual

SCSI (3:0)

30GB

REDO_DG

6

REDO_DISK01

CRS Disk 1

Paravirtual

SCSI (3:1)

20GB

CRS_DG

7

CRS_DISK01

VOTE Disk 1

Paravirtual

SCSI (3:2)

10GB

VOTE_DG

8

VOTE_DISK01

VOTE Disk 2

Paravirtual

SCSI (3:3)

10GB

VOTE_DG

9

VOTE_DISK02

VOTE Disk 3

Paravirtual

SCSI (3:4)

10GB

VOTE_DG

10

VOTE_DISK03

VOTE Disk 4

Paravirtual

SCSI (3:5)

10GB

VOTE_DG

11

VOTE_DISK04

VOTE Disk 5

Paravirtual

SCSI (3:6)

10GB

VOTE_DG

12

VOTE_DISK05

 

 

 

 

 

 

 

New Disks

 

 

 

 

 

 

DATA Disk 2

Paravirtual

SCSI (1:1)

200GB

DATA_DG

13

DATA_DISK02

FRA Disk 2

Paravirtual

SCSI (2:2)

100GB

FRA_DG

14

FRA_DISK02

GIMR Disk 2

Paravirtual

SCSI (2:3)

50GB

GIMR_DG

15

GIMR_DISK02

REDO Disk 2

Paravirtual

SCSI (3:8)

30GB

REDO_DG

16

REDO_DISK02

CRS Disk 2

Paravirtual

SCSI (3:9)

20GB

CRS_DG

17

CRS_DISK02

VOTE Disk 6

Paravirtual

SCSI (3:10)

10GB

VOTE_DG

18

VOTE_DISK06

VOTE Disk 7

Paravirtual

SCSI (3:11)

10GB

VOTE_DG

19

VOTE_DISK07

VOTE Disk 8

Paravirtual

SCSI (3:12)

10GB

VOTE_DG

20

VOTE_DISK08

VOTE Disk 9

Paravirtual

SCSI (3:13)

10GB

VOTE_DG

21

VOTE_DISK09

VOTE Disk 10

Paravirtual

SCSI (3:14)

10GB

VOTE_DG

22

VOTE_DISK10

 

Table : Oracle RAC VM prddg02 Disk Layout

Below table provides Oracle RAC VM prddg02 disk layout:

Name

SCSI TYPE

SCSI ID (Controller, LUN)

Size

Type

Hard Disk

Disk Name

Operating System /

Paravirtual

SCSI (0:0)

50GB

ext4 file system

1

/dev/sda1

Oracle binary disk /u01

Paravirtual

SCSI (0:1)

50GB

ext4 file system

2

/dev/sdb1

 

 

In addition to the two VMDKs shown in the table above , attach the previously created prddg01 shared VMDKs to prddg02 on the same SCSI X:Y slot with the multiwriter attribute set.

The following is a list of Oracle RAC prddg Oracle ASM disk groups:

 

 

Below command output shows a list of Oracle RAC prddg cluster services:

 

[root@prddg01 ~]# /u01/app/12.2.0/grid/bin/crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State Details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.DATA_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.FRA_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.GIMR_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.LISTENER.lsnr
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.OCR_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.REDO_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.VOTE_DG.dg
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.chad
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.net1.network
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.ons
     ONLINE ONLINE prddg01 STABLE
     ONLINE ONLINE prddg02 STABLE
ora.proxy_advm
     OFFLINE OFFLINE prddg01 STABLE
     OFFLINE OFFLINE prddg02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
   1 ONLINE ONLINE prddg02 STABLE
ora.LISTENER_SCAN2.lsnr
   1 ONLINE ONLINE prddg01 STABLE
ora.LISTENER_SCAN3.lsnr
   1 ONLINE ONLINE prddg01 STABLE
ora.MGMTLSNR
   1 ONLINE ONLINE prddg01 169.254.108.242 192.168.138.210,STABLE
ora.asm
   1 ONLINE ONLINE prddg01 Started,STABLE
   2 ONLINE ONLINE prddg02 Started,STABLE
   3 OFFLINE OFFLINE STABLE
ora.cvu
   1 ONLINE ONLINE prddg01 STABLE
ora.mgmtdb
   1 ONLINE ONLINE prddg01 Open,STABLE
ora.prddg01.vip
   1 ONLINE ONLINE prddg01 STABLE
ora.prddg02.vip
   1 ONLINE ONLINE prddg02 STABLE
ora.qosmserver
   1 ONLINE ONLINE prddg01 STABLE
ora.scan1.vip
   1 ONLINE ONLINE prddg02 STABLE
ora.scan2.vip
   1 ONLINE ONLINE prddg01 STABLE
ora.scan3.vip
   1 ONLINE ONLINE prddg01 STABLE
--------------------------------------------------------------------------------
[root@prddg01 ~]#