Oracle RAC RDS reference:
Oracle Real Application Clusters (RAC) allows Oracle Database to run any packaged or custom application, unchanged across a set of clustered servers. This provides high availability and flexible scalability. If a clustered server fails, then database continues running on the rest of the servers. When you need additional processing power, you can add additional servers to the cluster online.
A key hardware component of this technology is the private interconnect network between the clustered servers. The interconnect is used by the cluster for inter-node messaging. The interconnect is also used by RAC to implement cache fusion technology.
Reliable Datagram Sockets (RDS) protocol provides reliable datagram services multiplexing UDP packets over InfiniBand connection improving performance to Oracle RAC. It provides high performance cluster interconnect for Oracle 10g RAC, utilizing InfiniBand which has 10X to 30X bandwidth advantage and 10X to 30X latency reduction vs. Gigabit Ethernet.
Only the database servers are configured with a 64K MTU. Presumably this is to benefit TCP/IP traffic between the database servers, and between the database servers and any external host that is linked to the IB switch. You may be surprised to know that the IB ports on the storage cells are configured with the standard 1,500 byte MTU size. The large MTU size is not necessary on the storage cells, because I/O between the database grid and the storage grid utilizes the RDS protocol, which is much more efficient for database I/O and bypasses the TCP/IP protocol stack altogether.
As you can see from the diagram, using the RDS protocol to bypass the TCP processing cuts out a portion of the overhead required to transfer data across the network. Note that the RDS protocol is also used for interconnect traffic between RAC nodes.
目前Exadata DBserver端与cell端 都采用了RDS协议 这使得Exadata 整体之间的通信变的更为流畅
CELL端可以看到日志：
CELL communication is configured to use 1 interface(s): 192.168.10.3 Memory reserved for cellsrv: 22355 MBMemory for other processes: 1600 MB. Auto Online Feature 1.2 CellServer MD5 Binary Checksum: a9f851b13ee6a072d7724e21c0c99118 OS Hugepage status: Total/free hugepages available=4013/4013; hugepage size=2048KB MS_ALERT HUGEPAGE CLEAR Cache Allocation: Num 1MB hugepage buffers: 8000 Num 1MB non-hugepage buffers: 0 Cache Allocation: BufferSize: 512. Num buffers: 5000. Start Address: 2AACA2E00000 Cache Allocation: BufferSize: 2048. Num buffers: 5000. Start Address: 2AACA3072000 Cache Allocation: BufferSize: 4096. Num buffers: 5000. Start Address: 2AACA3A37000 Cache Allocation: BufferSize: 8192. Num buffers: 10000. Start Address: 2AACA4DC0000 Cache Allocation: BufferSize: 16384. Num buffers: 5000. Start Address: 2AACA9BE1000 Cache Allocation: BufferSize: 32768. Num buffers: 5000. Start Address: 2AACAEA02000 Cache Allocation: BufferSize: 65536. Num buffers: 5000. Start Address: 2AACB8643000 Cache Allocation: BufferSize: 10485760. Num buffers: 23. Start Address: 2AACCBEC4000 CELL communication is configured to use 1 interface(s): 192.168.10.3 IPC version: Oracle RDS/IP (generic) IPC Vendor 1 Protocol 3 Version 4.1
database server 之间同样使用RDS
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL>
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/edw/edw1/trace/edw1_ora_81207.trc
[oracle@dm01db01 ~]$ more /u01/app/oracle/diag/rdbms/edw/edw1/trace/edw1_ora_81207.trc |grep RDS
SKGXP:[2ba9e2cb3e08.43]{ctx}: SSKGXPT 0x2ba9e2cb5548 flags 0×5 { READPENDING } sockno 5 IP 192.168.10.1 RDS 38133 lerr 0
SKGXP:[2ba9e2cb3e08.96]{ctx}: SSKGXPT 0x2ba9e3033e50 flags 0×0 sockno 11 IP 192.168.10.1 RDS 49737 lerr 0
SKGXP:[2ba9e2cb3e08.97]{ctx}: SKGXPGPID Internet address 192.168.10.1 RDS port number 49737, mask 22
[oracle@dm01db01 ~]$[root@dm01cel01 ExadataRDS]# rds-info -I
RDS IB Connections:
LocalAddr RemoteAddr LocalDev RemoteDev
192.168.10.3 192.168.10.3 fe80::21:2800:1ef:6ba3 fe80::21:2800:1ef:6ba3
192.168.10.3 192.168.10.2 fe80::21:2800:1ef:6ba3 fe80::21:2800:1ef:783f
192.168.10.3 192.168.10.1 fe80::21:2800:1ef:6ba3 fe80::21:2800:1ef:788b
