DatabaseVerse: Journey into the World of Databases

1. Context
2. The importance of the tooling
3. Error
4. Fixing the issue
5. Is there any pre check?
6. Conclusion

If you are reading the title you will be like “eh?”, well, let me explain you the context and you will get a better idea (I hope so)

Context

You already know the first clue, this happened in an Exadata Cloud at Customer (ExaDB-C@C) gen2. I have created a database using the dbaascli and everything was smooth, but then when I tried to create the Data Guard (DG) using the OCI console… something happened… it failed, and the message was not enough for me and maybe for anybody who read it:

The importance of the tooling

Most of the automations rely on dbaascli, dbaasapi, dbcs agent (as the tools check if is running and what is the version) and for dg creation there are others like Data Guard Deployer (DGdeployer), Oracle Cloud Deployment Engine (ODCE)

Error

I didn’t want to open a case but I did anyway, in the meanwhile I was looking for the logs in the server and I found something interesting in /var/opt/oracle/cstate/*

"CDG-50629" : [
"dbaastools_exa version does not match between primary and standby clusters",
"Install same version of dbaastools_exa on primary and standby clusters",
"dbcs-agent-update-exacc version does not match between primary and standby clusters",
"Install same version of dbcs-agent-update-exacc on primary and standby clusters"
],

AHA!, I have more information now that I can work with, let’s see our rpm versions (the case was similar for primary and standby clusters):

[opc@exaccprimnode1]$ rpm -qa --last | egrep -i 'dbaas|dbcs'
dbcs-agent-exacc-2.8OL7-22.3.1.1.0_221013.0027.x86_64 Thu 18 Jan 2024 01:01:35 PM CET
dbaastools_exa-1.0-23.4.1.0.0_231213.2304.x86_64 Thu 18 Jan 2024 12:11:32 PM CET
[opc@exaccprimnode2~]$ rpm -qa --last | egrep -i 'dbaas|dbcs'
dbaastools_exa-1.0-23.4.1.0.0_231213.2304.x86_64 Thu 18 Jan 2024 12:11:43 PM CET
dbcs-agent-update-exacc-23.4.1.0.0-231219.1834.x86_64 Thu 18 Jan 2024 06:57:12 AM CET
dbcs-agent-exacc-2.8OL7-22.3.1.1.0_221013.0027.x86_64 Wed 07 Dec 2022 10:53:57 AM CET
[opc@exaccprimnode3]$ rpm -qa --last | egrep -i 'dbaas|dbcs'
dbaastools_exa-1.0-23.4.1.0.0_231213.2304.x86_64 Thu Jan 18 12:11:55 2024
dbcs-agent-update-exacc-23.4.1.0.0-231219.1834.x86_64 Thu Jan 18 06:57:00 2024
dbcs-agent-exacc-2.8OL7-23.3.1.0.0_231028.0918.x86_64 Tue Dec 12 11:33:55 2023
[opc@exaccprimnode1~]$ sudo dbaascli admin showLatestStackVersion
DBAAS CLI version 23.4.1.0.0
Executing command admin showLatestStackVersion
Job id: xxxxxx
Session log: /var/opt/oracle/log/admin/showLatestStackVersion/dbaastools_2024-01-18_01-53-19-PM_186006.log
{
"description" : "dbaastools for Exadata cloud service EXACC",
"version" : "23.4.1.0.0_231219.1834",
"baseVersion" : "23.4.1.0.0"
}
dbaascli execution completed

Looking at the output and comparing the output between nodes, I found two things:

• dbcs-agent-update-exacc* rpm was missing from node1
• dbcs-agent-exacc-2* rpm on node3 was newer than on node1 and node2:
  • dbcs-agent-exacc-2.8OL7-23.3.1.0.0_231028.0918.x86_64 > dbcs-agent-exacc-2.8OL7-22.3.1.1.0_221013.0027.x86_64

Fixing the issue

Ok, let’s see if updating the dbaascli will update the DBCS RPMs as well. For that, I have executed:

dbaascli admin updateStack --version LATEST

It didn’t update my DBCS rpm.. so, where to download the rpm and update them? That was the matter here.

After doing so research I found this note: How to install DBCS Admin and DBCS Agent Exadata Cloud at Customer (ExaCC Gen2) (Doc ID 2940756.1)

The note is ok but is not completely up to date. It mentioned the following:

ExaCC DBCS AGENT RPM Format:

• DBCS Admin: dbcs-agent-exacc-2.8OL7-x.x.x.0.x_Date.xxxx.x86_64.rpm
• DBCS Agent: dbcs-agent-update-exacc-xx.x.x.0.x_Date.xxxx.x86_64.rpm

DBCS Admin Information

DBCS Admin RPM is NOT downloadable from the CPS. You need to use the following PAR URL to download RPM.

And the URL is for an old RPMversion… dbcs-agent-exacc-2.8OL7-22.4.1.0.1_221214.1249.x86_64.rpm

It also mentioned that the DBCS Agent can be downloaded from the CPS while accessing the OSS URL. So I followed the steps to download the Agent and the interesting part is that I found the DBCS Admin RPM as well.

To find the OSS URL you can check the info in the /var/opt/oracle/exapatch/exadbcpatch.cfg

grep oss_container /var/opt/oracle/exapatch/exadbcpatch.cfg

Example:

oss_container_url="http://169.254.200.5:2080/dbaas_patch/shome"

In case that the oss_container is empty (it is possible), you can find out the ip address executing:

ip a s | grep '169.254'

Example:

ip a s | grep '169.254'
inet 169.254.200.6/30 brd 169.254.200.7 scope global eth0 

NAT IP for the OSS will be “-1” from the above IP 169.254.200.6 which is 169.254.200.5

It is good practice to modify the file /var/opt/oracle/exapatch/exadbcpatch.cfg (do a previous backup of course) and set the oss_container variable once you have identify the correct value.

Now, you know what is the OSS URL but you need to know what is the latest version of the RPM for that as root user:

dbaascli admin showLatestStackVersion

You will see an output like this:

[root@exaccprimnode1~]$ dbaascli admin showLatestStackVersion
DBAAS CLI version 23.4.1.0.0
Executing command admin showLatestStackVersion
Job id: xxxxxx
Session log: /var/opt/oracle/log/admin/showLatestStackVersion/dbaastools_2024-01-18_01-53-19-PM_186006.log
{
"description" : "dbaastools for Exadata cloud service EXACC",
"version" : "23.4.1.0.0_231219.1834",
"baseVersion" : "23.4.1.0.0"
}
dbaascli execution completed

The two important things here are:

"version" : "23.4.1.0.0_231219.1834",
"baseVersion" : "23.4.1.0.0"

To download the RPM we simply execute the curl command to the OSS container:

• DBCS ADMIN:

curl -O http://169.254.200.X:2080/dbaas_patch/dbcsagent/<BASE_VERSION>/dbcs-admin-update-exacc-<VERSION>.x86_64.rpm -vvv

• DBCS Agent:

curl -O http://169.254.200.X:2080/dbaas_patch/dbcsagent/<BASE_VERSION>/dbcs-agent-update-exacc-<VERSION>.x86_64.rpm -vvv

• Where X in this example is 5, so the whole IP should be: 169.254.200.5
• Base Version: 23.4.1.0.0
• Version: 23.4.1.0.0_231219.1834

Once you have the RPM you can just install the DBCS Admin and DBCS Agent like the note mentioned:

1. List installed DBCS RPMs 

rpm -qa | grep 'dbcs'

2. Remove above-listed RPMs from the DomU (Do Not perform to update only DBCS Agent)

rpm -ev dbcs-agent-update-exacc dbcs-agent-exacc-2.8OL7

3. Install DBCS Admin RPM (check the version as it might be a newer one)

rpm -ivh dbcs-admin-update-exacc-23.4.1.0.0_231219.1834.rpm

4. Install / Upgrade DBCS Agent RPM (Run the following command in the case of DBCS Agent only) (check the version as it might be a newer one)

rpm -Uvh dbcs-agent-update-exacc-23.1.1.0.1_230302.2348.x86_64.rpm --force

5. Verify Installed RPM with the following

rpm -qa --last | egrep 'dbaas_exa|dbcs'

6. Verify DBCS Admin as well as DBCS Agent Services are up and running

systemctl status dbcsadmin
systemctl status dbcsagent

7. If Services are not up and running then start with the following and verify it. 

systemctl start dbcsadmin
systemctl start dbcsagent

Is there any pre check?

You can perform a pre check for the DG using dbaascli before trying to execute the workflow to create it executing the following command:

dbaascli dataguard verify --dbname XXX

This solved my issue with the RPM version mismatch between nodes and primary and standby cluster, the automation failed again with a different message that I will leave below, I will post about later and another post so stay tuned 😉

+-----------+---------------------------------------------------------------------------------------------------------------------------------------------+

| EXCEPTION | DETAILS |

+-----------+---------------------------------------------------------------------------------------------------------------------------------------------+

| CDG-50638 | Standby Environment already has the CREG resource name being used |

| | Check errors reported in logs |

| dg_api | CDG-50107 : DataGuard prechecks failed for stage VERIFY_DG_STANDBY |

| | Refer the exceptions raised and fix the issues |

| | File: dg_api, Line#: 1749, Log: /var/opt/oracle/log/XXXX/dbaasapi/db/dg/dbaasapi_VERIFY_DG_STANDBY_2024-01-26_10:05:40.869076_166150.log |

Conclusion

It is crucial to prioritize the regular maintenance and updates of essential toolings such as dbaastools, dbaascli, dbcs agent, AHF (tfa, exachk), syslen, and others within your environment. Ensuring that these tools are kept up to date is fundamental in preventing a multitude of issues related to the automation processes within Oracle Cloud Infrastructure and your engineering system like ExaDB-C@C. Furthermore, in the event of any unexpected failures, it is advisable to promptly open a case with Oracle Support. By doing so, you can leverage their expertise and assistance in troubleshooting and resolving any challenges that may arise, thereby ensuring the smooth operation of your environment.