vCloud Startup Issue

By Karthic K 12 Dec, 2013

vCD service fails to start for numerous reasons, if you understand what exactly it requires to start the vCD service then you may resolve the issues. I tried explaining what happens when you start the vCloud director services. Below is the workflow that I understood from my experience

Start-up Flow:

Startup

vCloud Director uses JAVA OSGI, which is Java Open Services Gateway Initiative (OSGi) defines an architecture for developing and deploying modular applications and libraries.

To know more about OSGI you may refer wiki article here. What we need to understand is, OSGI is a Container or Folder which has its own sub/internal components. OSGI should be completely up and running for vCloud to properly function.

To begin with check the troubleshooting checklist first. Detailed information can be found below.

Verifying IP/FDQN:

Check if the RHEL cell is configured with proper FQDN and make sure point to a valid DNS address. This is the initial requirement, 2 IPs, one for vCloud HTTP traffic and 2nd one for console proxy, of-course it wouldn’t allow you to install vCloud director application if you do not have 2 nics / ips. There may be chances that NIC might have removed by mistake

 Certificate:

SSL certificate is a crucial requirement for vCloud director since it is a enterprise application and it is used largely by public service providers. The customers / clients / end-user should be confident that their connectivity is secured and encrypted so that a intruder cannot access the packets.Check if the certificates are valid and updated. you may run the below command in the vCloud director server (RHEL / Centos with vCloud Installed).

[code]#opt/vmware/vcloud-director/jre/bin/keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -list[/code]

To know more about creating / validating certificates, check below VMware KB.

 

http://kb.vmware.com/kb/1026309

NTP Issues:
Make sure all the  below components are properly synced with NTP server.
  • vCenter Server
  • vShield Manager
  • All vCloud Cells
  • vCloud DB
 DB Connectivity  Issues:
vCD stores all its information in the database, the main reason behind this is clustering and load balancing. you can have more than one vCloud director cell for load balancing and faster access. So if you have more than one vCloud cell then all the cells must have proper connectivity to the Database. At present vCD support Oracle and MSSQL, check the vmware interop matrix for supported versions.Verify the vCD cell can communicate with DB server, may be try a basic test.To know the configured DB for vCloud, you may  check in the global.properties file, there are lot of information in global.properties but we need only the database properties, you may run below command

[code]#grep database.jdbcUrl /opt/vmware/vcloud-director/etc/global.properties[/code]

After you get the DB FQDN /IP, perform telnet with respective port to check the connectivity 

[code]#telnet <DB_SERVER> 1433 (MSSQL)[/code]

[code]#telnet <DB_SERVER> 1521 (Oracle)[/code]

Sometimes vCD can connect to the DB server but it is not enough, the user must have DB_Owner privileges to the vCloud Database. if there is permission issue for the DB user, the OSGI container may start but you cannot access the vCloud UI, it shows a blank grey screen. To find such errors, you can verify cell.log for detailed information.

Reviewing vCloud cell.log
Sometimes the vCD services are fully started but you may see a blank/black screen in the vCD GUI, the first place to look vCD startup issue is

[code]$VCLOUD_HOME/logs/cell.log.[/code]

Below is an example of a fully started cell.log file so you can use it to compare.  This log re-writes every time the application restart, you may use this as reference.

 

Application startup begins: MM/DD/YY HH:MM AM
Successfully bound network port: 80 on host address: 192.168.110.xxx
Successfully bound network port: 443 on host address: 192.168.110.xxx
Application Initialization: 9% complete. Subsystem 'com.vmware.vcloud.common.core' started
Successfully connected to database: jdbc:oracle:thin:@Oracle01.test.local:1521/orcl
Successfully bound network port: 443 on host address: 192.168.110.yyy
Successfully bound network port: 61616 on host address: 192.168.110.xxx
Successfully bound network port: 61613 on host address: 192.168.110.xxx
Application Initialization: 18% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application Initialization: 27% complete. Subsystem 'com.vmware.vcloud.consoleproxy' started
Application Initialization: 36% complete. Subsystem 'com.vmware.vcloud.vlsi-core' started
Application Initialization: 45% complete. Subsystem 'com.vmware.vcloud.vim-proxy' started
Successfully verified transfer spooling area: /opt/vmware/cloud-director/data/transfer
Application Initialization: 54% complete. Subsystem 'com.vmware.vcloud.backend-core' started
Application Initialization: 63% complete. Subsystem 'com.vmware.vcloud.imagetransfer-server' started
Application Initialization: 72% complete. Subsystem 'com.vmware.vcloud.rest-api-handlers' started
Application Initialization: 81% complete. Subsystem 'com.vmware.vcloud.ui.configuration' started
Application Initialization: 90% complete. Subsystem 'com.vmware.vcloud.jax-rs-servlet' started
Application Initialization: 100% complete. Subsystem 'com.vmware.vcloud.ui-vcloud-webapp' started
Application Initialization: Complete. Server is ready in 0:46 (minutes:seconds)
Successfully initialized ConfigurationService session factory
Successfully started scheduler
Successfully started remote JMX connector on port 8999