View source for Setup manual lab environment for vCloud Foundation using nested ESXi hosts

[[Main_Page|Home]] > [[VMWare platform]] > [[VCloud Foundation|vCloud Foundation]] > [[Setup manual lab environment for vCloud Foundation using nested ESXi hosts]]

It is possible to create nested four ESXi hosts and then use them to deploy vCloud Foundation.  To do this easily we can refer to [[Setup automated lab environment for vCloud Foundation using VLC]].  But to have more control on environment and to understand things better we can try the same without using VLC.  To do that use following steps

# First build nested ESXi template of desired ESXi version as explained in particular vCF version release notes.  For nested setup see [[Install nested ESXi on top of ESXi in a VM]]
#: Perhaps create host with 40GB disk for ESXi installaation, 10GB disk for cache and 3x200GB disks for capacity
#: If there is no direct ISO for the mentioned version then download the lower version available.  Then update ESXi host using [[Install ESXi patch via depot zip file]]
#: Base nodes should have CPU capable of running ESXi required for particular vCF version.  For example if the underlying physical CPU can only support till ESXi 6.5U3 and then we cannot deploy ESXi 7.0 in nested VM on top of same CPU.
# Create NSX port-group for all VLAN trunk using [[Create standard port-group for trunking all VLANs to VM]].  Each nested ESXi VM should have 4 vmxnet3 uplinks connecting to ALL VLAN trunk port-group
# '''On both switch and at port-group level in security settings allow Promiscious mode, MAC address change and forge transmits by setting all three to Accept.'''
# Change the MTU of vSwitch (Standard or DVS) to at least 9000
# To build nested VSAN node refer [[Create nested ESXi for vSAN experiments with SSD disk]]
#: Basically download the VM .vmx file.  Edit it to have '<tt>scsi0:1.virtualSSD = "1"</tt>' for 10GB and 3x200GB disks.  The 40GB disk can be left as it is without making it SSD.
# To build template of ESXi host so that it can be cloned to get 4 identical ESXi hosts refer [[Create template of nested ESXi for cloning and repeat use]]
# Build networks and parameters as explained in NSX-T article [[Configure NSX-T 3.0 from scratch with edge cluster and tier gateways]]
#* Compared to NSX article linked here note that vCF does not allows ToR and VMWare T0 gateway on edge to have same BGP AS number.  Hence change AS number on ToR to 65001 while keeping rest settings same to deploy vCF
#* Since NSX article was not created with vSAN network in mind, re-use DHCP (host-VTEP network) for vSAN also.  Choose IPs near the end such as 10.1.3.210 to 10.1.3.240 for vSAN as anyway DHCP will not release as many IPs for host-VTEP.
# For nested VMs created using templates there would be issue with SSL Certificates.  That can be resolving using:
## Set correct hostname with .domain-suffix 
## '''Do not have any custom suffix (It should be not set)'''
## Run the /sbin/generate-certificate command from ESXi shell (SSH)
## Reboot host (Restarting services as explained in below article is not enough)
#: Refer: https://my-cloudy-world.com/2021/02/10/cloud-builder-validation-ssl-certificate-common-name-doesnt-match-esxi-fqdn/
# Prepare ESXi hosts for vCF using:
## Configure Management IP (10.1.1.11-14), Managment VLAN (201), DNS (10.1.1.2), DNS suffix as vcfrnd.com - https://docs.vmware.com/en/VMware-Cloud-Foundation/4.2/com.vmware.vcf.ovdeploy.doc_42/GUID-E22AA0E2-98F3-479F-A983-02D83D2FBF17.html
## VM network VLAN ID same as management (201) - https://docs.vmware.com/en/VMware-Cloud-Foundation/4.2/com.vmware.vcf.ovdeploy.doc_42/GUID-1FCA9839-02D7-437C-8666-04DD6F022083.html
## Start and enable ssh and ntp services - https://docs.vmware.com/en/VMware-Cloud-Foundation/4.2/com.vmware.vcf.ovdeploy.doc_42/GUID-9AFB9772-8B23-4015-A751-ED04B7CC305D.html
# Build a jump host with IP 10.1.1.2 for accessing cloud builder and for providing NTP and DNS services.   Example configuration files are mentioned at end of this article assuming domain name to be vcfrnd.com
## Configure NTP as explained with [[#Example_ntp_configuration_file]] below
## Configure DNS as explained with [[#Example_DNS_configuration_files]] '''including reverse entries''' below
# Fill the deployment sheet.  Add appropriate DNS entries.  Configure BGP values appropriately.  
#: Example deployment sheet is available at [[Media:2021-04-16-vcf-manual-deployment.xlsx|2021-04-16-vcf-manual-deployment.xlsx]]
# Deploy cloud builder appliance with IP 10.1.1.31 - VLAN 201 - hostname cb - Domain name vcfrnd.com
# Open cloud builder interface in '''private browser''' window ( https://10.1.1.31/ ).  Upload the filled sheet and run validation
#: '''If after uploading sheet you get "Internal server error".  Best option is to delete cloud builder VM and deploy a new one.'''
# If you are doing this for lab setup and have limited resources, considering building json from xlsx file and removing two network managers to save resources using:
## scp xlsx from jump box to /home/admin in cloud builder
## ssh as admin to cloud builder
## sudo su -
## cd /opt/vmware/sddc-support
## ./sos --jsongenerator --jsongenerator-input /home/admin/poclab-ems-deployment-parameter.xlsx --jsongenerator-design vcf-public-ems
## cd /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-public-ems
##  cp vcf-public-ems.json /home/admin
## chown admin:vcf /home/admin/vcf-public-ems.json
## scp json from /home/admin to jump-box
## In jump box edit json and remove network manager specs for 10.1.1.6 (nsx01b) and 10.1.1.7 (nsx01c).   Same can be seen in example json at [[Media:Vcf-public-ems.json|Vcf-public-ems.json]]
#: Refer https://greatwhitetec.com/2020/08/05/vcf-lab-tips-nsx-cluster-size/ and
#: https://greatwhitetec.com/2020/07/28/vcf-generate-json-file-from-excel-spreadsheet/
# '''Ideally once validate is successful.  Shutdown all four nested ESXi hosts and take a snapshot.  This helps in getting back to validation succeed stage quickly.'''
# Perform build up
#: Note that build up may fail multiple times.  It should be possible to restart cloud builder or VMs (vCenter, etc.) running inside nested ESXi host to solve problem.  If all fails you can revert to snapshot when validation succeeded and try again.
#: The lab deployment works when done using a 2TB SSD USB as datastore [[Configure VMFS filesystem on USB disk and mount it as datastore on ESXi host]] with host with 256GB RAM where each nested ESXi host is created with 8 cores and 48GB RAM 
#: You can monitor various logs at /opt/vmware/bringup/logs with tail -f * in cloud builder appliance and see what is going on.  Any specific log file can be opened in another terminal
#: See [[#Bring_up_failed_due_to_authentication_failures_from_Cloud_builder_to_vCF_SDDC_appliance]]
# '''Change "vSAN Default Storage Policy" to have FTT=0, Disable object checksum, Enable force provisioning'''.  This should ensure that data is written only once and not twice.
# You can now try to deploy vRealize Suite components by referring [[Deploy vRealize LifeCycle Manager on vCloud Foundation]]


'''Note that in nested environments with NSX ping responses might appear to be duplicate.  This only happens in nested environment where the nested VMs are on same physical host.  '''
See: https://communities.vmware.com/t5/VMware-NSX-Discussions/Duplicated-Ping-responses-in-NSX/td-p/969774



==Bring up failed due to authentication failures from Cloud builder to vCF SDDC appliance==
In case of Retry bringup you can possibly see following log lines in file '<tt>/opt/vmware/bringup/logs/vcf-bringup.2021-04-21.0.log</tt>'
<pre>
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Rollback failed for configure base install repo task
...
caused by: com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to gbb-vcf01.vcfrnd.com
</pre>

After this even local login on nested vCF SDDC VM via nested vCenter console fails.  To solve this:
# Reboot the vcf appliance
# After reboot login into vCF appliance console via nested vCenter / ESXi host.   You should enter corrrect password in first attempt.
# Then edit '<tt>/etc/pam.d/system-auth</tt>' to change deny, root_unlock_time and unlock_time values:
#:<pre>
#::    auth    required       pam_tally2.so file=/var/log/tallylog deny=30 onerr=fail even_deny_root unlock_time=4 root_unlock_time=3
#:</pre>
# Edit '<tt>/etc/ssh/sshd_config</tt>' to allow more login attempts:
#:<pre>
#::    MaxAuthTries 200
#::    PermitRootLogin yes
#:</pre>
# Restart ssh server
#:<pre>
#:: systemctl restart sshd
#:</pre>
# After this generated ssh-key pair on cb '<tt>ssh-keygen</tt> root account and copied the keys as authorized to root of vcf '<tt>ssh-copy-id root@10.1.1.8</tt>'.
# Restarted the bring-up process.  Found following in logs this time in file <tt>/opt/vmware/bringup/logs/vcf-bringup.log</tt>
#:<pre>
#::  2021-04-22T06:12:07.411+0000 [bringup,558d56060e59839d,9e4f] INFO  [c.vmware.vcf.secure.ssh.SshExecuter,bringup-exec-8] Creating directory /nfs/vmware/vcf/nfs-mount/base-install-images/nsxt_ova on host: gbb-vcf01.vcfrnd.com
#::  2021-04-22T06:12:07.411+0000 [bringup,558d56060e59839d,0d58] INFO  [c.vmware.vcf.secure.ssh.SshExecuter,bringup-exec-7] Creating directory /nfs/vmware/vcf/nfs-mount/base-install-images/vcenter_ova on host: gbb-vcf01.vcfrnd.com
#:</pre>
# Note that there is also
#:<pre>
#::    pam_tally2 -u root
#::    pam_tally2 -u root -r
#:</pre>
#: but this does not helps as no. of authentication attempts is too small.  That count ends up getting used by ssh-keys itself.

Refer:
* https://davidring.ie/2019/11/25/vcf-sddc-manager-reset-vcf-user-a-c-due-to-failed-login-attemtps/




==Example ntp configuration file==

See [[CentOS 8.x chronyc ntp client configuration]] for chrony configuration on CentOS 8.  Along with the steps use below options in configuration file:
<pre>
server time.google.com

allow 192.168.0.0/16
allow 10.0.0.0/8
allow 172.16.0.0/12

local stratum 10
</pre>



==Example DNS configuration files==
See [[Configuring basic DNS service with bind]] for DNS server setup.  Then have following additional lines/files:

===named.conf===
After zone "." definition insert:
<pre>
zone "vcfrnd.com." IN {
        type master;
        file "vcfrnd.com.forward";
};

zone "1.10.in-addr.arpa." {
	type master;
	file "10.1.reverse.db";
};
</pre>


===/var/named/vcfrnd.com.forward===
<pre>
$TTL 3600
@ SOA ns.vcfrnd.com. root.vcfrnd.com. (1 15m 5m 30d 1h)
	NS ns.vcfrnd.com.
	A 10.1.1.2
ns              IN      A       10.1.1.2
ntp		IN	A	10.1.1.2
gbb01-m01-esx01	IN	A	10.1.1.11
gbb01-m01-esx02	IN	A	10.1.1.12
gbb01-m01-esx03	IN	A	10.1.1.13
gbb01-m01-esx04	IN	A	10.1.1.14
gbb-m01-vc01	IN	A	10.1.1.3
gbb-m01-nsx01	IN	A	10.1.1.4
gbb-m01-nsx01a	IN	A	10.1.1.5
gbb-m01-nsx01b	IN	A	10.1.1.6
gbb-m01-nsx01c	IN	A	10.1.1.7
gbb-vcf01	IN	A	10.1.1.8
gbb-m01-en01	IN	A	10.1.1.21
gbb-m01-en02	IN	A	10.1.1.22
cb		IN	A	10.1.1.31
vrslcm		IN	A	10.1.8.11
wsa		IN	A	10.1.8.12
vrli		IN	A	10.1.8.13
vrops		IN	A	10.1.8.14
</pre>


===/var/named/10.1.reverse.db===
<pre>
$TTL 3600
@ SOA	ns.vcfrnd.com. root.vcfrnd.com. (1 15m 5m 30d 1h)
	NS ns.vcfrnd.com.

1.1		PTR	l3router.vcfrnd.com.
2.1		PTR	ntp.vcfrnd.com
2.1		PTR	ns.vcfrnd.com
3.1		PTR	gbb-m01-vc01.vcfrnd.com.
4.1		PTR	gbb-m01-nsx01.vcfrnd.com.
5.1		PTR	gbb-m01-nsx01a.vcfrnd.com.
6.1		PTR	gbb-m01-nsx01b.vcfrnd.com.
7.1		PTR	gbb-m01-nsx01c.vcfrnd.com.
8.1		PTR	gbb-vcf01.vcfrnd.com.
11.1		PTR	gbb01-m01-esx01.vcfrnd.com.
12.1		PTR	gbb01-m01-esx02.vcfrnd.com.
13.1		PTR	gbb01-m01-esx03.vcfrnd.com.
14.1		PTR	gbb01-m01-esx04.vcfrnd.com.
21.1		PTR	gbb-m01-en01.vcfrnd.com.
22.1		PTR	gbb-m01-en02.vcfrnd.com.
31.1		PTR	cb.vcfrnd.com.
11.8		PTR	vrslcm.vcfrnd.com.
12.8		PTR	wsa.vcfrnd.com.
13.8		PTR	vrli.vcfrnd.com.
14.8		PTR	vrops.vcfrnd.com.
</pre>





[[Main_Page|Home]] > [[VMWare platform]] > [[VCloud Foundation|vCloud Foundation]] > [[Setup manual lab environment for vCloud Foundation using nested ESXi hosts]]