Difference between revisions of "CentOS 7.x Configure rekallcm1 (high-end desktop) with software raid, lxc, kvm and other utilities"

From Notes_Wiki
m
m
Line 224: Line 224:
===Configure nautlius===
===Configure nautlius===
'''From this point onwards we can work directly on rekallcm1, instead of working remotely'''
'''From this point onwards we can work directly on rekallcm1, instead of working remotely'''
# Open nautilus
# Open nautilus  
# Remove /home/saurabh/{Music,Pictures,Videos,Templates}
# Remove /home/saurabh/{Music,Pictures,Videos,Templates}
# Create bookmarks for   
# Create bookmarks for  (Open corresponding folder and use Ctrl+D for shortcut)
#* /mnt/data1/workspace/gbb/owncloud/saurabh
#* /mnt/data1/workspace/gbb/owncloud/saurabh
#* /mnt/data1
#* /mnt/data1

Revision as of 13:19, 9 December 2019

<yambe:breadcrumb>CentOS_7.x_New_machine_configuration|CentOS 7.x New machine configuration</yambe:breadcrumb>

CentOS 7.x Configure rekallcm1 (high-end desktop) with software raid, lxc, kvm and other utilities

Backup before formatting

Before formatting ensure that there is backup for on external drive

  • /etc
  • /home
  • /root
  • /mnt/data1 appropriate sub-folders
  • Last hourly.0 backup of cloud VMs excluding local backups


For backup to external disk, use steps similar to:

  mount /dev/sd<n>1 /mnt/usb
  mkdir /mnt/usb/2019-08-07-rekallcm1-backup
  rsync -axH /etc/ /mnt/usb/2019-08-07-rekallcm1-backup/etc/
  rsync -axH /home/ /mnt/usb/2019-08-07-rekallcm1-backup/home/
  cd /mnt/data1
  mkdir -p /mnt/usb/2019-08-07-rekallcm1-backup/mnt/data1
  rsync -axH plain_folders workspace /mnt/usb/2019-08-07-rekallcm1-backup/mnt/data1/
  rsync -axH large_files --exclude=large_files/rsnapshot_backups /mnt/usb/2019-08-07-rekallcm1-backup/mnt/data1/
  rsync -axH large_files/rsnapshot_backups/hourly.0 --exclude=hourly.0/rekallcm1 /mnt/usb/2019-08-07-rekallcm1-backup/rsnapshot_backups

Ensure that there is enough space in external disk (=df -h=) as required for various folders (=du -sh=). It is also important to check health of backup disk (=smartctl=) against failed self-test or other pre-failure alert signals.


OS installation

Install OS using following options (Assuming 3 disks of 2TB each):

  • If necessary validate hardware information eg processor, disks, RAM, etc. You can look at =smartctl= information of disks for results of last self-tests and any possible pre-failure alerts.
  • Partitioning:
    /boot/efi
    1GB - EFI filesytem - RAID 1
    /
    50GB - ext4 - RAID 1
    swap
    18GB - swap - LVM with Volume group using RAID 5
    /mnt/data1
    Remaining - ext4 - LVM with Volume group using RAID 5
  • Disable kdump
  • Do not apply any security profile
  • Server with GUI without Infiband, smart card, FTP related modules
  • Enable networking and configure appropriate hostname (Eg rekallcm1)
  • Create saurabh user with administrative privileges

Note that OS installation would be very slow while the RAID is building on large 3.8TB /mnt/data1 partitioning. It would take close to 5-6 hours for the installation to complete when installing through USB stick.


Initial configuration and package installations

Enable NOPASSWD based sudo access

  1. Login with saurabh user
  2. Enter saurabh user password after typing =sudo su -=
  3. Run 'visudo' and comment below line
    %wheel ALL=(ALL) ALL
  4. Uncomment below line
    # %wheel ALL=(ALL) NOPASSWD: ALL
  5. Exit root terminal and test by typing =sudo su -=. This time there should not be any password prompt.


IP addressing

  1. Configure static IP address using Network Manager GUI.
  2. Add the same IP in other hosts /etc/hosts files for easy access via name.
  3. Validate network connectivity
    ping 4.2.2.2
  4. Validate DNS resolution
    ping www.google.co.in

Note that this temporary. We will configure software bridge later for VMs and containers and disable access to interface via Network Manager (NM_CONTROLLED=no)


Configure history retention

Refer Storing date / time along with commands in history

After this exit all running shells and reopen new shell. Validate history is stored with date/time information.


Full update

  1. Edit /etc/yum.conf and set =keepcache=1=
  2. Do full update. This is required so that if any packages that were used before formatting were updated and have updated db/config files, those things can be read properly without version related issues:
    yum -y update --skip-broken
  3. Reboot into latest stable kernel

If the RAID partition rebuild is still not finished the full-update would run very slow, same as OS installation. Check software RAID rebuild status using

   cat /proc/mdstat


Disable screen lock

Start gnome-control-center and under Privacy, disable automatic screen-lock


Disable SELinux

  1. setenforce 0
  2. edit /etc/sysconfig/selinux
    SELINUX=disabled

This is required as SELinux related errors are very hard to troubleshoot as the issues happen elsewhere apache, image editor, etc. and it is hard to connect that something is not working due to SELinux.


Restore backup to /mnt/data1

  1. Install byobu for running tasks in background
    yum -y install epel-release
    yum -y install byobu
  2. Mount external drive and copy files to /mnt/data1 after the raid sync operation has finished (Could take 4-5 hours for 3.8TB):
    mkdir /mnt/usb
    mount /dev/sd<n>1 /mnt/usb
    byobu
    #Inside byobu shell
    rsync -axH /mnt/usb/2019-08-06-rekallcm1-backup /mnt/data1/
    #Press F6 to exit, Run byobu again to check rsync has completed or not
  3. After backup is complete
    umount /mnt/usb
  4. Move backup folders to appropriate place Be careful with relative vs absolute path while doing below operations
    cd /mnt/data1/2019-08-06-rekallcm1-backup
    mv /mnt/data1/2019-08-06-rekallcm1-backup/mnt/data1/* /mnt/data1/
    rmdir -p mnt/data1
    #move rsnapshot_backups inside /mnt/data1/large_files with latest cloud VM backups inside hourly.0 folder
    mv home/* /home/*
    mv -f home/saurabh/* home/saurabh/.* /home/saurabh/
    ls -al home/saurabh #Ensure empty except local, gnupg, dbus, config, cache
    #For example if firefox was opened as saurabh user before restoring backup then
    #rm -rf /home/saurabh/.mozilla
    #mv home/saurabh/.mozilla
    #After home/saurabh is empty
    rm -rf home/saurabh
    rmdir -p home
    ls -al /home/saurabh #Ensure .emacs, .thunderbird, .mozilla are symlinks
    ls -l /home
    useradd jitender #UID 1001
    useradd achal #UID 1002
    passwd jitender
    passwd achal
    mv -f root/* root/.* /root/
    ln -s /mnt/data1/plain_folders/documents/ /documents
    #exit and ssh again to rekallcm1
    #Validate older history from backup is visible
    history | less


Install and configure required packages

Yum based packages

    yum -y remove bind-chroot
    yum -y install htop   #Colorful command-line top
    yum -y install gimp   #Advanced image editor useful for editing pdf as images
    yum -y install kile  #Latex editor
    yum -y install wireshark wireshark-gnome #Packet capturing
    yum -y install erlang yaws #Programming
    yum -y install fdupes      #Duplicate file detection
    yum -y install dia         #Diagram such as flowcharts or network design editor
    yum -y install libreoffice #Libreoffice
    yum -y install p7zip       #Support for 7zip
    yum -y install kolourpaint #Basic image editor
    yum -y install expect      #For shell automation.  Includes mkpasswd utility
    yum -y install git bzr kdiff3  #For versioning and comparing files
    yum -y install bind bind-utils   #For DNS related tasks
    yum -y install unetbootin liveusb-creator  #For creating live bootable USB from iso files
    yum -y install xfreerdp    #RDP based Remote Desktop


Install and configure owncloud

  1. Install owncloud client using CentOS 7.x Owncloud desktop client installation
  2. ssh -X saurabh@rekallcm1
  3. owncloud &
    • Local folder - /mnt/data1/workspace/gbb/owncloud
  4. Ensure that sync is happening properly
  5. Resolve all conflicts
    • Might require deleting older renamed files and restarting owncloud client
  6. exit from ssh


Install and configure emacs

  1. yum -y install emacs
  2. Ensure /home/saurabh/.emacs and /home/saurabh/.emacs.d are symlink to appropriate folders of application-data
  3. Validate things are working by opening to-do.org file
  4. Configure nautilus to open .org file via emacs by default


Install vlc

Refer CentOS 7.x Installing VLC


Install and configure fail2ban


Configure nautlius

From this point onwards we can work directly on rekallcm1, instead of working remotely

  1. Open nautilus
  2. Remove /home/saurabh/{Music,Pictures,Videos,Templates}
  3. Create bookmarks for (Open corresponding folder and use Ctrl+D for shortcut)
    • /mnt/data1/workspace/gbb/owncloud/saurabh
    • /mnt/data1
    • /mnt/data1/plain_folders/documents/Desktop/rekall
  4. Configure nautilus to open .org file via emacs by default, if not done already


Configure firefox

  1. Ensure that /home/saurabh/.mozilla is symlink to application-data folder
  2. Starting firefox and see that various profiles are appearing properly.
  3. If required use following to restore .mozilla again from backup:
    mount /mnt/sd<n>1 /mnt/usb
    cd /mnt/usb/2019-08-06-rekallcm1-backup/mnt/data1/workspace/application-data/mozilla
    rsync -axH --delete ./ /mnt/data1/workspace/application-data/mozilla/
    umount /mnt/usb
  4. Validate history and stored passwords are working as expected


Install and configure thunderbird

  1. yum -y install thunderbird
  2. Launch thunderbird and validate that various emails accounts are visible


Install anydesk

  1. Refer CentOS 7.x installing anydesk
  2. As such below should work:
    yum -y localinstall /mnt/data1/large_files/setup/anydesk/anydesk-5.1.1-1.el7.x86_64.rpm
  3. Start anydesk and validate older saved IDs are coming properly


Configure SSH

  1. Validate /home/saurabh/.ssh folder has properly restored backup
  2. ssh root@sbarjatiya.com
    • Enter passphrase and enable automatic unlock during login


Install Google chrome

  1. Refer CentOS 7.x Installing Google chrome
  2. Start chrome
  3. Do not make chrome default browser
  4. Disable submitting of usage information
  5. Test by opening www.google.com


Install flash-plugin

  1. Open https://get.adobe.com/flashplayer/
  2. Choose download of "Yum for Linux"
  3. As such below should work:
    yum -y localinstall /mnt/data1/large_files/setup/flash/adobe-release-x86_64-1.0-1.noarch.rpm
  4. Close all browsers (Esp. firefox)
  5. yum -y install flash-plugin
  6. Start firefox and check "about:plugins"


Configure git

  1. Configure various git parameters:
    git config --global user.name "Saurabh Barjatiya"
    git config --global user.email "saurabh@sbarjatiya.com"
    git config --global color.ui true
    git config --global color.status auto
    git config --global color.branch auto
    git config --global core.editor vim
  2. Configure to use simple push by default
    git config --global push.default simple
    Refer https://stackoverflow.com/questions/21839651/git-what-is-the-difference-between-push-default-matching-and-simple


Configure bazaar

  1. Configure bazaar parameters:
    bzr whoami "Saurabh Barjatiya <saurabh@sbarjatiya.com>"


Enable bind

  1. Configure bind for starting automatically:
    systemctl start named
    systemctl enable named
  2. Change DNS of machine in /etc/sysconfig/network-scripts/ifcfg-br0 file to 127.0.0.1
  3. Optionally edit /etc/named.conf as follows:
    1. Update first listen to 'listen-on port 53 { 127.0.0.1; any; };'
    2. Comment listen-on-v6 port 53 { ::1; };' using '//' as prefix
    3. Update allow-query to 'allow-query { localhost; 10.0.0.0/8; 192.168.0.0/12; 172.16.0.0/16; };'
  4. CentOS 7.x Disabling IPv6 lookups in bind
  5. Restart named if not done already


Setup various servers and services

Configure software bridge

  1. Refer Creating bridge interfaces (br0) for virtual hosts to use shared interface section "Bridge interfaces in CentOS 7.0"
  2. Validate that bridge is having ip address and not interface:
    ip addr show
    brctl show


Install and configure lxc

  1. Install lxc by referring CentOS 7.x installing lxc


Import various /etc/hosts entries

  1. Open file /mnt/data1/2019-08-06-rekallcm1-backup/etc/hosts and copy valid entries and paste them in /etc/hosts file


Re-add various older VMs

Add the VMs via disk or config file

  1. Start virt-manager as root user. Add connection to Qemu/KVM if not present.
    virt-manager
  2. Go to File -> New virtual machine
    1. Choose "Import existing disk image"
    2. Select VM disk (Eg =/mnt/data1/large_files/vms/tally_vm/tally_vm.qcow2= )
    3. Select OS type "Linux" and Version "CentOS 7.0"
    4. Give required RAM and CPU (Eg 2048, 2 vCPU)
    5. Ensure that bridge br0 is selected
    6. Give appropriate VM name (Eg tally_vm)
    7. Select "Customize configuration before install"
    8. For "Disk 1" select "Disk Bus" as "IDE" instead of virtio
    9. For CPU select copy host CPU configuration option
    10. For Windows XP on NIC choose "e1000" instead of default virtio
    11. Click "Begin installation" to boot the VM.
  3. Download Intel Pro/1000 MT drivers for 32-bit and 64-bit windows from: https://downloadcenter.intel.com/download/18717/Network-Adapter-Drivers-for-Windows-XP-Final-Release?product=50485 As such 32-bit drivers from above link works for Tally_VM. But for other OS / VMs consider using one of the below as required:
  4. Copy the downloaded setup to pen drive and make it available to VM using USB redirection.
  5. Install the setup and Ethernet should get detected.


Take latest snapshot of VM and delete older snapshots via qemu-img

  1. Start virt-manager and take at least one snapshot of all important VMs.
  2. If there are VM folders in /mnt/data1/large_files/vms which are not useful to be added then delete those folders
  3. Go to vms folder and list snapshots of existing vm disks:
    cd /mnt/data1/large_files/vms/tally_vm
    qemu-img snapshot -l tally_vm.qcow2
    qemu-img snapshot -d '<Old-snapshot-name>' tally_vm.qcow2
    Be careful and do not delete recent snapshot taken
  4. Boot the VM and validate that it is still booting properly


Postfix and logwatch

  1. Configure postfix gmail relay using Sending email via Gmail relay through postfix
  2. Edit /etc/aliases and add barjatiya.saurabh [at] gmail.com as alias for root.
  3. Install and configure logwatch


Setup apache, mariadb including DB restore and php for wikis

  1. Replace httpd.conf from configuration folder:
    mv /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf-old
    ln -s /documents/room-documents/documents/general/configuration_files/httpd.conf /etc/httpd/conf/
  2. Start and enable mariadb
    systemctl start mariadb
    systemctl enable mariadb
  3. Check old wiki passwords at following locations:
    • /documents/public_html/notes_wiki/LocalSettings.php
    • /documents/room-documents/documents/databases/mysql/notes_wiki/mediawiki/LocalSettings.php
    • /documents/room-documents/documents/databases/mysql/research_wiki/mediawiki/LocalSettings.php
    • /documents/room-documents/documents/databases/mysql/readme_wiki/mediawiki/LocalSettings.php
  4. Login as root user (=mysql -u root=) and create other database accounts:
    create database wikidb_notes;
    grant all on wikidb_notes.* to wikidb_notes@localhost identified by '<password>';
    create database wikidb_res;
    grant all on wikidb_res.* to wikidb_res@localhost identified by '<password>';
    create database wikidb_readme;
    grant all on wikidb_readme.* to wikidb_readme@localhost identified by '<password>';
    create database notes_wiki;
    grant all on notes_wiki.* to notes_wiki@localhost identified by '<password>';
    flush privileges;
    \q
  5. Restore various database backups using:
    cd /documents/public_html/
    bunzip2 -k notes_wiki.sql.bz2
    cat notes_wiki.sql | mysql -u notes_wiki -p<password> notes_wiki
    rm -f notes_wiki.sql
    cd /documents/room-documents/documents/databases/mysql/notes_wiki/
    bunzip2 -k wikidb_notes.sql.bz2
    cat wikidb_notes.sql | mysql -u wikidb_notes -p<password> wikidb_notes
    rm -f wikidb_notes.sql
    cd /documents/room-documents/documents/databases/mysql/research_wiki
    bunzip2 -k wikidb_res.sql.bz2
    cat wikidb_res.sql | mysql -u wikidb_res -p<password> wikidb_res
    rm -f wikidb_res.sql
    cd /documents/room-documents/documents/databases/mysql/readme_wiki
    bunzip2 -k wikidb_readme.sql.bz2
    cat wikidb_readme.sql | mysql -u wikidb_readme -p<password> wikidb_readme
    rm -f wikidb_readme.sql
  6. Install php7 for latest notes_wiki
    yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
    yum -y install yum-utils
    yum-config-manager --enable remi-php72
    yum -y update php
  7. Install required packages for php and php-mysql:
    yum -y install php-mysql php php-xml httpd mariadb-server mariadb mod_ssl php-mbstring
  8. Restart and enable httpd:
    systemctl restart httpd
    systemctl enable httpd


Setup postgreSQL databases

  1. Install postgreSQL and start DB
    yum -y install postgresql postgresql-server
    postgresql-setup initdb
    systemctl start postgresql
    systemctl enable postgresql
  2. Restore postgreSQL configuration file to allow password based logins:
    mv /var/lib/pgsql/data/pg_hba.conf /var/lib/pgsql/data/pg_hba.conf-old
    sudo ln -s /documents/room-documents/documents/general/configuration_files/pg_hba.conf /var/lib/pgsql/data/
    sudo chown postgres:postgres /documents/room-documents/documents/general/configuration_files/pg_hba.conf
    systemctl restart postgresql
  3. Read existing DB password in
    • /documents/room-documents/public_html/bookworm/global/library/global_common.php
  4. Use below to create required databases:
    sudo su - postgres
    psql
    CREATE USER saurabh WITH NOSUPERUSER NOCREATEDB LOGIN ENCRYPTED PASSWORD '<password>';
    CREATE DATABASE saurabh WITH OWNER=saurabh;
    CREATE USER sen WITH NOSUPERUSER NOCREATEDB LOGIN ENCRYPTED PASSWORD '<password>';
    CREATE DATABASE sen WITH OWNER=sen;
    \q
    exit
  5. Restore various database backups using:
    cd /documents/room-documents/documents/databases/pgsql/
    psql -U saurabh -d saurabh < saurabh_saurabh.sql
    # cd /documents/room-documents/documents/databases/pgsql/
    # bunzip2 -k sen.sql.bz2
    # psql -U sen -d sen < sen.sql
    # rm -f sen.sql
  6. Install php postgresql libraries:
    yum -y install php-pgsql
    systemctl restart httpd
    Note that localhost/bookworm does not seems to work with latest php packages


Configure backups for localhost and AWS VMs

  1. Install required packages
    yum -y install rsnapshot sshpass
  2. Copy rsnapshot backup file to /etc
    mv /etc/rsnapshot.conf /etc/rsnapshot.conf-old
    cd /mnt/data1/2019-08-06-rekallcm1-backup
    cp etc/rsnapshot.conf /etc/
  3. Create rsnapshot root folder:
    mkdir -p /mnt/data1/large_files/rsnapshot_backups/
  4. Edit /etc/rsnapshot.conf and comment backups which are not required or incorrect
  5. Test backups using
    rsnapshot hourly
  6. Edit root crontab using =crontab -e= and use:
    0 0,4,8,12,16,20 * * * /usr/bin/rsnapshot hourly
    45 23 * * * /usr/bin/rsnapshot daily
    30 23 * * 0 /usr/bin/rsnapshot weekly
    15 23 1 * * /usr/bin/rsnapshot monthly


Also see:


<yambe:breadcrumb>CentOS_7.x_New_machine_configuration|CentOS 7.x New machine configuration</yambe:breadcrumb>