CentOS 8.x Systemd based reverse ssh tunnel service

From Notes_Wiki

Home > CentOS > CentOS 8.x > CentOS 8.x Security tools > CentOS 8.x Systemd based reverse ssh tunnel service

In case anydesk/teamviewer are proving unreliable there is option to connect to an internal server behind NAT using reverse ssh. For this we create service on internal server which does SSH to a public server via public key and creates reverse ssh tunnel from public server to itself. Anyone who has SSH access to public server, can now also connect to the internal server via this reverse tunnel. Thus, internal server without public IP can be accessed from anywhere (Internet) provided the user has access to public server where reverse tunnel is created.

To setup above use following steps to set it up:

  1. Assuming not only SSH but graphical access is also required install nomachine on both source and destination machines. See CentOS 8.x nomachine
    This is optional. The steps would also work only for SSH reverse tunnel without nomachine setup.
  2. Create ssh-keys on internal server using:
    ssh-keygen
  3. Configure the public key of internal server as authorized on public server using appropriate mechanism.
  4. Connect to public server from internal server using below command:
    /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R <remote-ssh-port>:127.0.0.1:22 -R <remote-nomachine-port>:127.0.0.1:4000 root@<server-ip-or-fqdn> -N
    Here replace <remote-ssh-port>, <remote-nomachine-port> and <server-ip-or-fqdn> apporpriately.
    If you are not doing nomachine reverse setup then ignore that particular "-R <remote-nomachine-port>:127.0.0.1:4000
    The output of above command should be blank. In case of tunneled SSH there might be some output lines related to use of proxy in above command.
    In case the command has output, we need to modify script ( /root/helpdesk-reverse-ssh.sh explained later) as follows
    status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1 | grep -v "<content>")
    to filter the content lines. This way $status would be ok during the ssh test part of script.
  5. Leave the ssh connected and in another tab check output of:
    ps aux | grep root@<server-ip-or-fqdn> | wc-l
    The above command output is 2 in CentOS 8 and the below script/steps are based on this value.
    The same command output is 3 in case of CentOS7. In that case in following shell script we should use '"if (( "$COUNT" > 2 )); then"'
  6. Create /etc/systemd/system/helpdesk-reverse-ssh.service with following contents on internal server:
    [Unit]
    Description=helpdesk-reverse-ssh service
    After=network.target
    
    [Service]
    Type=simple
    ExecStart=/bin/sh /root/helpdesk-reverse-ssh.sh
    Restart=always
    User=root
    Group=root
     
    [Install]
    WantedBy=multi-user.target
    
    No need to redirect output or capture Pid of processes. All of that is taken care of by systemd.
    Replace helpdesk-reverse-ssh with appropriate name based on your environment'
  7. Make service executable
    chmod 744 /etc/systemd/system/helpdesk-reverse-ssh.service
  8. Create shell script /root/helpdesk-reverse-ssh.sh on internal server which does reverse SSH tunnel to remote server using:
    #!/bin/bash
    
    SERVER="<service-ip-or-fqdn>"
    RPORT_SSH=3001
    RPORT_NOMACHINE=3002
    
    while sleep 60; do
       #Count number of existing connections
       COUNT=$(ps aux | grep root@$SERVER | wc -l)
       if (( "$COUNT" >= 2 )); then
           echo "SSH already connected, not doing anything extra"
       else
           # Check SSH Connction is available or not	
           status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1)
        
           #If available connect to remote server
           if [[ $status == ok ]] ; then
                echo "Going to start new SSH connection in background"
                /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R $RPORT_SSH:127.0.0.1:22 -R $RPORT_NOMACHINE:127.0.0.1:4000 root@$SERVER -N &
           elif [[ $status == "Permission denied"* ]] ; then
               echo "Not able to connect due to permission denied error"
           else
               echo "Error: $status"
        fi
    fi
    done     
    exit 0
    
    Here incase of CentOS 8 the condition should be >=2, while in case of CentOS 7 it should be >2. This is because of CentOS 7 while doing ssh "ps aux | grep root@<server>" results into three lines of output. On CentOS 8 same command results into only two lines of output.
    Values of RPORT_NOMACHINE, RPORT_SSH can be changed, if required
    <service-ip-or-fqdn> should be replaced with public server details
    If nomachine or SSH are not listening on their default ports 4000, 22 then replace those port numbers in -R argument value appropriately.
  9. Set required permissions and start service
    chmod 744 /root/helpdesk-reverse-ssh.sh
    systemctl daemon-reload
    systemctl enable helpdesk-reverse-ssh
    systemctl start helpdesk-reverse-ssh
    systemctl status helpdesk-reverse-ssh
  10. Now watch for the command to get executed after 60 seconds. Also after 120 seconds ensure that command is not executed again if ssh is running using:
    watch "systemctl status helpdesk-reverse-ssh"
    You can see systemd keeping track of shell script, sleep as well as ssh command
    If SSH fails then the error output would be part of status command logs lines shown at bottom
  11. Now open connection from a different client machine Not internal server to public server using:
    ssh -L 9000:127.0.0.1:3002 root@<server-ip-or-fqdn>
    You can use any other local port instead of 9000 in above command.
    In the script if you have changed RPORT_NOMACHINE then use that value instead of 3002 in above command
  12. Connect using nomachine to 127.0.0.1:9000 using nx protocol.
    Ignore warning about infinite loop while connecting to localhost and proceed.

There is older article on this at CentOS 7.x Systemd based reverse SSH service


Home > CentOS > CentOS 8.x > CentOS 8.x Security tools > CentOS 8.x Systemd based reverse ssh tunnel service