CentOS 8.x Systemd based reverse ssh tunnel service
In case anydesk/teamviewer are proving unreliable there is option to connect to an internal server behind NAT using reverse ssh. For this we create service on internal server which does SSH to a public server via public key and creates reverse ssh tunnel from public server to itself. Anyone who has SSH access to public server, can now also connect to the internal server via this reverse tunnel. Thus, internal server without public IP can be accessed from anywhere (Internet) provided the user has access to public server where reverse tunnel is created.
To setup above use following steps to set it up:
- Assuming not only SSH but graphical access is also required install nomachine on both source and destination machines. See CentOS 8.x nomachine
- This is optional. The steps would also work only for SSH reverse tunnel without nomachine setup.
- Create ssh-keys on internal server using:
- Configure the public key of internal server as authorized on public server using appropriate mechanism.
- Connect to public server from internal server using below command:
- /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R <remote-ssh-port>:127.0.0.1:22 -R <remote-nomachine-port>:127.0.0.1:4000 root@<server-ip-or-fqdn> -N
- Here replace <remote-ssh-port>, <remote-nomachine-port> and <server-ip-or-fqdn> apporpriately.
- If you are not doing nomachine reverse setup then ignore that particular "-R <remote-nomachine-port>:127.0.0.1:4000
- The output of above command should be blank. In case of tunneled SSH there might be some output lines related to use of proxy in above command.
- In case the command has output, we need to modify script ( /root/helpdesk-reverse-ssh.sh explained later) as follows
- status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1 | grep -v "<content>")
- to filter the content lines. This way $status would be ok during the ssh test part of script.
- Leave the ssh connected and in another tab check output of:
- ps aux | grep root@<server-ip-or-fqdn> | wc-l
- The above command output is 2 in CentOS 8 and the below script/steps are based on this value.
- The same command output is 3 in case of CentOS7. In that case in following shell script we should use '"if (( "$COUNT" > 2 )); then"'
- Create /etc/systemd/system/helpdesk-reverse-ssh.service with following contents on internal server:
[Unit] Description=helpdesk-reverse-ssh service After=network.target [Service] Type=simple ExecStart=/bin/sh /root/helpdesk-reverse-ssh.sh Restart=always User=root Group=root [Install] WantedBy=multi-user.target
- No need to redirect output or capture Pid of processes. All of that is taken care of by systemd.
- Replace helpdesk-reverse-ssh with appropriate name based on your environment'
- Make service executable
- chmod 744 /etc/systemd/system/helpdesk-reverse-ssh.service
- Create shell script /root/helpdesk-reverse-ssh.sh on internal server which does reverse SSH tunnel to remote server using:
#!/bin/bash SERVER="<service-ip-or-fqdn>" RPORT_SSH=3001 RPORT_NOMACHINE=3002 while sleep 60; do #Count number of existing connections COUNT=$(ps aux | grep root@$SERVER | wc -l) if (( "$COUNT" >= 2 )); then echo "SSH already connected, not doing anything extra" else # Check SSH Connction is available or not status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1) #If available connect to remote server if [[ $status == ok ]] ; then echo "Going to start new SSH connection in background" /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R $RPORT_SSH:127.0.0.1:22 -R $RPORT_NOMACHINE:127.0.0.1:4000 root@$SERVER -N & elif [[ $status == "Permission denied"* ]] ; then echo "Not able to connect due to permission denied error" else echo "Error: $status" fi fi done exit 0
- Here incase of CentOS 8 the condition should be >=2, while in case of CentOS 7 it should be >2. This is because of CentOS 7 while doing ssh "ps aux | grep root@<server>" results into three lines of output. On CentOS 8 same command results into only two lines of output.
- Values of RPORT_NOMACHINE, RPORT_SSH can be changed, if required
- <service-ip-or-fqdn> should be replaced with public server details
- If nomachine or SSH are not listening on their default ports 4000, 22 then replace those port numbers in -R argument value appropriately.
- Set required permissions and start service
- chmod 744 /root/helpdesk-reverse-ssh.sh
- systemctl daemon-reload
- systemctl enable helpdesk-reverse-ssh
- systemctl start helpdesk-reverse-ssh
- systemctl status helpdesk-reverse-ssh
- Now watch for the command to get executed after 60 seconds. Also after 120 seconds ensure that command is not executed again if ssh is running using:
- watch "systemctl status helpdesk-reverse-ssh"
- You can see systemd keeping track of shell script, sleep as well as ssh command
- If SSH fails then the error output would be part of status command logs lines shown at bottom
- Now open connection from a different client machine Not internal server to public server using:
- ssh -L 9000:127.0.0.1:3002 root@<server-ip-or-fqdn>
- You can use any other local port instead of 9000 in above command.
- In the script if you have changed RPORT_NOMACHINE then use that value instead of 3002 in above command
- Connect using nomachine to 127.0.0.1:9000 using nx protocol.
- Ignore warning about infinite loop while connecting to localhost and proceed.
There is older article on this at CentOS 7.x Systemd based reverse SSH service