Run long-lived processes on a remote server while you are logged out, using the Screen utility
Introduction
Consider the following scenario: you’re on a long-distance multi-day bike ride, you have your laptop with you and you need to run a long process on a remote Ubuntu server, something like importing a huge database, which might take upwards of several hours.
Let’s complicate things and imagine you have two databases to import, and you want to do these imports concurrently.
One way of doing this is to launch two terminal windows, and log into your server and run one import in each window, something like:
ssh me@example.com
> ./some-super-long-process-one.sh
And then in another window:
ssh me@example.com
> ./some-super-long-process-two.sh
The problem here is that if you turn off your laptop, your sessions will terminate and so will your processes.
Enter Screen
Screen is a handy utility to run persistent processes on a remote server without requiring you to be running a terminal window. Here’s how it works:
- You log into your remote server and start a screen session.
- You start your long process.
- You can log out of your session on the remote server, but the screen session keeps on running.
- Later on, you can log back onto your remote server, and back into the screen session, and notice that any task you started earlier has kept on running even when your laptop is turned off.
Let’s get started
I’m assuming you have screen installed; I have found it to be installed by default on most servers, for example Ubuntu. If you don’t have screen installed, well, you can keep doing things the old way!
Start by confirming that screen works, and that there are no screen sessions:
> screen -ls
No Sockets found in /run/screen/S-root.
We will start a screen session where we’ll run a long task:
> screen
Press enter after the intro text. You will see a shell prompt. Let’s start a process which never ends, just for illustration purposes. It will keep printing decremental numbers until we kill it.
> COUNTER=0; while [ $COUNTER -lt 1 ]; do echo The counter is $COUNTER; let COUNTER=COUNTER-1; done;
Now leave the screen session with the very intuitive sequence of commands:
> control-a, then d
It may take a 10 seconds or so to get back to your original shell. You will see something like this:
[detached from 1795.pts-0.example.com]
Your process is still running in the background, trust me.
We will start another screen session to print an infinite list of incremental numbers, just because we can.
> screen
Press enter after the intro text. As earlier, you will see a shell, where you can enter a variation of our earlier command:
> COUNTER=1; while [ $COUNTER -gt 0 ]; do echo The counter is $COUNTER; let COUNTER=COUNTER+1; done;
Again, quit the screen:
> control-a, then d
After 10 seconds or so, as earlier, you will see something like this:
[detached from 1807.pts-0.example.com]
At this point you can log out of your server, turn off your computer, go for a jog, meditate, and have a kale smoothie, or whatever it is you do.
This is where the magic happens
All the while your server is still crunching numbers. Don’t believe me? Let’s check in by logging back into your remote server and running:
screen -ls
You will see output that looks like this:
There are screens on:
30199.pts-1.example.com (12/04/18 01:15:49) (Detached)
30176.pts-1.example.com (12/04/18 01:12:56) (Detached)
2 Sockets in /run/screen/S-root.
This tells you there are two screens running, one should be displaying decremental numbers, the other should be displaying incremental numbers. Let’s revisit one of our screens:
> screen -r 30199.pts-1.example.com
The counter is 1418453
The counter is 1418454
The counter is 1418455
...
We can now leave the screen again if we want it to keep running (see above), but there is only so much fun to be had watching numbers unfurl, so I’d suggesting quitting this process:
> control-c
And exit the screen
> exit
This leaves us with one screen:
> screen -ls
There is a screen on:
30176.pts-1.example.com (12/04/18 01:12:55) (Detached)
1 Socket in /run/screen/S-root.
I’ll leave it as an exercise for the reader to visit that screen, terminate its process, and exit the screen.
After you have done that, you should no longer have any active screens:
> screen -ls
No Sockets found in /run/screen/S-root.
I’ve found this technique to be useful, with one caveat: if you use ssh agent forwarding to allow you to log into server A using your SSH key pair, and then from server A, log into server B transparently using your original key pair (without the private key existing on server A), this might fail. If you log out of your main session, any running screen sessions will cease to have access to your original ssh key and won’t be able to perform operations on remote servers. When you think about it, this does make sense from a security perspective, but alas! you can’t rely on ssh agent forwarding in detached screens when we’re logged out of our server.
Happy multitasking!