How to Automate and Schedule GCP VM Instance

Suppose you have a script in the Google Cloud’s VM Instance that needs to run every day/week/month at a particular time ( e.g., a data pipeline for Machine Learning model training, data crawling, etc.). In that case, this article will guide you through automating the task.

Different ways you can automate a script on GCP

Your script/program can be in any language, but let’s assume we are trying to automate a Python script for simplicity.

Let’s say you are working on Google Compute Engine’s VM instance. You have a Python script main.py that does some specific task ( e.g., scrape data from multiple sites, train Machine Learning model, etc.). You need to do this same task (running the script) every Friday at 12:00 AM; then, you have a couple of options to automate this process.

1. Use cloud functions with pub/sub.
2. Add a startup-script in your VM instance to run the program automatically.

You can consider the first option if your job takes less than 540 seconds. Otherwise, it’s ideal to go with the second approach. The first option won’t work for long-running processes because the Google Cloud Functions can run a maximum of 940 seconds¹.

But what is a ‘startup-script’?

A startup script is a file that performs tasks during the startup process of a virtual machine (VM) instance. Startup scripts can apply to all VMs in a project or to a single VM⁴.

Configure the VM Instance and Environment

If you try to add a startup-script to your VM instance², your startup-script will run in the root user mode. When you connect your VM instance remotely with an SSH connection to your local machine, you log in as a different user from the root mode. You can enter the root user mode by the following command:

1$ sudo su -

This command will take you to the root user mode. You will not find your coding resources in root mode, which you probably worked on while connecting the VM instance via SSH to your local machine. So, I suggest maintaining a git repo to maintain your code and cloning the git repo in the root user. You should also install program dependencies into the root user. For example, if your program runs on Anaconda Virtual Environment, then install Anaconda in the root user. Ensure you can run your program correctly by logging into the root user mode. Let’s say you can perfectly run your program by the following command in your virtual environment:

1$ python main.py

There might be different python interpreters existing in your VM. Just make sure which python interpreter you are using to run your program, you can run the following command in the terminal to see which interpreter you are using. If you are working inside a virtual environment, then activate it, and run the following command.

1$ which python

This will give your python interpreter path, like usr/bin/python3 .

Configure the The startup-script on VM Instance Settings

To attach a startup-script in your instance, go to the Compute Engine > VM Instances page. Then click on your target VM Instance. Now, Edit the instance.

Click on Edit. after that, scroll down the page, and you will find a section called ‘Metadata’. There, under the Automation section, you can add your startup-script .

For example: let’s say our driver code exists in the /root/data_pipeline/src/main.py file. So, to run the program automatically when VM starts, we can write the following startup-script .

1#! /bin/bash
2
3/usr/bin/python3 /root/data_pipeline/src/main.py

We want to run the program to keep running without any kind of interruption, even if the configuration changes in some other VMs and the main GCP project. To run the program uninterruptedly, we should enable shielded-learn-integrity-policy policy³. To do that, we can add the following command before calling the main.py script.

1#! /bin/bash
2gcloud compute instances update <instance-name> --zone <instance-zone-name> --shielded-learn-integrity-policy
3
4/usr/bin/python3 /root/data_pipeline/src/main.py

When the main.py program execution gets completed, we want our VM instance gets stopped automatically. To do that, we can add the following command at the end of the startup-script :

1#! /bin/bash
2
3gcloud compute instances update <your-instance-name> --zone <instance-zone-name> --shielded-learn-integrity-policy
4
5# run the main script that you want to run
6/usr/bin/python3 /root/data_pipeline/src/main.py
7
8# turn of the instance
9gcloud compute instances stop <your-instance-name> --zone <instance-zone-name>

You can also add other commands in thestartup-script as per your requirements. For example:

 1#! /bin/bash
 2sudo service tor restart
 3
 4gcloud compute instances update <your-instance-name> --zone <instance-zone-name> --shielded-learn-integrity-policy
 5
 6# set the file open limit
 7ulimit -n 100000
 8
 9# use the python interpreter to run the main script
10/usr/bin/python3 /root/data_pipeline/src/main.py
11
12# stop the instance automatically after the program finishes its execution
13gcloud compute instances stop <your-instance-name> --zone <instance-zone-name>

The above script makes our program to gets executed automatically when the VM instance gets started and stops the VM instance after the program execution gets completed.

Now, we need to schedule the instance to automatically starts the VM instance.

Schedule the VM instance

Go to the Navigation menu > Compute Engine > VM Instances page. There is a section called INSTANCE SCHEDULES .

Go to that tab. Now, you will find a section in the top bar called CREATE SCHEDULE

If you click on the Create Schedule, you will see a page like the below:

Give any name you want. The region area must match the VM instance’s region. Define a Start time and Frequency (like daily / weekly / monthly, etc.). You don’t need to define any Stop time, as we already have a command to stop the instance automatically after our main.py program ends its execution in the startup-script .

You can also use CRON expressions to define the start time for instance. Finally, submit the page. This will create a scheduler page. Go to that page, and you will find options to add instances in it.

Just add your target instance to it, and you’re done.

This will allow your instance to automatically start at your defined start time on the scheduled page, call the main.py file from the startup-script , after the program finishes its execution, the VM instance will be automatically turned off.

References Author: Sadman Kabir Soumik

[1]. https://cloud.google.com/functions/quotas

[2]. https://cloud.google.com/compute/docs/instances/startup-scripts/linux

[3]. https://cloud.google.com/compute/shielded-vm/docs/integrity-monitoring#updating-baseline

[4]. https://cloud.google.com/compute/docs/instances/startup-scripts/