Comprehensive Guide for Creating and Managing Cron Jobs in Nodejs / Adonisjs Applications
Table of contents
- What are Cron Jobs?
- System Cron Jobs
- In-Process Cron Jobs
- Advantages of In-Process Cron Jobs
- Disadvantages of In-Process Cron Jobs
- Example of In-Process Cron Jobs with AdonisJs
- Step 1: Clone the repository and install dependencies
- Step 2: Create the .env file
- Step 3: Create strong type for the ENABLE_DB_BACKUPS variable
- Step 4: Setup the database
- Step 5: Improve .gitignore file
- Step 6: Install node-schedule package
- Step 7: Create the Backup Handler
- Step 7: Create the api/app/Cron/index.ts scheduling file with the following contents:
- Step 8: Run the schedule
In this article, we will discuss in great details the strategies which you can adopt for creating and managing cron jobs for your Nodejs applications. Some of the examples will be demonstrated with the Adonisjs framework.
What are Cron Jobs?
Traditionally, cron jobs are regular shell scripts which are setup/scheduled to run via the operating system cron at predefined intervals. The interval could be daily, twice a day, once a month, one a week, etc.
While the operating system cron can be used to schedule jobs via the crontab file, there now exists JavaScript/Nodejs libraries which allows one to execute scripts at predefined intervals without the dependence on the operating system cron. Such libraries are installed within an application, so that the jobs are executed within the process of the currently-running Nodejs application.
For the sake of differentiation, in this article, we will call jobs set up via the operating system cron, System Cron Jobs, and jobs set up within an application, In-process Cron Jobs. So, the two strategies we will discuss for creating and scheduling cron job within our Nodejs/Adonisjs application are:
- System Cron Jobs, and
- In-Process Cron Job.
System Cron Jobs
Our Cron job discussion will be focused on the Linux operating system.
System cron jobs are scheduled with the operating system Crontab file. Let's have a look at some basics of the System cron.
Crontab file
A Crontab file is just a simple text file where instructions are defined for the Cron
daemon. The Cron
daemon is the persistent background service which runs your cron instructions. Each user of a Linux machine has a separate Crontab file.
Creating or Editing the Crontab file
You can maintain the Crontab file of the current (logged in) user with the command:
crontab -e
If you are accessing for the first time, a new crontab file will be created and opened with your default text editor such as
nano
orvim
.
If you are a privileged user (with sudo
access), you can edit the Crontab file of other users with the command:
sudo crontab -u other-user -e
Displaying the Crontab file
If you need to simply display the content of the Crontab file over standard output, use the command:
crontab -l
For other users, do:
sudo crontab -u other-user -l
Examples of Cron instructions
# Define the variable `APP_DIR`
APP_DIR=/var/www/my_app
# Delete temporary files at five minutes after midnight, every day
# Uses a Bash script
5 0 * * * $APP_DIR/jobs/delete_tmp_files.sh > $APP_DIR/logs/cron.log 2>&1
# Generate monthly report for the application at 2:00pm on the first of every month
# Uses Adonisjs Ace command
0 14 1 * * cd $APP_DIR && node ace generate:monthly_report > $APP_DIR/logs/cron.log 2>&1
# Send newsletters at 8 am on weekdays
# Uses a JavaScript script
0 8 * * 1-5 $APP_DIR/jobs/send_newsletters.js > $APP_DIR/logs/cron.log 2>&1
You can use crontab.guru, do generate Cron schedules
Logging the Output of a Cron job
In the Cron instructions below:
0 8 * * 1-5 $APP_DIR/jobs/send_newsletters.js > $APP_DIR/logs/cron.log 2>&1
You will notice the output redirection: > $APP_DIR/logs/cron.log 2>&1
Let's break it down:
- The redirection operator
>
sends the standard output from the script or command on the left-hand side into the log file on the right-hand side. In a typical shell script, contents for the standard output are generated with either theecho
orprintf
commands. In a typical JavaScript script, contents for the standard output are generated with theconsole.log
method. The standard output is typically the console. The second redirection
2>&1
is used to redirect the standard error (indicated as the file descriptor2
into the standard output (indicated as the file descriptor1
). In a typical shell script, contents for the standard error are generated when a script exits with an error. In a typical JavaScript script, contents for the standard error are generated with theconsole.error
method.The
&
is used to signify the the1
on the right-side of the redirection operator>
is a file descriptor and not a file name. The&
is not necessary when a file descriptor is used on the left-side of a redirection operator. Also notice that there are no space in the redirection instruction2>&1
between two file descriptors. Read more that the shell redirections and Bash One-Liners Redirections.Bonus: A
File Descriptor
is a reference to an open file within the filesystem of an operating system. When you open a file for reading or writing using any command or function provided by the language you are using, the opened file is known to the operating system via itsfile descriptor
. File descriptors are basically numerical pointers such as1, 2, 3, etc
. For any operating system, three files are always opened by default and assigned the file descriptors:1
,2,
and3
. There are: the standard input (with file descriptor0
), standard output (with file descriptor1
), and standard error (with file descriptor2
).
The redirection discussed above can be simplified with:
0 8 * * 1-5 $APP_DIR/jobs/send_newsletters.js &>$APP_DIR/logs/cron.log
Again read this excellent article on Bash One-Liners Redirections
Advantages of Using the System Cron
- Since System Cron jobs are scheduled and ran with the
Cron daemon
, the jobs will run even when the application is not running. - You can easily pause or discontinue a system cron job by commenting out the instructions on the Crontab file.
- Since the system cron jobs are not running within the process of the application, you can scale the application without worrying about duplication of the jobs.
Disadvantages of Using the System Cron
- Cannot be used in edge deployments which do not provide access to the operation system Crontab.
- Difficult to use in some serverless deployments unless the deployment service provides Cron scheduling interface.
- Requires tampering with the system crontab
- Crontab file are not automatically moved when you migrate your application to another server. You will need to copy your Crontab file manually to the new server.
Security Considerations When Using the System Cron
It is important to adopt security/safety precautions when working with system cron jobs since the jobs are executed by the system shell.
Use the least privileged user. If your application root is located at
/var/www/my_app
and the/var/www
directory and subdirectories are owned by thewww-data
user (the default web user for Ubuntu, you should create the Crontab instructions for the application with the
www-data` user by executing:sudo crontab -u www-data -e
When the cron jobs for the
www-data
user are executed, the privileges of the scripts will be limited to the privileges of thewww-data
user. On very rare reasons should you create cron jobs with your own user (log in) account or theroot
user account.Thoroughly test your scripts and understand every bit of what it does before creating cron jobs with them.
Strategies for Creating System Cron Jobs
We will discuss three (3) strategies which can be adopted for Creating System Cron Job:
- Use of Bash shell scripts
- Use of Nodejs shell scripts
- Use of Adonis Ace commands
Use of Bash shell scripts for Cron Jobs
Learn how to write Bash shell scripts.
The following steps should typically be followed when working with a Bash shell script:
Bash shell scripts typically begin with a shebang. A
shebang
for theBash
shell looks like:#!/bin/bash
The
shebang
tells the shell which interpreter to use for interpreting the file. Theshebang
is respected by all programming languages and will be ignored when it appears on the first line of the script.It is common practice to save Bash shell script files with the extension
.sh
. E.g.my_backup_script.sh
.After the saving the file and as a security practice, set the owner, group, and permissions for the file. It is important to make the file executable. See the commands below:
APP_DIR=/var/www/my_app
# Equivalent of `cd /var/www/my_app`
cd $APP_DIR
# Set the user and group for the script
sudo chown www-data:www-data jobs/delete_tmp_files.sh
# Set the permissions for the script
# Here we want only the user and group to have `execution` (`x`) permission
sudo chmod ug+x jobs/delete_tmp_files.sh
The commands above are also applicable when working with JavaScript shell scripts.
Use of Nodejs Shell Scripts for Cron Jobs
As a JavaScript developer, you might not be as proficient in Bash
as you are with JavaScript. Instead of struggling to learn Bash, you could use JavaScript combined with standard Nodejs API to write your shell scripts. Also, by writing your shell script with JavaScript, you have full access to all NPM modules including JavaScript SDKs for 3rd party platforms you might want to interact with within your cron job.
The following steps should typically be followed when working with a Nodejs shell script:
Nodejs shell scripts typically begin with a shebang and the
shebang
for theNodejs
looks like:#!/usr/bin/env node
As you already know, you would save Nodejs file with the extension
.js
.Follow the commands in the previous section to set the user and group for the script and also set the execution permissions.
Example of a Nodejs Shell Script
The script below is a working script for backing up a PostgreSQL database
#!/usr/bin/env node
"use strict";
require("dotenv").config();
const path = require("path");
const { DateTime } = require("luxon");
const { exec } = require("child_process");
const { existsSync, mkdirSync } = require("fs");
class DbBackupHandler {
constructor() {
this.DB_DATABASE = process.env.PG_DB_DATABASE;
this.DB_USER = process.env.PG_BACKUP_USER;
this.DB_PASSWORD = process.env.PG_BACKUP_USER_PASSWORD;
if (!this.DB_DATABASE || !this.DB_USER || !this.DB_PASSWORD) {
throw new Error("Invalid credentials");
}
}
get dbCredentials() {
return {
db: this.DB_DATABASE,
user: this.DB_USER,
password: this.DB_PASSWORD
};
}
get now() {
return DateTime.now().toFormat("yyyy-LL-dd HH:mm:ss");
}
run() {
return new Promise(async (resolve, reject) => {
try {
console.info(`DbBackupHandler: DB Backup started at: ${this.now}`);
const fileName = `${this.dbCredentials.db}-${DateTime.now().toFormat(
"yyyy-LL-dd-HH-mm-ss"
)}.gz`;
const relativeDirName = "backups";
const fullDirName = `${path.join(process.cwd(), relativeDirName)}`;
const fullFilePath = `${fullDirName}/${fileName}`;
// Create the backup directory if not exists
if (!existsSync(relativeDirName)) {
mkdirSync(relativeDirName);
}
exec(
`PGPASSWORD=${this.dbCredentials.password} pg_dump -U ${this.dbCredentials.user} -Fc -w ${this.dbCredentials.db} | gzip > ${fullFilePath}`,
async (error, _stdout, stderr) => {
if (stderr) {
console.error(`DbBackupHandler: exec stderr:`, stderr);
}
if (error) {
console.error(`DbBackupHandler: exec error:`, error);
reject(error);
} else {
console.info(`DbBackupHandler: Local backup created.`);
return resolve(
"DbBackupHandler: Backup completed at: " + this.now
);
}
}
);
} catch (error) {
reject(error);
}
});
}
}
async function main() {
return new DbBackupHandler().run();
}
main().then(console.log).catch(console.error);
module.exports = DbBackupHandler;
The script above can be scheduled to run every midnight with the Cron instruction:
APP_DIR=/var/www/my_app
0 0 * * * $APP_DIR/jobs/db_backup_script.js > $APP_DIR/logs/cron.log 2>&1
Use of the Adonisjs Ace Commands for Cron Jobs
The Adonisjs Framework comes with an in-built CLI call Ace. With the Ace CLI, you can run in-built commands or create your own custom commands. The custom commands are written in the familiar JavaScript language. The Ace CLI allows you to define command description, arguments, flags, prompts, and craft beautiful command UI for feedback while your command is running.
To use the Adonisjs Ace CLI for cron jobs, follow the steps below:
Within the directory of your Adonisjs application, create a new command by running:
node ace make:command DbBackup
This should create a new command file named:
commands/DbBackup.ts
.Regenerate the Ace command manifest file:
node ace generate:manifest
Now, if you run
node ace
, you will see the new command listed among the available commands for your application asdb:backup
. This means that you can run the command withnode ace db:backup
Go ahead and develop the command script located in
commands/DbBackup.ts
. See the example below for a working Ace command.Then schedule the command to run with the Cron instruction:
APP_DIR=/var/www/my_app
# Generate daily backup for the application at 01:00am
0 1 * * * cd $APP_DIR && node ace db:backup &>$APP_DIR/logs/cron.log
Example of a Working Adonisjs Ace Command for Cron Job
Below is an example of a working Ace command for PostgreSQL backup to local disk. Ensure that you run: node ace generate:manifest
to regenerate the manifest file. Afterwards, node ace
will show the update listing of the command. In the script below, I avoided loading the Adonisjs application and relying on the IoC container which requires starting up the application.
import('dotenv').then((file) => file.config())
import path from 'path'
import { DateTime } from 'luxon'
import { exec } from 'child_process'
import { existsSync, mkdirSync } from 'fs'
import { BaseCommand } from '@adonisjs/core/build/standalone'
export default class DbBackup extends BaseCommand {
/**
* Command name is used to run the command
*/
public static commandName = 'db:backup'
/**
* Command description is displayed in the "help" output
*/
public static description = 'Simple backup script for your PostgreSQL database'
public static settings = {
/**
* We do want to load the Adonisjs application
* when this command is ran
*/
loadApp: false,
/**
* We want the process within which the command is run to exit
* immediately the command completes execution
*/
stayAlive: false,
}
public async run() {
const NOW = DateTime.now().toFormat('yyyy-LL-dd HH:mm:ss')
const DB_CREDENTIALS = {
db: process.env.PG_DB_NAME,
user: process.env.PG_USER,
password: process.env.PG_PASSWORD,
}
if (!DB_CREDENTIALS.db || !DB_CREDENTIALS.user || !DB_CREDENTIALS.password) {
throw new Error('Invalid credentials')
}
try {
console.info(`DbBackupHandler: DB Backup started at: ${NOW}`)
const fileName = `${DB_CREDENTIALS.db}-${DateTime.now().toFormat('yyyy-LL-dd-HH-mm-ss')}.gz`
const relativeDirName = 'backups'
const fullDirName = `${path.join(process.cwd(), relativeDirName)}`
const fullFilePath = `${fullDirName}/${fileName}`
// Create the backup directory if not exists
if (!existsSync(relativeDirName)) {
mkdirSync(relativeDirName)
}
exec(
`PGPASSWORD=${DB_CREDENTIALS.password} pg_dump -U ${DB_CREDENTIALS.user} -Fc -w ${DB_CREDENTIALS.db} | gzip > ${fullFilePath}`,
async (error, _stdout, stderr) => {
if (stderr) {
console.error(`DbBackupHandler: exec stderr:`, stderr)
}
if (error) {
throw error
} else {
console.info(`DbBackupHandler: Local backup created.`)
console.info('DbBackupHandler: Backup completed at: ' + NOW)
}
}
)
} catch (error) {
console.error('error: %o', error)
}
}
}
In-Process Cron Jobs
In-Process Cron Jobs are created by using special packages to create schedules within the same process as the currently-running Nodejs application. A handler function or method is then attached to each schedule so they are executed when the scheduled time arrives.
One example of such packages which can be used to create In-Process Cron Jobs
is node-schedule.
It is possible to execute Ace commands within In-Process Cron Jobs
via packages like execa or Nodejs Child Process
Advantages of In-Process Cron Jobs
- Simple. In-process cron jobs run within the same Nodejs process as the application server so does not require special setup. Great for applications which won't be scaled horizontally.
- Fast. Since in-process cron jobs do not boot-up the application server. Instead they execute as a regular functions or methods would within the application.
- Portable. Because the schedules are created within the source code of the application. So, the schedules go wherever the source code goes.
- Can be used on edge deployments which do not provide access to the operation system Crontab.
Disadvantages of In-Process Cron Jobs
- Job Duplication. If multiple instances of the application are running (like when deploying the application with process managers like PM2 in
cluster
mode), each instance of the application will execute theIn-Process Cron Job
. - Not reliable. In-process cron jobs will only run when the application is running. Therefore, the cannot be used for mission-critical jobs.
- Not easily pause-able. It might be difficult to pause the execution of in-process cron jobs unless you make use of environment variables, in which case, you will have to restart the application each time you toggle the variables.
Example of In-Process Cron Jobs with AdonisJs
In the example below, we are going to implement an In-Process Cron Job for database backup.
Step 1: Clone the repository and install dependencies
git clone https://github.com/ndianabasi/google-contacts.git
# We focus on the API server
cd api
yarn install
Step 2: Create the .env
file
cp .env.example .env
Add an entrance variable to the .env
file.
MYSQL_USER=lucid
MYSQL_PASSWORD=
MYSQL_DB_NAME=lucid
+ ENABLE_DB_BACKUPS=true
Step 3: Create strong type for the ENABLE_DB_BACKUPS
variable
Open api/env.ts
file and add:
MYSQL_PASSWORD: Env.schema.string.optional(),
MYSQL_DB_NAME: Env.schema.string(),
+ ENABLE_DB_BACKUPS: Env.schema.boolean(),
Step 4: Setup the database
- Create a MySQL database for the application.
- Replace the value of the database environment variables in
.env
with the real database credentials. Most especially:MYSQL_USER
,MYSQL_PASSWORD
, andMYSQL_DB_NAME
. - Migrate the database with:
node ace migration:run
. - Seed the database with:
node ace db:seed
Step 5: Improve .gitignore
file
We want to ignore the database backup files which will be stored in api/backups
folder.
Open api/.gitignore
file and add:
.env
tmp
+ backups
Step 6: Install node-schedule
package
yarn add node-schedule && yarn add @types/node-schedule -D
Step 7: Create the Backup Handler
Create a new file api/app/Cron/Handlers/DailyDbBackupHandler.ts
with the following contents:
import path from 'path'
import { DateTime } from 'luxon'
import { exec } from 'child_process'
import Env from '@ioc:Adonis/Core/Env'
import { existsSync, mkdirSync } from 'fs'
import Logger from '@ioc:Adonis/Core/Logger'
export default class DailyDbBackupHandler {
private logger: typeof Logger
constructor() {
this.logger = Logger
}
public run() {
return new Promise(async (resolve, reject) => {
try {
const NOW = DateTime.now().toFormat('yyyy-LL-dd HH:mm:ss')
const DB_CREDENTIALS = {
db: Env.get('MYSQL_DB_NAME'),
user: Env.get('MYSQL_USER'),
password: Env.get('MYSQL_PASSWORD'),
}
if (!DB_CREDENTIALS.db || !DB_CREDENTIALS.user) {
throw new Error('Invalid credentials')
}
this.logger.info('DbBackupHandler: DB Backup started at: %s', NOW)
const fileName = `${DB_CREDENTIALS.db}-${DateTime.now().toFormat('yyyy-LL-dd-HH-mm-ss')}.gz`
const relativeDirName = 'backups'
const fullDirName = `${path.join(process.cwd(), relativeDirName)}`
const fullFilePath = `${fullDirName}/${fileName}`
// Create the backup directory if not exists
if (!existsSync(relativeDirName)) {
mkdirSync(relativeDirName)
}
exec(
`mysqldump -u${DB_CREDENTIALS.user} -p${DB_CREDENTIALS.password} --compact ${DB_CREDENTIALS.db} | gzip > ${fullFilePath}`,
async (error, _stdout, stderr) => {
if (stderr) {
this.logger.info('DbBackupHandler: %s', stderr)
}
if (error) {
return reject(error)
}
this.logger.info('DbBackupHandler: Local backup created at: %s', NOW)
this.logger.info('DbBackupHandler: Backup completed at: %s', NOW)
resolve('done')
}
)
} catch (error) {
reject(error)
}
})
}
}
Step 7: Create the api/app/Cron/index.ts
scheduling file with the following contents:
import scheduler from 'node-schedule'
import Env from '@ioc:Adonis/Core/Env'
import Logger from '@ioc:Adonis/Core/Logger'
import DailyDbBackupHandler from './Handlers/DailyDbBackupHandler'
/**
* Runs every 12 hours
*/
scheduler.scheduleJob('0 */12 * * *', async function () {
const isDbBackupsEnabled = Env.get('ENABLE_DB_BACKUPS')
if (isDbBackupsEnabled) {
await new DailyDbBackupHandler()
.run()
.catch((error) => Logger.error('DailyDbBackupHandler: %o', error))
}
})
Logger.info('In-process Cron Jobs Registered!!!')
Here you can change how frequent you want to backup to be made. For testing purposes, you might want to change the schedule to every minute:
scheduler.scheduleJob('* * * * *', async function () {
const isDbBackupsEnabled = Env.get('ENABLE_DB_BACKUPS')
if (isDbBackupsEnabled) {
await new DailyDbBackupHandler()
.run()
.catch((error) => Logger.error('DailyDbBackupHandler: %o', error))
}
})
Step 8: Run the schedule
Import the schedule entry file into api/providers/AppProvider.ts
and execute the schedules when the application is ready
public async boot() {
// IoC container is ready
}
public async ready() {
// App is ready
+ import('App/Cron/index')
}
public async shutdown() {
// Cleanup, since app is going down
}
When the server restarts, you should see a log in the console:
[1661263670963] INFO (google-contacts-clone-api/55793 on xxxx): In-process Cron Jobs Registered!!!
[1661263670967] INFO (google-contacts-clone-api/55793 on xxxx): started server on 0.0.0.0:3333
Congratulations. You have setup an In-process Cron Job
for database backup.
See all relevant files here: github.com/ndianabasi/google-contacts/compa..
Cover image background from Freepik.