Purpose of this tutorial

In this tutorial we'll learn how to set up an action sequence which allows your task to perform various operations in a chain, one after the other. We'll also see how to set up the task so that if anything goes wrong, you are informed about the problems in order to react in a timely manner.

Scenario

Let's assume we have the following scenario: We have a workflow to analyse and publish data about the Covid-19 pandemic. The workflow is using a locally stored data file with the most up to date data. Every day fresh data is made available by the European Union which we want to download and use as new data to update our workflow. After publishing the updated report we want to send an email to interested stakeholders that an updated report is available. Furthermore since the data is coming from the internet, and human and technical errors can occur while the data is published or downloaded, we want to be informed about any problems that happened during this process.

Setting up a sequence of actions

A sequence of actions can be used to define actions that need to be executed in chain, i.e. one after the other. Its advantage to simply creating multiple normal actions is that the sequence is treated as a single action with special configuration options. These options e.g. control how long the whole sequence is allowed to execute before it times out, or allows to create performance timings to analyse how long each sub-action in the sequence takes to be executed.

Creating a Task

Say we have the workflow to the left in which a locally stored file called "current_covid_data.csv" is read, processed in some way, and published via a report block.

Let's create a task to refresh "current_covid_data.csv" and send an email. Go to into the scheduler and create a new Task.

Creating a sequence of actions

Now it's time to setup a sequence of actions. 

Go inside the task and click on "+ Add Action". At the very top you should see "Sequence of Actions". Click on it to add it to the list of actions. Afterwards press on the newly added "Sequence of actions" list item to get into its configuration page.

First we change the name to something more appropriate.

Next, we have to add and setup three actions:

1. Download the most current Covid19 data.

2. Re-execute the workflow so that the new data is read and processed.

3. Send the email.

Download the most current Covid19 data

Click on "+ Add Action" and find "Resource action". Click on it to add it into the sequence.

You should see "Resource action" in the list of actions:

Click on Resource action to get into its configuration page.

Firstly, we rename the action to make clear what its purpose it.

 Secondly, we check "Replace existing files?" to make sure the old data is replaced by the most up to date.

Up to date Covid19 case information is available at https://coronavirus.data.gov.uk/details/cases. 

A csv file of the data can be downloaded by right-clicking on "Download in CSV" to copy the url.

The url itself is this:

https://opendata.ecdc.europa.eu/covid19/casedistribution/csv/data.csv

In the "Source" configuration of the Resource action, choose URL and paste the url into the URL field:

As destination, we choose the file on the local disk to be replaced by updated data.

Refresh Covid19 Report

We'll now insert another action into the sequence. Go back to the sequence of actions configuration page. The bread crumbs for this page were these:

After having setup the file replacement action, we move to using the updated data to refresh the report. For this purpose we add a "Project (IOX file) action".

Click on the newly added action to configure it. We change the name to "Refresh Covid19 report" and choose our workflow file.

Then, we click at "+ Add File Action" and choose "Execute blocks".

We enter the configuration page for the new file action, rename it to "Refresh report" and configure it so that the file and report blocks are re-executed.

Send email

Now there is just one action left to create: One that sends an email once the report is re-published. Go back to the sequence of actions configuration page. The bread crumbs for this page were these:

Click on "+ Add Action" and choose "Email action"

There should now be 3 actions in the sequence. Click on "Email action" to configure it

We change the name of the action to "Send report to stakeholders"

Next we need to configure the mail server. Insert values for the host / port, username and password. In case you use a gmail account, use the following settings:

Your username should be your email address and the password needs to be an app-password. A guide how to create one can be found here: https://support.google.com/accounts/answer/185833

In the "Email" section, we set up the actual email: its text and recipients.

Once this is configured the sequence of actions is completed.

Failure handling

There is much that can go wrong in an automated process. If it does, we want to first try if we can recover gracefully, and if this fails, notify ourselves that something has gone wrong. 

We'll do the following things:

1. Do a test execution of the task

2. Retry actions that might fail

3. Configure a realistic timeout

3. Setup another email action to notify us in case there are unrecoverable failures

4. Check the logs to figure out what went wrong

Do a test execution of the task

Let's go back all the way to the scheduler config. Where we see the task list

After pressing the "play" button, we are asked to confirm our action and  should then be presented with a dialog that informs us about the status of the execution.

Once it is done, we should see a success message. If there were any issues, we should already see some information about what went wrong.

Retry actions that might fail

We'll go into the "Refresh Covid19 data and send email" sequence action. Your bread crumbs should look like this: 

This sequence might actually fail, because we rely on the availability of a resource located on the internet. E.g. the resource might be temporarily unavailable. In order to recover from a failed download, we will instruct the scheduler to retry the sequence three times, and to wait 5 minutes between each retry.

We scroll down to "Failure retry options" and configure it appropriately.

Configure a realistic timeout

All scheduled tasks are initially configured with a 5 minute timeout by default. If we see that our task might need more than that, we have to adjust this value, otherwise the task will be prematurely interrupted. We do that in the task configuration. The bread crumbs are these:

If you scroll down to the bottom of the page you should see a section called "Timeout and delay"

Here you also see the previously mentioned timeout of 5 minutes. If the timeout is changed to a 0, it means there is no timeout and the task is never stopped. 

We will instead set it to 30 minutes which should be plenty for our purpose.

The two other options have to do with delays. We can control here whether we want to skip the task if it is delayed too much. E.g. if we wanted to executed our task every hour, and the task is delayed by 45 minutes, we might want to skip it in order to not to create a backlog of old tasks.

Setup another email action to notify us in case there are unrecoverable failures

In case we were not able to recover from a failed action sequence, we need to make sure we know about the failure. For this purpose, we'll set up yet another email which we'll send in this case.

First, we need to make sure that the failure is actually signalled. 

Just above the list of actions in your sequence is a checkbox "Propagate failures". It is checked by default. We will leave it like this.

If it is checked, it means that a failure in any part of the sequence is propagated to the next level below, which in our case is the scheduler task itself. So the action sequence fails -> the scheduled task failed. If "Propagate failures" were unchecked, then the action sequence fails -> the scheduled task succeeds, which is not want we want.

Next, we go back to the task configuration. In terms of bread crumbs, that's here:

There is a section here called "Failure action". Which are actions to be performed in case the task fails.

We choose "Email action" and setup another email informing us that the Covid19 task has failed.

Check the logs to figure out what went wrong

Let's assume now the task has been completely setup, time and period for its executions specified and it has been running successfully for a while. Suddenly there is an email in the inbox notifying us about a Covid19 task failure.

It is then time to check the logs in order to understand what happened. For this purpose we go into the scheduler config, i.e. the scheduler welcome screen.

Here, we see a couple of icons on the upper right:

They give us quick access to the log files:

	The "Log" is a text file that is continuously populated by the scheduler with human-readable status information.
	The "Task Status" is a csv file that shows which tasks are scheduled for execution next.
	The "Tasks Log" is a csv file with information about previously executed tasks. You will find here information about when it was executed, how long it took, and whether any problems were encountered.
	The "Event Log" is a csv file that is akin with the scheduler log in Omniscope Classic.

For our purposes, we want to investigate the "Log". At the very bottom of the log you find the the most recent events. All you have to do is to check the date range of when the scheduled task was supposed to be executed and see if you can determine what went wrong. In our case we see that the action sequence failed with the error: "Unknown host, The server you are accessing does not exist or is not currently accessible.". So it seems there were problems with our internet connection. 

2020-11-10T11:53:00.020+0000 INFO  [omniscope.service.scheduler.main] - Executing task "Covid19" (Task action, every day from 09-Nov-2020 11:53, forever)...
2020-11-10T11:53:15.078+0000 ERROR [omniscope.service.scheduler.main] - Could not execute action
2020-11-10T11:53:15.101+0000 ERROR [omniscope.service.scheduler.main] - Action ended with failure
2020-11-10T11:53:15.101+0000 ERROR [omniscope.service.scheduler.main] - Action generated warnings/errors:
2020-11-10T11:53:15.101+0000 ERROR [omniscope.service.scheduler.main] - -------------------------------
Failed to execute the chain action: (Refresh Covid19 data and send email). Retry count remaining: 3
-------------------------------
Unknown host
The server you are accessing does not exist or is not currently accessible.

Host name: opendata.ecdc.europa.eu
Sleeping for 5 before retrying chain action: (Refresh Covid19 data and send email)
-------------------------------
Failed to execute the chain action: (Refresh Covid19 data and send email). Retry count remaining: 2
-------------------------------
Unknown host
The server you are accessing does not exist or is not currently accessible.

Host name: opendata.ecdc.europa.eu
Sleeping for 5 before retrying chain action: (Refresh Covid19 data and send email)
-------------------------------
Failed to execute the chain action: (Refresh Covid19 data and send email). Retry count remaining: 1
-------------------------------
Unknown host
The server you are accessing does not exist or is not currently accessible.

Host name: opendata.ecdc.europa.eu
Sleeping for 5 before retrying chain action: (Refresh Covid19 data and send email)
-------------------------------
Failed to execute the chain action: (Refresh Covid19 data and send email) - with no more remaining retries...
-------------------------------
Chain action "Refresh Covid19 data and send email" failure:
Unknown host
The server you are accessing does not exist or is not currently accessible.

Host name: opendata.ecdc.europa.eu
Propagating chain action "Refresh Covid19 data and send email" failure
Chain action "Task action" failure:
Chain action "Refresh Covid19 data and send email" failure
Propagating chain action "Task action" failure
Could not execute action:
Chain action "Task action" failure