Custom block execution in Docker containers

Omniscope Evo allows the user to execute custom workflow blocks in Docker containers. By using Docker containers you can get more flexible and robust execution environment for your R and Python custom blocks, as well as improved security in your server installation.

When Omniscope is configured to execute custom block in Docker containers, each time a custom block is executed a new container will be instantiated. A container provides a sandboxed environment where all the libraries and system packages that might be required by the script can be installed without the need to share them across your entire server installation. 

By default, Omniscope Evo will use Docker images provided by Visokio, containing both the language interpreter, as well as the Omniscope api libraries. If your script has a more complex setup (e.g., it requires the use of custom binaries), it is possible to use a custom image to execute the custom script. 

This article will give an overview on how to enable and configure Omniscope to run custom blocks in Docker containers. You can find more information on how to build custom images here.

The second main reason to use docker containers is improved security: when scripts execute within Docker containers the host filesystem won't be visible, hence the script won't be able to access resources it was not supposed to. The only resources the script is allowed to access are the files and folders that have been specified in the options. You can control if users are allowed to use resources outside the sharing folder via the the option in the settings page:

Enabling custom block execution in Docker containers

Docker needs to be installed on the system, and, on linux systems, the user used to run Omniscope must be a member of the docker group. Omniscope will need access to the docker client command line interface, and it assumes the docker daemon runs on the same machine as Omniscope.

You can find more information about installing Docker for your system here https://docs.docker.com/get-docker/. 

Once installed, docker must be running in order to use Docker functionality from Omniscope:

• On Mac and Windows, ensure that the "Docker Desktop" app is running.
• On Linux, the "docker" daemon/service must be started (usually automatically. 

Once docker is installed and running, you need to enable custom block docker container by changing the custom block execution environment setting in the R-Python page of the admin app.

Note: This setting will affect the execution of all custom blocks. It is not possible to configure the  execution environment on a per block basis.

Custom block options

When used in the Docker execution environment, there are some differences in the Custom block configuration options, also in the way they behave.

Here we'll discuss differences for a custom block configured to use Python. Similar considerations can be made for a custom block configured to use R.

Firstly, it is not possible to select the version of the interpreter. The interpreter will be contained within the docker image, and not the interpreters installed on the machine. 

As previously noted - the Dependencies field behaves a bit differently: instead of installing the dependencies on the host machine, where they will be shared across all custom blocks, these dependencies will only be available exclusively to this block. This allows multiple blocks to have a different set of potentially conflicting dependencies (e.g., two versions of the same dependency required by two different blocks).

In the Advanced section it is possible to specify a custom image. 

The custom image can be available in the local registry, or it needs to be available in a private or public registry as needed where it can be pulled from. 

In order for the custom image to be compatible with Omniscope it needs to satisfy some requirements. For more information on how you can create your custom images see this article.

If the custom image does not use one of the Visokio images as its base, Omniscope needs to install the library, so that the script can communicate with Omniscope.  To achieve that, you can set the option "Install Ominscope Api libraries".

It is possible to install some system packages that should also be available in the container. This feature requires a base image based on Debian, and the system packages are assumed to be names of Debian packages available in the repositories of the base image.  

Custom block executions in Docker containers

The first time Omniscope executes a custom block using docker container, it will need to pull the base image for the selected runtime. This operation requires internet access and it can take some time.

Alternatively you can manually add Visokio images directly to your local registry. As of this writing the latest images for R and Python can be pulled from

europe-west2-docker.pkg.dev/visokiowebsite/visokio-custom-block/visokio-r-custom-block:0.0.1

and 

europe-west2-docker.pkg.dev/visokiowebsite/visokio-custom-block/visokio-python-custom-block:0.0.1

respectively.

If dependencies or system packages are required by the custom script, Omniscope will generate an image for the custom block. The first time the custom block executes the image will be built, and the libraries/packages installed as part of the Docker image. This operation can be slow, depending on which operations are needed. The image is stored in the local registry, so that it can be used by future block executions.

N.B. Images will be reused whenever possible. If you are not frequently changing system packages or libraries to install, you should not see an increase of the number of images used by Omniscope. 

Images generated by Omniscope are labelled with "visokio-image-type"="custom-block-generated" in their Dockerfile. See Disk clean-up section for more information on how to manage the disk space used by Visokio docker images. 

The script itself will run within the container, which provides a sandboxed environment for its execution. This also means that the script won't be able to access file system resources with the only exception of files and folders that have been specified in the options: those will be mounted as volumes. 

N.B. by default volumes will be mounted as read only, but you can change this behaviour from the options:

If your script needs to change its  behaviour, depending on whether or not it is running in a docker container, it can query an Omniscope api instance

Python:

omniscope_api.is_docker()

R:

is.docker(omni.api)

Images clean-up

As mentioned in the previous section, Omniscope will generate custom images to run custom blocks. Although images will be reused wherever possible, and additional disk space only accounting for the difference with respect to the base image, it is possible for images generated by Omniscope to take up a significant amount of disk space over time.  

Depending on the user's system setup - it may be sensible to set a threshold on the disk space that custom images are allowed to use. If this threshold is exceeded - Omniscope will start deleting Focker images as needed. Images will be rebuilt next time a block is executed.

You can configure this threshold in the R-Python page in the admin app:

This value will be compared to an estimation of the size taken up by the images generated by Omniscope.

N.B. Base images (either custom base images or Visokio-provided base images) will never be removed.

If you want your custom images do be deleted when a cleanup is triggered you can add the label 

visokio-image-retention=prune-on-cleanup in the dockerfile