Using Batch Append and Append files blocks to import multiple files

Omniscope toolbox contains blocks to allow users to connect to data sources in a variety of formats and locations. The source blocks can connect to one file, in a specific location, connect to a folder and extract contents of multiple files or connect to multiple files in different folders.

Batch Append block

If you are working with data which is gathered in a particular folder (e.g. data for Week 1, Week 2, Week 3 etc.) you can use a single Omniscope block to append the records for the whole series.

Another benefit of this approach is that every time a new file is saved in this folder a manual or scheduled data source refresh will drive the new records through the pre-defined transformation process and publish it in the desired format, such as an interactive dashboard.

The block will prompt you to select a target folder, where you can use the wildcard character to pick the word in the file name OR an extension. You could for example use "Sales*" to append all files that contain the word "sales" or "*.csv" to append all csv files in the folder, regardless of their name.

NOTE: If you are working with files in Excel format and wish to extract data from only some sheets, from multiple files, you can un-tick the 'Auto detect file type' box and manually pick the file format (here Excel) and the sheets in the workbook you wish to use. (see image)

In the same dialogue you can also specify which row contains the headers with the field names and from-to record section, in case you wish to pick only rows 10-25 and ignore the rest.

It is worth noting that in the data Sources section you will also find "Append files" block, which will allow you to append multiple files from different folders.

Append files block

This block facilitates the extraction of multiple files from any location (could be mix of machine, network or cloud locations) and in any format (not just Excel – CSV, JSON, XML, etc). 

By default Omniscope will auto-detect the file type and read the contents of the file according to the default options. The user can configure how Omniscope will read the files by un-ticking ‘Auto detect file type’, then selecting the file type and further options.

Below are shown configuration options for all three file connection options.