Workflow

Overview

Scheduled daily in the morning, a sequence of tasks are executed to extract data from Google BigQuery, merge data points with metadata, calculate metrics, loop through attributes, save data to a table, export the data to an FTP server and Google Cloud Storage, email the process owner of process status, and distribute an HTML report via email to a distribution group.  All of these actions are available and made possible because of Magnus Workflow.

Magnus Workflow manages Tasks, scheduling, and ownership of processes that primary bridge the gap between Google BigQuery data and ETL tools.  Workflows can consist of any of the available Tasks in any combination to perform complex, scheduled, automated functions.  Workflows are scheduled by specifying frequency and time period of run.  Workflow control is in the hand of creators who own the process and are able to delegate and share with other users.

Workflows feature shortcuts, search/replace, and parameters to make process design easier.  Shortcuts allow designers to quickly jump to tasks.  Search and Replace features an easy way to update any text within a workflow.  Parameters help generalize values so that they can be used in any task though out the Workflow.

Header

A workflow has the following properties:

Workflow ID	Unique name of the workflow
Description	Used to describe purpose of workflow (optional)
Owner	Account under which workflow is run when scheduled Primarily the user who created workflow User account is case-sensitive!
Delegates	Users who can edit and run workflow Added by Owner or other Delegates Optional  User accounts are case-sensitive
Shared With	Users who can run workflow via Execute Workflow Task Added by Owner or Delegates Optional  User accounts are case-sensitive
Lifetime	Time period during which scheduled workflow is executed
Schedule	When to execute a workflow Requires Lifetime to be set and Workflow to be enabled Please consider: Try not to schedule on the exact hour, half hour, or quarter hour.  Spread the time out such as 9:04 or 9:32 instead of 9:00 or 9:30 Optional Can be set using drop down options Or by setting Quartz Cron (Advanced)
Time Zone	By default, the Workflow is executed in Magnus’ server time zone, which is US/Eastern.  You can change the time zone in which to execute the Workflow.  This affects when the Workflow is executed and the time-related built-in parameters. If the Workflow with US/Eastern time zone is scheduled to run at each day at 5 AM from 6/1/2018 to 6/2/2018, then it will run on 6/1/2018 and 6/2/2018 at 5 AM US/Eastern time.  The following built-in parameters will reflect the date and time in US/Eastern time zone during the execution: var_date, var_daydate, var_year, var_month, var_day, var_hour, var_yesterday, var_timezone.  Note:  - The time zone selected in not applied to Script Task, so you will need to utilize the built-in parameter <var_timezone> in your script body to do any date time manipulations. - The time zone selected is not inherited by the child Workflow in Execute Workflow Task.  For example, if the parent Workflow has US/Eastern time zone and the child Workflow has US/Pacific time zone, then during execution the parent Workflow will be executed with US/Eastern time zone and thus its built-in parameters will reflect the date and time in US/Eastern time zone, while the child Workflow will be executed with US/Pacific time zone and its built-in parameters will reflect the date and time in US/Pacific time zone. -The time zone selected only applies to this Workflow. If you want to change the time zone for other Workflows, you will have to update each of them individually.
Enabled	Enable/Disable workflow schedule.
Billing Project From	Owner or Delegate User can choose what Billing Project a specific workflow should use Run-User: use the Magnus-assigned billing project of the user who is running the Workflow.  Owner: use the Magnus-assigned billing project of the Workflow Owner.  Custom: Specify a billing project. Parameter is not allowed
Cascade Billing Project	If checked, during workflow execution, the billing project ID that is derived from the setting “Billing Project From” will be passed down to child and descendent workflows. During the execution of the descendent workflows, the cascaded billing project will be used. This applies for descendent workflows that are executed via Execute Workflow Task or Remote Workflow Execution. The user who is running the workflows must have required permission in the cascaded billing project. For example, if this is a workflow chain w1 > w2 > w3 > w4 > w5, where the top level workflow w1 calls w2, and w2 calls w3, and so on. If you want all 5 workflows to be executed in a custom billing project, then you will set the custom billing project and cascade option in w1. Then all 5 workflows will be run under the billing project my-custom-billing-project.
Public	Checked: the Workflow is viewable by all users who have same or set to have visibility to Magnus-assigned billing project as the Workflow Owner  Unchecked: the workflow is viewable only by its owners/delegates/shared users
Save	Save new or modified Workflow
Exit	Exit workflow.  Prompts to save if there are unsaved changes
Run	Runs workflow.  Automatically saves if there are unsaved changes Workflow will run under Owner Account when scheduled Workflow will run under User Account who Manually Run Workflow
Workflow Header	If clicked twice Tasks will expand/collapse
Show/Hide Disabled Tasks	Clicking icon will show/hide disabled Tasks
Archived	Option becomes available when Archive icon is clicked from the Workflows Browser Check indicates to Archive the workflow on save Archiving a workflow removes the workflow from listing in Workflow Browser To see archived workflows check Show Archived from Workflow Browser

Left Vertical Toolbar

Toolbar is available when adding new or editing existing workflow

	Go back to Magnus home page
	Run workflow manually
	Save workflow changes
	Exit workflow
	Jump to task Names of tasks in workflow will display in list Selecting option will jump to task within workflow
	Collapse/Expand all tasks
	Hide/Show disabled tasks
	Show Workflow Execution History
	Show Workflow Versions Modification date and the user who did the modification will display in a list. Click on a particular version to view it

Find and Replace

Text within workflows are searchable and can be replaced by using Find/Replace feature

Find/Replace Collapsed

Find/Replace Expanded

Find input box	The target keyword to be searched for within workflow
Replace input box	Text used to replace found keyword
Match case	Checked keyword search is case sensitive Unchecked keyword search is case insensitive
Match whole word	Checked entire keyword matches are displayed Unchecked partial keyword matches are displayed
Find button	Clicking initiates Find Results will be displayed Selections can be checked from Results to determine which matches to Replace Select all and Unselect all are available for quick selections

Parameters

Parameters can be used throughout the workflow.
To add a custom parameter, follow the steps below:

1. Click to open Parameters tool 

2. Built-in parameters are system defined parameters
3. Custom parameters are user defined parameters

a. Click Add to create custom parameter 

b. Parameter is always created with var_ prefix 

c. Parameter can be deleted if x is clicked
d. Parameter type can be string, number, Boolean, or record

Workflow footer

Add Task container	List of individual tasks that can be added to workflow Clicking on a specific Task button will add that task to the workflow
Add to position	Displays list of existing Tasks from which a selection can be made to insert added Task into that position
Save	Save new or modified Workflow
Exit	Exit workflow.  Prompts to save if changes have been made
Run	Runs workflow.  Prompts to save if changes have been made Workflow will run under Owner Account when scheduled Workflow will run under User Account who Manually Run Workflow
Copy	Creates copy of saved workflow Available to Owner and Admin
Share Link	Workflow shortcut link is a link that is auto generated by the system and can be used to access the workflow immediately from the browser

 

Share Workflow Link (Permalink)

At the bottom of the Workflow Editor, there is a button titled as “Share Link” as shown below:

If the button was clicked then the system will show a text box that will appear with the shortcut link highlighted/selected: 

Now, user can copy and use this link to reference this workflow, share it with anyone who entitled 

On-Save Notification 

Any time – someone saves workflow – email notification can be sent to the owner of that workflow and cc to user who saved workflow.

There is a magic parameter available that allows enable/disable this feature for given workflow - var_sys_save_notification_enabled – by default it is false.

To set the recipients, add and set parameter var_sys_save_notification_recipients to a comma-separated list of e-mail addresses (default is workflow owner and user who did the change)

You can add this parameter and set it to false to disable notification

Notification Email will have details on Tasks that were modified , deleted or added

Ownership Change Notification

Magnus administrators have the ability to change workflow ownership.  When there is a change in workflow ownership, previous owner and new owner will be notified via email about the change.

Failure Notification

It is important to have your respective workflows properly set with graceful alerting/notification toward its owners and consumers.

Magnus has all you need for this:

To fully control workflow notification options follow below instructions:

To disable notification, add and set boolean parameter var_sys_notification_enabled to false (default is true) 

To set the recipients, add and set parameter var_sys_notification_recipients to a comma-separated list of e-mail addresses (default is workflow owner)

To set when notification is sent, add and set parameter var_sys_notification_event to failure, success or completion (default is failure)

 To get this notification on manual runs and/or scheduled runs, set var_sys_notification_scope to manual, scheduled or all (default is scheduled)

To get this notification on all other modes of executions (via Execute Workflow Task, Trigger or Remote), set var_sys_notification_scope to all

Please make sure you use this features when needed, especially for business critical workflows

Size Limitation

Workflow size is limited to 5 MB; user cannot save a Workflow that is more than 5 MB in size.  To keep Workflows within the size limit, it is recommended to clear any large Workflow-owned custom parameters at the end of the Workflow execution.