Workflows History

Overview

Scheduled and manually run workflows display in Workflows History.
Workflows History lists owned, delegated, shared, and public workflows.
The list indicates if the Workflow was run manually, via schedule, via remote, via trigger or executed by another workflow, the running start and end time, duration of the run, cost, maximum billing tier, who ran the workflow, who owns it, a link to Workflow details, status of the workflow, and any messages on failure or task notification.

Content of Workflows can be searched and front end Workflow information can be filtered.
The Workflow list can also be filtered by displaying only owned, owned and delegated, owned and delegated and shared, public Workflows, or all of the above.
An optional Hierarchy View is available to consolidate listing of Workflows executed by another Workflow.

Toolbar

1. Home icon: take user to Magnus main landing page
2. Refresh icon: Refresh the page to the latest results
3. Workflow search: get history from the backend for those workflows with matching Id
4. Scope selector:
   1. Owned workflows: show only the workflows that user is owner of
   2. Owned + Delegated: show only the workflows that user is owner or delegate of
   3. Owned + Delegated + Shared: Owned +delegated , plus the workflows that are shared with the user
   4. Public workflows: show only History from public workflows from user's billing projects and viewable projects.
      When this option is selected, History from the following public workflows are returned:
      1. Workflows with Owners from the same billing project as the user
      2. Workflows with Owners from viewable projects of the user
         If the user has viewable projects, user's projects are listed in the options in the order of billing project and viewable projects
         
   5. Get Started Examples: show only History from examples.
   6. All Above: Owned + Delegated + Shared + Public + Examples
5. Filter: Filter history based on the input text in the client side
   1. In Hierarchal Mode it will only apply to parent workflows
6. Hierarchy View: By default it is unchecked, once it is checked all workflows that were executed by a parent workflow will be gathered under the parent. For example

Normal view

Hierarchical view

1. If user clicks on the arrow on the left hand side, parent workflow and all workflow executed by it will listed sorted by start date.
2. Also Hierarchy mode will roll up cost by adding all costs and highest billing tier by displaying the maximum.

Sorting

User can sort ascending or descending by clicking the small arrow near the column title.

Row example

1. Run mode:
   1.  The workflow was run manually
   2.  The workflow was run via scheduler
   3.  The workflow was run via an Execute Workflow Task
   4.  The workflow was run via an Execute Workflow Task and the workflow is a shared workflow
   5.  The workflow was run remotely, not via queue (remote API v1)
   6.  The workflow was run remotely via queue (remote API v2)
   7.  The workflow was run via trigger
2. Run duration
3. Cost for running the workflow
4. Highest billing tier of BigQuery tasks in workflow
5.  action button to view Workflow Run Details
6. Status:
   1.  The workflow is still running
   2.  The workflow completed successfully
   3.  The workflow failed
   4. The workflow run was rejected because another instance is already running

   5. The workflow was killed by admin, or the workflow failed because a BigQuery job was canceled.

   6.   The workflow is waiting to be executed.
7. Error message

Paging

The first line will show a summary how many workflows history were fetched, and only 30 workflows history per page will be displayed. The second line will show the number of pages and by clicking different numbers it will show the next 30 workflows history and so on. Exit button will take user to the last page that history was opened from, this page might be Magnus landing page, or workflow browser

 

Max Results

User can adjust how many history items to retrieve.  Whenever Max Results is adjusted or the Search button is clicked, a new set of results is retrieved from the metadata store, ordered by the current sorting order.  Except for these four columns: BQ Cost, BQ Max Billing Tier, Status, and Message, sorting is supported from the metadata store.  If the current sorting order is any of these four columns, when Max Results is adjusted or the Search button is clicked, a new set of results is retrieved, ordered by Start Time.

Workflow Run Details

	User can view the above popup by either clicking on the workflow Id in history browser or this action button  It gives details about the workflow, and list all the tasks within this workflow even the task was disabled Click "Run Info" to see some run-time details like who ran the workflow , mode and parameter overrides if any. It shows what task caused failure if any, and if the task is a BQ task, it will show BQ Job ID. BQ Job ID is clickable and if you click it, it will show task details, in the example shown above, aggregate_data workflow had two enabled tasks: b and b2
	Run mode: it describes how the workflow execution was initiated. Manual: the workflow was run manually Schedule: the workflow was run via schedule Execute Workflow Task: the workflow was run via Execute Workflow Task Remote: the workflow was run via Remote Workflow Execution Run by: it describes who ran the workflow.   Parameters overrides: these are parameter overrides passed to the workflow during execution
	Click  to navigate between task details Click  to go back to Workflow Run Details main page Cancel BQ job: User can cancel only a running or pending BQ job that the user owns. User can achieve that by clicking on workflow name or icon  from history browser. Then click on the desired BQ Job ID that is still running, then at the bottom of the popup window, there should be a cancel button as show below. Once cancel button is clicked a confirmation message will be disabled as below Once the user confirms by clicking yes, the following message will displayed. The user needs to refresh the history browser to see updates, if the task was cancelled successfully and not finished yet then this message will be displayed with workflow status as failure.

Tasks Run Details

BigQuery Task

From Workflow Run Details, a BigQuery Task is displayed BQ Job ID: The BQ job ID that was submitted Cost: Cost of the BQ job Billing Tier: Billing tier of the BQ job

Click on the BQ Job ID link to view BQ job details. destinationTable: Destination table of the query results state: State of the BQ job totalBytesProcessed:  Number of bytes processed by the BQ job totalBytesBilled: Number of bytes there were billed billingTier: Billing tier of the BQ job referencedTables: BQ tables that were referenced by the BQ query user_email: The user that executed this BQ job query: The query text If the BQ job has state “RUNNING” or “PENDING” and you are either super-admin or the BQ job creator (listed under user_email), then you will be able to click on the “Cancel Job” button to submit a job cancellation request to Google

 

GS Export Task

	From Workflow Run Details, a GS Export Task is displayed BQ Job ID: The BQ job ID that was submitted
	Click on the BQ Job ID link to view BQ job details. sourceTable: This is the source BQ table being extracted destinationUris: This is the destination GS URI state: State of the BQ job destinationUriFileCounts: The number of destination files user_email: The user that executed this BQ job

E-Mail Task

	From Workflow Run Details, an E-Mail Task is displayed Email: This is the email subject
	Click on the email subject link to view details. Subject: email subject Sent: The date and time the email was sent successfully. To:  To email address Cc:  Cc email address Bcc: Bcc email address Attachment: Attachment file name and file size if there was any attachment file sent

 

Workflow Task

	From Workflow Run Details, a Workflow Task is displayed Workflow: The child workflowID
	Click on the child workflow ID link to view details. Workflow ID:  The child workflow ID Is Shared: If the child workflow a shared workflow? Parameter list: Parameter name Parameter type Parameter value: It is the design time value, run time value can be different. Overwrite action

Script Task

	From Workflow Run Details, a Script Task is displayed: Script: The script language used
	Click on the script language link to view details. Language: The script language used Script: The script body Return parameter: Name of the return parameter Return value: The value that is returned from the script execution Standard output: The standard output from the script.  Only the last output call is displayed

 

API Task

	From Workflow Run Details, an API Call Task is displayed API Request: This is the request URL
	Click on the API Request link to view details. Request URL Request Method Request Body: This is the request body if any Response Body: This is the response body if any

Go-To Task

From Workflow Run Details, a Go-To Task is displayed

Click on the link condition to view details. Script: The script language used Condition: This is the Left Expression, Comparison Operator, and the Right Expression Evaluated to: This is the Left Value (after parameter substitution and script execution if any), Comparison Operator, and the Right Value (after parameter substitution and script execution if any).  The last columns shows comparison outcome. Proceeded to:  This is ID of the task proceeded to, based on the comparison outcome

Misc Task

From Workflow Run Details, a Misc Task is displayed Message: This is the message from Raise Failure

FTP-GS Task

	From Workflow Run Details, a FTP-GS Task is displayed FTP to GS info: This is the source FTP location
	Click on the source FTP location link to view details. Source: This is the source FTP location Destination: This is the destination GS location Files loaded: The number of files loaded with total file size. Files list: This is the list of loaded files

 

Loop Task

From Workflow Run Details, a Loop Task is displayed Iterations: This is the number of iterations executed so far. Select an iteration from drop down to view respective tasks. Last iteration: These are the tasks executed from the most recent iteration. Task/iterations: These are the task IDs and the number of iterations executed per task

 

FTP Task

From the Workflow Run Details, a FTP Task is displayed FTP: This is the BQ job ID submitted

Click on the BQ job ID link to view details. BQ Job info: This is the first step of a FTP Task.  It extracts the BQ table to a temporary Google Cloud Storage location. sourceTable: This is the source BQ table state: This is the state of the BQ job user_email: This is the user who executed this BQ job FTP info: This is the last step of a FTP Task.  It copies the file from Google Cloud Storage to the destination FTP location. Destination: This is the FTP destination location with the delivered file size

MySQL Task

From the Workflow Run Details, a MySQL Task is displayed

Click on the export link to view details. Source BQ table: This is the source BQ table Destination DB: This is the destination MySQL database Destination table: This is the destination MySQL table Write preference: This is the write preference option selected Replace data where: For write preference "Replace data", this is the replace filter provided. Records: This is the number of records exported.