Workflow Triggers - Cloud Storage Events (beta)

Overview

Magnus provides Workflow Triggers as a way to initiate a Workflow execution based on some predefined cloud events.  When the watched event is fired, the Workflow is triggered to execute via Remote Workflow Execution v2 via queue.  That is:

1. the request to execute is queued and processed by the Fair Scheduler
2. the quota for max enqueued requests per user applies
3. multiple instances of the Workflow is allowed
4. the Workflow state is not saved after execution

Cloud Storage Event

When a file is created, updated, or deleted in the watched Google Cloud Storage bucket, the Workflow is triggered to execute with information about the file and event.

Prerequisites before creating the Trigger

1. Trigger must be enabled for the organization.  Please contact your organization admins.
2. Before creating the Trigger, user must have already granted offline access to Magnus.  Otherwise, saving of the trigger will fail.
3. The service account service-[Project Number of Organization’s Trigger Cloud Project]@gcf-admin-robot.iam.gserviceaccount.com must be Storage Admin of the project that the watched Google Cloud Storage (GCS) bucket belongs to.  Otherwise, saving of the trigger will fail.

Add a Workflow Trigger

1. Make sure that prerequisites are met.
2. From Magnus Homepage, click on Triggers to go to the Triggers Browser.
   
   
   
   
3. Click on the Add Trigger icon on the upper left corner.
   
   
   
   
4. Specify Trigger properties.
   
   
   

   A Trigger has the following properties:

   Trigger ID	[Required] Unique name of the Trigger
   Description	[Optional] Description of the Trigger
   Owner	[Required] Account under which the Workflow is run when triggered if the Workflow is not locked to run always as the Workflow owner. This is case-sensitive.
   Lifetime	[Optional] Period during which the Trigger is enabled. If not specified, the Trigger is always enabled.
   Time Zone	Time zone of the Trigger lifetime.  For example, if the Trigger lifetime is 6/1/2018 to 6/2/2018 and the time zone is US/Eastern, then the Trigger will be enabled from 6/1/2018 to 6/2/2018 US/Eastern time.  The Trigger time zone does not affect Workflow time zone.

5. Choose GCS as Trigger type
   
   
   
   

6. Specify GCS event properties.
   
   
   
   A GCS event has the following properties:

   Description	[Optional] Description of the event
   Source GS Bucket	[Required] The Google Cloud Storage bucket to put a watch on
   Source GS Folder	[Required] The folder or file pattern to watch for. For example, to put a watch on gs://myBucket/fruits/apple_*.txt, specify /fruits/apple_*.txt as the Source GS Folder. More examples: /a/b: This matches the object b in folder a only, such as /a/b. /a/b/: This matches any objects in folder /a/b/, like /a/b/c or /a/b/c.txt. It doesn't match objects in further child subfolders, like /a/b/c/d.txt /a/b/*.txt: This matches any .txt objects in folder /a/b/, like /a/b/c.txt. It doesn't match objects in further child subfolders, such as /a/b/c/d.txt /a/*/*.txt: This matches any .txt objects in immediate child folders of a, like /a/b/c.txt or /a/d/c.txt.  It doesn’t match objects in further child subfolders, such as /a/b/e/c.txt /a/*/*/*.txt: This matches any .txt objects in immediate grandchild folders of a, like /a/b/e/c.txt or /a/d/f/c.txt
   State	[Required] The qualifying file state to trigger the Workflow.

   
   
   
7. Specify the Workflow.
   
   
   
   

   It has the following properties:

   Description	[Optional] Description of the response.
   Workflow ID	[Required] The Workflow to trigger.  Only owned or delegated Workflows are allowed.

8. Specify parameter overrides to pass to the Workflow during execution. 
   1. Click on the Arguments link.
      
      
      
      
   2. Select a value:
      
      
      
      
      1. inherit-var: inherit the parameter value from the Workflow
      2. set-value: set a custom value for the parameter
      3. var_data_*: these are built-in parameters that contain information about the GCS object that triggered the event.  See below for definitions.
      4. var_event_*: these are built-in parameters that contain information about the event.  See below for definitions.
         
         
9. Click Save to save the trigger.  This could take up to a few minutes.
   
   You will end up with something like below:
   
   
   
   

The Trigger will be enabled few minutes after saving.

Triggers Browser

All Triggers are listed in Triggers Browser. Triggers Browser lists Triggers by Trigger Type, Name, Owner, Workflow to be execute,  Created By and Created At
Action Icons are available for each Trigger providing a quick way to view, show Workflow to execute history, and delete.
Trigger information can be filtered. 

Toolbar

1. Take user to Magnus main landing page.
2.  Add new Trigger.
3. Refresh icon: Refresh the page to the latest results.
4. Content Search is disabled and not supported for now
5. Scope selector: Owned Triggers option only.
6. Filter: filter Triggers based on the input text on client side.

Row example and action buttons

Results can be sorted ascending or descending by clicking the small arrow next to each column title

1. Trigger Type
2. Trigger Name.
3. Owner for the trigger.
4.  View Trigger , read only mode.
5. Delete Trigger.
6. View the history of the Workflow.
7. The name of the workflow that is going to be executed by the Trigger.
8. Who created Trigger.
9. When Trigger was created.
10. Status of the Trigger:
1. failed: Trigger creation failed and by hovering over the status more information will be displayed.
2. deploying: Trigger is being saved and deployed and during that it can’t be deleted (Delete action icon will be disabled).
3. deleting:  Trigger is being deleted.
4. ready: Trigger was a successfully created.
11. Icon indicating GCS Event based Trigger 
12. Icon indicating Pub/Sub Event based Trigger

Paging

• The first line will show a summary how many Triggers were fetched, and only 30 Triggers per page will be displayed.
• The second line will show the number of pages and by clicking different numbers it will show the next 30 Triggers and so on.
• Exit button will take user to the Magnus main landing page.

 

Definitions of Built-in Parameters

Parameter name	Description
var_data_kind	The kind of item this is.
var_data_id	The ID of the object, including the bucket name, object name, and generation number.
var_data_name	The name of the object.
var_data_bucket	The name of the bucket containing this object.
var_data_generation	The content generation of this object. Used for object versioning.
var_data_contentType	Content-Type of the object data
var_data_timeCreated	The creation time of the object in RFC 3339 format.
var_data_updated	The modification time of the object metadata in RFC 3339 format.
var_data_size	Content-Length of the data in bytes.
var_data_md5Hash	MD5 hash of the data; encoded using base64
var_data_timeDeleted	The deletion time of the object in RFC 3339 format.
var_event_id	A unique ID for the event
var_event_timestamp	The date/time this event was created in ISO 8601 format
var_event_type	The type of the event.

Internals

Workflow Triggers are backed by Google Cloud Functions.  When a Workflow Trigger is created, a respective GCF is created in the organization's Google Cloud project.  When event invokes the GCF, the GCF in turns makes a call to remotely execute the Workflow.  Per the nature of GCF, it is guaranteed to be invoked at least once for each incoming event (https://cloud.google.com/functions/docs/concepts/exec#execution_guarantees).  It is recommended to make the target Workflow idempotent so that when an incoming event invokes it multiple times, the desired results will still be achieved.