Savepoints

Cloudomation automatically creates savepoints of your running executions. Savepoints are used to scale your automations and can be used to recover an execution after an error.

Use Cases

Savepoints are used to

persist the state of executions when they are idle.
protect executions from workspace process downtimes.
transfer executions between workspace processes to allow for scaling.
mark a position in your code which can be used to jump back to.

Cloudomation automatically handles the persisting, scaling and restoring for you.

Concept

Savepoints contain all information needed for the execution to resume at exactly the same instruction and going forward. Additionally the line and name of the Flow-API call and the timestamp is stored.

It is possible to revert an execution to the state of a savepoint for it to resume operation from the instruction. The complete state of the execution is restored, including all variables used in the execution.

Restoring an execution to a savepoint does not however affect other objects, like Cloudomation settings, Cloudomation files, or third party systems. Only the execution is restored. Objects which the execution already modified will remain modified.

Types of Savepoints

Additionally to the automatically created implicit savepoints, a Flow script can explicitly request a savepoint to be created. Such explicit savepoints can also be assigned a name.

Using Savepoints

Persisting Idle Executions

When an execution is idle for more than 60 seconds Cloudomation will store it's state in the database and unload it from memory. See SLEEP_MAX_SECONDS in Performance Settings

When the execution resumes Cloudomation will automatically restore the state.

Since savepoints are created implicitly with each Flow-API call, executions are resistent to workspace process crashes, server reboots, software upgrades, or network interruptions. As soon as a workspace process becomes online again, it will resume the execution from the last state.

Transfer Executions for Scaling

Cloudomation workspace processes are stateless workers. They consume executions from the database, process them, and and save the state in the database. Multiple workspace processes can be active at the same time. Whenever possible the least-busy workspace process is chosen to process an execution.

When deploying Cloudomation in an auto-scaling environment like Kubernetes, the CPU usage of workspace processes can be used to determine the need for up- or downscaling.

Create a Named Savepoint

The Flow-API command create_dump creates an explicit, named savepoint.

Example

import flow_api
def handler(system: flow_api.System, this: flow_api.Execution):
    # fetch some data
    data = this.connect(
        'database',
        fetch='SELECT * FROM data',
    ).get('output_value')['result']
    # create a named savepoint
    # in case of an error the execution can be reverted
    # to this state and resume normal operations
    this.save_dump('with-fetched-data')
    # process the data
    # in case the remote-api call fails it is possible to
    # revert to the savepoint above and retry the remote-api call
    # without having to fetch the data again
    this.connect(
        'remote-api',
        method='post',
        path='/upload',
        json=data,
    )
    return this.success('all done')

Restore an Execution to a Savepoint

It is possible to restore a savepoint using the Flow-API:

this.load_dump('with-fetched-data', by='name')
# or by ID
this.load_dump('8f47bbb1-6a04-435f-952f-1cbb84f71b4a')

Be careful to avoid loops when restoring a Savepoint using load_dump.

Savepoint lifetime

When an execution is archived all its savepoints will be deleted.

Learn more

Exceptions

Workspace configuration

Knowledge Base — Previous

Plugins

Next — Knowledge Base

Scheduling