Implementation of the xOpera orchestrator REST API

xOpera REST API uses git backend to store blueprints. It supports github.com and gitlab (both gitlab.com and private gitlab based servers). If GIT server is not set it uses computer's filesystem to store git repos.

To connect REST API to github.com:

• github.com user with enough private repository is needed. Every blueprint will be saved to its own repository, so unlimited account with private repositories is recommended.
• obtain github access token with repo end delete_repo permissions
• export following environmental variables:
  • XOPERA_GIT_TYPE=github
  • XOPERA_GIT_AUTH_TOKEN=[your_github_access_token]
• optionally set some of optional git config settings

To connect REST API to gitlab server:

• obtain Gitlab's Personal Access Token with api scope
• export following environmental variables:
  • XOPERA_GIT_TYPE=gitlab
  • XOPERA_GIT_URL=[url_to_your_gitlab_server]
  • XOPERA_GIT_AUTH_TOKEN=[your_personal_access_token]
• optionally set some of optional git config settings

If non of options above is available, xOpera REST API can use its internal filesystem as git server (MockConnector). This option has a big limitation of not enabling users to inspect blueprints as on github.com, gitlab.com or any other gitlab server. To connect REST API with MockConnector to internal filesystem:

• export following environmental variables:
  • XOPERA_GIT_TYPE=mock
• optionally set some of optional git config settings

Note: Mock connector is default option which is used in case of missing XOPERA_GIT_TYPE environmental variable.

Beside obligatory settings following settings can be configured:

• XOPERA_GIT_WORKDIR - workdir for git on REST API server
• XOPERA_GIT_REPO_PREFIX (default: gitDB_) - repo prefix. Blueprint with token 963d7c94-34f9-498d-b122-472dbd9a8681 would be saved to repository gitDB_963d7c94-34f9-498d-b122-472dbd9a8681
• XOPERA_GIT_COMMIT_NAME (default: SODALITE-xOpera-REST-API) - user.name to author git commit
• XOPERA_GIT_COMMIT_MAIL (default: no-email@domain.com) - user.mail to author git commit
• XOPERA_GIT_GUEST_PERMISSIONS (default: reporter) - role, assigned to user, added to repository. See access to blueprints.

See example config for example on how to export variables.

In most applications, REST API needs docker registry to store docker images. It can be run locally or remotely. See docker docs for more details.

xOpera REST API uses OAuth 2.0 for authentication.

It can be overridden by setting AUTH_API_KEY env var in xopera-rest-api container to key_name of choice. This key must be added to requests as -H "X-API-Key: [key_name]"

xOpera needs SSH key pair with xOpera substring in name in /root/.ssh (or other) dir. It can be generated using

sudo ./Installation/ssh_keys.sh [common name] [SSH_DIR | default='/root/.ssh'] 

where common name is desired name for SSH key (usually computer's IP).

To run locally, use docker compose or local TOSCA template with compliant orchestrator.

REST API can be deployed remotely using TOSCA template with compliant orchestrator, for instance xOpera.

Check openapi spec and sample api requests.

Standard scenarios of using REST api:

xOpera uses SSH key to connect to instance VMs. Its public key can be obtained with GET to /ssh/keys/public. Public key must be registered with cloud provider (e.g. OpenStack).

Blueprint consist of TOSCA service template with all corresponding artifacts, neatly packed into The TOSCA Cloud Service Archive (CSAR). xOpera REST API leverages GIT server (it supports Github and Gitlab at the moment) for storing CSARs.

A new blueprint can be uploaded with POST to /blueprint. CSAR must be a valid TOSCA CSAR archive (in zip format). Optionally, user can add several other parameters:

• revision_msg --> commit message for git
• blueprint_name --> human-readable blueprint_name
• aadm_id --> End-to-end debugging id
• username --> User's username
• project_domain --> domain of blueprint

If successful, response in form of BlueprintVersion schema that will include blueprint_id and version_id, which can later be used for accessing blueprint (version).

This endpoint allows querying blueprints by username, project_domain or both. It returns [Blueprint] where Blueprint consist of (blueprint_id, project_domain, username, timestamp)

A new version can be added to existing blueprint with POST to /blueprint/{blueprint_id}. CSAR must be a valid TOSCA CSAR archive (in zip format). Optionally, user can add revision_msg, or specify project_domain for blueprint. If successful, response in form of BlueprintVersion schema will include blueprint_id and version_id, which can later be used for accessing blueprint (version).

Blueprint can be deleted with DELETE to /blueprint/{blueprint_id}. Before deleting, REST API will check if blueprint is a part of any existing deployment and block deletion. This behaviour can be overridden with force=true parameter.

Blueprint version can be deleted with DELETE to /blueprint/{blueprint_id}/version/{version_id}. Before deleting, REST API will check if blueprint version is a part of any existing deployment and block deletion. This behaviour can be overridden with force=true parameter.

List of git users with access to Repository with blueprint can be obtained with GET to /blueprint/{blueprint_id}/user.

Git (Github or Gitlab) user can be added to blueprint with POST to /blueprint/{blueprint_id}/user/{user_id}. User will be assigned developer access and will be able to modify blueprint.

Git user can be removed with POST to /blueprint/{blueprint_id}/user/{user_id}.

Blueprint metadata from the most recent version of blueprint can be obtained with GET to /blueprint/{blueprint_id}/meta. Metadata consist of:

• blueprint_id
• version_id
• name
• project_domain
• timestamp
• url
• commit_sha
• users
• deployments

Blueprint metadata from specific version of blueprint can be obtained with GET to /blueprint/{blueprint_id}/version/{version_id}/meta. Endpoint returns the same fields as /blueprint/{blueprint_id}/meta. Note that some parameters are blueprint-version-specific (every version can have its own values) and some are not (every version of single blueprint has the same values).

Version-specific metadata:

• version_id
• timestamp
• commit_sha

Although blueprint name is part of blueprint meta it can also be obtained via dedicated endpoint with GET to /blueprint/{blueprint_id}/name.

Blueprint name is usually set upon blueprint creation (POST to /blueprint), but it can also be changed later, with POST to /blueprint/{blueprint_id}/name.

List of deployments, created from current blueprint can be obtained with GET to /blueprint/{blueprint_id}/deployments. Every list item consist of:

• deployment_id
• state
• operation
• timestamp

List of deployments is part of blueprint metadata and can also be obtained via metadata endpoint.

History of changes, made by xOpera REST API to git repository with blueprint, can be obtained with GET to /blueprint/{blueprint_id}/git_history. Changes can be either update (addition of blueprint version) or delete (deletion of blueprint or blueprint version).

Blueprint from database can be validated with PUT to /blueprint/{blueprint_id}/validate. Optionally, file with inputs can be added. If blueprint has multiple versions, the last will be validated.

Blueprint version from database can be validated with PUT to /blueprint/{blueprint_id}/version/{version_id}/validate. Optionally, file with inputs can be added.

Any blueprint in The TOSCA Cloud Service Archive (CSAR) form can be validate with PUT to /blueprint/validate. After validation, blueprint will be discarded.

Deployment is xOpera REST API's internal representation of current instance state, deployed on cloud platform

To initialize deployment with first deploy, user can POST to /deployment/deploy. If version_id is not specified, last version is used. Inputs are optional, depending on blueprint. Workers is a maximum number of concurrent workers, leveraged by xOpera. deployment_label is a human-readable label, which can be given to deployment.In case of a successful invocation, REST API returns Invocation schema, with deployment_id params, which must be used for any further interactions with current deployment.

Status of deployment can be obtained with GET to /deployment/{deployment_id}/status. State of deployment can be one of [ pending, in_progress, success, failed ]. After invocation is done, user can inspect stdout, stderr, instance_state and outputs (if defined within service template).

Entire history of deployment (list of all invocations) can be obtained with GET to /deployment/{deployment_id}/history.

In case of deployment failure, deploy invocation can be continued (optionally with new inputs) with POST to /deployment/{deployment_id}/deploy_continue. Opera will continue, where previous deploy failed. Optionally, it can also start from beginning (clean_state=True).

Diff between current deployment state and new blueprint (with inputs) can be obtained by PUT to /deployment/{deployment_id}/diff. Blueprint (version) can be another version of the previously used blueprint or some version of another blueprint.

Deployment can be updated from new blueprint (version) with POST to /deployment/{deployment_id}/update. Opera will calculate the difference between deployed instance and new blueprint and (un)deploy it. Blueprint (version) can be another version of the previously used blueprint or some version of another blueprint.

Deployment can be undeployed with POST to /deployment/{deployment_id}/undeploy. This is the last invocation in deployment lifecycle, and antoher invocation will not be possible, but logs will be preserved.

xOpera REST API uses CSAR format as input format for uploading blueprints to REST API server.

Blueprint can be transformed to CSAR format with blueprint2CSAR script. Check help:

python3 scripts/blueprint2CSAR.py --help

blueprint2CSAR can also be used as python library to be included to your python application.

For details about CSAR format structure, visit TOSCA Simple Profile in YAML Version 1.3

When deploying the xOpera Rest API, intended to target Openstack, make sure to pass correct environment variables for openstack orchestration (section # XOPERA OPENSTACK DEPLOYMENT FALLBACK SETTINGS in xopera_env field in inputs-local.yaml or inputs-openstack.yaml)

To run docker image registry locally, run the following command:

docker run -d --restart=always --name registry -v /mnt/registry:/var/lib/registry registry:2

If you are using ubuntu, just run the following script:

sudo ./Installation/docker_certs.sh [computer_IP] [registry_IP] [path_to_CA_dir]

If you choose to preform steps manually, follow the steps below:

• Ubuntu: add root certificate ca.crt to /usr/share/ca-certificates/ folder, add the filename ca.crt to /etc/ca-certificates.conf and run sudo update-ca-certificates to update certificates.
• CentOS: add root certificate ca.crt to /etc/pki/ca-trust/source/anchors/ folder and run sudo update-ca-trust to update certificates.

Generate client cert and key, get it signed using docker registry's root CA certificate.

Now create a new directory in the docker certificate folder (/etc/docker/certs.d/) that has the same name as the IP/domain of your docker registry. Afterwards, copy the docker registry's ca.crt to newly created folder. You also need to copy client certificate and private key you generated and got signed.

The file structure should be as follows:

/etc/docker/certs.d/
└── my_registry_ip
   ├── client.cert
   ├── client.key
   └── ca.crt

See docker docs for more details.

Rest API is using PostgreSQL database. It is deployed with REST API as part of docker-compose template and TOSCA template. REST API can be configured to connect to any PostgreSQL instance by following environmental variables:

• XOPERA_DATABASE_IP=[database_ip]
• XOPERA_DATABASE_PORT=[database_port]
• XOPERA_DATABASE_NAME=[database_name]
• XOPERA_DATABASE_USER=[database_username]
• XOPERA_DATABASE_PASSWORD=[database_password]
• XOPERA_DATABASE_TIMEOUT=[database_timeout], optional

See example config.

PostgreSQL can be run as docker container.