Perform CAS Management, Jobs, and Compute server tasks

compute_and_jobs

2023.10

2024.03

2024.06

2024.07

2024.08

2024.09

The Job Execution API is used to execute and manage asynchronous jobs across environments. <details> <summary>Overview</summary> The Job Execution API acts as a gateway to execute jobs. Jobs are created to be executed by this API. The API then decides which execution provider can best execute the job based on the type of job. To execute a job, the client creates a JobRequest. When a job request is submitted, execution starts and the URI of the newly created job is returned. From this job, the user can query for job completion, status, and output. The provider executes the job and reports the job's status to the Job Execution API. The lifecycle of this job is managed by the Job Execution API. The consumer uses this API to query for the status of this job as well as to delete the job. A job is run on a suitable job execution provider that is determined by the type of the job. </details> <details> <summary>Events</summary> Events are generated to indicate changes in the execution state of jobs created by the Job Execution API. Typical consumers for such events are other APIs or applications interested in changes in job state (for example, the completion of an executing job). Because both the submitter and creator are part of the event-routing key, an API (such as Workflow) has the option of listening to job events from only the jobs that it has submitted. Alternatively, an application component (such as Job Monitor) can listen to all submitters. Events are written to the sas.job exchange. The AMQP routing key for all job event payloads is: `job.[type].[jobId].[submitter].[creator].[stateValue]` For example: job.state.d37ba178-a115-4162-8289-7132491e1306.workflow.jobExecution.completed
</details> <details> <summary>State Details</summary> An additional member called `stateDetails` of type string has been added to the Job representation object to supplement the `state` member (also a string) when needed. `stateDetails` supplements by providing an indication that additional information is available. It is an indicator and not the actual information. The purpose of the `stateDetails` member is for providers to provide a hint, when applicable, about the state set and for that hint to be used by components that convey state information to clients. The Job Execution API itself does not use or validate the hint values. Because the hint values are not validated, hint values can be added (as the need arises) without having to publish a new version of Job Execution Service (JES), the rep JAR or any of the other JES libraries. Only the `completed` state supports a value for `stateDetails`. The following set of hints are applicable to the `completed` state: <ul> <li>info</li> <li>warning</li> <li>error</li> <li>fatal</li> <li>diagnostic</li> </ul> The `info`, `warning`, `error`, and `fatal` values are intended to correspond to different severity levels of messages that are displayed in the log. There is an implied severity hierarchy, with `info` being the lowest and `fatal` being the highest. If the provider messages use a similar severity model, and you do not have to scan the log to determine the level of the most severe message, then set that level in the field. The `diagnostic` value is meant to be used when the above model, for whatever reason, cannot be used and there is something noteworthy to report. Information is conveyed either via the log or in the `error` member. Existing providers setting the `stateDetails` member for a completed job should set the same state as before (`completed`) in the `state` member, thereby maintaining backward compatibility for tools that are not updated to handle the new `stateDetails` member.
</details> <details> <summary>Security</summary> <ul> <li>AuthorizationAny authenticated user can `GET` or `HEAD` the root of the API and can create a Job. All other permissions (read, update, delete, add, and remove for the Job object) are granted only to the user who created the job. Other SAS APIs and applications are authorized to load job requests via the bootstrap mechanism. The Authorization API automatically protects any endpoints of any API that includes dependencies on it. This dependency intercepts calls to the endpoints and builds an authorization context, which is then sent to the Authorization API for evaluation. <ul> <li>If the context is granted permission to perform the request action on that endpoint, it returns an HTTP status code of 200 and allows the request to proceed.</li> <li>If the context is prohibited, it returns an HTTP status code of 403 and the request does not continue.</li> </ul> This access is controlled by authorization rules that can be created through bootstrapping, by an administrative user, or by other methods. </li> <li>Authentication The endpoints that result in a job being executed are invoked with an access token. This can be an access token obtained on behalf of a user or an access token obtained from client credentials. If invoked with an access token on behalf of a user, the Job Execution API attempts to obtain a new access token on behalf of the user and to use this new access token in all requests made by JES, including requests to the provider to execute the job. If JES is not able to obtain a new access token on behalf of the user, the original access token continues to be used. JES-obtained access tokens have the same lifetime as those obtained at the application level, except that the clock starts when JES obtains the token and not when the user signs in. If a job invoked with an access token on behalf of a user times out because the default lifetime is too small, the Job might fail. The lifetime of the JES-obtained tokens can be increased by setting the Consul property config/jobExecution/oauth2.client.accessTokenValidity to an appropriate value, which represents the lifetime expressed in seconds. For example, setting it to 86400 results in the tokens obtained by JES lasting for 86400 seconds (or 24 hours). Job Execution should be restarted after the consul property is set. After the Job completes, JES does not revoke tokens that it created. Instead, tokens expire at the end of their lifetime. If Job Execution is not able to obtain an access token on behalf of a user, it propagates the token from the request made to execute the job. </li></ul> </details> <details> <summary>Terminology</summary> <ul> <li>job definition: a batch execution. Job definitions are managed by the Job Definitions API. </li> <li>job request: information about the job definition to be executed and the list of run-time arguments used for this particular request. The job definition can be either included in the job request or referenced. </li> <li>job: a container for an asynchronous operation. A job is created when a job request is submitted. It represents an instance of a job execution. </li> <li>job execution provider: an execution environment, usually an independent API, that is called by the job execution API to execute a job request. The provider takes the created job instance and executes it in an environment that is appropriate for the job type. </li> </details> <details> <summary>Error Codes</summary> <table> <tr> <th>HTTP Status Code</th> <th>Error Code</th> <th>Description</th> </tr> <tr> <td>400</td> <td>31502</td> <td>The job request members cannot be updated on the job (application/vnd.sas.job.execution.job)</td> </tr> <tr> <td>400</td> <td>31504</td> <td>The user-specified timestamp in the request is not a valid timestamp of the format <code>yyyy-MM-ddTHH:mm:ss.SSSZ</code>.</td> </tr> <tr> <td>400</td> <td>31505</td> <td>The state was not specified in the value-query parameter or the body of the request.</td> </tr> <tr> <td>400</td> <td>31506</td> <td>The specified state is not a valid value.</td> </tr> <tr> <td>400</td> <td>31507</td> <td>Could not find a provider object in the JES provider cache for the job-definition type in the request submitted for execution.</td> </tr> <tr> <td>400</td> <td>31508</td> <td>Attempting to access the provider service to execute the submitted request returned an HTTP status of 503 (Service Unavailable).</td> </tr> <tr> <td>405</td> <td>31501</td> <td>The job already completed and is now read only.</td> </tr> <tr> <td>409</td> <td>31503</td> <td>The job has not yet completed. The expiration timestamp cannot be set.</td> </tr> </table> Jobs that do not complete successfully might have one of the following error codes in the Job object: <table> <tr> <th>Error Code</th> <th>Description</th> </tr> <tr> <td>31500</td> <td>The job timed out after failing to receive a heartbeat.</td> </tr> <tr> <td>31508</td> <td>Attempting to access the provider service to execute the submitted request returned an HTTP status of 503 (Service Unavailable).</td> </tr> <tr> <td>31509</td> <td>Execution in the provider service resulted in an error. Its text, if available, is included in the error object.</td> </tr> <tr> <td>31510</td> <td>Set by the provider library in the Job object if an exception in the provider is not caught by the provider but gets caught by the library.</td> </tr> </table> </details> </details>

Job Execution

Viya_35

2022.09

2023.03

The Job Execution API is used to execute and manage asynchronous jobs across environments. <details> <summary>Overview</summary> The Job Execution API acts as a gateway to execute jobs. Jobs are created to be executed by this API. The API then decides which execution provider can best execute the job based on the type of job. To execute a job, the client creates a JobRequest. When a job request is submitted, execution is kicked off and the URI of the newly created job is returned. From this job, the user can query for job completion, status, and outputs. The provider executes the job, reporting status back to the Job Execution API for the given job. The lifecycle of this job is managed by the Job Execution API. The consumer would use this API to query for the status of this job as well as to delete the job. A job is run on a suitable job execution provider, determined by the type of the job. </details> <details> <summary>Events</summary> Events are generated to indicate changes in execution state of jobs created by the Job Execution API. Typical consumers for such events are other APIs or applications interested in changes in job state (e.g., the completion of an executing job). Since both the submitter and creator are part of the event routing key an API (e.g. Workflow) has the option of listening to job events from just those jobs that it has submitted; alternatively, an application component (e.g. Job Monitor) can listen to all submitters. Events are written to the sas.job exchange. The AMQP routing key for all job event payloads is: `job.[type].[jobId].[submitter].[creator].[stateValue]` e.g. job.state.d37ba178-a115-4162-8289-7132491e1306.workflow.jobExecution.completed
</details> <details> <summary>State Details</summary> An additional member called `stateDetails` of type string has been added to the Job representation object to "supplement" the `state` (also a string) member when needed. It supplements by providing an indication that additional information is available. It is an indicator and not the actual information. The purpose of this member is for providers to provide a hint, when applicable, about the state set and for that hint to be used by components that convey state information to clients. Job Execution API itself will not use or validate the hint values in any way. Not validating ensues that, as the need arises, hint values can be added without the need to publish a new version of JES, the rep JAR or any of the other JES libraries. Only the `completed` state supports a value for `stateDetails`. The following set of hints are applicable to the `completed` state: <ul> <li>info</li> <li>warning</li> <li>error</li> <li>fatal</li> <li>diagnostic</li> </ul> The `info`, `warning`, `error`, and `fatal` values are intended to correspond to different severity levels of messages that are displayed in the log. There is an implied severity hierarchy, with `info` being the lowest and `fatal` being the highest. If the provider messages use a similar severity model, and you do not have to scan the log to determine the level of the most "severe" message, then set that level in the field. The `diagnostic` value is meant to be used when the above model, for whatever reason, cannot be used and there is something noteworthy to report. Information is conveyed either via the log or in the error member. Existing providers setting the `stateDetails` member for a completed job should set the same state as before (`completed`) in the `state` member, thereby maintaining backward compatibility for tools that are not updated to handle the new `stateDetails` member.
</details> <details> <summary>Security</summary> <ul> <li>AuthorizationAny authenticated user can GET or HEAD the root of the API and can create a Job. All other permissions - read, update, delete, add, and remove for the Job object - are granted only to the user who created the job. Other SAS APIs and applications are authorized to load job requests via the bootstrap mechanism. The Authorization API automatically protects any endpoints of any API that includes dependencies on it. This dependency intercepts calls to the endpoints and builds an authorization context, which is then sent to the Authorization API for evaluation. If the context is granted permission to perform the request action on that endpoint, it returns an HTTP status code of 200 and allows the request to proceed. If the context is prohibited, it returns an HTTP status code of 403 and the request does not continue further. This access is controlled by authorization rules that can be created through bootstrapping, by an administrative user, or by other methods. </li> <li>Authentication The endpoints that result in a job being executed are invoked with an access token and this may be an access token obtained on behalf of a user or an access token obtained from client credentials. If invoked with an access token on behalf of a user, the Job Execution API attempts to obtain a new access token on behalf of the user and use this new access token in all requests made by JES, including that to the provider to execute the job. If JES is not able to obtain a new access token on behalf of the user the original access token continues to be used. JES-obtained access tokens have the same lifetime as those obtained at the application level, except that the clock starts when JES obtains the token and not when the user signs in. If a job invoked with an access token on behalf of a user times out because the default lifetime is too small, the Job may fail. The lifetime of the JES-obtained tokens can be increased by setting the Consul property config/jobExecution/oauth2.client.accessTokenValidity to an appropriate value, which represents the lifetime expressed in seconds. For example, setting it to 86400 results in the tokens obtained by JES lasting for 86400 seconds or 24 hours. Job Execution should be restarted after the consul property is set. JES does not revoke tokens that it creates once the Job completes - they expire at the end of their lifetime. If Job Execution is not able to obtain an access token on behalf of a user, it propagates the token from the request made to execute the job. </li></ul> </details> <details> <summary>Terminology</summary> <ul> <li>job definition: a batch execution. Job definitions are managed by the Job Definitions API. </li> <li>job request: information about the job definition to be executed and the list of run-time arguments used for this particular request. The job definition can either be included in the job request or referred to </li> <li>job a container for an asynchronous operation. A job is created when a job request is made. It represents an instance of a job execution. </li> <li>job execution provider: an execution environment, usually an independent API, that is called by the job execution API to execute a job request. It takes the created job instance and executes it in an environment that is appropriate for the job type. </li> </details> <details> <summary>Error Codes</summary> <table> <tr> <th>HTTP Status Code</th> <th>Error Code</th> <th>Description</th> </tr> <tr> <td>400</td> <td>31502</td> <td>The job request members cannot be updated on the job (application/vnd.sas.job.execution.job)</td> </tr> <tr> <td>400</td> <td>31504</td> <td>The user-specified time stamp in the request is not a valid time stamp of the format <code>yyyy-MM-ddTHH:mm:ss.SSSZ</code>.</td> </tr> <tr> <td>400</td> <td>31505</td> <td>The state was not specified in the value query parameter or the body of the request.</td> </tr> <tr> <td>400</td> <td>31506</td> <td>The specified state is not a valid value.</td> </tr> <tr> <td>400</td> <td>31507</td> <td>Could not find a provider object in the JES provider cache for the job definition type in the request submitted for execution.</td> </tr> <tr> <td>400</td> <td>31508</td> <td>Attempting to access the provider service to execute the submitted request returned an http status of 503 (Service Unavailable).</td> </tr> <tr> <td>405</td> <td>31501</td> <td>The job already completed and is now read-only.</td> </tr> <tr> <td>409</td> <td>31503</td> <td>The job has not yet completed, so the expiration time stamp cannot be set.</td> </tr> </table> Jobs that do not complete successfully might have one of the following error codes in the Job object: <table> <tr> <th>Error Code</th> <th>Description</th> </tr> <tr> <td>31500</td> <td>The job timed out after not receiving a heartbeat.</td> </tr> <tr> <td>31508</td> <td>Attempting to access the provider service to execute the submitted request returned an http status of 503 (Service Unavailable).</td> </tr> <tr> <td>31509</td> <td>Execution in the provider service resulted in an error whose text, if available, is included in the error object.</td> </tr> <tr> <td>31510</td> <td>Set by the provider library in the Job object if an exception in the provider is not caught by the provider but gets caught by the library.</td> </tr> </table> </details> </details>

Returns the header information for the heartbeat timestamp job.

jobId

The `heartbeatTimeStamp` has not been set yet.

Check whether the heartbeat timestamp exists

Job Definition Parameter Schema

Job Definition Schema

Job Request Schema

Execution Job Schema

A page-able collection of job-request resources.

Job Request Collection

Job Collection

Resource Property

Provider Summary

Provider

A page-able collection of provider resources.

Provider Summary Collection

A collection of job or error objects. When submitting multiple jobs, the items in the response MUST match 1-to-1 with the items in the request. The corresponding item is either a job object or an error object.

Collection of job objects and possibly error objects.

Job or error object

A collection of resource summary or error objects. When updating the state of multiple jobs, the items in the response MUST match 1-to-1 with the items in the request. The corresponding item is either a resource-summary object or an error object.

Collection of resource summary and possibly error objects.

A resource-summary object or error object.

Resource summary or error object

Resource Summary Schema

This is a base schema used to define paginated collections of resources. This base schema is extended by other schemas in APIs by adding an 'items' array property. These extensions define the `application/vnd.sas.collection` media type (version 2).

Base Collection

A link to a related operation or resource.

Link

Error

The representation of a selection of zero or more resources.

Resource Selection

The list of links to top-level resources and operations available from the root of the API.

Status	Meaning	Description
200	OK	The request succeeded.
204	No Content	The `heartbeatTimeStamp` has not been set yet.
404	Not Found	No job exists at the requested path.

Check whether the heartbeat timestamp exists

Request Samples

Response Samples

Path Parameters

Responses