Overview

TA functional description with some example on how to call REST APIs with the REST Subscriber Adapter in SAS ESP including calling the REST API of SAS Intelligent Decisioning.

The objective is to complement the official SAS documentation of the REST Subscriber Adapter.

.

Overview

To help system administrators navigate through the nuances of deploying and managing SAS Viya in the cloud, SAS has launched and maintains a set of Github projects called SAS Viya4 Deployment, SAS Viya4 Infrastructure as Code (IaC), and SAS Viya4 Orders CLI.

The tool was developed by the IoT Enablement Team with the objective to create a hub to administer deployments of separate Viya4 environments across different Cloud providers. Viya_Manager automates the entire deployment cycle by leveraging each of the three tools mentioned earlier.

Overview

Let’s say you would like to know if the price of a stock is above or below the current average price, and by how much. Because the data is streaming, you would like to use SAS Event Stream Processor (ESP) to accomplish this but are not sure how it can be done.

The ESP model would need to process the incoming stock prices, calculate the average, and then join the original price with its average. It would then calculate the difference between the current price and the average price. The model would output an event with the current price, average price, and the difference between the two. The model also needs to be fast and should perform at maximum efficiency.

This tutorial discuss how this can be done in ESP. This repository includes a completed model that you can run, and a starter model with just the Source window. The starter model allows you to construct the model yourself.

Overview

A SAS ESP data stream can be stored in a SQLite table in a couple different ways:

• Through the Database Adapter
• Using a combination of Python and the SAS Micro Analytic Service

This tutorial focuses on the latter solution.

Overview

A SAS ESP data stream can be stored in a CAS table in a couple different ways:

• Through the CAS Adapter (with Viya4, available only on secure environments)
• Programmatically, via the loadStream CAS actionset (with Viya4, available only on unsecure environments)

This tutorial illustrates an additional techinique based on Python and the SAS Micro Analytic Service.

Overview

This project will show how to collect live accelerometer and gyroscope data from a mobile device and pass it to Event Stream Processing (ESP) using an MQTT protocol. Once in ESP the JSON data is decoded and passed to the unsupervised machine learning models k-means and subspace tracking (SST). These unsupervised machine learning models are able to find patterns in data and are trained dynamically from streaming data.

In this use case we consider the mobile device as a sensor placed on any stationary piece of equipment on a factory floor. For demo purposes our anomaly detection time frame is 30 seconds. In a real world scenario, this time frame would be much longer. That is to say as long as the equipment continues to generate 30 seconds of consistent readings, this will be considered the baseline. Once that 30 second baseline is established, new observations are compared against it and any deviation will be considered an anomaly. KPIs determine the severity of the anomaly. With this type of self-training the baseline is dynamically set and does not have to be calculated separately for each piece of equipment. If the sensor is placed on a rotating object, the accelerometer data becomes the norm, as long as it is consistent.

Imagine a factory floor with 1000 machines, each oscillating at different amounts. Now imagine how much time would be saved if the sensors placed on these machines were able to self-train their own unique anomaly detection models. Very powerful.

Overview

You can integrate the streaming analytics available with SAS Event Stream Processing with Microsoft's Power Automate flows. This can be accomplished simply by using Microsoft Azure EventHubs to connect the two computing environments. SAS Event Stream Processing provides two ways to connect to EventHubs: the EventHubs connector and the Kafka connector. Each of these connectors can connect to Power Automate with Microsoft's EventHubs Connector.

This example uses a simple model that covers a common use case. The model monitors a sensor and sends events that exceed a specific threshold value to Microsoft Azure in order to generate a maintenance request. The model contains two windows, a Source window and a Filter window. Data streams into SAS Event Stream Processing through the Source Window and the Azure connector subscribes to output from the Filter window.

Overview

This is a very small streaming demo built to showcase pattern matching, temporal sliding windows and data aggregation for simple AML / fraud alerts on transactions in a streaming banking use case. There is no advanced analytics in here, it is all simple data comparison and analysis with rules, predominently in sliding, temporal windows.

If you want to take advantage of the SMTP notifications then use a third-party SMTP service, there are several free ones, I used (sendinblue.com and a personal gmail email account).

Overview

Open Neural Network Exchange (ONNX) is an open standard format to represent machine learning models. ONNX Runtime is an accelerator for machine learning models with support for multiple platforms and the flexibility to be integrated with a variety of frameworks. This enables developers to approach more complex use cases while maintaining efficient building and testing efforts.

This demo showcases the integration of ONNX Runtime with SAS Event Stream Processing on a Tiny YOLOv2 model, a real-time object detection network. This demo leverages an Intel CPU with (or without, as per user specifications) a CUDA TensorRT CPU on Intel hardware. It is based on a Docker configuration, so having Docker installed is required. The sample UI was developed specifically for this demo with a separate API, SAS ESP Connect.

Overview

In this project we will configure an Azure Logic application to query the Azure Maps Weather service once an hour. The logic app will then format the data returned from the weather API into the required JSON string and send it to an Azure Event Hub. Once in the Event Hub, it is available to other Azure resources and applications. We will create an ESP project that receives the events from the Event Hub using the ESP Azure Event Hub Connector.

Overview

In this project we will configure a simulated IoT Device to send sensor data to an IoT Hub, configure the IoT Hub to expose the message stream via an event hub endpoint, and then connect a SAS Event Stream Processing project to process the messages.

Overview

The IoT analytical life cycle expands traditional analysis processing beyond investigation of stored data (i.e., at rest) to analytical investigation of streaming data (i.e., in motion) or at the edge. The combined capability of data management, streaming analytics execution, and intelligent decisioning enables fast and confident decision making from data, to discovery, to deployment.

SAS Analytics for IoT offers an optimized IoT solution ecosystem and addresses the entire analytical life cycle.

In this tutorial we are focusing on the integrated process of model management and analytics deployment.

Overview

The Processing Streaming Trade Data model is an XML model included in the examples that are installed with SAS Event Stream Processing (ESP). It includes five ESP window types that perform various tasks on the stream.

1. Data Source Windows
There are two data source windows. The Trades window inputs the transactional records for each trade. The Traders window inputs the names of the traders which is joined with the stream. Neither window includes an input data connector. Therefore, a file/socket adapter command must be used to connect the files to the windows.

2. Filter Window
Filter windows use expressions, user-defined functions, and registered plug-in functions to set up a filter condition. The LargeTrades window uses the simple expression, quantity >= 100, to filter out small trades.

3. Join Window
Streaming joins take event streams from two windows and combine them into a single stream based on a key field in each stream. This model performs a one-to-many join because one trader name affects many trade events. Therefore, this is a left-outer join.

4. Compute Window
Compute windows take the input stream and create an output stream using computational manipulation. New output field values are created using expressions, user-defined functions, or plug-in functions. Fields from the input stream can be passed to the output stream without manipulation, as well. This model passes through all input fields and creates field TotalCost by using the simple expression price*quantity.

5. Aggregate Window
Aggregate windows place input events into groups based on one or more key fields. Output values of non-key fields are aggregated using available functions. In our model we use the sum function to output total quantity and cost by security code.

Overview

The Aggregating Stock Transactions model is a simple XML model included in the examples that are installed with SAS Event Stream Processing (ESP). It includes a Source window with an Input Data Connector and an Aggregate window to perform aggregate functions on the stream. 

The model reads stock transactions for a set of stock symbols. The input stream includes the following for each transaction:
     • stock symbol
     • number of shares
     • price of the stock

The model then aggregates the total number of shares traded for each symbol. This repository includes the files required to execute the example and video demonstrations that include the following topics:
     • viewing and editing the model using a text editor
     • viewing and editing the model in SAS ESP Studio (optional)
     • executing the model using the SAS ESP XML Server
     • subscribing to the output using the file/scoket adapter command and writing the results to a csv file
     • subscribing to the output using SAS ESP Streamviewer (optional)

Overview

The purpose of this tutorial is to provide a guideline on how to convert and score SAS Tiny YoloV2 computer vision models leveraging the Intel OpenVINO framework and ONNX format.

We will also highlight some of the functional difference between ONNX format and native SAS ASTORE formats to take into consideration.

OpenVINO is an inferencing (scoring) toolkit provided by Intel that aims to ensure quick deployment on Intel hardware of applications and solutions that emulate human vision by:
     • enabling Convolutional Neural Networks (CNN) based deep learning Inference (Scoring) on the edge
     • supporting heterogeneous execution across Intel CPU, Intel Integrated Graphics, Intel FPGA, Intel Movidius Neural Compute Stick, Intel Neural Compute Stick 2 and Intel Vision Accelerator Design with Intel Movidius VPUs

Additional information on the toolkit might be found on the OpenVINO toolkit page.

Overview

The purpose of this tutorial is to demonstate how to stream live weather data into any SAS Event Stream Processing (ESP) model. The GitHub repository also includes an examples directory that contains sets of configuration, model, and output files.

This example works with any weather service that returns a JSON message by way of an API call. A configuration file contains the a URL connector. It requests the data using an API call, transforms the data, and publishes events to the model.

After starting the model, the ESP XML Server uses the output data connector to create a csv file from the returned data.

Overview

The Event Retention and Calculating Throughput model is an XML model included in the examples that are installed with SAS Event Stream Processing (ESP). It includes a source, compute, three copy, and three aggregate windows that perform various tasks on the stream. 

The model is based on the following components:

1. Data Source Window
There is a single source window named source_win. The window inputs the transactional records using an input data connector.

2. Compute Window
Compute windows take the input stream and create an output stream using computational manipulation.  This model uses a user-defined function to calculate throughput rate. User-defined functions contain two parts. An initializer function executes when the model starts and the main portion of the function executes each time an event passes through the stream.

3. Copy Window 
There are three copy windows that retain events based on a retention technique. The retained events are then passed to the aggregate windows.

4. Aggregate Window
Aggregate windows place input events into groups based on one or more key fields. In this model there are three aggregate windows, each tied to one of the copy windows. The aggregate windows are identical. The difference is the input stream which is controlled by the retention values in the copy windows. Each aggregate window has an output data connector to create a csv file of the output.

Once setup is complete and data is streaming through the model, you can subscribe to the four output windows using SAS ESP Streamviewer. 

Overview

This is a fun one! The Zambretti algorithm is based on an instrument developed by Negretti and Zambra in the mid 1800s to forecast local weather. The output of the instrument or algorithm is one of 26 forecast statements, such as becoming unsettled. The forecast can be as much as 94% accurate in the Northern Hemisphere.

The forecast is determined based on four parameters:
    •  Value of the pressure at sea level
    •  Whether the pressure is falling, rising, or steady
    •  Pressure meets range requirements
    •  Wind direction

Follow the steps in this tutorial to set up and run the project based on input weather data.

Overview

Beam me up! Welcome to the Tracking the International Space Station, an Introduction to Geofences ESP example. In this example you will get an introduction to the Geofence window including:
    •  How to define areas of interest for a geofence
    •  How to use the geofence window to detect a position within a circle
    •  How to use position proximity analysis to detect a position’s proximity to a polygon
    •  How to create a GEO Map in SAS ESP Streamviewer

ISS background

The International Space Station (ISS) is a modular space station in low Earth orbit. It maintains an altitude of 251 miles and completes an orbit of the Earth in about 93 minutes. The ISS makes 15.54 orbits per day and is travelling at 7.66 km/s (27,600 km/h; 17,100 mph). The API used by this project publishes the latitude and longitude of the ISS every 10 seconds.

Overview

When building an ESP project to consume streams of data, we frequently encounter situations in which it makes sense to have certain events take one path and for other events to take other paths. Sometimes, we want to have different branches in an ESP project that will have their own logic, and we need to route events based on that. Another case would be when we are not achieving the performance we need, so we will want to parallelize the processing in order to increase throughput. In all these cases, we need to create a splitter to help us route the events. Let’s see how to split a stream of data in ESP. We’ll first go over a few simple cases, and then something a bit more intricate.

We will look at two basic splitters and three more complex splitters.

Overview

Prometheus is quickly becoming the de-facto monitoring tool for Kubernetes, in part because of the simplicity of its installation and configuration processes, but also because of the integration with Grafana, an open source visualization and analytics software that allows you to query, visualize, alert on, and explore metrics no matter where they are stored.

The documents found in this project provides a high-level introduction to Prometheus and Grafana, as well as a description of the steps required to set up a monitoring stack for a Kubernetes cluster that is running the SAS ESP operator. The goal is to make the whole deployment easy to understand and perform, while ultimately providing the ability to monitor resources with the goal of minimizing performance degradation.

Overview

When building an ESP project to consume streams of data, we frequently encounter situations in which it makes sense to have certain events take one path and for other events to take other paths. Sometimes, we want to have different branches in an ESP project that will have their own logic, and we need to route events based on that. Another case would be when we are not achieving the performance we need, so we will want to parallelize the processing in order to increase throughput.

In all these cases, we need to create a splitter to help us route the events. Let’s see how to split a stream of data in ESP. We’ll first go over a few simple cases, and then something a bit more intricate.

Overview

The database connector in SAS Event Stream Processing (ESP) can be utilized in two ways: with DataDirect Drivers or with SAS Threaded Kernel Drivers. This repository will explain how to utilize the Database Connector with DataDirect Drivers in ESP to read and write data to relational databases like Microsoft SQL Server and PostgreSQL on Azure.

Currently, the DataDirect ODBC drivers are only certified for the following databases: Oracle, MySQL, IBM DB2, Greenplum, PostgreSQL, SAP Sybase ASE, Teradata, Microsoft SQL Server, IBM Informix and Sybase IQ. In this tutorial we are connecting to databases on Azure, these steps could also be utilized to connect to an on-premise database.

Overview

SAS Event Stream Processing provides multiple ways to work with dates and times. Based on how the overall data stream is structured, ESP uses different internal languages and engines to handle the dates and times accordingly.

This document talks about various ways of processing dates and times in ESP with example models. We have also provided explanations to help you to navigate through some of the common scenarios one can face when working with such diverse data.

Overview

In this project, we will configure an Azure Logic application to query the Azure Maps Weather service once an hour. The logic app will then format the data returned from the weather API into the required JSON string and send it to an Azure Event Hub. Once in the Event Hub, it is available to any application running a Kafka listener. An ESP project will be created which will retrieve these events from the Event Hub using an ESP Kafka connector. Once in ESP a function window will be used to parse the incoming JSON into an ESP schema.

Overview

The Functional window in ESP is very useful and also not widely understood. There are too many things that can be done with the Functional window to cover in just one repository, so we will just focus on parsing simple JSON objects and processing more complex JSON objects.

Overview

In this tutorial, learn how to use the Transpose window in SAS ESP Studio and follow a few examples to illustrate functionality. Explore the source window configuration and view examples of both long-to-wide and wide-to-long transpose configurations.