History

SDK APIs

You need to understand what a pipe and stream step in that pipe is AND nothing comes for free. The cost of working with large amounts of data in near real-time environments with RStreams is you have to think about what you are doing and what it means with respect to reading and writing. It is strongly recommended you read the Read/Write at Scale article at some point.

A note on typescript types in pipe stream steps. Great effort was made so that pipe steps can infer types based on what’s been defined in previous pipe steps. Follow the examples closely to understand the minimum types necessary to keep unknown types from cropping up.

Person types referenced in the examples
 1export interface Person {
 2    gender: string;
 3    firstName: string;
 4    lastName: string;
 5    email: string;
 6    birthDate: string;
 7    nationality: string;
 8    addr: {
 9        addr1: string;
10        city: string;
11        state: string;
12        country: string;
13        countryCode?: string;
14        postcode: number;
15        longitude: string;
16        latitude: string;
17        tzOffset: string;
18        tzDesc: string;
19    }
20}
21
22export interface PersonRaw {
23    gender: string;
24    name: {
25        title: string;
26        first: string;
27        last: string;
28    }
29    location: {
30        street: {
31            number: number;
32            name: string;
33        }
34        city: string;
35        state: string;
36        country: string;
37        postcode: number;
38        coordinates: {
39            longitude: string;
40            latitude: string;
41        }
42        timezone: {
43            offset: string;
44            description: string;
45        }
46    }
47    email: string;
48    dob: {
49        date: string;
50        age: number;
51    }
52    nat: string;
53}
54
55export interface PersonRawResults {
56    results: PersonRaw[];
57}

Overview

The RStreams Node SDK includes a simple utility function to create pipes and nearly every kind of stream you will need to handle the massive amounts of continuously generated data in an instance of the RStreams bus. The RStreams Node SDK also includes functions that allow you to send and retrieve data to and from the RStreams Bus without dealing with pipes or streams.

Standalone Operations

RStreams Node SDK functions that allow you to send and receive events to and from the RStreams bus without the need to implement pipes and streams.

put Operation
A function that lets you write a single event to the specified RStreams queue
enrich Operation
A function that reads from the specified source RStreams queue, lets you transform the events and then sends the modified events to the specified destination RStreams queue
offload Operation
A function that reads from the specified RStreams queue and lets you do something with the events retrieved, perhaps save them in a DB

Source Stream Functions

These functions create a source stream for you, acting as the first stream in a pipe. Each source stream feeds a pipe with data that you specify, allowing it to flow through to the next stream step in your pipe.

Read Function
A function that creates a source stream that gets events from the specified queue and feeds them into the pipe.
Create Source Function
A function that creates a source stream that gets events from the specified queue and feeds them into the pipe.

Transform Stream Functions

RStreams Node SDK functions that create a transform stream for you, acting as a pipe step sitting between a source and sink. Each transform stream feeds accepts data from the previous pipe stream step, does something with it and then sends the resulting data on to the next pipe stream step.

Stringify Function
A function that creates a transform stream that takes in an upstream event, turns it into a string and tacks on a newline character to help in creating JSON lines files
Through Function
A function that creates a transform stream that takes in un upstream event and allows the developer to modiy/enrich/aggregate/reduce events and then send them on to the next stream step in the pipe
ToCSV Function
A function that creates a transform stream that helps build a csv file by taking each upstream event that comes in and formatting it as a line to put in a CSV file which it outputs to the next pipe stream step

Sink Stream Functions

RStreams Node SDK functions that create a sink for you, the last step in a pipe.

Load Function
A function that creates a sink that takes in an upstream event and pushes it to an RStreams queue on the bus
Devnull Function
A function that creates a sink stream that takes in un upstream event and does absolutely nothing with it, except log it if you ask it to