Summary

Learn how to install a new RStreams Bus instance.

AWS Cost

Be aware that following these instructions will result in AWS resources being created that cost between 50 and 100 dollars a month.

Of course, after creating the bus, you can dial down the capacity of the resources and get that down well below ten dollars a month. The more data you push into and read out of the bus, the more it costs. During testing and evaluation, all bus resources plus read/write capacity will easily remain in the 50 - 100 dollar a month range. You should pay close attention to cost as you ramp up use. The greatest expense is DynamoDB reads/writes which powers queues. The second greatest expense is DynamoDB storage. Kinesis and S3 costs are relatively minor by comparison.

The Stack of Stacks

As stated by AWS, “A stack is a collection of AWS resources that you manage as a single unit.” Installing RStreams involves creating a new stack that itself contains four additional stacks:

• The Bus Stack : the actual RStreams bus instance itself
• The Auth Stack : sets up authentication tables for managing access to bots/queues
• The Cognito Stack : used for access to RStreams resources
• The Botmon Stack : the monitoring, tracing and visual debugging website that accompanies an RStreams Bus instance

From AWS CloudFormation’s designer.

Create the RStreams Stack

Create a new stack using this RStreams Stack CloudFormation Template URL.

https://leo-cli-publishbucket-abb4i613j9y9.s3.amazonaws.com/leo/2.0.0/cloudformation-1652216325999.json

How do you access the new RStreams Bus instance?

If you are on a new version of the RStreams Bus, then creating the RStreams Stack will have published a new Secrets Manager secret that contains all the resources necessary for the RStreams Node SDK to connect to and use the new bus. It will be named like this: rstreams-{busStackName} where {busStackName} is the name you gave your RStreams bus instance plus -Bus-<random stuff>. Here’s how you can find the name of the bus stack.

If I have a bus installation named PlaygroundBus then there will be a stack named PlaygroundBus-Bus-<random-characters-AWS-puts-on>. Go to AWS CloudFormation and click on Stacks and then in the search field search on the name of your bus. In my case, I searched on the word playground and the five stacks showed up, the main stack and the five child stacks. I found the exact name of the bus stack which was PlaygroundBus-Bus-1JX7JSIIUQRAO.

Check SecretsManager for the new Secret

1. Go to AWS Secrets Manager
2. In the filter text area type “rstreams” and hit enter
3. If it’s there click on it and then click the Retrieve secret value button and you should see something like this:

{
  "LeoStream":         "PlaygroundBus-Bus-123456-LeoStream-123456",
  "LeoCron":           "PlaygroundBus-Bus-123456-LeoCron-123456",
  "LeoEvent":          "PlaygroundBus-Bus-123456-LeoEvent-123456",
  "LeoSettings":       "PlaygroundBus-Bus-123456-LeoSettings-123456",
  "LeoSystem":         "PlaygroundBus-Bus-123456-LeoSystem-123456",
  "LeoKinesisStream":  "PlaygroundBus-Bus-123456-LeoKinesisStream-123456",
  "LeoFirehoseStream": "PlaygroundBus-Bus-123456-LeoFirehoseStream-123456",
  "LeoS3":             "playgroundbus-bus-123456-leos3-123456",
  "Region":            "us-east-1"
}

Make your own RStreams Config Secret

If your secret isn’t there, feel free to create your own to make working with the SDK easier. The RStreams Flow deployment process will automatically look for the secret name specified by the developer for a given environment and associate the correct environment-specific RStreams config JSON structure with your bot when it deploys. Note that you’ll see environment called stage in RStreams Flow since that’s what the Serverless Framework calls it.

Get Your RStreams Config Values

The bus stack, one of the four stack, contains all the configuration as exported stack values. Go get them.

1. Go to CloudFormation
2. Filter by the name you selected for the RStreams bus stack, I chose PlaygroundBus
3. You should see five entries in the filtered results, one each for the five stacks that were created and here are mine
   1. PlaygroundBus : the main RStreams Bus stack that caused the other four stacks to be created
   2. PlaygroundBus-Botmon : the monitoring, tracing, debugging web app stack
   3. PlaygroundBus-Auth : the authentication stack
   4. PlaygroundBus-Bus : the actual RStreams bus instance stack itself which created kinesis, firehose, s3 and other resources
   5. PlaygroundBus-Cognito : the cognito stack
4. Select your {myBusName}-Bus stack that is the actual RStreams bus instance stack to see its detail
5. Select the Outputs tab along the top
6. You are now looking at all the values the SDK needs to connect to your bus instance

The SDK needs a structure that looks like this in order to connect to your RStreams bus instance where the keys in the JSON structure match the keys on the Outputs tab of the Bus stack we just landed on and the values come from the corresponding Value column.

 1{
 2  "LeoStream":         "abc",
 3  "LeoCron":           "abc",
 4  "LeoEvent":          "abc",
 5  "LeoSettings":       "abc",
 6  "LeoSystem":         "abc",
 7  "LeoKinesisStream":  "abc",
 8  "LeoFirehoseStream": "abc",
 9  "LeoS3":             "abc",
10  "Region":            "abc"
11}

Create the Secret

1. Go to secrets manager
2. Click Store a new secret
3. For Secret type select Other type of secret
4. For Key/value pairs select Plaintext and populate the text area with the SDK RStreams Config JSON Structure listed above
5. Follow the instructions in the previous section to populate the values in that JSON structure with the correct values for you bus instance
6. Name your secret exactly this: rstreams-{busName} where {busName} is the name of your bus (the bus I created was named PlaygroundBus so my secret should be named rstreams-PlaygroundBus)
7. Save it and continue on to use your bus below

Check Parameter Store for the new Parameter

1. Go to AWS Systems Manager -> Application Management -> Parameter Store
2. In the filter filter on the stack name you provided above when uploading the template to cloudformation.
3. Select your bus parameter, and you should notice a value similar to the name of the nested bus stack.
   This value is important to using (RStreamsFlow)["../../rstreams-flow/getting-started"] AND must match the name of the bus stack. This should also align with the name of the Secret created above.

Make your own Parameter Store entry

If you plan to use RStreams Flow and the parameter does not exist, you will need to create it. In AWS Systems Manager -> Application Management -> Parameter Store select Create Parameter. The name should be the name of the parent stack you provided when configuring CloudFormation. Type should always be set to String with a data type set to text. The value should be the name of your nested Bus stack.

What do you Have?

The next article explains in detail what was created but if you’re ready to give it a spin, head over to the RStreams Flow Getting Started Guide and Running Locally articles which will demonstrate using the RStreams Bus, creating RStreams queues on the bus instance and bots that interact with them.