Summary

This guide explains RStreams Smart Config, it’s typical use case, and various commands to get up and running.

Prerequisites

serverless framework is installed globally.
This assumes you are using Visual Studio Code as your IDE, however everything should be possible in your favorite IDE such as IntelliJ or whatever it may be
If you want to follow along, you should go through the Getting Started Guide

You will need the following specific dependencies for smart config to work:

serverless-leo >= 3.x
leo_sdk >= 6.0.16

Things you need to know

Your AWS Account Credentials

You’ll need to have credentials setup to connect to your AWS account where the service will be deployed / hosted. In most instance, you’ll want to use a credential management tool such as kerb-sts, but you may also setup your .aws/credentials file with static credentials to connect to AWS.

Running the smart config

There are three commands you can run from the command line to interact with the config file definition.

sls edit-config. This will modify (or create) the definition file which is used to generate the typescript file you will be working with.

By default, the file will be called project-config.def.json. If you prefer a different name, simply edit the serverless.yml file and specify the name you prefer (see below for details). Follow the command line prompts to add or remove configuration entries. You may also manually edit the configuration file if you know the AWS resource name. See below for information on the structure of the config entries.

It is a CLI application which will provide a menu to allow you to search, add, and remove entries from the config file. Don’t be concerned it it seems slow when first launching. It’s using your AWS credentials to query all the services in your account to which you have access so they can be used in the config.

sls generate-config. This command will generate or update the typescript output file which you will use in your code. Although edit-config will do this for you, it normally doesn’t add type information (everything starts as “undefined”), and if you manually change anything in the config file (such as specifying a datatype), you can run this to regenerate the typescript interface.

sls watch-config. Running this keeps the definition and typescript files in sync automatically when either one is changed.

Adding typescript information

Once the config file is created, you’ll want to edit the definition file to add type information. You can either specify a simple or complex type inline, or you can create a types.d.ts file to hold more complex types. For example project-config.def.json:

Typescript Types

{
    "rstreams": {
        "prodBusCronTableName": "secret::rstreams-${Stage}Bus.LeoCron::string",
    },
    "es_endpoints": "secret::es-endpoints-${stage}::EsEndpoints",
    "my_custom_type": "secret::foo::{key1:number,key2:string,key3:boolean}::"
}

Notice how the prodBusCronTableName has an inline type of string, where es_endpoints has an external type of EsEndpoints, and my_custom_type uses an inline complex type. types.d.ts:

Another thing you might have noticed is that the inline type for my_custom_type ends with a :: (indicating there are no optional fields). This is sometimes necessary because of the way smartconfig is parsing the type under the covers. If the type ends with a }" then the regular expression will get confused and the parsing will fail. By adding the :: afterwards, the parser is satisfied and it’s still a valid smartconfig shortform (see below for the shortorm syntax). This will result in the following typescript file: project-config.ts:

Project Config

/* Generated by serverless-leo */
import { ConfigurationBuilder } from "leo-sdk/lib/configuration-builder";
import { EsEndpoints } from "types";

export interface ProjectConfig {
  rstreams: {
    prodBusCronTableName: string;
    prodChubBusCronTableName: string;
    prodStreamBusCronTableName: string;
    prodBusEventTableName: string;
    prodChubBusEventTableName: string;
    prodStreamBusEventTableName: string;
  };
  es_endpoints: EsEndpoints;
  my_custom_type: {key1:number,key2:string,key3:boolean};
}

export default new ConfigurationBuilder<ProjectConfig>(process.env.RSF_CONFIG ?? "").build({});

Customization via the serverless config file

With the serverless file you can change the filename, specify where the config is stored when deployed, and add custom deployment targets.

Serverless Config

custom:
  leo:
    # when edit-config runs, it will tokenize the resources and do a replacement on the rsfConfigStages to find matching resources.
    # there are many default stages built-in: dev, test, stage, staging, production, prod
    # but what if you have your own stage named "stage1"? this will allow you to create that stage.
    rsfConfigStages:
      - stage1

    # specifies where the generated config will be stored
    # `environment` - include the generated config as an environment variable that you can see in your lambda config section
    # `secretsManager` - put the generated config in a secret, retrieve the secret at runtime and place in an environment variable
    rsfConfigType:
      environment # this is the default

    # informs how the secretsManager secret replicates (assuming rsfConfigType is set to `secretsManager`)
    # specified as from:to (where `to` is either a single value or a list)
    # there are a number of defaults already built-in, so in all likelihood you won't need to specify this
    rsfConfigReplicationRegion:
      us-east-1: us-west-1
      us-east-2:
        - us-west-1
        - us-west-2

    # the name of the project config file (by default: `project-config.def.json`)
    configurationPath: custom_config.json

The structure of a config element

There are three forms for the config elements. By default the builder will create the short form, but hand-crafted long-form is also supported. The third form is hard-coded name/value pairs.

Short Form

${service}::${key}[::${type}[::${options}]]

service can be:

cf => exported cloud formation stack variables (from other projects)
secret => secrets manager
ssm => parameter store
stack => the current project’s local cloud formation stack variables

key is the name of the resource.

type is either the inline or referenced typescript type.

options are additional options that might be useful in certain cases. Each option is separated by a semicolon in the short form, or is a key/value pair in the long form.

resolve = deploy|runtime => (default=deploy). runtime ONLY applies to secrets
optional = true|false => (default=false). adds a question mark ? to the typescript type

Long Form

This contains the same information as the short-form, but it’s just spelled out as an object with key/value pairs. A long form entry must have a service, key, and type entry, and it may have an options entry.

Here’s an entry in both short and long form:

Short and Long Form

{
    "es_endpoints_short": "secret::es-endpoints-${stage}::EsEndpoints::resolve=deploy;optional=false",
    "es_endpoints_long": {
        "service": "secret",
        "key": "es-endpoints-${stage}",
        "type": "EsEndpoints",
        "options": {
            "resolve": "deploy",
            "optional": false
        }
    }
}

Standard and custom replacements

Within the config file, there are a few standard replacements you can specify:

${stage} => test, staging, prod, … (or anything you put into rsfConfigStages in serverless.yml)
${Stage} => the same as stage, but “proper cased” (Test, Staging, Prod, …)
${region} => us-east-1, etc

If you have defined any custom stack parameters in your serverless.yml, you can use those as well. For example, if you added

You can then use it in your configuration entries:

How do i use it in my code?

Import the config object and use it.

import config from '../path/to/project-config'

const esOrderEndpoint = config.es_endpoints.orderEndpoint;

That’s it if you’re running your project via serverless-leo or when deployed. Doing so will populate the RSF_CONFIG environment variable. However, if you’re running the project some other way and the RSF_CONFIG variable isn’t being set automatically, you may get a strange error about an invalid variable name. To get around this you will need to explicitly populate the RSF_CONFIG environment variable when your application starts up.

process.env.RSF_CONFIG = JSON.stringify(require('./path/to/project-config.def.json'));

It also gets slightly more complex if you want to use any of the standard replacements, because running local doesn’t do the replacement (except for ${stage}). So, for example, if you use ${Stage} (proper cased), you’ll need to do this instead. Let’s assume you have a variable called stage which contains the current deployment stage (’test’, ‘staging’, or ‘prod’, for example).

process.env.RSF_CONFIG = JSON
    .stringify(require('./path/to/project-config.def.json'))
    .replace(/\${Stage}/g, stage.charAt(0).toUpperCase()+stage.slice(1));

Revisiting sls-edit

To use Smart Config simply pass the edit-config command to serverless.

  sls edit-config #serverless edit-config

You’ll be prompted for the region you wish to search within your authenticated account (default is us-east-1).

Smart Config is pulling and caching AWS resources behind the scenes locally in your project. You might notice a new file /.rsf/resource-cache.json become available in your project. This cache is used to reduce making multiple calls to your AWS account looking for resources; you can safely add it to your .gitignore.

search

By default entering a term into the prompt for edit-config will search AWS for items similar to that term. In the example below, a search for the term my secret in the Smart Config prompt yields a response from the cache. Terms do not have to be exact matches.

Where can I pull data from? Currently, Smart Config can pull from four locations to gather data:

add Command

Adding a resource to your Smart Config is done by entering the numerical value of the term from your search into the prompt. In the example above, only one result was returned against the search term. To add that resource simple supply it’s value back to the prompt, as below. If successful, you sould see that the resource was added to your config.

You can add as many items during a session as you like. To continue adding more resources, you can continue going through the Search -> Add flow, as shown below.

done Command

When you have added all the resources you want, simply provide done to the Smart Config prompt. You should see an output in the console that shows the readout of a file called project-config-new.def.json. This file is used to generate the project-config-new.ts file.

Smart Config Done

Navigation

History

Smart Config