Summary

This guide explains RStreams Smart Configcopy, it’s typical use case, and various commands to get up and running.

Prerequisites

serverless framework is installed globally.
This assumes you are using Visual Studio Code as your IDE, however everything should be possible in your favorite IDE such as IntelliJ or whatever it may be
If you want to follow along, you should go through the Getting Started Guide

You will need the following specific dependencies for smart config to work:

serverless-leocopy >= 3.x
leo_sdkcopy >= 6.0.16

Things you need to know

Your AWS Account Credentials

You’ll need to have credentials setup to connect to your AWS account where the service will be deployed / hosted. In most instance, you’ll want to use a credential management tool such as kerb-sts, but you may also setup your .aws/credentialscopy file with static credentials to connect to AWS.

Running the smart config

There are three commands you can run from the command line to interact with the config file definition.

sls edit-configcopy. This will modify (or create) the definition file which is used to generate the typescript file you will be working with.

By default, the file will be called project-config.def.jsoncopy. If you prefer a different name, simply edit the serverless.yml file and specify the name you prefer (see below for details). Follow the command line prompts to add or remove configuration entries. You may also manually edit the configuration file if you know the AWS resource name. See below for information on the structure of the config entries.

It is a CLI application which will provide a menu to allow you to search, add, and remove entries from the config file. Don’t be concerned it it seems slow when first launching. It’s using your AWS credentials to query all the services in your account to which you have access so they can be used in the config.

sls generate-configcopy. This command will generate or update the typescript output file which you will use in your code. Although edit-configcopy will do this for you, it normally doesn’t add type information (everything starts as “undefined”), and if you manually change anything in the config file (such as specifying a datatype), you can run this to regenerate the typescript interface.

sls watch-configcopy. Running this keeps the definition and typescript files in sync automatically when either one is changed.

Adding typescript information

Once the config file is created, you’ll want to edit the definition file to add type information. You can either specify a simple or complex type inline, or you can create a types.d.tscopy file to hold more complex types. For example project-config.def.jsoncopy:

Typescript Types

{
    "rstreams": {
        "prodBusCronTableName": "secret::rstreams-${Stage}Bus.LeoCron::string",
    },
    "es_endpoints": "secret::es-endpoints-${stage}::EsEndpoints",
    "my_custom_type": "secret::foo::{key1:number,key2:string,key3:boolean}::"
}
copy

Notice how the prodBusCronTableNamecopy has an inline type of stringcopy, where es_endpointscopy has an external type of EsEndpointscopy, and my_custom_typecopy uses an inline complex type. types.d.tscopy:

Another thing you might have noticed is that the inline type for my_custom_typecopy ends with a :: (indicating there are no optional fields). This is sometimes necessary because of the way smartconfig is parsing the type under the covers. If the type ends with a }" then the regular expression will get confused and the parsing will fail. By adding the :: afterwards, the parser is satisfied and it’s still a valid smartconfig shortform (see below for the shortorm syntax). This will result in the following typescript file: project-config.tscopy:

Project Config

/* Generated by serverless-leo */
import { ConfigurationBuilder } from "leo-sdk/lib/configuration-builder";
import { EsEndpoints } from "types";

export interface ProjectConfig {
  rstreams: {
    prodBusCronTableName: string;
    prodChubBusCronTableName: string;
    prodStreamBusCronTableName: string;
    prodBusEventTableName: string;
    prodChubBusEventTableName: string;
    prodStreamBusEventTableName: string;
  };
  es_endpoints: EsEndpoints;
  my_custom_type: {key1:number,key2:string,key3:boolean};
}

export default new ConfigurationBuilder<ProjectConfig>(process.env.RSF_CONFIG ?? "").build({});
copy

Customization via the serverless config file

With the serverless file you can change the filename, specify where the config is stored when deployed, and add custom deployment targets.

Serverless Config

custom:
  leo:
    # when edit-config runs, it will tokenize the resources and do a replacement on the rsfConfigStages to find matching resources.
    # there are many default stages built-in: dev, test, stage, staging, production, prod
    # but what if you have your own stage named "stage1"? this will allow you to create that stage.
    rsfConfigStages:
      - stage1

    # specifies where the generated config will be stored
    # `environment` - include the generated config as an environment variable that you can see in your lambda config section
    # `secretsManager` - put the generated config in a secret, retrieve the secret at runtime and place in an environment variable
    rsfConfigType:
      environment # this is the default

    # informs how the secretsManager secret replicates (assuming rsfConfigType is set to `secretsManager`)
    # specified as from:to (where `to` is either a single value or a list)
    # there are a number of defaults already built-in, so in all likelihood you won't need to specify this
    rsfConfigReplicationRegion:
      us-east-1: us-west-1
      us-east-2:
        - us-west-1
        - us-west-2

    # the name of the project config file (by default: `project-config.def.json`)
    configurationPath: custom_config.json
copy

The structure of a config element

There are three forms for the config elements. By default the builder will create the short form, but hand-crafted long-form is also supported. The third form is hard-coded name/value pairs.

Short Form

${service}::${key}[::${type}[::${options}]]copy

servicecopy can be:

cf => exported cloud formation stack variables (from other projects)
secretcopy => secrets manager
ssmcopy => parameter store
stackcopy => the current project’s local cloud formation stack variables

keycopy is the name of the resource.

typecopy is either the inline or referenced typescript type.

optionscopy are additional options that might be useful in certain cases. Each option is separated by a semicolon in the short form, or is a key/value pair in the long form.

resolve = deploy|runtimecopy => (default=deploy). runtimecopy ONLY applies to secrets
optional = true|falsecopy => (default=false). adds a question mark ? to the typescript type

Long Form

This contains the same information as the short-form, but it’s just spelled out as an object with key/value pairs. A long form entry must have a servicecopy, keycopy, and typecopy entry, and it may have an optionscopy entry.

Here’s an entry in both short and long form:

Short and Long Form

{
    "es_endpoints_short": "secret::es-endpoints-${stage}::EsEndpoints::resolve=deploy;optional=false",
    "es_endpoints_long": {
        "service": "secret",
        "key": "es-endpoints-${stage}",
        "type": "EsEndpoints",
        "options": {
            "resolve": "deploy",
            "optional": false
        }
    }
}
copy

Standard and custom replacements

Within the config file, there are a few standard replacements you can specify:

${stage}copy => test, staging, prod, … (or anything you put into rsfConfigStagescopy in serverless.yml)
${Stage}copy => the same as stage, but “proper cased” (Test, Staging, Prod, …)
${region}copy => us-east-1copy, etc

If you have defined any custom stack parameters in your serverless.yml, you can use those as well. For example, if you added

You can then use it in your configuration entries:

How do i use it in my code?

Import the config object and use it.

import config from '../path/to/project-config'

const esOrderEndpoint = config.es_endpoints.orderEndpoint;
copy

That’s it if you’re running your project via serverless-leo or when deployed. Doing so will populate the RSF_CONFIG environment variable. However, if you’re running the project some other way and the RSF_CONFIG variable isn’t being set automatically, you may get a strange error about an invalid variable name. To get around this you will need to explicitly populate the RSF_CONFIG environment variable when your application starts up.

process.env.RSF_CONFIG = JSON.stringify(require('./path/to/project-config.def.json'));
copy

It also gets slightly more complex if you want to use any of the standard replacements, because running local doesn’t do the replacement (except for ${stage}copy). So, for example, if you use ${Stage}copy (proper cased), you’ll need to do this instead. Let’s assume you have a variable called stagecopy which contains the current deployment stage (’test’, ‘staging’, or ‘prod’, for example).

process.env.RSF_CONFIG = JSON
    .stringify(require('./path/to/project-config.def.json'))
    .replace(/\${Stage}/g, stage.charAt(0).toUpperCase()+stage.slice(1));
copy

Revisiting sls-edit

To use Smart Configcopy simply pass the edit-configcopy command to serverlesscopy.

  sls edit-config #serverless edit-config
copy

You’ll be prompted for the region you wish to search within your authenticated account (default is us-east-1copy).

Smart Configcopy is pulling and caching AWS resources behind the scenes locally in your project. You might notice a new file /.rsf/resource-cache.jsoncopy become available in your project. This cache is used to reduce making multiple calls to your AWS account looking for resources; you can safely add it to your .gitignorecopy.

search

By default entering a term into the prompt for edit-configcopy will search AWS for items similar to that term. In the example below, a search for the term my secretcopy in the Smart Configcopy prompt yields a response from the cache. Terms do not have to be exact matches.

Where can I pull data from? Currently, Smart Configcopy can pull from four locations to gather data:

add Command

Adding a resource to your Smart Configcopy is done by entering the numerical value of the term from your search into the prompt. In the example above, only one result was returned against the search term. To add that resource simple supply it’s value back to the prompt, as below. If successful, you sould see that the resource was added to your config.

You can add as many items during a session as you like. To continue adding more resources, you can continue going through the Search -> Addcopy flow, as shown below.

done Command

When you have added all the resources you want, simply provide donecopy to the Smart Configcopy prompt. You should see an output in the console that shows the readout of a file called project-config-new.def.jsoncopy. This file is used to generate the project-config-new.tscopy file.

Navigation

History

Smart Config