A Useful Framework for Interpreting Success Stories

Recently, I had the pleasure of reading Work Is Work, an essay by Coda Hale on organizational design. Aside from providing a thought-provoking perspective on scaling organizational efforts, the post makes reference to two terms from anthropological field research that were new to me: emic and etic. Below, I’ll describe how these terms provide a useful framework for interpreting success stories.

Emic and Etic

When we read success stories, we often do so to help narrow down the solution space for a problem we’re facing. During that process, it can sometimes be easy to lose track of how important details of the story (its plot, setting, actors, etc.) are different from ours.

Emic and etic help describe behaviors or beliefs from the actor’s perspective (emic) vs. behavior or beliefs observed by an outsider (etic). Continuing with the success story example, writing about how I had great success with a new JavaScript framework is an emic account. You reading my story as research for selecting a JavaScript framework to use for your project is an etic account.

This framework has been valuable to me in two ways. It:

1. Helps heighten my awareness; prompting an additional level of scrutiny toward the solutions I consider (e.g., you had success, but the project you used the JavaScript framework on was small and mine is large).
2. Provides shorthand terms for what are otherwise relatively difficult concepts to communicate.

Scheduling Lambda Functions with AWS SAM

A few days ago, I spent some time learning how to use Amazon’s Serverless Application Model (SAM) to schedule the recurring execution of Lambda functions. To help better cement my understanding, I assembled an overview of all the SAM template components necessary to schedule the periodic execution of a Go-based Lambda function. I also made note of how I used the SAM CLI to package and deploy everything to AWS.

Serverless Application Model

Amazon’s Serverless Application Model is a specification for translating SAM templates into CloudFormation templates. Much like macro expansion, it works through a textual transformation of the input SAM template into a template the CloudFormation engine can make sense of.

There are several components that make up a SAM template, but in this example we only use four: Format Version, Description, Transform, and Resources.

• Format Version equates to AWSTemplateFormatVersion in the template, which identifies its capabilities
• Description is optional, but provides a way to give the template a high-level description
• Transform can map to multiple things, but here it maps to the AWS::Serverless-2016-10-31 transform, which is a version of the SAM specification

As far as Resources go, this template defines two: TestFunction and TestRole.

AWSTemplateFormatVersion: '2010-09-09'
Description: A scheduled Amazon Lambda function.
Resources:
  TestFunction:
    Properties:
      CodeUri: .
      Events:
        Testy:
        Properties:
          Schedule: rate(1 hour)
        Type: Schedule
      Handler: main
      Role: !GetAtt TestRole.Arn
      Runtime: go1.x
    Type: AWS::Serverless::Function
  TestRole:
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Action:
          - sts:AssumeRole
        Effect: Allow
        Principal:
          Service:
          - lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Type: AWS::IAM::Role
Transform: AWS::Serverless-2016-10-31

TestRole is a resource of type AWS::IAM::Role, which is a top-level CloudFormation resource. It creates an Identity and Access Management (IAM) role containing the permissions necessary for our Lambda function to do its thing. In this case, it simply encapsulates a canned IAM policy, AWSLambdaBasicExecutionRole. This policy allows the Lambda function to use the following CloudWatch API calls to log function output to CloudWatch Logs.

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents

The next resource, TestFunction, is of type AWS::Serverless::Function. This is not a top-level CloudFormation resource. Instead, it is a SAM resource that expands into multiple top-level CloudFormation resources. Based on our usage, it expands into three:

• AWS::Lambda::Function
• AWS::Lambda::Permission
• AWS::Events::Rule

AWS::Lambda::Function is the top-level CloudFormation resource to define an Amazon Lambda function. Because we want to schedule the function’s periodic execution, we include an Events property on our AWS::Serverless::Function resource. This allows us to define the function execution schedule within the context of the function’s properties. Behind-the-scenes, the Events property expands into a AWS::Events::Rule resource with an invocation rate of once per hour.

Lastly, in order for the CloudWatch Events API to invoke our function, it needs permissions to do so. AWS::Lambda::Permission grants CloudWatch Events the permission to invoke our function.

Package and ship

The AWS SAM CLI builds on top of the SAM specification by providing a single tool to manage the packaging and deployment of serverless applications. Installation is a bit out-of-scope for this post, but once you’ve managed to install the sam tool, the application deployment process occurs in three phases.

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func HandleRequest(ctx context.Context, e events.CloudWatchEvent) (string, error) {
    return fmt.Sprintf("Hello, world."), nil
}

func main() {
    lambda.Start(HandleRequest)
}

First, compile your Go-based Lambda function into a Linux compatible binary.

GOOS=linux go build -o main main.go

Once the binary exists, use sam to upload the binary to S3 and reference it in a newly created packaged.yaml CloudFormation configuration.

$ sam package --s3-bucket test-global-config-us-east-1 \
              --template-file template.yaml \
              --output-template-file packaged.yaml
Uploading to 7001c68762c2fcda61de373e0a30563d  29187040 / 29187040.0  (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.

Before using sam to deploy using the contents of packaged.yaml, run a quick diff to see what changed.

$ diff template.yaml packaged.yaml
<       CodeUri: .
---
>       CodeUri: s3://test-global-config-us-east-1/7001c68762c2fcda61de373e0a30563d

Lastly, use sam again to deploy the template through a CloudFormation stack named Test.

$ sam deploy --template-file packaged.yaml \
             --stack-name Test \
             --capabilities CAPABILITY_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - Test

Within an hour or so (it only takes a few minutes to deploy—the wait is for the function schedule to trigger), you should see something like the following in your function’s CloudWatch Logs log stream.

START RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1 Version: $LATEST
"Hello, world."
END RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
REPORT RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
Duration: 1.59 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 5 MB

Haskell Code Katas: Counting Duplicates

For the past few weeks, I’ve been starting off my days with Haskell flavored code katas from Codewars. Today I started with the kata below and figured it would be a good exercise to walk through my solution.

Write a function that will return the count of distinct case-insensitive alphabetic characters and numeric digits that occur more than once in the input string. The input string can be assumed to contain only alphabets (both uppercase and lowercase) and numeric digits.

To help clarify the specifications for this kata, the Hspec test suite is below:

module Codwars.Kata.Duplicates.Test where

import Codwars.Kata.Duplicates (duplicateCount)
import Data.List (nub)
import Test.Hspec
import Test.QuickCheck

main = hspec $ do
  describe "duplicateCount" $ do
    it "should work for some small tests" $ do
      duplicateCount ""                         =?= 0
      duplicateCount "abcde"                    =?= 0
      duplicateCount "aabbcde"                  =?= 2
      duplicateCount "aaBbcde"                  =?= 2
      duplicateCount "Indivisibility"           =?= 1
      duplicateCount "Indivisibilities"         =?= 2
      duplicateCount ['a'..'z']                 =?= 0
      duplicateCount (['a'..'z'] ++ ['A'..'Z']) =?= 26
    it "should work for some random lists" $ do
      property $ forAll (listOf $ elements ['a'..'z']) $ \x ->
        let xs = nub x
        in duplicateCount (concatMap (replicate 2) xs) =?= length xs
  where (=?=) = shouldBe

Sorting & Grouping

To start things off, we are given the following snippet:

module Codwars.Kata.Duplicates where

duplicateCount :: String -> Int
duplicateCount = undefined

My first step is to figure out how to deal with case-insensitivity. Within Data.Char is toLower, which can be used to map over each character in the input String.

Prelude> x = "aaBbcde"
Prelude> x
"aaBbcde"
Prelude> import Data.Char
Prelude Data.Char> :t toLower
toLower :: Char -> Char
Prelude Data.Char> map toLower x
"aabbcde"

Next, I want to group like characters together. To do that, I need to sort and then group the characters together.

Prelude Data.Char> import Data.List
Prelude Data.Char Data.List> :t sort
sort :: Ord a => [a] -> [a]
Prelude Data.Char Data.List> sort . map toLower $ x
"aabbcde"

The sort doesn’t do very much in this case because the input string was already sorted. Either way, now we can work on grouping like characters with group:

Prelude Data.Char Data.List> :t group
group :: Eq a => [a] -> [[a]]
Prelude Data.Char Data.List> group . sort . map toLower $ x
["aa","bb","c","d","e"]

Home Stretch

Now, how do we go from a list of [Char] to an Int length that can be used for filtering characters that only occur once? filter, with a >1 condition applied to the length, should get us there.

Prelude Data.Char Data.List> z = group . sort . map toLower $ x
Prelude Data.Char Data.List> filter ((>1) . length) z
["aa","bb"]

Here, the . allows us to compose length and >1 together so that both can be applied to the [Char] provided to filter. The result rids the list of any characters that only occur once in the original input.

Lastly, we need the count of distinct characters from the input String that occur more than one, which is as simple as getting the length of the filtered list.

Prelude Data.Char Data.List> f = filter ((>1) . length) z
Prelude Data.Char Data.List> length f
2

Putting it all together, and breaking out some of the pipelined functions into a variable in the where clause, we get the duplicateCount function below.

module Codwars.Kata.Duplicates where

import Data.List (group, sort)
import Data.Char (toLower)

duplicateCount :: String -> Int
duplicateCount = length . filter ((>1) . length) . grouped
  where
    grouped = group . sort . map toLower