Scheduling Lambda Functions with AWS SAM

14 Jun 2018

A few days ago, I spent some time learning how to use Amazon’s Serverless Application Model (SAM) to schedule the recurring execution of Lambda functions. To help better cement my understanding, I assembled an overview of all the SAM template components necessary to schedule the periodic execution of a Go-based Lambda function. I also made note of how I used the SAM CLI to package and deploy everything to AWS.

Serverless Application Model

Amazon’s Serverless Application Model is a specification for translating SAM templates into CloudFormation templates. Much like macro expansion, it works through a textual transformation of the input SAM template into a template the CloudFormation engine can make sense of.

There are several components that make up a SAM template, but in this example we only use four: Format Version, Description, Transform, and Resources.

Format Version equates to AWSTemplateFormatVersion in the template, which identifies its capabilities
Description is optional, but provides a way to give the template a high-level description
Transform can map to multiple things, but here it maps to the AWS::Serverless-2016-10-31 transform, which is a version of the SAM specification

As far as Resources go, this template defines two: TestFunction and TestRole.

AWSTemplateFormatVersion: '2010-09-09'
Description: A scheduled Amazon Lambda function.
Resources:
  TestFunction:
    Properties:
      CodeUri: .
      Events:
        Testy:
        Properties:
          Schedule: rate(1 hour)
        Type: Schedule
      Handler: main
      Role: !GetAtt TestRole.Arn
      Runtime: go1.x
    Type: AWS::Serverless::Function
  TestRole:
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Action:
          - sts:AssumeRole
        Effect: Allow
        Principal:
          Service:
          - lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Type: AWS::IAM::Role
Transform: AWS::Serverless-2016-10-31

TestRole is a resource of type AWS::IAM::Role, which is a top-level CloudFormation resource. It creates an Identity and Access Management (IAM) role containing the permissions necessary for our Lambda function to do its thing. In this case, it simply encapsulates a canned IAM policy, AWSLambdaBasicExecutionRole. This policy allows the Lambda function to use the following CloudWatch API calls to log function output to CloudWatch Logs.

logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents

The next resource, TestFunction, is of type AWS::Serverless::Function. This is not a top-level CloudFormation resource. Instead, it is a SAM resource that expands into multiple top-level CloudFormation resources. Based on our usage, it expands into three:

AWS::Lambda::Function
AWS::Lambda::Permission
AWS::Events::Rule

AWS::Lambda::Function is the top-level CloudFormation resource to define an Amazon Lambda function. Because we want to schedule the function’s periodic execution, we include an Events property on our AWS::Serverless::Function resource. This allows us to define the function execution schedule within the context of the function’s properties. Behind-the-scenes, the Events property expands into a AWS::Events::Rule resource with an invocation rate of once per hour.

Lastly, in order for the CloudWatch Events API to invoke our function, it needs permissions to do so. AWS::Lambda::Permission grants CloudWatch Events the permission to invoke our function.

Package and ship

The AWS SAM CLI builds on top of the SAM specification by providing a single tool to manage the packaging and deployment of serverless applications. Installation is a bit out-of-scope for this post, but once you’ve managed to install the sam tool, the application deployment process occurs in three phases.

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func HandleRequest(ctx context.Context, e events.CloudWatchEvent) (string, error) {
    return fmt.Sprintf("Hello, world."), nil
}

func main() {
    lambda.Start(HandleRequest)
}

First, compile your Go-based Lambda function into a Linux compatible binary.

$ GOOS=linux go build -o main main.go

Once the binary exists, use sam to upload the binary to S3 and reference it in a newly created packaged.yaml CloudFormation configuration.

$ sam package --s3-bucket test-global-config-us-east-1 \
              --template-file template.yaml \
              --output-template-file packaged.yaml
Uploading to 7001c68762c2fcda61de373e0a30563d  29187040 / 29187040.0  (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.

Before using sam to deploy using the contents of packaged.yaml, run a quick diff to see what changed.

$ diff template.yaml packaged.yaml
<       CodeUri: .
---
>       CodeUri: s3://test-global-config-us-east-1/7001c68762c2fcda61de373e0a30563d

Lastly, use sam again to deploy the template through a CloudFormation stack named Test.

$ sam deploy --template-file packaged.yaml \
             --stack-name Test \
             --capabilities CAPABILITY_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - Test

Within an hour or so (it only takes a few minutes to deploy—the wait is for the function schedule to trigger), you should see something like the following in your function’s CloudWatch Logs log stream.

START RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1 Version: $LATEST
"Hello, world."
END RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
REPORT RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
Duration: 1.59 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 5 MB

Haskell Code Katas: Counting Duplicates

17 Dec 2017

For the past few weeks, I’ve been starting off my days with Haskell flavored code katas from Codewars. Today I started with the kata below and figured it would be a good exercise to walk through my solution.

Write a function that will return the count of distinct case-insensitive alphabetic characters and numeric digits that occur more than once in the input string. The input string can be assumed to contain only alphabets (both uppercase and lowercase) and numeric digits.

To help clarify the specifications for this kata, the Hspec test suite is below:

module Codwars.Kata.Duplicates.Test where

import Codwars.Kata.Duplicates (duplicateCount)
import Data.List (nub)
import Test.Hspec
import Test.QuickCheck

main = hspec $ do
  describe "duplicateCount" $ do
    it "should work for some small tests" $ do
      duplicateCount ""                         =?= 0
      duplicateCount "abcde"                    =?= 0
      duplicateCount "aabbcde"                  =?= 2
      duplicateCount "aaBbcde"                  =?= 2
      duplicateCount "Indivisibility"           =?= 1
      duplicateCount "Indivisibilities"         =?= 2
      duplicateCount ['a'..'z']                 =?= 0
      duplicateCount (['a'..'z'] ++ ['A'..'Z']) =?= 26
    it "should work for some random lists" $ do
      property $ forAll (listOf $ elements ['a'..'z']) $ \x ->
        let xs = nub x
        in duplicateCount (concatMap (replicate 2) xs) =?= length xs
  where (=?=) = shouldBe

Sorting & Grouping

To start things off, we are given the following snippet:

module Codwars.Kata.Duplicates where

duplicateCount :: String -> Int
duplicateCount = undefined

My first step is to figure out how to deal with case-insensitivity. Within Data.Char is toLower, which can be used to map over each character in the input String.

Prelude> x = "aaBbcde"
Prelude> x
"aaBbcde"
Prelude> import Data.Char
Prelude Data.Char> :t toLower
toLower :: Char -> Char
Prelude Data.Char> map toLower x
"aabbcde"

Next, I want to group like characters together. To do that, I need to sort and then group the characters together.

Prelude Data.Char> import Data.List
Prelude Data.Char Data.List> :t sort
sort :: Ord a => [a] -> [a]
Prelude Data.Char Data.List> sort . map toLower $ x
"aabbcde"

The sort doesn’t do very much in this case because the input string was already sorted. Either way, now we can work on grouping like characters with group:

Prelude Data.Char Data.List> :t group
group :: Eq a => [a] -> [[a]]
Prelude Data.Char Data.List> group . sort . map toLower $ x
["aa","bb","c","d","e"]

Home Stretch

Now, how do we go from a list of [Char] to an Int length that can be used for filtering characters that only occur once? filter, with a >1 condition applied to the length, should get us there.

Prelude Data.Char Data.List> z = group . sort . map toLower $ x
Prelude Data.Char Data.List> filter ((>1) . length) z
["aa","bb"]

Here, the . allows us to compose length and >1 together so that both can be applied to the [Char] provided to filter. The result rids the list of any characters that only occur once in the original input.

Lastly, we need the count of distinct characters from the input String that occur more than one, which is as simple as getting the length of the filtered list.

Prelude Data.Char Data.List> f = filter ((>1) . length) z
Prelude Data.Char Data.List> length f
2

Putting it all together, and breaking out some of the pipelined functions into a variable in the where clause, we get the duplicateCount function below.

module Codwars.Kata.Duplicates where

import Data.List (group, sort)
import Data.Char (toLower)

duplicateCount :: String -> Int
duplicateCount = length . filter ((>1) . length) . grouped
  where
    grouped = group . sort . map toLower

Installing Tor on FreeBSD 11

12 Nov 2016

Tor is a piece of free software and an open network that enables anonymous communication. Combined, these two components help defend against various forms of traffic analysis and network surveillance. Trying to re-explain Tor in a comprehensive way is outside the scope of this post, but please read about it via the literature provided by the project site and The Electronic Frontier Foundation (EFF) before installing.

Installation

The first step toward installing Tor on FreeBSD is deciding whether you want to install the precompiled package with pkg, or you want to compile it yourself from the FreeBSD Ports Collection. The tradeoffs between these two approaches are well-explained within the FreeBSD Handbook. I chose the package because customizing the installation configuration beyond the defaults didn’t seem necessary.

With all of that said, from inside a root shell install the Tor package with:

# pkg install tor

Configuration

From there, copy the sample Tor configuration file into its default location and open it inside your editor:

# cp /usr/local/etc/tor/torrc.sample /usr/local/etc/tor/torrc
# vim /usr/local/etc/tor/torrc

Once inside the file, there are three settings that we want to make explicit. All should be commented out by default (SOCKSPort,Log, and Log again), so we simply need to uncomment them. Below is a diff of the changes between the sample and our desired configuration file:

18c18
< SOCKSPort 9050
---
> #SOCKSPort 9050 # Default: Bind to localhost:9050 for local connections.
38c38
< Log notice file /var/log/tor/notices.log
---
> #Log notice file /var/log/tor/notices.log
42c42
< Log notice syslog
---
> #Log notice syslog

The SOCKSPort setting ensures that we’re binding Tor to 127.0.0.1 on its default port of 9050. The two Log settings ensure that notice level log messages are written to a specific log file, as well as syslog.

Now, we can launch Tor using the tor command to see if things are working properly:

% tor
[notice] Tor v0.2.8.9 running on FreeBSD with Libevent 2.0.22-stable, OpenSSL 1.0.2j-freebsd and Zlib 1.2.8.
[notice] Tor cant help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
[notice] Read configuration file "/usr/local/etc/tor/torrc".
[notice] Opening Socks listener on 127.0.0.1:9050
[notice] Parsing GEOIP IPv4 file /usr/local/share/tor/geoip.
[notice] Parsing GEOIP IPv6 file /usr/local/share/tor/geoip6.
[notice] Bootstrapped 0%: Starting
[notice] Bootstrapped 80%: Connecting to the Tor network
[notice] Bootstrapped 85%: Finishing handshake with first hop
[notice] Bootstrapped 90%: Establishing a Tor circuit
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Bootstrapped 100%: Done

Once satisfied, CTRL+C the process so that control is returned to your shell.

Lastly, let’s enable the Tor service so that it starts on its own after the system boots. To achieve that, all we have to do is ensure that /etc/rc.conf contains the following line:

tor_enable="YES"

Afterwards, launch the Tor service through the service manager if you want it running prior to the next boot cycle:

# service tor start

That’s it. You should now have a fully functional installation of Tor running on FreeBSD.