Centralized Scala Steward with GitHub Actions

Keeping project dependencies up-to-date is a challenging problem. Services like GitHub’s automated dependency updating system, Dependabot, go a long way to help make things easier, but that is only helpful if your package manager’s ecosystem is supported. In the case of Scala based projects, it is not.

Enter Scala Steward.

Scala Steward provides a similar, low-effort way to keep project dependencies up-to-date. You simply open a pull request against the Scala Steward repository and add a reference to your project’s GitHub repository inside of a specially designated Markdown file. After that, Scala Steward (which manifests itself as a robot user on GitHub) keeps your project dependencies up-to-date via pull requests.

Unfortunately, this easy-mode option requires that your repository be publicly accessible. There are options for running Scala Steward as a service for yourself, but that path is less trodden and requires a bit more effort.

Scala Steward and GitHub Actions

So what other options do you have if your Scala project is inside a private repository? Well, if your project is on GitHub, then you likely have access to their workflow automation service, GitHub Actions. Scala Steward’s maintainers created a GitHub Action that lowers the bar to adding Scala Steward support to projects via the GitHub Actions execution model.

By default, the Action supports dependency detection through a workflow defined inside of your project’s repository. This approach makes it easy to simulate the public instance of Scala Steward on a per repository basis. But, there is also a centralized mode that allows you to mimic the way the centrally managed instance of Scala Steward works.

This centralized mode gives us an opportunity to have the best of both worlds: a low-effort way to keep multiple project dependencies up-to-date (similar to the public instance of Scala Steward), and the ability to do so across both public and private repositories!

Putting things together

First, create a GitHub repository for your instance of Scala Steward and put a file in it at .github/workflows/scala-steward.yml with the following contents:

name: Scala Steward

on:
  schedule:
    # Schedule to run every Sunday @ 12PM UTC. Replace this with
    # whatever seems appropriate to you.
    - cron: "0 0 * * 0"
  # Provide support for manually triggering the workflow via GitHub.
  workflow_dispatch:

jobs:
  scala-steward:
    name: scala-steward
    runs-on: ubuntu-latest
    steps:
      # This is necessary to ensure that the most up-to-date version of
      # REPOSITORIES.md is used.
      - uses: actions/checkout@v2

      - name: Execute Scala Steward
        uses: scala-steward-org/scala-steward-action@vX.Y.Z
        with:
          # A GitHub personal access token tied to a user that will create
          # pull requests against your projects to update dependencies. More
          # on this under the YAML snippet.
          github-token: ${{ secrets.SCALA_STEWARD_GITHUB_TOKEN }}
          # A Markdown file with a literal Markdown list of repositories
          # Scala Steward should monitor.
          repos-file: REPOSITORIES.md
          author-email: scala-steward@users.noreply.github.com
          author-name: Scala Steward

Hopefully, the inline comments help minimize any ambiguity in the GitHub Actions workflow configuration file. For completeness, below is an example of the Markdown file as well:

- organization/repository1
- organization/repository2
- organization/repository3

The last step is to ensure that any private repositories add the user associated with the GitHub personal access token as a collaborator with the Write role permissions. Also, to slightly improve usability and maintainability, consider the following suggestions:

• Add Dependabot support to your Scala Steward repository to keep the Scala Steward GitHub Action up-to-date.
• Avoid tying Scala Steward to an individual user GitHub account. Consider creating a bot account first, then create a personal access token with it to use with Scala Steward.
• Create a custom Scala Steward team (e.g., @organization/scala-steward) and add the bot account above to it. Now, instead of remembering to add the bot account to your Scala project repository as a collaborator, you can add the more intuitive Scala Steward team.

A Useful Framework for Interpreting Success Stories

Recently, I had the pleasure of reading Work Is Work, an essay by Coda Hale on organizational design. Aside from providing a thought-provoking perspective on scaling organizational efforts, the post makes reference to two terms from anthropological field research that were new to me: emic and etic. Below, I’ll describe how these terms provide a useful framework for interpreting success stories.

Emic and Etic

When we read success stories, we often do so to help narrow down the solution space for a problem we’re facing. During that process, it can sometimes be easy to lose track of how important details of the story (its plot, setting, actors, etc.) are different from ours.

Emic and etic help describe behaviors or beliefs from the actor’s perspective (emic) vs. behavior or beliefs observed by an outsider (etic). Continuing with the success story example, writing about how I had great success with a new JavaScript framework is an emic account. You reading my story as research for selecting a JavaScript framework to use for your project is an etic account.

This framework has been valuable to me in two ways. It:

1. Helps heighten my awareness; prompting an additional level of scrutiny toward the solutions I consider (e.g., you had success, but the project you used the JavaScript framework on was small and mine is large).
2. Provides shorthand terms for what are otherwise relatively difficult concepts to communicate.

Scheduling Lambda Functions with AWS SAM

A few days ago, I spent some time learning how to use Amazon’s Serverless Application Model (SAM) to schedule the recurring execution of Lambda functions. To help better cement my understanding, I assembled an overview of all the SAM template components necessary to schedule the periodic execution of a Go-based Lambda function. I also made note of how I used the SAM CLI to package and deploy everything to AWS.

Serverless Application Model

Amazon’s Serverless Application Model is a specification for translating SAM templates into CloudFormation templates. Much like macro expansion, it works through a textual transformation of the input SAM template into a template the CloudFormation engine can make sense of.

There are several components that make up a SAM template, but in this example we only use four: Format Version, Description, Transform, and Resources.

• Format Version equates to AWSTemplateFormatVersion in the template, which identifies its capabilities
• Description is optional, but provides a way to give the template a high-level description
• Transform can map to multiple things, but here it maps to the AWS::Serverless-2016-10-31 transform, which is a version of the SAM specification

As far as Resources go, this template defines two: TestFunction and TestRole.

AWSTemplateFormatVersion: '2010-09-09'
Description: A scheduled Amazon Lambda function.
Resources:
  TestFunction:
    Properties:
      CodeUri: .
      Events:
        Testy:
        Properties:
          Schedule: rate(1 hour)
        Type: Schedule
      Handler: main
      Role: !GetAtt TestRole.Arn
      Runtime: go1.x
    Type: AWS::Serverless::Function
  TestRole:
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Action:
          - sts:AssumeRole
        Effect: Allow
        Principal:
          Service:
          - lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Type: AWS::IAM::Role
Transform: AWS::Serverless-2016-10-31

TestRole is a resource of type AWS::IAM::Role, which is a top-level CloudFormation resource. It creates an Identity and Access Management (IAM) role containing the permissions necessary for our Lambda function to do its thing. In this case, it simply encapsulates a canned IAM policy, AWSLambdaBasicExecutionRole. This policy allows the Lambda function to use the following CloudWatch API calls to log function output to CloudWatch Logs.

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents

The next resource, TestFunction, is of type AWS::Serverless::Function. This is not a top-level CloudFormation resource. Instead, it is a SAM resource that expands into multiple top-level CloudFormation resources. Based on our usage, it expands into three:

• AWS::Lambda::Function
• AWS::Lambda::Permission
• AWS::Events::Rule

AWS::Lambda::Function is the top-level CloudFormation resource to define an Amazon Lambda function. Because we want to schedule the function’s periodic execution, we include an Events property on our AWS::Serverless::Function resource. This allows us to define the function execution schedule within the context of the function’s properties. Behind-the-scenes, the Events property expands into a AWS::Events::Rule resource with an invocation rate of once per hour.

Lastly, in order for the CloudWatch Events API to invoke our function, it needs permissions to do so. AWS::Lambda::Permission grants CloudWatch Events the permission to invoke our function.

Package and ship

The AWS SAM CLI builds on top of the SAM specification by providing a single tool to manage the packaging and deployment of serverless applications. Installation is a bit out-of-scope for this post, but once you’ve managed to install the sam tool, the application deployment process occurs in three phases.

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func HandleRequest(ctx context.Context, e events.CloudWatchEvent) (string, error) {
    return fmt.Sprintf("Hello, world."), nil
}

func main() {
    lambda.Start(HandleRequest)
}

First, compile your Go-based Lambda function into a Linux compatible binary.

GOOS=linux go build -o main main.go

Once the binary exists, use sam to upload the binary to S3 and reference it in a newly created packaged.yaml CloudFormation configuration.

$ sam package --s3-bucket test-global-config-us-east-1 \
              --template-file template.yaml \
              --output-template-file packaged.yaml
Uploading to 7001c68762c2fcda61de373e0a30563d  29187040 / 29187040.0  (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.

Before using sam to deploy using the contents of packaged.yaml, run a quick diff to see what changed.

$ diff template.yaml packaged.yaml
<       CodeUri: .
---
>       CodeUri: s3://test-global-config-us-east-1/7001c68762c2fcda61de373e0a30563d

Lastly, use sam again to deploy the template through a CloudFormation stack named Test.

$ sam deploy --template-file packaged.yaml \
             --stack-name Test \
             --capabilities CAPABILITY_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - Test

Within an hour or so (it only takes a few minutes to deploy—the wait is for the function schedule to trigger), you should see something like the following in your function’s CloudWatch Logs log stream.

START RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1 Version: $LATEST
"Hello, world."
END RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
REPORT RequestId: 5886a0f4-50a1-1cca-10b2-67f512fd83b1
Duration: 1.59 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 5 MB