A few weeks ago, I set out to upgrade the version of Go (1.6 to 1.15) used to build an old command-line utility I developed, named Heimdall. Heimdall provides a way to wrap an executable program inside of an exclusive lock provided by a central PostgreSQL instance via pg_try_advisory_lock.
Now, Heimdall is nice little utility and all (if you’re intrigued, check out the README), but the most interesting part of the upgrade process came after I got everything working and started to think about how to create a new release. That’s when I came across GoReleaser.
GoReleaser
GoReleaser is a release automation tool specifically for Go projects. With a few bits of YAML configuration, GoReleaser provided me with:
Hooks into the Go module system for managing library dependencies
The ability to easily produce a set of build artifacts for multiple operating systems and computer architectures
This year was my first participating in Advent of Code—and I’m glad I did, because solving one of the challenges exposed me to an excellent data validation library for Python named Cerberus.
What’s in a valid passport
Below are some excerpts from the challenge, along with specific field level validation rules:
You arrive at the airport only to realize that you grabbed your North Pole Credentials instead of your passport. While these documents are extremely similar, North Pole Credentials aren’t issued by a country and therefore aren’t actually valid documentation for travel in most of the world.
It seems like you’re not the only one having problems, though; a very long line has formed for the automatic passport scanners, and the delay could upset your travel itinerary.
…
The line is moving more quickly now, but you overhear airport security talking about how passports with invalid data are getting through. Better add some data validation, quick!
You can continue to ignore the cid field, but each other field has strict rules about what values are valid for automatic validation:
byr (Birth Year) - four digits; at least 1920 and at most 2002.
iyr (Issue Year) - four digits; at least 2010 and at most 2020.
eyr (Expiration Year) - four digits; at least 2020 and at most 2030.
hgt (Height) - a number followed by either cm or in:
If cm, the number must be at least 150 and at most 193.
If in, the number must be at least 59 and at most 76.
hcl (Hair Color) - a # followed by exactly six characters 0-9 or a-f.
ecl (Eye Color) - exactly one of: ambblubrngrygrnhzloth.
pid (Passport ID) - a nine-digit number, including leading zeroes.
cid (Country ID) - ignored, missing or not.
Your job is to count the passports where all required fields are both present and valid according to the above rules.
For completeness, here are some invalid passports (delimited by \n\n):
Step two involved converting the passports into Cerberus documents. This was mostly an exercise in parsing uniquely assembled text into Python dictionaries.
# Split the batch file records by double newline.
forrecordinbatch_file.read().split("\n\n"):# Split the fields within a record by a space or newline.
record_field_list=[tuple(field.split(":"))forfieldinre.compile(r"\s").split(record.strip())]
Equipped with a better understanding of what’s possible with Cerberus, and a list of Python dictionaries representing passports, below is the schema I put together to enforce the passport validation rules of the challenge. Only one of the rules (hgt) required a custom function (compare_hgt_with_units).
SCHEMA={"byr":{"min":"1920","max":"2002"},"iyr":{"min":"2010","max":"2020"},"eyr":{"min":"2020","max":"2030"},"hgt":{"anyof":[{"allof":[{"regex":"[0-9]+cm"},{"check_with":compare_hgt_with_units}]},{"allof":[{"regex":"[0-9]+in"},{"check_with":compare_hgt_with_units}]},]},"hcl":{"regex":"#[0-9a-f]{6}"},"ecl":{"allowed":["amb","blu","brn","gry","grn","hzl","oth"]},"pid":{"regex":"[0-9]{9}"},"cid":{"required":False},}# Provide a custom field validation function for a height with units.
defcompare_hgt_with_units(field:str,value:str,error:Callable[...,str])->None:ifvalue.endswith("cm"):ifnot(150<=int(value.rstrip("cm"))<=193):error(field,"out of range")elifvalue.endswith("in"):ifnot(59<=int(value.rstrip("in"))<=76):error(field,"out of range")else:error(field,"missing units")
With a schema in place, all that’s left to do is instantiate a Validator and validate each document:
Keeping project dependencies up-to-date is a challenging problem. Services like GitHub’s automated dependency updating system, Dependabot, go a long way to help make things easier, but that is only helpful if your package manager’s ecosystem is supported. In the case of Scala based projects, it is not.
Scala Steward provides a similar, low-effort way to keep project dependencies up-to-date. You simply open a pull request against the Scala Steward repository and add a reference to your project’s GitHub repository inside of a specially designated Markdown file. After that, Scala Steward (which manifests itself as a robot user on GitHub) keeps your project dependencies up-to-date via pull requests.
Unfortunately, this easy-mode option requires that your repository be publicly accessible. There are options for running Scala Steward as a service for yourself, but that path is less trodden and requires a bit more effort.
Scala Steward and GitHub Actions
So what other options do you have if your Scala project is inside a private repository? Well, if your project is on GitHub, then you likely have access to their workflow automation service, GitHub Actions. Scala Steward’s maintainers created a GitHub Action that lowers the bar to adding Scala Steward support to projects via the GitHub Actions execution model.
By default, the Action supports dependency detection through a workflow defined inside of your project’s repository. This approach makes it easy to simulate the public instance of Scala Steward on a per repository basis. But, there is also a centralized mode that allows you to mimic the way the centrally managed instance of Scala Steward works.
This centralized mode gives us an opportunity to have the best of both worlds: a low-effort way to keep multiple project dependencies up-to-date (similar to the public instance of Scala Steward), and the ability to do so across both public and private repositories!
Putting things together
First, create a GitHub repository for your instance of Scala Steward and put a file in it at .github/workflows/scala-steward.yml with the following contents:
name:Scala Stewardon:schedule:# Schedule to run every Sunday @ 12PM UTC. Replace this with# whatever seems appropriate to you.-cron:"00**0"# Provide support for manually triggering the workflow via GitHub.workflow_dispatch:jobs:scala-steward:name:scala-stewardruns-on:ubuntu-lateststeps:# This is necessary to ensure that the most up-to-date version of# REPOSITORIES.md is used.-uses:actions/checkout@v2-name:Execute Scala Stewarduses:scala-steward-org/scala-steward-action@vX.Y.Zwith:# A GitHub personal access token tied to a user that will create# pull requests against your projects to update dependencies. More# on this under the YAML snippet.github-token:${{ secrets.SCALA_STEWARD_GITHUB_TOKEN }}# A Markdown file with a literal Markdown list of repositories# Scala Steward should monitor.repos-file:REPOSITORIES.mdauthor-email:scala-steward@users.noreply.github.comauthor-name:Scala Steward
Hopefully, the inline comments help minimize any ambiguity in the GitHub Actions workflow configuration file. For completeness, below is an example of the Markdown file as well:
The last step is to ensure that any private repositories add the user associated with the GitHub personal access token as a collaborator with the Write role permissions. Also, to slightly improve usability and maintainability, consider the following suggestions:
Add Dependabot support to your Scala Steward repository to keep the Scala Steward GitHub Action up-to-date.
Avoid tying Scala Steward to an individual user GitHub account. Consider creating a bot account first, then create a personal access token with it to use with Scala Steward.
Create a custom Scala Steward team (e.g., @organization/scala-steward) and add the bot account above to it. Now, instead of remembering to add the bot account to your Scala project repository as a collaborator, you can add the more intuitive Scala Steward team.