Experimenting with CI tools — Git Presubmit Linter

The opinions stated here are my own, not those of my company.

I enjoy writing code. In fact it’s the easiest part of my job. Knowing what code to write, and getting reviewers to agree, is certainly a more human and less fun part.

These reviews may go through several layers including style, functionality, and sometimes even the title of my commit. Many years ago there was a lot of debate we had over the way we write commit titles, whether to past tense, present tense, or imperative tense.

• Past: “Added new field”
• Present: “Adds new field”
• Imperative: “Add new field”

We finally agreed on a format, but it made me think a lot about how much time we spend on these nitpicks and how to save time in the future. It’s not good for productivity when I ask for a code review and an hour later am told to rewrite my commit message.

This was the inception for an open-source project I started, Git Presubmit Linter. While it originally started as a way to review git messages, it has since evolved with a broader collection of tools.

When I started looking in the industry, I found a number of pre-existing tools. If I had a Python project I could install a package using pip. For Node.js there were other packages. As we worked on a number of languages, I could not find anything that was suitably cross-language. I couldn’t justify installing NPM in my Python CI environment for something this trivial.

Instead I selected Bash as the primary language and used pipes as the primary input mechanism. This also means that these tools will work for other subsystems without much difficulty.

set -e # Exit script with error if a rule fails
GIT_PRESUBMIT_LINTER='git-presubmit-linter'
RULES="${GIT_PRESUBMIT_LINTER}/rules"
if [ ! -d ${GIT_PRESUBMIT_LINTER} ]; then
    # Clone the git presubmit linter repository
    git clone https://github.com/google/git-presubmit-linter.git
fi
cd ${GIT_PRESUBMIT_LINTER}
git pull origin master
cd ../
# Check git commit for style
git log -1 --pretty=%B | ${RULES}/verb-tense.sh imperative
git log -1 --pretty=%B | ${RULES}/no-second-line.sh
git log -1 --pretty=%B | ${RULES}/line-length.sh 100
# Check git diff lines that have been modified
git diff HEAD~1 | grep '^\+' | ${RULES}/trailing-whitespace.sh

Installing the tool in your environment is going to look something like that. You can clone using git and then pass git values into each of the rules. These rules may have additional parameters. The project README includes documentation for each.

In the snippet above, we enforce four different policies in a cross-language way:

• The verb tense in the commit message must be imperative, ie. “Add”
• The commit message should have nothing on the second line
• The commit title should be less than 100 characters long
• No modified file should have any trailing spaces

These may be solved using a disparate set of existing tools, or part of general rules of thumb, but cementing this in formal rules will prevent any mistakes from getting through and help newcomers learn standard practices.

Over time I began to see value in using a number of tools that went along with git and I added them to this repo because I couldn’t find a better place for them. These are also command-line tools that can be employed in the CI environment or outside of it.

./tools/changelog v1.0.0 v1.0.1 > CHANGELOG.md

One that I use the most frequently is the changelog script, which generates a list of changes in Markdown between two tags or commits. Without any parameters, it will select all the changes between HEAD and the most recent tag. This has come in handy for quickly creating release notes.

tar -tf myfiles.tar.gz | ./tools/filelist.sh ./expected-files.txt

There are also some tools for checking archives to ensure there aren’t unintended files that were bundled. This has been used a few times as an additional safeguard before publishing a new release.

cat README.md | ./rules/path-exists.sh

When reformatting a few READMEs I wrote this tool to catch broken links. It will scan through every URL in the provided files and try to cURL to it. If it cannot get a response, it will flag that section of the file so that you can fix it.

There are a bunch of other rules and tools that I didn’t introduce here. If you are looking to catch certain strings like “TODO”s, ensure a certain file gets updated, or make sure that all your dependencies use a particular license, you will find the full repository has what you need.

I’ve also been studying the Conventional Commits format and the Keep a Changelog format, with scripts for both. They can ensure every commit fits the pattern and generate high-quality changelogs in a way that is easy to setup and not dependent on anything else including a certain programming language.

Everything is available at https://github.com/google/git-presubmit-linter