I have a Jekyll based builder with site contents in a private github repo. I use Obsidian to edit it, because Obsidian.

When I’m happy with a page, I push it to my private repo. I have an action set up that will then build the site with the following docker file:

FROM ruby:3.3 as builder  
WORKDIR /site  
ADD  Gemfile Gemfile.lock ./  
RUN bundle install

and then I publish it using Github Pages, using this workflow:

name: Makefile CI  
  
on:  
  push:  
    branches: [ "main" ]  
  pull_request:  
    branches: [ "main" ]  
  
jobs:  
  build:  
  
    runs-on: ubuntu-latest  
  
    steps:  
    - uses: actions/checkout@v4  
  
    - name: Build Site  
      run: make  
  
    - name: Push directory to another repository  
      uses: cpina/github-action-push-to-another-repository@v1.7.2  
      env:  
        SSH_DEPLOY_KEY: $  
      with:  
          user-email: 'dewey@deweysasser.com'  
          destination-repository-name: 'deweysasser.github.io'  
          destination-github-username: 'deweysasser'  
          source-directory: '_site'  
          target-branch: 'main'

Obviously, I have a Makefile to stitch this all together, but that only does a docker build, then a docker run with the current path mapped into the container, so it can produce the site output. I could do that in straight up ruby on GitHub Actions, but I want the same build process locally as in GHA, and this keeps it nice and consistent.

Once the generated site is committed to the public site repo, I use the default Action to publish a static site.

Potential improvements

• use the Obsidian Enveloppe plugin to allow me to use all of the rich set of Obsidian rendering in my blog.
• use the Obsidian Git Plugin to automate the commits

These two approaches are likely in conflict, so I probably won’t do both. I’ll likely go with the latter choice until I need something like Dataview (which is utterly awesome for doing things in Obsidian).

• deploy the site directly from my private repo
• make my private repo the public one

I’m currently on a free GitHub plan, so I can’t have pages associated with a private site, but…there’s really nothing private in my public site (though I should scrutinize that before committing to it), so I will quite probably at some point make the source public, and not just the site.

For some years, this site has run in a Kubernetes cluster with several others, and used Traefik Proxy, which has been an excellent piece of code, to multiplex sites through one load balancer.

For better or worse, my provider has just forced a kubernetes upgrade…which removed some deprecated APIs…on which traefik 1.7 depended.

And, instead of maintaining the same standard API of reading annotations from standard Ingress objects, traefik has gone the “custom resource definition” route, destroying the simple and elegant (and standard!) way of using it in favor of complicated and proprietary ways of specifying everything.

Of course, it’s always possible that Traefik is doing this to contribute to monetizing their software. My opinion on that is a topic for another day.

I will update this page later with some details, but for now, the result is that TLS on this (and several other) sites are broken until I can either reconfigure everything to use their new system, or replace them with other methods.

I apologize for the inconvenience. TLS is very useful, and right now we don’t have it. Gagh!

• put the processing where the knowledge is, don’t move the knowledge to the processing
• Where there’s 2, there’s N
• Be more general
• keep it in the problem space as long as possible
• Sufficiently detailed documentation is executable
• DRY is good
• always align authority and responsibility

(more will be added, along with explanations)

Runbook-framework

“Any sufficiently detailed documentation is executable” – me

What is it?

A shell (bash) framework to implement runbook scripts that help move from fully manual to fully automated processes.

Based on the clever concept of a Do Nothing Script, provide some structure for the script which allows & encourages documentation and useful tracking at run time.

You write this code:

runbook() {
  step "Step One"
  step "Step Two, on $env" echo "set $env=$version"
  step "Wait for something to happen" sleep 10
  step "Run a function" run-function
  step "Last thing we can do automatically" echo "Version is $version"
  step "bring it on home"
  step "this one will fail" fail_me
}

And you can run it and get

Or if you just need to produce some documentation, you can get

$ ./example-runbook --version=1.2.3 -doc
1: Step One
2: Step Two, on staging
3: Wait for something to happen
4: Run a function
5: Last thing we can do automatically
6: bring it on home
7: this one will fail
$

Background

I love automation, but often getting something “automated” is a long process, and you actually need to get something done now and don’t have the time to fully automate it.

A few years ago I read a blog post presenting a clever way to move from unautomated to automated processes.

The clever key: start with a script that does nothing other than tell you what to do. A checklist. Then, as time evolves, you can automate one little piece. Since the script is walking you through a checklist, this becomes one step that’s done for you instead of you having to do it. This is one of those simple and amazingly insightful ways of getting things done.

This repository implements a small library of BASH functions to support runbooks in a consistent and useful way.

Quickstart

Be sure to read the “important details” section.

Writing the script

Create a script that looks like this:

#!/bin/bash

source runbook-framework.sh

parameters p1 p2
p1="some default value"

# Here's where you put your actual runbook steps
runbook() {
  step "This is a manual step and you'll get a prompt"
  step "This is an automated step" some-command
}

# This must be the last line in your file
main "$@"

Of course, the steps should be doing whatever it is you need. some-command can be a function you define (I suggest later in the file, so the file starts with your runbook).

If you leave out the command, you’ve just made a prompt. If you include the command, it will be executed. The status of all commands will be printed as they are executed.

Running the script

The framework provides some overall features, including

• -help or -h – show how to run this runbook, including the parameters
• -doc – show the checklist without executing it
• -verbose or -v – show the output of each command as it happens

What happened to the output?

The output of each step is stored away and shown only on error or on request.

Important Details

The script sets up bash to be pretty paranoid, specifically:

set -ueo pipefail

The short form is: unexpanded variables will be an error, any error (that’s not in a conditional) will stop the script, and it counts errors in the middle of a pipeline, not just the result of the last command in the pipeline.

Passing Parameters

You can set the parameters variable to a list of parameters you want to use in your script. They will be set as (global) environment variables, exactly as you type them. All names that begin with an underscore (‘_’) are reserved.

When you call the script, you may pass these variables on the command line as command line options. If any of them are unset, the script will stop. So, if you want a default value, make sure you set it.

Examples

Just try to run it

$ ./example-runbook
Parameter 'version' required
Usage: ./example-runbook [-h|-help] [-show] [--env=VALUE] --version=VALUE 
Defaults:
   env = staging

Get some documentation on the process

$ ./example-runbook  --version=1.2.3 -doc
1: Step One
2: Step Two, on staging
3: Wait for something to happen
4: Run a function
5: Last thing we can do automatically
6: bring it on home
7: this one will fail

Actually run the runbook

$ ./example-runbook  --version=1.2.3
 1: (00:52) Step One...(press enter when done) (5 s) DONE
 2: (00:52) Step Two, on staging... (0 s) DONE
 3: (00:52) Wait for something to happen... (10 s) DONE
 4: (00:52) Run a function... (0 s) DONE
 5: (00:52) Last thing we can do automatically... (0 s) DONE
 6: (00:52) bring it on home...(press enter when done) (12 s) DONE
 7: (00:53) this one will fail... (0 s) FAILED
    >> it's time we stop
    >> Hey, what's that sound
    >> Everybody look, what's going down?

The Future

• the -doc argument should extract some documentation from the comment blocks and include it.
• can we do some kind of time estimate and make that useful information? (discussion here)

Documentation

I’d like to make a “generate a fairly complete markdown document describing the process” be a feature. The question is how to do that without obscuring the actual logic. See the github discussion

Managing state

Would it be interesting to manage the current state of “a run” in a persistent way, particularly so it could be resumed later if e.g. a step failed or something? (discussion here)

So, I’m moving it form Drupal to something else.

I’m trying out Jekyll to statically generate, and it’s pretty cool.

I’m not sure if that’s the right long term solution or not. We’ll see. But, I needed something now, and it’s something.

If you have an option on what I should use (or, better yet, what features I should support), please let me know.

Some thoughts are:

• discussions (could use disqus.com or commento.io)
• a more wiki-like structure with nice cross-links (probably dokuwiki)

This project contains a script and makefile to maintain a table of contents in your markdown documents, along with an optional git pre-commit hook to keep it up to date.

I find it useful for either long documents or long READMEs for github projects.

# Markdown-doc

Add tables of contents to all markdown files in this repo.

## Contents

* [Markdown-doc](#markdown-doc)
  * [Contents](#contents)
  * [Features](#features)
  * [Usage](#usage)
    * [Generate tables of contents](#generate-tables-of-contents)
    * [Bulk html generation](#bulk-html-generation)
  * [Author and Bug Reports](#author-and-bug-reports)
  * [Notes](#notes)
  * [TODO](#todo)

## Features

* Generate Tables of Contents
* bulk translate to HTML
* ...

## Usage

### Generate tables of contents
When run on a file with a section (any level) titled "Contents" (as above), the scripts here will enable you to automatically generate and insert a Table of Contents.

If no Contents section is found, the files will not be updated.

This also provides git hooks to update the ToC on commit (WARNING:  the way it does so will make `git add -p` pretty useless.  Feel free to modify the hook to better suit your needs if this 
one doesn't work for you.)

To (manually) update the table of contents for all markdown files in this directory, run

     make contents

(or just run `make`)

To set up the git hooks for this repository so that ToCs are updatted on commit, run.

     make hooks

### Bulk html generation

`make html`

all markdown files will be rendered into HTML using 'markdown' and placed in `generated/html`

## Author and Bug Reports

Please use GitHub issues for any bug reports or change requests

## Notes

The script uses `sed` and `awk` for maximum system compatability.  It was developed on MacOS, so it's expected to run anywhere with a `sed` and `awk` more recent than 1990 or so.

It's also about the gnarliest piece of `sed` that I ever want to write, and could be used as the example on a slide for "why modern interpreted languages exist".

## TODO

* use pandoc to generate arbitrary formats
* use a static web site generator to generate a site?

It contains what is probably the most gnarly sed script I’ve ever written. I am so glad we’ve moved beyond sed.

(However, sed is also everywhere without having to install other tools on almost any system, which is why I used sed and awk instead of something more modern.)

Anyway, for your reading pleasure (or torture), here it is:

/^#/ {
        s/^\(#*\) */\1|/
        p
        }
        /^--*/ {
        x
        s/^/##|/
        p
        }
        /^==*/ {
        x
        s/^/#|/
        p
        }
        h

Technical competence is table stakes.

To actually play the game, it’s all about communication, advocacy, communication, evangelism, communication, persuasion, communication, relationships, and communication.

There’s no point in designing the perfect cat tree if you can’t convince the cats to use it. It’s really about herding cats.

git-crypt is pretty awesome, but I’ve managed to do enough evil things to bork the decryption keys. Here’s how to fix it.

The Background

git-crypt allows you to encrypt some of the files in a git repository. It supports adding other users’ GPG keys so they will be able to encrypt/decrypt the file. It’s pretty clever, and I like it.

The basic operation is that it creates a single symmetric key (which it stores in .git/git-crypt/keys/default), then on request will use GPG to encrypt that key to the public key IDs you specify, and those are stored in .git-crypt//keys/default/0.

However, if you do sufficiently evil things in git repos, like using git filter-repo (another pretty awesome tool), it’s possible to drive git crypt into a pathological state where your keys are wrong and you can’t do anything.

In order to recover from this situation, make sure you have unencrypted copies of the files around somewhere, then:

The Fix

• create a fresh clone of your repo. In that clone, do…
• save the IDs of all keys

keys=$(ls .git-crypt/keys/default/0/*.gpg | xargs -n 1 -I F basename F .gpg)

• Remove the existing encrypted keys

git rm -r .git-crypt
rm -rf .git/git-crypt

• Reinitialize the encryption

git crypt init

• Re-add the public keys which should access the encrypted files

for k in $keys; do git crypt add-gpg user $k; done

• COPY in fresh, unencrypted copies of all previously encrypted files, however you do that

• (Optional) rebase so that all of this stuff occurs in a single commit

git rebase -i origin/master

Note that the VI magic to “fixup” all commits into the first one is 2,$ s/pick/fixup/