Skip to content

Commit dfaca3f

Browse files
blaggacaoblaggacao
committed
docs: update readme with better examples
1 parent c2ca7a2 commit dfaca3f

File tree

2 files changed

+194
-103
lines changed

2 files changed

+194
-103
lines changed

‎README.md

Lines changed: 193 additions & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -1,155 +1,246 @@
1-
# Usage
1+
# Standard Action
22

3-
- Works with https://github.com/divnix/std
4-
- Since GitHub CI doesn't support yaml anchors, explode your file with: `yq '. | explode(.)' ci.raw.yaml > ci.yaml`
5-
- To set up AWS Credentials for an S3 Cache, find details [here](https://github.com/aws-actions/configure-aws-credentials)
6-
- **Warning:** This is still under active development and testing. You're likely better off waiting a little while, still.
7-
- But it's already being used with success :smile:
3+
_for [Standard] & [Paisano]_.
4+
5+
[Paisano]: https://github.com/paisano-nix
6+
[Standard]: https://github.com/divnix/std
7+
8+
Don't waste any time on extra work. Use Standard Action to automatically
9+
detect CI targets that need re-doing; implemented on top of familiar GH Actions.
10+
11+
## Features
12+
13+
- Evaluate once and distribute final build instructions to workers
14+
- Once configured, `discovery` picks up new targets automatically
15+
- Optional `proviso` script can detect if work needs to be done
16+
17+
> **Note on `proviso`**: one example is the oci block type which
18+
> [checks if the image] is already in the registry and only schedules
19+
> a build if its missing. If `proviso` queries private remote state
20+
> then the `discovery` environment must provide all authentication
21+
> prior to running the discovery step.
22+
23+
[checks if the image]: https://github.com/divnix/std/blob/main/src/std/fwlib/blockTypes/containers-proviso.sh
24+
25+
## Usage
26+
27+
**Minimumn nix version `v2.16.1`**
28+
29+
Tip! Since GitHub CI doesn't support `yaml` anchors, explode your file with:
30+
31+
```
32+
yq '. | explode(.)' ci.raw.yaml > ci.yaml
33+
```
34+
35+
### Standalone
36+
37+
```nix
38+
{
39+
/* ... */
40+
outputs = {std, ...}@inputs: std.growOn {
41+
/* ... */
42+
cellBlocks = with std.blockTypes; [
43+
(installables "packages" {ci.build = true;})
44+
(containers "oci-images" {ci.publish = true;})
45+
(kubectl "deployments" {ci.apply = true;})
46+
];
47+
/* ... */
48+
};
49+
}
50+
```
51+
52+
<details><summary><h4>GH Action file</h4></summary>
853

954
```yaml
10-
# .github/workflows/ci.yml
11-
name: Standard CI
55+
# yq '. | explode(.)' this.yml > .github/workflows/std.yml
56+
name: CI/CD
1257

1358
on:
59+
pull_request:
60+
branches:
61+
- main
1462
push:
1563
branches:
1664
- main
17-
workflow_dispatch:
1865

1966
permissions:
67+
id-token: write
2068
contents: read
2169

70+
concurrency:
71+
group: std-${{ github.workflow }}-${{ github.ref }}
72+
cancel-in-progress: true
73+
2274
jobs:
2375
discover:
2476
outputs:
2577
hits: ${{ steps.discovery.outputs.hits }}
26-
nix_conf: ${{ steps.discovery.outputs.nix_conf }}
27-
2878
runs-on: ubuntu-latest
29-
concurrency:
30-
group: ${{ github.workflow }}
3179
steps:
32-
- name: Standard Discovery
33-
uses: divnix/std-action/discover@main
34-
id: discovery
80+
# Important: use this as it also detects flake configuration
81+
- uses: blaggacao/nix-quick-install-action@detect-nix-flakes-config
82+
# if you want to use nixbuild
83+
- uses: nixbuild/nixbuild-action@v17
3584
with:
36-
github_pat: ${{ secrets.HUB_PAT }}
85+
nixbuild_ssh_key: ${{ secrets.SSH_PRIVATE_KEY }}
86+
generate_summary_for: job
87+
# significantly speeds up things in small projects
88+
- uses: DeterminateSystems/magic-nix-cache-action@main
89+
- uses: divnix/std-action/discover@main
90+
id: discovery
3791

38-
build-packages: &run-job
92+
build: &job
3993
needs: discover
94+
name: ${{ matrix.target.jobName }}
95+
runs-on: ubuntu-latest
96+
if: fromJSON(needs.discover.outputs.hits).packages.build != '{}'
4097
strategy:
4198
matrix:
4299
target: ${{ fromJSON(needs.discover.outputs.hits).packages.build }}
43-
name: ${{ matrix.target.cell }} - ${{ matrix.target.name }}
44-
runs-on: ubuntu-latest
45100
steps:
46-
- name: Configure AWS Credentials
47-
uses: aws-actions/configure-aws-credentials@v1-node16
101+
# Important: use this as it also detects flake configuration
102+
- uses: blaggacao/nix-quick-install-action@detect-nix-flakes-config
103+
# if you want to use nixbuild
104+
- uses: nixbuild/nixbuild-action@v17
48105
with:
49-
role-to-assume: arn:aws:iam::123456789100:role/my-github-actions-role
50-
aws-region: us-east-2
106+
nixbuild_ssh_key: ${{ secrets.SSH_PRIVATE_KEY }}
107+
generate_summary_for: job
108+
- uses: DeterminateSystems/magic-nix-cache-action@main
51109
- uses: divnix/std-action/run@main
52-
with:
53-
extra_nix_config: |
54-
${{ needs.discover.outputs.nix_conf }}
55-
json: ${{ toJSON(matrix.target) }}
56-
# optional:
57-
github_pat: ${{ secrets.HUB_PAT }}
58-
nix_key: ${{ secrets.NIX_SECRET_KEY }}
59-
nix_ssh_key: ${{ secrets.NIXBUILD_SSH }}
60-
cache: s3://nix?endpoint=sfo3.digitaloceanspaces.com
61-
builder: ssh-ng://eu.nixbuild.net
62-
ssh_known_hosts: "eu.nixbuild.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIQCZc54poJ8vqawd8TraNryQeJnvH1eLpIDgbiqymM"
63-
64-
build-devshells:
65-
<<: *run-job
110+
111+
images:
112+
<<: *job
113+
needs: [discover, build]
114+
if: fromJSON(needs.discover.outputs.hits).oci-images.publish != '{}'
66115
strategy:
67116
matrix:
68-
target: ${{ fromJSON(needs.discover.outputs.hits).devshells.build }}
69-
70-
publish-containers:
71-
<<: *run-job
117+
target: ${{ fromJSON(needs.discover.outputs.hits).oci-images.publish }}
118+
119+
deploy:
120+
<<: *job
121+
needs: [discover, images]
122+
environment:
123+
name: development
124+
url: https://my.dev.example.com
125+
if: fromJSON(needs.discover.outputs.hits).deployments.apply != '{}'
72126
strategy:
73127
matrix:
74-
target: ${{ fromJSON(needs.discover.outputs.hits).containers.publish }}
128+
target: ${{ fromJSON(needs.discover.outputs.hits).deployments.apply }}
75129
```
76130
77-
## Notes & Explanation
131+
</details>
78132
79-
### Notes on the Build Matrix
133+
### Persistent Discovery Host
80134
81-
Hits from the discovery phase are namespaced by Block and Action.
135+
#### Requirements
82136
83-
That means:
137+
- `nix` >= v2.16.1
138+
- `zstd`
139+
- (gnu) `parallel`
140+
- `jq`
141+
- `base64`
142+
- `bash` > v5
84143

85-
- In: `target: ${{ fromJSON(needs.discover.outputs.hits).packages.build }}`
86-
- `packages` is the name of a Standard Block
87-
- `build` is the name of an Action of that Block
144+
The persistent host must also implement the `nixConfig` detection capabilities
145+
implemented by [this script][script].
88146

89-
This example would be defined in `flake.nix` as such
147+
[script]: https://github.com/nixbuild/nix-quick-install-action/blob/5752d21669438be20da4de77327ae963e98c82a3/read-nix-config-from-flake.sh
90148

91149
```nix
92150
{
93151
/* ... */
94152
outputs = {std, ...}@inputs: std.growOn {
95153
/* ... */
96154
cellBlocks = with std.blockTypes; [
97-
(installables "packages" {ci.build = true;})
98-
(containers "containers" {ci.publish = true;})
155+
(devshells "envs" {ci.build = true;})
156+
(containers "oci-images" {ci.publish = true;})
99157
];
100158
/* ... */
101159
};
102160
}
103161
```
104162

105-
An example schema of the json returned by the dicovery phase:
163+
<details><summary><h4>GH Action file</h4></summary>
106164

107-
```json
108-
{
109-
"containers": {
110-
"publish": [
111-
{
112-
"action": "publish",
113-
"actionDrv": "/nix/store/6b0i2ww5drcdfa6hgxijx39zbcq57rwl-publish.drv",
114-
"actionFragment": "\"__std\".\"actions\".\"x86_64-linux\".\"_automation\".\"containers\".\"vscode\".\"publish",
115-
"block": "containers",
116-
"blockType": "containers",
117-
"cell": "_automation",
118-
"name": "vscode",
119-
"targetDrv": "/nix/store/4hs8x5lgb9nkvjfrxj7azv95hi77avxn-image-std-vscode.json.drv",
120-
"targetFragment": "\"x86_64-linux\".\"_automation\".\"containers\".\"vscode\""
121-
}
122-
]
123-
},
124-
"devshells": {
125-
"build": [
126-
{
127-
"action": "build",
128-
"actionDrv": "/nix/store/zmlva6xlngzj098znyy47p72rxjzgka3-build.drv",
129-
"actionFragment": "\"__std\".\"actions\".\"x86_64-linux\".\"_automation\".\"devshells\".\"default\".\"build",
130-
"block": "devshells",
131-
"blockType": "devshells",
132-
"cell": "_automation",
133-
"name": "default",
134-
"targetDrv": "/nix/store/xq4sl7pf51gp0a036garz56kkr160n5c-Standard.drv",
135-
"targetFragment": "\"x86_64-linux\".\"_automation\".\"devshells\".\"default\""
136-
}
137-
]
138-
},
139-
"packages": {
140-
"build": [
141-
{
142-
"action": "build",
143-
"actionDrv": "/nix/store/l4y4gzpgym5wbvn42avsaf24nqj0d27y-build.drv",
144-
"actionFragment": "\"__std\".\"actions\".\"x86_64-linux\".\"std\".\"packages\".\"adrgen\".\"build",
145-
"block": "packages",
146-
"blockType": "installables",
147-
"cell": "std",
148-
"name": "adrgen",
149-
"targetDrv": "/nix/store/mwidj7li8b7zypq83ap0fmmwxqx58qn6-adrgen-2022-08-08.drv",
150-
"targetFragment": "\"x86_64-linux\".\"std\".\"packages\".\"adrgen\""
151-
}
152-
]
153-
}
154-
}
165+
```yaml
166+
# yq '. | explode(.)' this.yml > .github/workflows/std.yml
167+
name: CI/CD
168+
169+
on:
170+
pull_request:
171+
branches:
172+
- main
173+
push:
174+
branches:
175+
- main
176+
177+
env:
178+
DISCOVERY_USER_NAME: gha-runner
179+
DISCOVERY_KNOWN_HOSTS_ENTRY: "10.10.10.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEOVVDZydvD+diYa6A3EtA3WGw5NfN0wv7ckQxa/fX1O"
180+
181+
permissions:
182+
id-token: write
183+
contents: read
184+
185+
concurrency:
186+
group: ${{ github.sha }}
187+
cancel-in-progress: true
188+
189+
jobs:
190+
discover:
191+
outputs:
192+
hits: ${{ steps.discovery.outputs.hits }}
193+
runs-on: [self-hosted, discovery]
194+
steps:
195+
- name: Standard Discovery
196+
uses: divnix/std-action/discover@main
197+
id: discovery
198+
# avoids transporting derivations via GH Cache
199+
with: { ffBuildInstructions: true }
200+
201+
image: &run-job
202+
needs: discover
203+
strategy:
204+
fail-fast: false
205+
matrix:
206+
target: ${{ fromJSON(needs.discover.outputs.hits).oci-images.publish }}
207+
if: fromJSON(needs.discover.outputs.hits).oci-images.publish != '{}'
208+
name: ${{ matrix.target.jobName }}
209+
runs-on: ubuntu-latest
210+
steps:
211+
# sets up ssh credentials for `ssh discovery ...`
212+
- uses: divnix/std-action/setup-discovery-ssh@main
213+
with:
214+
ssh_key: ${{ secrets.SSH_PRIVATE_KEY_CI }}
215+
user_name: ${{ env.DISCOVERY_USER_NAME }}
216+
ssh_known_hosts_entry: ${{ env.DISCOVERY_KNOWN_HOSTS_ENTRY }}
217+
- uses: divnix/std-action/run@main
218+
# avoids retreiving derivations via GH Cache and uses `ssh discovery ...` instead
219+
with: { ffBuildInstructions: true }
220+
221+
build:
222+
<<: *run-job
223+
strategy:
224+
matrix:
225+
target: ${{ fromJSON(needs.discover.outputs.hits).envs.build }}
226+
if: fromJSON(needs.discover.outputs.hits).envs.build != '{}'
155227
```
228+
229+
</details>
230+
231+
## Notes & Explanation
232+
233+
### Notes on the Build Matrix
234+
235+
Hits from the discovery phase are namespaced by Block and Action.
236+
237+
That means:
238+
239+
- In: `target: ${{ fromJSON(needs.discover.outputs.hits).packages.build }}`
240+
- `packages` is the name of a Standard Block
241+
- `build` is the name of an Action of that Block
242+
243+
### Debugging
244+
245+
Watch out for `base64`-encoded blobs in the logs, you can inspect the
246+
working data of that context by doing: `base64 -d <<< copy-blob-here | jq`.

‎discover/eval.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ echo "::group::📞️ Pass artifacts to the build matrix ..."
142142
if [[ "$SKIP_DRV_EXPORT" == "false" ]]; then
143143
command mkdir -p "$EVALSTORE_EXPORT"
144144
for drv in $(command jq --compact-output --raw-output '.[].actionDrv' <<<"$PROVISIONED"); do
145-
nix-store --query --requisites "$drv" | command nix-store --stdin --export | command zstd > "$EVALSTORE_EXPORT/$(basename $drv).zst"
145+
command nix-store --query --requisites "$drv" | command nix-store --stdin --export | command zstd > "$EVALSTORE_EXPORT/$(basename $drv).zst"
146146
done
147147
fi
148148
}

0 commit comments

Comments
 (0)