Skip to content

Commit f100f67

Browse files
daroradarora
committed
Fix build and speed it up.
Addresses current breakage by modifying perms on the /etc/postgresql-common dir to allow adminapi to write to it. Additionally, speeds up the build by applying a few fixes: - using parallelism where applicable (make/maven parallelism) - not building postgres docs - using our own s3 bucket for postgis sources (origin is extremely slow) - using a 16 core machine instead of 8 cores: plv8 is the slowest task right now and is mostly CPU constrained on v8 compilation - links postgres binaries using a shell loop rather than ansible (saves a minute) - switch to use1 (closer to a lot of external resources used) - enable ssh pipelining Old timing info (incomplete as the build was broken, but based on GH actions it completed it in ~1hr 40mins; 20mins of that was likely post-ansible AMI copying etc): 2022-03-21T17:05:46Z: amazon-ebs: Monday 21 March 2022 17:05:46 +0000 (0:00:02.675) 1:15:00.775 ********** 2022-03-21T17:05:46Z: amazon-ebs: =============================================================================== 2022-03-21T17:05:46Z: amazon-ebs: postgis - build SFCGAL ----------------------------------------------- 1101.32s 2022-03-21T17:05:46Z: amazon-ebs: plv8 - build ---------------------------------------------------------- 801.52s 2022-03-21T17:05:46Z: amazon-ebs: Postgres - build ------------------------------------------------------ 419.36s 2022-03-21T17:05:46Z: amazon-ebs: pgRouting - build ----------------------------------------------------- 364.44s 2022-03-21T17:05:46Z: amazon-ebs: pljava - build -------------------------------------------------------- 319.36s 2022-03-21T17:05:46Z: amazon-ebs: postgis - download latest release ------------------------------------- 219.28s 2022-03-21T17:05:46Z: amazon-ebs: System - apt update and apt upgrade ----------------------------------- 138.52s 2022-03-21T17:05:46Z: amazon-ebs: postgis - build ------------------------------------------------------- 126.25s 2022-03-21T17:05:46Z: amazon-ebs: pljava - download & install dependencies ------------------------------- 63.43s 2022-03-21T17:05:46Z: amazon-ebs: libsodium - build ------------------------------------------------------ 61.62s 2022-03-21T17:05:46Z: amazon-ebs: postgis - download & install dependencies ------------------------------ 61.46s 2022-03-21T17:05:46Z: amazon-ebs: Install other useful tools --------------------------------------------- 30.84s 2022-03-21T17:05:46Z: amazon-ebs: Download LLVM & Clang -------------------------------------------------- 26.68s 2022-03-21T17:05:46Z: amazon-ebs: Postgres dependencies -------------------------------------------------- 18.55s 2022-03-21T17:05:46Z: amazon-ebs: Postgres - configure --------------------------------------------------- 18.29s 2022-03-21T17:05:46Z: amazon-ebs: plv8 - download & install dependencies --------------------------------- 17.32s 2022-03-21T17:05:46Z: amazon-ebs: Download GCC 10 -------------------------------------------------------- 16.87s 2022-03-21T17:05:46Z: amazon-ebs: pgAudit - download & install dependencies ------------------------------ 16.45s 2022-03-21T17:05:46Z: amazon-ebs: install Vector for logging --------------------------------------------- 11.14s 2022-03-21T17:05:46Z: amazon-ebs: libsodium - configure -------------------------------------------------- 10.69s New timing info: 2022-03-22T09:36:08Z: amazon-ebs: Tuesday 22 March 2022 09:36:08 +0000 (0:00:01.587) 0:36:11.036 ********* 2022-03-22T09:36:08Z: amazon-ebs: =============================================================================== 2022-03-22T09:36:08Z: amazon-ebs: plv8 - build ---------------------------------------------------------- 513.34s 2022-03-22T09:36:08Z: amazon-ebs: Postgres - build ------------------------------------------------------ 355.08s 2022-03-22T09:36:08Z: amazon-ebs: wal-g - additional go dependencies ------------------------------------ 118.28s 2022-03-22T09:36:08Z: amazon-ebs: postgis - build SFCGAL ------------------------------------------------ 101.59s 2022-03-22T09:36:08Z: amazon-ebs: System - apt update and apt upgrade ------------------------------------ 87.44s 2022-03-22T09:36:08Z: amazon-ebs: postgis - download & install dependencies ------------------------------ 58.85s 2022-03-22T09:36:08Z: amazon-ebs: libsodium - build ------------------------------------------------------ 52.69s 2022-03-22T09:36:08Z: amazon-ebs: wal-g - build ---------------------------------------------------------- 41.63s 2022-03-22T09:36:08Z: amazon-ebs: pljava - download & install dependencies ------------------------------- 35.37s 2022-03-22T09:36:08Z: amazon-ebs: pgRouting - build ------------------------------------------------------ 25.96s 2022-03-22T09:36:08Z: amazon-ebs: Install other useful tools --------------------------------------------- 22.59s 2022-03-22T09:36:08Z: amazon-ebs: pljava - build --------------------------------------------------------- 21.98s 2022-03-22T09:36:08Z: amazon-ebs: Download LLVM & Clang -------------------------------------------------- 20.20s 2022-03-22T09:36:08Z: amazon-ebs: postgis - build -------------------------------------------------------- 19.40s 2022-03-22T09:36:08Z: amazon-ebs: Remove build dependencies ---------------------------------------------- 16.02s 2022-03-22T09:36:08Z: amazon-ebs: ensure services are stopped -------------------------------------------- 15.34s 2022-03-22T09:36:08Z: amazon-ebs: Postgres dependencies -------------------------------------------------- 14.54s 2022-03-22T09:36:08Z: amazon-ebs: Postgres - configure --------------------------------------------------- 14.18s 2022-03-22T09:36:08Z: amazon-ebs: PgBouncer - build ------------------------------------------------------ 13.92s 2022-03-22T09:36:08Z: amazon-ebs: ensure services are stopped and disabled for first boot ---------------- 12.86s
1 parent 09869ab commit f100f67

12 files changed

+29
-28
lines changed

‎ansible.cfg

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[defaults]
2+
3+
callback_whitelist = profile_tasks

‎ansible/playbook.yml

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -157,9 +157,5 @@
157157

158158
- name: Create symbolic links for Postgres binaries to /usr/bin/
159159
become: yes
160-
file:
161-
src: "{{ item.path }}"
162-
path: "/usr/bin/{{ item.path | basename }}"
163-
state: link
164-
force: yes
165-
with_items: "{{ postgresql_bin.files }}"
160+
shell:
161+
cmd: "for fl in /usr/lib/postgresql/bin/* ; do ln -sf $fl /usr/bin/$(basename $fl) ; done"

‎ansible/tasks/internal/admin-api.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- name: adminapi - system user
22
user:
33
name: adminapi
4-
groups: root,admin
4+
groups: root,admin,postgres
55
append: yes
66

77
- name: disk script

‎ansible/tasks/postgres-extensions/01-postgis.yml

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
- name: postgis - build SFCGAL
5050
make:
5151
chdir: /tmp/SFCGAL-v{{ sfcgal_release }}
52+
jobs: 16
5253
become: yes
5354

5455
- name: postgis - install SFCGAL
@@ -58,11 +59,8 @@
5859
become: yes
5960

6061
- name: postgis - download latest release
61-
get_url:
62-
url: "https://download.osgeo.org/postgis/source/postgis-{{ postgis_release }}.tar.gz"
63-
dest: /tmp/postgis-{{ postgis_release }}.tar.gz
64-
checksum: "{{ postgis_release_checksum }}"
65-
timeout: 60
62+
shell:
63+
cmd: "curl -sf -L https://supabase-public-artifacts-bucket.s3.amazonaws.com/postgis-{{ postgis_release }}.tar.gz -o /tmp/postgis-{{ postgis_release }}.tar.gz"
6664

6765
- name: postgis - unpack archive
6866
unarchive:
@@ -80,10 +78,11 @@
8078
- name: postgis - build
8179
make:
8280
chdir: /tmp/postgis-{{ postgis_release }}
81+
jobs: 16
8382
become: yes
8483

8584
- name: postgis - install
8685
make:
8786
chdir: /tmp/postgis-{{ postgis_release }}
8887
target: install
89-
become: yes
88+
become: yes

‎ansible/tasks/postgres-extensions/02-pgrouting.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,11 @@
3636
- name: pgRouting - build
3737
make:
3838
chdir: /tmp/pgrouting-{{ pgrouting_release }}/build
39+
jobs: 16
3940
become: yes
4041

4142
- name: pgRouting - install
4243
make:
4344
chdir: /tmp/pgrouting-{{ pgrouting_release }}/build
4445
target: install
45-
become: yes
46+
become: yes

‎ansible/tasks/postgres-extensions/12-pljava.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
- name: pljava - build
2727
become: yes
2828
shell:
29-
cmd: mvn clean install
29+
cmd: mvn -T 1C clean install -Dmaven.test.skip -DskipTests -Dmaven.javadoc.skip=true
3030
chdir: /tmp/pljava-{{ pljava_release }}
3131

3232
- name: pljava - install
@@ -61,4 +61,4 @@
6161
become: yes
6262
file:
6363
path: ~/.m2
64-
state: absent
64+
state: absent

‎ansible/tasks/postgres-extensions/13-plv8.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,4 +46,4 @@
4646
make:
4747
chdir: /tmp/plv8
4848
target: install
49-
become: yes
49+
become: yes

‎ansible/tasks/setup-postgres.yml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,12 +63,12 @@
6363

6464
- name: Postgres - build
6565
make:
66-
target: world
66+
target: world-bin
6767
chdir: /tmp/postgresql-{{ postgresql_release }}
6868

6969
- name: Postgres - install
7070
make:
71-
target: install-world
71+
target: install-world-bin
7272
chdir: /tmp/postgresql-{{ postgresql_release }}
7373

7474
# Create postgres user
@@ -99,7 +99,7 @@
9999
state: directory
100100
owner: postgres
101101
group: postgres
102-
mode: 0664
102+
mode: 0775
103103
with_items:
104104
- '/etc/postgresql-custom'
105105

@@ -109,6 +109,7 @@
109109
state: touch
110110
owner: postgres
111111
group: postgres
112+
mode: 0664
112113
with_items:
113114
- 'generated-optimizations.conf'
114115
- 'custom-overrides.conf'

‎ansible/tasks/setup-wal-g.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
make:
4545
chdir: /tmp/wal-g
4646
target: pg_build
47+
jobs: 16
4748
params:
4849
GOBIN: "/usr/local/bin"
4950
PATH: "{{ ansible_env.PATH }}:/usr/local/go/bin"

‎ansible/vars.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,10 @@ node_exporter_release_checksum:
3434
arm64: sha256:eb5e7d16f18bb3272d0d832986fc8ac6cb0b6c42d487c94e15dabb10feae8e04
3535
amd64: sha256:8c1f6a317457a658e0ae68ad710f6b4098db2cad10204649b51e3c043aa3e70d
3636

37-
adminapi_release: 0.16.1
37+
adminapi_release: 0.16.2
3838
adminapi_release_checksum:
39-
arm64: sha256:96ad2a67063882c77ff63b73a2bf82e9af1bae837a33070484593eaca28b05b5
40-
amd64: sha256:5e82ac02ef43b9c962f840a0fb5e93b11969ab58e11b087afd8685d25d4c8cdb
39+
arm64: sha256:94fc9d916ce7b54c20e4cb9e0898a2aed205436ee604663636f38a40183163f3
40+
amd64: sha256:134be79a98416e8fcefe2bebf85c6fad56d23c0f99252c8566889c879b9651b5
4141

4242
# Postgres Extensions
4343
postgis_release: "3.1.4"

‎common.vars.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
{
2-
"postgres-version": "14.1.0.15-rc4"
2+
"postgres-version": "14.1.0.15-rc7"
33
}

‎development-arm.vars.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{
2-
"ami_regions": "ap-southeast-1",
2+
"ami_regions": "us-east-1",
33
"arch": "arm64",
44
"environment": "dev",
5-
"instance-type": "t4g.2xlarge",
6-
"region": "ap-southeast-1",
7-
"ubuntu-2004": "ami-077adae4d983338da"
5+
"instance-type": "c6g.4xlarge",
6+
"region": "us-east-1",
7+
"ubuntu-2004": "ami-0b49a4a6e8e22fa16"
88
}

0 commit comments

Comments
 (0)