Skip to content

Commit 06fec96

Browse files
jacksonkasi0jacksonkasi0
committed
initial commit
0 parents  commit 06fec96

26 files changed

+3856
-0
lines changed

‎.dockerignore

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Ignore logs
2+
# logs
3+
# *.log
4+
# *.out
5+
6+
# Ignore backups and generated files
7+
backups/
8+
*.tar
9+
10+
# Ignore Deno's cache and config files
11+
.deno/
12+
deno.lock
13+
deno_cache/
14+
15+
# Ignore Git files
16+
.git
17+
.gitignore
18+
19+
# Ignore editor files
20+
*.swp
21+
*.swo
22+
.vscode/
23+
.idea/
24+
25+
# Ignore build outputs (if any)
26+
dist/
27+
build/

‎.env.example

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
AUTOMATIC_BACKUP=true # true | false
2+
3+
BACKUP_DB_URL=
4+
RESTORE_DB_URL=
5+
6+
# R2 Folder Dir
7+
BACKUP_DIR="backups/<project-name>"
8+
9+
# R2
10+
BUCKET_NAME=
11+
ACCESS_KEY_ID=
12+
SECRET_ACCESS_KEY=
13+
ACCOUNT_ID=
14+

‎.todo

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# TODO:
2+
3+
- [ ] Support multiple database auto-backups with customizable backup policies
4+
- [ ] Retry on failure with customizable settings
5+
- [ ] Trigger an email notification if a failure occurs
6+
- [ ] Make it serverless (AWS Lambda – cost-effective)

‎.vscode/settings.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"deno.enable": true
3+
}

‎Dockerfile.backup

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
2+
# Use Deno runtime as base image
3+
FROM denoland/deno:alpine-2.1.0
4+
5+
# Set working directory
6+
WORKDIR /app
7+
8+
# Copy source files and deno.json explicitly
9+
COPY . .
10+
COPY deno.json /app/deno.json
11+
12+
# Update package index and install required packages
13+
RUN apk update && apk add --no-cache postgresql16-client bash curl cronie dos2unix
14+
15+
# Add cron job for daily execution of the backup task
16+
RUN echo "0 0 * * * /app/run_automate.sh >> /var/log/cron.log 2>&1" > /etc/crontabs/root
17+
18+
# Expose port for monitoring (if needed)
19+
EXPOSE 8000
20+
21+
# Copy the startup and wrapper scripts
22+
COPY start.sh /app/start.sh
23+
COPY run_automate.sh /app/run_automate.sh
24+
25+
# Ensure scripts have executable permissions and correct line endings
26+
RUN chmod +x /app/start.sh /app/run_automate.sh && dos2unix /app/start.sh /app/run_automate.sh
27+
28+
# Use the startup script as the entrypoint
29+
ENTRYPOINT ["/bin/bash", "/app/start.sh"]

‎Dockerfile.restore

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Use Deno runtime as base image
2+
FROM denoland/deno:alpine-2.1.0
3+
4+
WORKDIR /app
5+
COPY . .
6+
7+
RUN apk update && apk add --no-cache postgresql16-client
8+
EXPOSE 8000
9+
10+
CMD ["deno", "task", "run:restore-r2", "--key=backups/<project-name>/backup_1732267131464.tar"]

‎README.md

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# PostgreSQL DB Backup & Restore via Docker & Deno
2+
3+
This project provides a system to automate the backup and restore of a PostgreSQL database using **Deno**, **Docker**, and **Cloudflare R2**. It supports local backups, restoring from local files, and restoring directly from Cloudflare R2.
4+
5+
---
6+
7+
## **Technologies Used**
8+
9+
- **Deno**: Runtime for the scripts and automation.
10+
- **Docker**: Containerization for portable backup/restore workflows.
11+
- **PostgreSQL**: Database used for backups and restores.
12+
- **PostgreSQL 16 Client**: Installed in the container for backup and restore operations. *(Note: Update the `postgresql<x>-client` version in the Dockerfile if your database uses a different version.)*
13+
- **Cloudflare R2**: Object storage for storing backup files.
14+
- **Winston**: Logging library for structured and color-coded logs.
15+
- **pg_restore**: PostgreSQL's native tool for restoring database dumps.
16+
- **Cronie**: Lightweight cron daemon for scheduling backup and restore tasks in the container.
17+
18+
## **Getting Started**
19+
20+
### **Requirements**
21+
22+
1. **Deno** installed ([Install Guide](https://deno.land/manual/getting_started/installation)).
23+
2. **Docker** installed ([Install Guide](https://docs.docker.com/get-docker/)).
24+
3. A `.env` file in the root directory containing:
25+
26+
```env
27+
AUTOMATIC_BACKUP=If set to `true`, it will trigger a cron job to perform a backup.
28+
BACKUP_DB_URL=your_postgresql_connection_url
29+
RESTORE_DB_URL=your_postgresql_connection_url
30+
ACCESS_KEY_ID=your_r2_access_key_id
31+
SECRET_ACCESS_KEY=your_r2_secret_access_key
32+
BACKUP_DIR=your_r2_bucket_db_backup_dir
33+
ACCOUNT_ID=your_r2_account_id
34+
BUCKET_NAME=your_r2_bucket_name
35+
```
36+
37+
4. **Cloudflare R2** storage setup with appropriate access credentials.
38+
39+
---
40+
41+
## **Available Tasks**
42+
43+
### **Backup Task**
44+
45+
Creates a backup of the database and uploads it to Cloudflare R2.
46+
47+
**Command:**
48+
49+
```sh
50+
deno task run:backup
51+
```
52+
53+
**Process:**
54+
55+
1. Generates a backup file in the local `db_backups/<folder>` directory.
56+
2. Uploads the backup to the specified Cloudflare R2 bucket.
57+
58+
---
59+
60+
### **Restore Task**
61+
62+
Restores the database from a local backup file.
63+
64+
**Command:**
65+
66+
```sh
67+
deno task run:restore
68+
```
69+
70+
**Process:**
71+
72+
1. Restores the database from a specified local backup file.
73+
2. Ensure the local backup file exists in the `db_backups/<folder>` directory.
74+
75+
---
76+
77+
### **Restore From R2 Task**
78+
79+
Restores the database by downloading a backup file from Cloudflare R2.
80+
81+
**Command:**
82+
83+
```sh
84+
deno task run:restore-r2 --key=backups/<folder>/backup_1732266xxxxxx.tar
85+
```
86+
87+
**Process:**
88+
89+
1. Downloads the specified backup file from Cloudflare R2 to a temporary location.
90+
2. Restores the database using the downloaded file.
91+
3. Deletes the temporary file after restoring.
92+
93+
---
94+
95+
### **Automate Task**
96+
97+
Automates the backup process, including uploading to R2 and cleanup.
98+
99+
**Command:**
100+
101+
```sh
102+
deno task automate
103+
```
104+
105+
**Process:**
106+
107+
1. Creates a backup of the database.
108+
2. Uploads the backup file to Cloudflare R2.
109+
3. Cleans up local backup files older than the retention period.
110+
4. Deletes old R2 backups based on the retention policy.
111+
112+
---
113+
114+
### **Restore From R2 Task (Script)**
115+
116+
Runs the restore process from R2 using Docker for containerized execution.
117+
118+
**Setup:**
119+
120+
1. Set `RESTORE_DB_URL` in the `.env` file.
121+
2. Modify the `run_restore.sh` script to include the desired R2 object key.
122+
123+
**Command:**
124+
125+
```sh
126+
chmod +x run_restore.sh
127+
./run_restore.sh
128+
```
129+
130+
**Process:**
131+
132+
1. Builds and runs a Docker container using `Dockerfile.restore`.
133+
2. Passes environment variables and R2 object key to the container.
134+
3. Restores the database and removes the temporary container after completion.
135+
136+
---
137+
138+
## **Folder Structure**
139+
140+
```plaintext
141+
.
142+
├── .dockerignore # Specifies files and directories to ignore in Docker builds
143+
├── .env # Environment variables file
144+
├── .env.example # Example environment variables file
145+
├── .vscode/
146+
│ └── settings.json # VS Code-specific configuration
147+
├── deno.json # Deno configuration and tasks
148+
├── deno.lock # Dependency lock file for Deno
149+
├── Dockerfile.backup # Dockerfile for the backup container
150+
├── Dockerfile.restore # Dockerfile for the restore container
151+
├── railway.toml # Railway platform configuration for deployment
152+
├── README.md # Documentation
153+
├── run_automate.sh # Script to trigger automate task
154+
├── run_restore.sh # Script to trigger restore from R2 task via Docker
155+
├── start.sh # Startup script for cron-based backups
156+
├── src/
157+
│ ├── backup/ # Backup-related functionality
158+
│ │ ├── cleanCloudBackups.ts # Cleans up old backups from R2
159+
│ │ ├── cleanLocalBackups.ts # Cleans up old backups locally
160+
│ │ └── createBackup.ts # Logic to create a PostgreSQL backup
161+
│ ├── restore/ # Restore-related functionality
162+
│ │ └── restoreBackup.ts # Logic to restore a PostgreSQL backup
163+
│ ├── utils/ # Utility functions and helpers
164+
│ │ ├── r2/
165+
│ │ │ ├── cloudflareR2.ts # Cloudflare R2 integration logic
166+
│ │ │ ├── index.ts # R2 helper functions
167+
│ │ │ └── multipartUploadToR2.ts # Logic for multipart uploads to R2
168+
│ │ ├── execCommand.ts # Helper for executing shell commands
169+
│ │ └── logger.ts # Custom logger with color-coded log levels
170+
│ ├── config.ts # Configuration file for environment variables
171+
│ ├── index.ts # Entry point for backup/restore tasks
172+
│ └── automate.ts # Automation script for scheduled backups
173+
```
174+
175+
---
176+
177+
## **Environment Variables**
178+
179+
Ensure the following environment variables are defined in the `.env` file:
180+
181+
```env
182+
# PostgreSQL Connection URLs
183+
BACKUP_DB_URL=your_postgresql_backup_url
184+
RESTORE_DB_URL=your_postgresql_restore_url
185+
186+
# Cloudflare R2 Configuration
187+
ACCESS_KEY_ID=your_r2_access_key_id
188+
SECRET_ACCESS_KEY=your_r2_secret_access_key
189+
ACCOUNT_ID=your_r2_account_id
190+
BUCKET_NAME=your_r2_bucket_name
191+
```
192+
193+
---
194+
195+
## **Cron Integration**
196+
197+
To automate backups with cron:
198+
199+
1. Add the `deno task automate` command to your crontab file.
200+
2. Example entry for a daily backup at midnight:
201+
202+
```cron
203+
0 0 * * * deno task automate >> /var/log/db_backup.log 2>&1
204+
```
205+
206+
---
207+
208+
## **Additional Notes**
209+
210+
- Automatic backup is configured in `Dockerfile.backup`. It is designed for deployment on the **Railway.app** platform via the `railway.toml` file to enable automatic PostgreSQL backups. If you are using a different platform, update the configuration to run `Dockerfile.backup` by default.
211+
- Ensure your R2 bucket permissions allow read and write operations.
212+
- Use the `run_restore.sh` script for containerized restore processes.
213+
- For schema-specific restores, customize the `restoreBackup.ts` function as needed.

‎deno.json

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"tasks": {
3+
"run:backup": "deno run --allow-read --allow-env --allow-net --allow-run src/index.ts --mode=backup",
4+
"run:restore": "deno run --allow-read --allow-env --allow-net --allow-run src/index.ts --mode=restore",
5+
"run:restore-r2": "deno run --allow-read --allow-env --allow-sys --allow-net --allow-run --allow-write src/index.ts --mode=restore-r2",
6+
"automate": "deno run --allow-run --allow-sys --allow-env --allow-net --allow-read --allow-write src/automate.ts"
7+
},
8+
"imports": {
9+
"@/":"./src/",
10+
"@aws-sdk/client-s3": "npm:@aws-sdk/client-s3@^3.697.0",
11+
"@std/fs": "jsr:@std/fs@^1.0.5",
12+
"winston": "npm:winston@^3.17.0"
13+
}
14+
}

0 commit comments

Comments
 (0)