[Examples] Add CLI tool for starting a WordPress site from static files (#5)

Adds a CLI tool to import static files into a new Playground site. For
example, this command:

```bash
> npx create-wp-site.tar.gz crawler https://adamadam.blog --output-dir=./site
Importing static files from https://adamadam.blog
Starting the import

Import finished! See your imported content at: 
http://127.0.0.1:9400/adam-s-perspective-wordpress-playground-and-more
```

Crawls my blog, imports the content to a new WordPress site stored in
`./site` directory, and starts a local server for that site.

## Supported files and data sources

* Supported formats: Markdown, HTML, XHTML, WXR, EPUB
* Supported data sources: Local drive, URL, remote Git repository, a
remote website

## Implementation

This PR ships a WordPress plugin with a CLI script that runs the import.

The plugin and its dependencies are installed via Blueprints and
WordPress Playground CLI (`@wp-playground/cli`).

The import is done via the streaming parsers and readers provided in
this repo. They follow the [Data
Liberation](WordPress/data-liberation#78)
assumptions and are PHP-only, dependency-free, streaming, re-entrant,
report progress, parallelize downloads, etc. etc. This code would work
on any WordPress host (except for maybe PHP 7 support – but it's easy to
add).

Finally, this PR also ships a "Playground runtime protocol" – a wrapper
over `post_message_to_js` that allows the PHP script to control the
Playground runtime.

This PR effectively ships the following import pipeline as a CLI script:

![CleanShot 2025-03-12 at 15 12
11@2x](https://github.com/user-attachments/assets/2203b29d-d1b1-43eb-bc60-df3d93002793)

The same logic could be easily reused in the browser via Playground.

## Usage examples

Import a subset of Gutenberg documentation focused on data basics:

```bash
bun index.js \
  git https://github.com/WordPress/gutenberg.git \
  --branch=trunk \
  --path-in-repo=docs/how-to-guides/data-basics/ \
  --media-url=https://developer.wordpress.org/files/ \
  --media-url=https://raw.githubusercontent.com/WordPress/gutenberg/HEAD/docs/ \
  --source-site-url=https://developer.wordpress.org/block-editor/how-to-guides/data-basics/ \
  --additional-site-urls=https://developer.wordpress.org/docs/how-to-guides/data-basics/
```

Import the complete Gutenberg documentation:

```bash
bun index.js \
  git https://github.com/WordPress/gutenberg.git \
  --branch=trunk \
  --path-in-repo=docs/ \
  --media-url=https://developer.wordpress.org/files/ \
  --media-url=https://raw.githubusercontent.com/WordPress/gutenberg/HEAD/docs/ \
  --source-site-url=https://developer.wordpress.org/block-editor/ \
  --additional-site-urls=https://developer.wordpress.org/docs/
```

Import content from Adam's blog using a crawler:

```bash
bun index.js crawler https://adamadam.blog
```

Import accessibility testing content from a WordPress XML export:

```bash
bun index.js wxr https://raw.githubusercontent.com/wpaccessibility/a11y-theme-unit-test/master/a11y-theme-unit-test-data.xml
```

Import content from an EPUB file:

```bash
bun index.js epub https://github.com/IDPF/epub3-samples/releases/download/20230704/childrens-literature.epub
```

Import Gutenberg documentation from a local repository checkout:

```bash
bun index.js path ../../../gutenberg/docs/how-to-guides/
```

Import documentation from the WordPress Playground project:

```bash
bun index.js \
  git https://github.com/WordPress/wordpress-playground.git \
  --branch=trunk \
  --path-in-repo=packages/docs/site/docs/blueprints/ \
  --media-url=https://wordpress.github.io/wordpress-playground/ \
  --source-site-url=https://wordpress.github.io/
```

Import Bootstrap 5.3 documentation:

```bash
bun index.js \
  git https://github.com/twbs/bootstrap.git \
  --branch=gh-pages \
  --path-in-repo=docs/5.3/ \
  --media-url=https://getbootstrap.com/docs/5.3/ \
  --source-site-url=https://getbootstrap.com/docs/5.3/ \
  --additional-site-urls=https://getbootstrap.com/docs/
```

Import Laravel documentation:

```bash
bun index.js \
  git https://github.com/laravel/docs.git \
  --path-in-repo=/ \
  --branch=12.x \
  --source-site-url=https://laravel.com/docs/
```

Import internal documentation from the CPython repository:

```bash
bun index.js \
  git https://github.com/python/cpython.git \
  --branch=main \
  --path-in-repo=InternalDocs/ \
  --source-site-url=https://raw.githubusercontent.com/python/cpython/refs/heads/main/InternalDocs/
```

Import content from the Fullstack GraphQL book repository:

```bash
bun index.js \
  git https://github.com/GraphQLCollege/fullstack-graphql.git \
  --branch=master \
  --path-in-repo=manuscript/ \
  --source-site-url=https://raw.githubusercontent.com/GraphQLCollege/fullstack-graphql/refs/heads/master/manuscript/
```

Import content from the CPP WASM book repository:

```bash
bun index.js \
  git https://github.com/3dgen/cppwasm-book.git \
  --branch=master \
  --path-in-repo=en/ \
  --source-site-url=https://raw.githubusercontent.com/3dgen/cppwasm-book/refs/heads/master/en/
```

Author: adamziel

.github/workflows/release.yml

@@ -80,3 +80,13 @@ jobs:
                   asset_path: ./dist/plugins/static-files-editor.zip
                   asset_name: static-files-editor.zip
                   asset_content_type: application/zip
+
+            - name: Upload Import Static Files Example Asset
+              uses: actions/upload-release-asset@v1
+              env:
+                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+              with:
+                  upload_url: ${{ steps.create_release.outputs.upload_url }}
+                  asset_path: ./dist/examples/create-wp-site.tar.gz
+                  asset_name: create-wp-site.tar.gz
+                  asset_content_type: application/gzip

.gitignore

@@ -1,6 +1,8 @@
 vendor/
 **/*/vendor/
 opfs/
+node_modules
+.cursor
 rest/
 dist/
 outdir/
@@ -26,4 +28,5 @@ buntest.php
 blueprint-url-updater.json
 components/DataLiberation/Tests/test-output-html/
 components/DataLiberation/Tests/test-output-md
-blueprint-dev.json
+blueprint-dev.json
+examples/create-wp-site/data-liberation.zip

bin/build-all.sh

@@ -2,3 +2,4 @@
 
 bash bin/build-libraries-phar.sh
 bash bin/build-plugins.sh
+bash bin/build-examples.sh

bin/build-examples.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+PROJECT_DIR=$SCRIPT_DIR/..
+
+rm -rf $PROJECT_DIR/dist/examples
+mkdir -p $PROJECT_DIR/dist/examples
+
+cp $PROJECT_DIR/dist/plugins/data-liberation.zip $PROJECT_DIR/examples/create-wp-site/data-liberation.zip
+cp -r $PROJECT_DIR/examples/create-wp-site/ $PROJECT_DIR/dist/package
+cd $PROJECT_DIR/dist
+tar -czvf examples/create-wp-site.tar.gz package/{*.js,*.json,*.php,*.zip,cli,playground-protocol,README.md}
+rm -rf package

blueprint-dev.json renamed to blueprint-dev.json.dist

@@ -140,6 +140,7 @@ private function next_url_attribute() {
 
 			if ( is_string( $url_maybe ) ) {
 				$parsed_url = WPURL::parse( $url_maybe, $this->base_url_string );
+
 				if ( false === $parsed_url ) {
 					return false;
 				}
@@ -177,31 +178,41 @@ private function next_url_block_attribute() {
 		return false;
 	}
 
-	public function set_raw_url( $new_url ) {
+	/**
+	 * Replaces the currently matched URL with a new one.
+	 *
+	 * @param string $raw_url The raw URL.
+	 * @param URL    $parsed_url The parsed version of the raw URL. It is required
+	 *                           as $raw_url might be a relative URL pointing to a different
+	 *                           host than this processor's base URL.
+	 * @return bool True if the URL was set, false otherwise.
+	 */
+	public function set_url( $raw_url, $parsed_url ) {
 		if ( null === $this->raw_url ) {
 			return false;
 		}
-		$this->raw_url = $new_url;
+		$this->raw_url    = $raw_url;
+		$this->parsed_url = $parsed_url;
 		switch ( parent::get_token_type() ) {
 			case '#tag':
 				$attr = $this->get_inspected_attribute_name();
 				if ( false === $attr ) {
 					return false;
 				}
-				$this->set_attribute( $attr, $new_url );
+				$this->set_attribute( $attr, $raw_url );
 
 				return true;
 
 			case '#block-comment':
-				return $this->set_block_attribute_value( $new_url );
+				return $this->set_block_attribute_value( $raw_url );
 
 			case '#text':
 				if ( null === $this->url_in_text_processor ) {
 					return false;
 				}
 				$this->url_in_text_node_updated = true;
 
-				return $this->url_in_text_processor->set_raw_url( $new_url );
+				return $this->url_in_text_processor->set_raw_url( $raw_url );
 		}
 	}
 
@@ -216,7 +227,7 @@ public function set_raw_url( $new_url ) {
 	 *        relative URLs in text nodes. On the other hand, the detection is performed
 	 *        by this WPURL_In_Text_Processor class so maybe the two do go hand in hand?
 	 */
-	public function replace_base_url( URL $to_url ) {
+	public function replace_base_url( URL $to_url, ?URL $base_url = null ) {
 		$updated_url = clone $this->get_parsed_url();
 
 		$updated_url->hostname = $to_url->hostname;
@@ -227,19 +238,22 @@ public function replace_base_url( URL $to_url ) {
 		$from_url      = $this->get_parsed_url();
 		$from_pathname = $from_url->pathname;
 		$to_pathname   = $to_url->pathname;
-		if ( $this->base_url_object->pathname !== $to_pathname ) {
-			$base_pathname_with_trailing_slash = rtrim( $this->base_url_object->pathname, '/' ) . '/';
+
+		$base_url = $base_url ?? $this->base_url_object;
+		if ( $base_url->pathname !== $to_pathname ) {
+			$base_pathname_with_trailing_slash = rtrim( $base_url->pathname, '/' ) . '/';
 			$decoded_matched_pathname          = urldecode_n(
 				$from_pathname,
 				strlen( $base_pathname_with_trailing_slash )
 			);
 			$to_pathname_with_trailing_slash   = rtrim( $to_pathname, '/' ) . '/';
-			$updated_url->pathname             =
-				$to_pathname_with_trailing_slash .
-					substr(
-						$decoded_matched_pathname,
-						strlen( $base_pathname_with_trailing_slash )
-					);
+			$remaining_pathname                =
+				substr(
+					$decoded_matched_pathname,
+					strlen( $base_pathname_with_trailing_slash )
+				);
+
+			$updated_url->pathname = $to_pathname_with_trailing_slash . $remaining_pathname;
 		}
 
 		/*
@@ -263,18 +277,8 @@ public function replace_base_url( URL $to_url ) {
 			return false;
 		}
 
-		$is_relative = (
-			// The URL-rewriting specific logic. We make an assumption that only
-			// absolute URLs are detected in text nodes.
-			// @TODO: Verify this assumption, evaluate whether this is the right
-			// place to place this logic. Perhaps this *method* could be
-			// decoupled into two separate *functions*?
-			$this->get_token_type() !== '#text' &&
-			! str_starts_with( $this->get_raw_url(), 'http://' ) &&
-			! str_starts_with( $this->get_raw_url(), 'https://' )
-		);
-		if ( ! $is_relative ) {
-			$this->set_raw_url( $new_raw_url );
+		if ( ! $this->is_url_relative() ) {
+			$this->set_url( $new_raw_url, $updated_url );
 			return true;
 		}
 
@@ -286,10 +290,31 @@ public function replace_base_url( URL $to_url ) {
 			$new_relative_url .= $updated_url->hash;
 		}
 
-		$this->set_raw_url( $new_relative_url );
+		$this->set_url( $new_relative_url, $updated_url );
 		return true;
 	}
 
+	/**
+	 * Returns true if the currently matched URL is relative.
+	 *
+	 * @return bool Whether the currently matched URL is relative.
+	 */
+	public function is_url_relative() {
+		return (
+			! WPURL::can_parse( $this->get_raw_url() ) &&
+			// only absolute URLs are detected in text nodes.
+			'#text' !== $this->get_token_type()
+		);
+	}
+	/**
+	 * Returns true if the currently matched URL is absolute.
+	 *
+	 * @return bool Whether the currently matched URL is absolute.
+	 */
+	public function is_url_absolute() {
+		return WPURL::can_parse( $this->get_raw_url() );
+	}
+
 	public function get_inspected_attribute_name() {
 		if ( '#tag' !== $this->get_token_type() ) {
 			return false;

components/DataLiberation/BlockMarkup/BlockMarkupUrlProcessor.php

@@ -30,7 +30,7 @@
  * array(
  *     'post_title' => array( 'My first post' ),
  * )
- * 
+ *
  * @TODO: Satisfy the same test suite as the Gutenberg paste/raw handler.
  *        Look for the test examples (input/output) in the Gutenberg repo.
  * @TODO: Consider option for "presentation" vs "semantics" mode. We can preserve
@@ -59,6 +59,9 @@ public function consume() {
 							break;
 						}
 						$this->append_rich_text( htmlspecialchars( $this->markup_processor->get_modifiable_text() ) );
+						if ( in_array( $this->markup_processor->get_tag(), array( 'H1', 'H2', 'H3', 'H4', 'H5', 'H6' ) ) ) {
+							$this->on_title_candidate( $this->markup_processor->get_modifiable_text() );
+						}
 						break;
 					case '#tag':
 						$this->handle_tag();
@@ -92,11 +95,16 @@ private function handle_tag() {
 		$is_void_tag = ! $html->expects_closer() && ! $html->is_tag_closer();
 		if ( $is_void_tag ) {
 			switch ( $tag ) {
+				case 'TITLE':
+					$this->on_title_candidate( $html->get_modifiable_text() );
+					break;
 				case 'META':
 					$key   = $html->get_attribute( 'name' );
 					$value = $html->get_attribute( 'content' );
 					if ( ! array_key_exists( $key, $this->metadata ) ) {
-						$this->metadata[ $key ] = array();
+						if ( $key ) {
+							$this->metadata[ $key ] = array();
+						}
 					}
 					switch ( $html->get_attribute( 'type' ) ) {
 						case 'integer':
@@ -107,7 +115,9 @@ private function handle_tag() {
 							break;
 						// @TODO: Discuss what would support for other types look like.
 					}
-					$this->metadata[ $key ][] = $value;
+					if ( $key ) {
+						$this->metadata[ $key ][] = $value;
+					}
 					break;
 				case 'IMG':
 					$template = new \WP_HTML_Tag_Processor( '<img>' );
@@ -286,6 +296,20 @@ private function handle_tag() {
 		}
 	}
 
+	private function on_title_candidate( $text ) {
+		if ( ! array_key_exists( 'post_title', $this->metadata ) ) {
+			$this->metadata['post_title'] = array(
+				$text,
+			);
+		}
+		if ( ! array_key_exists( 'post_name', $this->metadata ) ) {
+			$this->metadata['post_name'] = array(
+				// @TODO: Slugify
+				$text,
+			);
+		}
+	}
+
 	/**
 	 * Checks whether the given tag is an inline formatting element
 	 * that we want to preserve when parsing rich text. For example,

components/DataLiberation/DataFormatConsumer/MarkupProcessorConsumer.php

@@ -46,6 +46,7 @@ public function next_entity() {
 		$all_metadata   = $this->metadata;
 		$post_fields    = array();
 		$other_metadata = array();
+
 		foreach ( $all_metadata as $key => $values ) {
 			if ( in_array( $key, ImportEntity::POST_FIELDS, true ) ) {
 				$post_fields[ $key ] = $values[0];
@@ -54,8 +55,9 @@ public function next_entity() {
 			}
 		}
 
-		$post_fields['post_id']      = $this->post_id;
-		$post_fields['post_content'] = $this->block_markup;
+		$post_fields['post_id']         = $this->post_id;
+		$post_fields['post_content']    = $this->block_markup;
+		$post_fields['parsed_metadata'] = $all_metadata;
 
 		// In Markdown, the frontmatter title can be a worse title candidate than
 		// the first H1 block. In block markup exports, it will be the opposite.