Initial commit

Author: jeanas

README.md

@@ -0,0 +1,62 @@
+# Pygments plugin examples
+
+This repository contains an example of the project scaffolding needed to write a
+plugin lexer/formatter/style/filter for the [Pygments](https://pygments.org)
+syntax highlighter.
+
+Roadmap:
+
+```
+.
+├── example_filter.py    # An example filter
+├── example_formatter.py # An example formatter
+├── example_lexer.py     # An example lexer
+├── example_style.py     # An example style
+├── pyproject.toml       # The Python package metadata file
+├── README.md            # You are here
+└── test.exmpl           # A test file in the mockup language of the example lexer
+```
+
+To be usable, plugins must be installed. If you are running the `pygmentize`
+command, you probably want to use a
+[virtual environment](https://docs.python.org/3/library/venv.html)
+to avoid installing packages globally. For example, here is how to create
+a virtual environment and install this set of plugins into it:
+
+```
+python -m venv venv/
+venv/bin/pip install . # install the project in current directory into the virtual environment
+venv/bin/pygmentize ... # use the pygmentize command from the virtual environment
+```
+
+Alternatively, since this example uses the [Hatch](https://hatch.pypa.io)
+tool, you may use
+
+```
+hatch run pygmentize ...
+```
+
+If you are using Pygments from Python, possibly through a tool like Sphinx,
+mkdocs, etc., then you just need to install the plugin in the same environment
+as the one where you installed Pygments.
+
+If you want to distribute your plugin on PyPI, you should read the
+[packaging user guide](https://packaging.python.org/en/latest/tutorials/packaging-projects).
+
+
+#### License for this template
+
+There isn't much copyrightable content here, but if you are worried about reuse:
+
+Copyright (C) 2023 by Jean Abou Samra <jean@abou-samra.fr>
+
+Permission to use, copy, modify, and/or distribute this software for any purpose
+with or without fee is hereby granted.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
+TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
+THIS SOFTWARE.

example_filter.py

@@ -0,0 +1,11 @@
+"""An example filter for Pygments."""
+
+from pygments.filters import Filter
+from pygments.token import Token
+
+class ExampleFilter(Filter):
+    # This filter replaces all tabs with "<tab>".
+    def filter(self, lexer, stream):
+        for ttype, value in stream:
+            parts = value.split("\t")
+            yield ttype, "<tab>".join(parts)

example_formatter.py

@@ -0,0 +1,41 @@
+"""An example plugin formatter for Pygments."""
+
+from pygments.formatter import Formatter
+
+class ExampleFormatter(Formatter):
+    # This should be the human-readable name of the format.
+    name = "Pygments Plugin Example Format"
+
+    # This is a list of names that can be used to select the formatter.
+    # In this example, we can call the pygmentize command as
+    #
+    #   pygmentize -f example-format
+    #
+    # Also, doing
+    #
+    #   pygments.formatters.find_formatter_class("example-format")
+    #
+    # in Python will find the formatter class.
+    aliases = ["example-format"]
+
+    # This is a list of file patterns that the formatter will
+    # typically be used to produce. In this example, calling
+    #
+    #   pygmentize -o out.exmplfmt
+    #
+    # will automatically select this formatter based on the output
+    # file name. Similarly,
+    #
+    #   pygments.formatters.get_formatter_for_filename("out.exmplfmt")
+    #
+    # will return an instance of this formatter class.
+    filenames = ["*.exmplfmt"]
+
+    def format_unencoded(self, tokensource, out):
+        # This formatter writes each token as [<color>]<string> .
+        for ttype, value in tokensource:
+            while not self.style.styles_token(ttype):
+                ttype = ttype.parent
+            color = self.style.style_for_token(ttype)['color']
+            out.write("[" + (color or "black") + "]")
+            out.write(value)

example_lexer.py

@@ -0,0 +1,66 @@
+"""An example plugin lexer for Pygments."""
+
+from pygments.lexer import RegexLexer
+from pygments.token import Keyword, Text
+
+class ExampleLexer(RegexLexer):
+    # This should be the human-readable name of the language.  In this example,
+    # doing
+    #
+    #   pygments.lexers.find_lexer_class("Pygments Plugin Example Language")
+    #
+    # will return the ExampleLexer class.
+    name = "Pygments Plugin Example Language"
+
+    # This is a list of names that can be used to select the language.
+    # In this example, we can call the pygmentize command as
+    #
+    #   pygmentize -l example-lang
+    #
+    # Also, doing
+    #
+    #   pygments.lexers.find_lexer_class_by_name("example-lang")
+    #
+    # in Python will find the lexer class. This is what many documentation or
+    # site generation tools implicitly do with code blocks, e.g., this in Sphinx
+    # reST:
+    #
+    #   .. code-block:: example-lang
+    #
+    #      your code here
+    #
+    # or this in Markdown:
+    #
+    #   ```example-lang
+    #   your code here
+    #   ```
+    #
+    aliases = ["example-lang"]
+
+    # This is a list of file patterns that will automatically be highlighted
+    # with this lexer. In this example, calling
+    #
+    #   pygmentize file.exmpl
+    #
+    # will automatically highlight file.exmpl with this lexer, without needing
+    # to pass `-l example-lang`. Similarly,
+    #
+    #   pygments.lexers.find_lexer_class_for_filename("file.exmpl")
+    #
+    # will return the ExampleLexer class.
+    filenames = ["*.exmpl"]
+
+    # This is a list of MIME types for the language. In this example,
+    #
+    #   pygments.lexers.get_lexer_for_mimetype("text/x-exmpl")
+    #
+    # will return the ExampleLexer class.
+    mimetypes = ["text/x-exmpl"]
+
+    # This lexer highlights lines that read "foo".
+    tokens = {
+        'root': [
+            (r'foo\n', Keyword),
+            (r'.*\n', Text),
+        ]
+    }

example_style.py

@@ -0,0 +1,10 @@
+"""An example plugin style for Pygments."""
+
+from pygments.style import Style
+from pygments.token import Keyword
+
+class ExampleStyle(Style):
+    styles = {
+        # This style merely highlights keywords in red.
+        Keyword: "#f00",
+    }

pyproject.toml

@@ -0,0 +1,76 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pygments-plugin-scaffolding" # change this to a package name for your plugin
+version = "0.0.1"
+dependencies = ["pygments"]
+readme = "README.md"
+
+
+# Declare plugin lexers in this table. The key (like `example_lexer` below) are
+# not significant. The value has the form `module_name:ClassName`. The lexer
+# class will be imported using `from module_name import ClassName`.
+
+# See the attributes declared in example_lexer.py for how to configure the
+# language names and file patterns.
+
+[project.entry-points."pygments.lexers"]
+example_lexer = "example_lexer:ExampleLexer"
+
+
+# Declare plugin formatters in this table. The key is not significant and the
+# value has the same format as for lexers. See example_formatter.py for
+# configuration options.
+
+[project.entry-points."pygments.formatters"]
+example_formatter = "example_formatter:ExampleFormatter"
+
+
+# Declare plugin styles in this table. The key *is* significant: it is the name
+# the style will be recognized as. In this example, we can use
+#
+#   pygmentize -Ostyle=example-style"
+#
+# on the command line. Also,
+#
+#   pygments.styles.get_style_by_name("example-style")
+#
+# will return the style class.
+
+[project.entry-points."pygments.styles"]
+example-style = "example_style:ExampleStyle"
+
+
+# Declare plugin filters in this table. The key *is* significant: it is the name
+# the filter will be recognized as. In this example, we can use
+#
+#  pygmentize -F example-filter
+#
+# on the command line. Also,
+#
+#   pygments.filters.find_filter_class("example-filter")
+#
+# will return the filter class.
+
+[project.entry-points."pygments.filters"]
+example-filter = "example_filter:ExampleFilter"
+
+
+# This is a test command. Running it should print:
+#
+# [ff0000]foo
+# [black]<tab><tab>bar # tabs here
+#
+# - Our custom lexer highlights "foo" as keyword,
+# - Our custom style uses red (ff0000) for keywords,
+# - Our custom formatter prints colors in brackets,
+# - Our custom filter replaces tabs with "<tab>".
+#
+# Run with
+#
+#   hatch run test
+
+[tool.hatch.envs.default.scripts]
+test = "pygmentize -l example-lang -f example-format -F example-filter -O style=example-style test.exmpl"

test.exmpl

@@ -0,0 +1,2 @@
+foo
+		bar # tabs here