Skip to content

Add strip_prefix parameter to py_image_layer #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add strip_prefix parameter to py_image_layer #515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e120c85
Add strip_prefix parameter to py_image_layer
Jan 26, 2025
f766cb2
Add unit test
Jan 26, 2025
e6229d0
Fix formatting
Jan 26, 2025
91e0622
Update docs
Jan 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/py_image_layer.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

37 changes: 19 additions & 18 deletions py/private/py_image_layer.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,40 +44,40 @@ default_layer_groups = {
"interpreter": "\\\\.runfiles/.*python.*-.*/",
}

def _split_mtree_into_layer_groups(name, root, groups, group_names, **kwargs):
def _split_mtree_into_layer_groups(name, root, strip_prefix, groups, group_names, **kwargs):
mtree_begin_blocks = "\n".join([
'print "#mtree" >> "$(RULEDIR)/%s.%s.manifest.spec";' % (name, gn)
'print "#mtree" >> "$(RULEDIR)/{name}.{gn}.manifest.spec";'.format(name = name, gn = gn)
for gn in group_names
])

# When an mtree entry matches a layer group, it will be moved into the mtree
# for that group.
ifs = "\n".join([
"""\
if ($$1 ~ "%s") {
print $$0 >> "$(RULEDIR)/%s.%s.manifest.spec";
if ($$1 ~ "{regex}") {{
print $$0 >> "$(RULEDIR)/{name}.{gn}.manifest.spec";
next
}""" % (regex, name, gn)
}}""".format(regex = regex, name = name, gn = gn)
for (gn, regex) in groups.items()
])

cmd = """\
awk < $< 'BEGIN {
%s
}
{
awk < $< 'BEGIN {{
{mtree_begin_blocks}
}}
{{
# Exclude .whl files from container images
if ($$1 ~ ".whl") {
if ($$1 ~ ".whl") {{
next
}
}}
# Move everything under the specified root
sub(/^/, ".%s")
sub(/^{strip_prefix}/, ".{root}")
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this line is the primary change. Here we append strip_prefix to the beginning matcher, so that it is taken out of the path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strip_prefix needs more than this i believe, see an example logic.

https://github.com/bazel-contrib/bazel-lib/blob/c6707f1de817296befc9c238ddb0c27ed43a7c46/lib/private/modify_mtree.awk#L3-L31

# Match by regexes and write to the destination.
%s
{ifs}
# Every line that did not match the layer groups will go into the default layer.
print $$0 >> "$(RULEDIR)/%s.default.manifest.spec"
}'
""" % (mtree_begin_blocks, root, ifs, name)
print $$0 >> "$(RULEDIR)/{name}.default.manifest.spec"
}}'
""".format(mtree_begin_blocks = mtree_begin_blocks, root = root, strip_prefix = strip_prefix, ifs = ifs, name = name)

native.genrule(
name = "{}_manifests".format(name),
Expand All @@ -90,7 +90,7 @@ awk < $< 'BEGIN {
**kwargs
)

def py_image_layer(name, binary, root = "/", layer_groups = {}, compress = "gzip", tar_args = [], compute_unused_inputs = 1, platform = None, **kwargs):
def py_image_layer(name, binary, root = "/", strip_prefix = "", layer_groups = {}, compress = "gzip", tar_args = [], compute_unused_inputs = 1, platform = None, **kwargs):
"""Produce a separate tar output for each layer of a python app

> Requires `awk` to be installed on the host machine/rbe runner.
Expand All @@ -115,6 +115,7 @@ def py_image_layer(name, binary, root = "/", layer_groups = {}, compress = "gzip
name: base name for targets
binary: a py_binary target
root: Path to where the layers should be rooted. If not specified, the layers will be rooted at the workspace root.
strip_prefix: Prefix to strip from files in the tar. This forms part of a regex, so certain characters must be escaped (notably `/`).
layer_groups: Additional layer groups to create. They are used to group files into layers based on their path. In the form of: ```{"<name>": "regex_to_match_against_file_paths"}```
compress: Compression algorithm to use. Default is gzip. See: https://github.com/bazel-contrib/bazel-lib/blob/main/docs/tar.md#tar_rule-compress
compute_unused_inputs: Whether to compute unused inputs. Default is 1. See: https://github.com/bazel-contrib/bazel-lib/blob/main/docs/tar.md#tar_rule-compute_unused_inputs
Expand All @@ -140,7 +141,7 @@ def py_image_layer(name, binary, root = "/", layer_groups = {}, compress = "gzip
groups = dict(groups, **default_layer_groups)
group_names = groups.keys() + ["default"]

_split_mtree_into_layer_groups(name, root, groups, group_names, **kwargs)
_split_mtree_into_layer_groups(name, root, strip_prefix, groups, group_names, **kwargs)

# Finally create layers using the tar rule
srcs = []
Expand Down
15 changes: 15 additions & 0 deletions py/tests/py_image_layer/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,18 @@ assert_tar_listing(
actual = [":my_app_layers"],
expected = ":my_app_layers.listing",
)

# Case 2: With `strip_prefix` and `root`
py_image_layer(
name = "my_app_layers_strip_prefix",
binary = ":my_app_bin",
platform = ":linux_amd64",
strip_prefix = "py\\/tests\\/py_image_layer\\/my_app_bin",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be possible to pass this string to awk -v strip_prefix= so that we don't have to escape slashes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

basically pass this information to awk using its -v strip_prefix=/py/test/my_app_bin and use strip_prefix variable within awk script.

root = "/app",
)

assert_tar_listing(
name = "my_app_layers_strip_prefix_test",
actual = [":my_app_layers_strip_prefix"],
expected = ":my_app_layers_strip_prefix.listing",
)
Loading
Loading