chore(hermetic-build): include .github template updates as part of generation #3723

diegomarquezp · 2025-03-26T21:03:18Z

Part of the fix for #3701 ☕

Approach

The templates come from https://github.com/googleapis/synthtool/tree/bf182cd41d9a7de56092cafcc7befe6b398332f6. The only update was to include generated-files-bot, which is already up to date in all the HW repos.

The .kokoro folder will be a separate follow up task. We will now focus on solving the update of update_generation_config yaml and sh files.

The fix to include the .github folder follows from https://setuptools.pypa.io/en/latest/userguide/datafiles.html:

Glob patterns do not automatically match dotfiles, i.e., directory or file names starting with a dot (.).
To include such files, you must explicitly start the pattern with a dot, e.g. .* to match .gitignore.

Interestingly, this is also the reason cp synthool/gcp/templates/java_library/* ... did not bring folders starting with dot (such as .kokoro) into #2884

Confirming effects in downstream repos

Demos show the results as of b66af92 in

There were no regressions on templated files that were manually modified.

The templates come from https://github.com/googleapis/synthtool/tree/bf182cd41d9a7de56092cafcc7befe6b398332f6 The fix follows from https://setuptools.pypa.io/en/latest/userguide/datafiles.html: ``` Glob patterns do not automatically match dotfiles, i.e., directory or file names starting with a dot (.). To include such files, you must explicitly start the pattern with a dot, e.g. .* to match .gitignore. ```

This reverts commit c25a14a.

diegomarquezp · 2025-03-27T20:07:20Z

hermetic_build/library_generation/generate_library.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash

-set -eo pipefail
+set -exo pipefail


xtrace was the only way I could debug my way out in local development. This decision may have taken longer for someone less familiar with the codebase.

I didn't add -x because I think the extra logs may substantial and this level of logs is not recommended for production.

Do you think it's necessary to add these logs?

I reverted this and explained how this would be useful in the dev guide.

JoeWang1127 · 2025-03-27T20:24:44Z

hermetic_build/DEVELOPMENT.md

@@ -78,7 +78,7 @@ as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/96999197
 2. Move the jar into its well-known location.

   ```shell
-   mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar"
+   mv ~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar "${HOME}/.library_generation/gapic-generator-java.jar"


Is this too much detail? What if a user change the default maven dir?

I thought it may be more convenient for copying the whole line and limit to making a replacement in {version}. I think a non-default maven setup is not a common use case, but still checks out, so I reverted this line.

JoeWang1127 · 2025-03-27T20:25:28Z

hermetic_build/DEVELOPMENT.md

@@ -130,6 +130,9 @@ owl-bot copy-code --version
 The key step is `npm link`, which will make the command available in you current
 shell session.

+If you get a permission denied error when running the command `owl-bot`, try
+relinking owl-bot by running `npm unlink -g` and re-running the steps above.


Why would rerun the same command can solve the permission issue?

npm link has effect in the current shell session. My guess is that what's in an old session may get its permissions overriden.

JoeWang1127 · 2025-03-27T20:29:29Z

hermetic_build/library_generation/setup.py

@@ -24,6 +24,8 @@
            "owlbot/src/poms/*.py",
            "owlbot/templates/clirr/*.j2",
            "owlbot/templates/poms/*.j2",
+            "owlbot/templates/java_library/.github/**/*",
+            "owlbot/templates/java_library/.kokoro/**/*",
            "owlbot/templates/java_library/**/*",


Do we still need this line?

yes, there are files outside .github and .kokoro that this line would include.

JoeWang1127 · 2025-03-27T20:30:07Z

hermetic_build/library_generation/utils/utilities.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash

-set -eo pipefail
+set -exo pipefail


Again, should we include extra logs in production?

I reverted this and explained how this would be useful in the dev guide.

diegomarquezp

Thanks for the review!

diegomarquezp · 2025-03-27T20:33:58Z

hermetic_build/DEVELOPMENT.md

@@ -130,6 +130,9 @@ owl-bot copy-code --version
 The key step is `npm link`, which will make the command available in you current
 shell session.

+If you get a permission denied error when running the command `owl-bot`, try
+relinking owl-bot by running `npm unlink -g` and re-running the steps above.


npm link has effect in the current shell session. My guess is that what's in an old session may get its permissions overriden.

diegomarquezp · 2025-03-27T20:34:44Z

hermetic_build/library_generation/setup.py

@@ -24,6 +24,8 @@
            "owlbot/src/poms/*.py",
            "owlbot/templates/clirr/*.j2",
            "owlbot/templates/poms/*.j2",
+            "owlbot/templates/java_library/.github/**/*",
+            "owlbot/templates/java_library/.kokoro/**/*",
            "owlbot/templates/java_library/**/*",


yes, there are files outside .github and .kokoro that this line would include.

diegomarquezp · 2025-03-27T20:46:39Z

hermetic_build/DEVELOPMENT.md

@@ -78,7 +78,7 @@ as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/96999197
 2. Move the jar into its well-known location.

   ```shell
-   mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar"
+   mv ~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar "${HOME}/.library_generation/gapic-generator-java.jar"


I thought it may be more convenient for copying the whole line and limit to making a replacement in {version}. I think a non-default maven setup is not a common use case, but still checks out, so I reverted this line.

diegomarquezp · 2025-03-27T20:47:15Z

hermetic_build/library_generation/generate_library.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash

-set -eo pipefail
+set -exo pipefail


I reverted this and explained how this would be useful in the dev guide.

diegomarquezp · 2025-03-27T20:47:35Z

hermetic_build/library_generation/utils/utilities.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash

-set -eo pipefail
+set -exo pipefail


I reverted this and explained how this would be useful in the dev guide.

JoeWang1127

Thanks for the fix.

suztomo · 2025-03-27T21:04:43Z

I don't think CODEOWNERS can be templated any more.

suztomo · 2025-03-27T21:07:55Z

Can you evaluate the templated, generated files are really needed? Seeing kokoro/nightly/java7.cfg, it seems you haven't checked that.

hermetic_build/library_generation/owlbot/templates/java_library/.github/generated-files-bot.yml

sonarqubecloud · 2025-04-04T17:44:27Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-04-04T17:48:08Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

diegomarquezp added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 26, 2025

product-auto-label bot added the size: l Pull request size is large. label Mar 26, 2025

diegomarquezp added wip Work in Progress and removed size: l Pull request size is large. labels Mar 26, 2025

update graal configs

4a59a65

product-auto-label bot added the size: l Pull request size is large. label Mar 26, 2025

include universe domain changes

c25a14a

This was referenced Mar 26, 2025

chore(do not merge): demo of template files fix googleapis/java-logging#1787

Closed

chore(do not merge): demo of template files fix googleapis/java-bigtable#2546

Closed

diegomarquezp added 2 commits March 26, 2025 21:38

Revert "include universe domain changes"

56c6f1c

This reverts commit c25a14a.

remove single line unwanted templates

a2effdc

This was referenced Mar 27, 2025

build(deps): update dependency com.google.cloud:google-cloud-shared-config to v1.15.1 #3724

Closed

chore(do not merge): demo of template files fix googleapis/java-pubsublite#1837

Closed

adapt to linter

364b79f

diegomarquezp commented Mar 27, 2025

View reviewed changes

diegomarquezp requested review from JoeWang1127, suztomo and blakeli0 March 27, 2025 20:18

diegomarquezp marked this pull request as ready for review March 27, 2025 20:20

diegomarquezp removed request for suztomo and blakeli0 March 27, 2025 20:20

JoeWang1127 reviewed Mar 27, 2025

View reviewed changes

diegomarquezp mentioned this pull request Mar 27, 2025

Add a wizard script for local development in hermetic_build/ #3725

Open

JoeWang1127 reviewed Mar 27, 2025

View reviewed changes

diegomarquezp added 2 commits March 27, 2025 20:42

use xtrace on local setups

df9a033

use abstract path to jar in instructions

c908f7b

diegomarquezp commented Mar 27, 2025

View reviewed changes

diegomarquezp requested a review from JoeWang1127 March 27, 2025 20:48

JoeWang1127 approved these changes Mar 27, 2025

View reviewed changes

diegomarquezp requested review from suztomo and blakeli0 March 27, 2025 20:50

diegomarquezp removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 27, 2025

diegomarquezp mentioned this pull request Mar 27, 2025

Downstream check should include changes from the generator #2685

Open

diegomarquezp added 3 commits March 28, 2025 16:03

restore .kokoro folder

755c0de

do not update .kokoro templates for now

cb5a1ca

remove CODEOWNERS template

b66af92

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Mar 28, 2025

diegomarquezp commented Mar 28, 2025

View reviewed changes

hermetic_build/library_generation/owlbot/templates/java_library/.github/generated-files-bot.yml Outdated Show resolved Hide resolved

diegomarquezp changed the title ~~chore(hermetic-build): include .kokoro and .github templates~~ chore(hermetic-build): include .github template updates as part of generation Mar 28, 2025

blakeli0 approved these changes Mar 28, 2025

View reviewed changes

chore: include cloud-java-bot in ignored authors of generated files

88e6edb

diegomarquezp mentioned this pull request Apr 2, 2025

There are no instructions to run the hermetic build ITs locally #3731

Open

format setup.py

b136655

diegomarquezp added the automerge Merge the pull request once unit tests and other checks pass. label Apr 4, 2025

Merge branch 'main' into fix-template-updates

b9a7d01

diegomarquezp merged commit 1c0841f into main Apr 7, 2025
52 of 53 checks passed

diegomarquezp deleted the fix-template-updates branch April 7, 2025 15:50

diegomarquezp mentioned this pull request May 1, 2025

update_generation_config.yaml is not updated in downstream repos #3701

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(hermetic-build): include .github template updates as part of generation #3723

chore(hermetic-build): include .github template updates as part of generation #3723

diegomarquezp commented Mar 26, 2025 •

edited

Loading

diegomarquezp Mar 27, 2025

JoeWang1127 Mar 27, 2025

diegomarquezp Mar 27, 2025

JoeWang1127 Mar 27, 2025

diegomarquezp Mar 27, 2025

JoeWang1127 Mar 27, 2025

diegomarquezp Mar 27, 2025

JoeWang1127 Mar 27, 2025 •

edited

Loading

diegomarquezp Mar 27, 2025

JoeWang1127 Mar 27, 2025

diegomarquezp Mar 27, 2025

diegomarquezp left a comment

diegomarquezp Mar 27, 2025

diegomarquezp Mar 27, 2025

diegomarquezp Mar 27, 2025

diegomarquezp Mar 27, 2025

diegomarquezp Mar 27, 2025

JoeWang1127 left a comment

suztomo commented Mar 27, 2025

suztomo commented Mar 27, 2025

sonarqubecloud bot commented Apr 4, 2025

sonarqubecloud bot commented Apr 4, 2025

chore(hermetic-build): include .github template updates as part of generation #3723

chore(hermetic-build): include .github template updates as part of generation #3723

Conversation

diegomarquezp commented Mar 26, 2025 • edited Loading

Approach

Confirming effects in downstream repos

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeWang1127 Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomarquezp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeWang1127 left a comment

Choose a reason for hiding this comment

suztomo commented Mar 27, 2025

suztomo commented Mar 27, 2025

sonarqubecloud bot commented Apr 4, 2025

Quality Gate passed for 'gapic-generator-java-root'

sonarqubecloud bot commented Apr 4, 2025

Quality Gate passed for 'java_showcase_integration_tests'

diegomarquezp commented Mar 26, 2025 •

edited

Loading

JoeWang1127 Mar 27, 2025 •

edited

Loading