fix: Podman compatibility and testing #142

angela-ko · 2025-04-29T20:08:29Z

Description of changes

Podman adds urls in front of all images

❯ podman image ls | grep docker_client
docker.io/local_host/foo/bar/docker_client_create_image_url_test  latest                        f499aed52e60  45 years ago   8.12 MB
docker.io/library/docker_client_create_image_test                 dummy                         f499aed52e60  45 years ago   8.12 MB

So we need to change the string matching to substring matching instead in tags.

Testing done

Tested locally with podman + Added ci testing with podman

License

[ x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
[ x] I sign the Developer Certificate of Origin below by adding my name and email address to the Signed-off-by line.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Signed-off-by: Angela Ko angela.ko@simulation.science

codecov · 2025-04-29T20:10:05Z

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Project coverage is 75.69%. Comparing base (4e2aed2) to head (fc03af6).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
tesseract_core/sdk/docker_client.py	81.81%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
+ Coverage   75.63%   75.69%   +0.05%     
==========================================
  Files          27       27              
  Lines        2910     2921      +11     
  Branches      446      449       +3     
==========================================
+ Hits         2201     2211      +10     
  Misses        512      512              
- Partials      197      198       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

angela-ko · 2025-04-30T17:57:48Z

oh wait i just realized this might lead to over matching of images

@heitorPB like matching "repo" with "repo_1234_dsjklfj:latest" , so maybe we should go back to your tag split method instead....

angela-ko · 2025-04-30T17:59:25Z

Also not sure why the images aren't being cleaned up in podman so need to fix that too in this pr

dionhaefner · 2025-04-30T18:07:17Z

👌 Ping me when the outstanding issues are fixed and I'll have a closer look. Agree on not abusing substring matching here in favor of something more robust.

And some food for thought (not necessarily implemented here): Can we test w/ podman on CI?

heitorPB · 2025-04-30T18:46:40Z

And some food for thought (not necessarily implemented here): Can we test w/ podman on CI?

GitHub runners have Podman installed, so we can. But, one of the tests will fail with Podman. Podman doesn't support port-ranges in the docker-compose file (containers/podman#15111) and we have a test for that particular case. ~~Can we skip that test when testing with Podman? How to do that?~~

angela-ko · 2025-04-30T21:37:56Z

@heitorPB I think the podman socket isn't correct:

DEBUG    tesseract:engine.py:94 #23 exporting config sha256:2db97d1217b95769fac0df0fcebf96a3f3b3961a1a9414db78161ff29a9045a3 done
DEBUG    tesseract:engine.py:94 #23 sending tarball
DEBUG    tesseract:engine.py:94 ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: read unix @->/run/user/1001/podman/podman.sock: read: connection reset by peer
WARNING  tesseract:engine.py:416 Build failed with logs:
WARNING  tesseract:engine.py:418 #0 building with "default" instance using docker-container driver

heitorPB · 2025-05-02T19:04:06Z

The podman socket (/run/user/1001/podman/podman.sock) is correct, I added a status command to the "set up podman" step and we can see the socket is active and the path is correct.

The error we get with a bad socket is different:

$  DOCKER_HOST=unix:///foo python -c 'import docker; print(docker.from_env().info())'
Traceback (most recent call last):
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
    ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<6 lines>...
        enforce_content_length=enforce_content_length,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 445, in request
    self.endheaders()
    ~~~~~~~~~~~~~~~^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1333, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1093, in _send_output
    self.send(msg)
    ~~~~~~~~~^^^^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1037, in send
    self.connect()
    ~~~~~~~~~~~~^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
        method=request.method,
    ...<9 lines>...
        chunked=chunked,
    )
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 493, in _make_request
    conn.request(
    ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<6 lines>...
        enforce_content_length=enforce_content_length,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 445, in request
    self.endheaders()
    ~~~~~~~~~~~~~~~^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1333, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1093, in _send_output
    self.send(msg)
    ~~~~~~~~~^^^^^
  File "/home/h/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/http/client.py", line 1037, in send
    self.connect()
    ~~~~~~~~~~~~^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/transport/unixconn.py", line 26, in connect
    sock.connect(self.unix_socket)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ~~~~~~~~~^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/api/client.py", line 246, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import docker; print(docker.from_env().info())
                         ~~~~~~~~~~~~~~~^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/client.py", line 94, in from_env
    return cls(
        timeout=timeout,
    ...<3 lines>...
        **kwargs_from_env(**kwargs)
    )
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/api/client.py", line 207, in __init__
    self._version = self._retrieve_server_version()
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/h/projects/pasteur/tesseract-core/.venv/lib/python3.13/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
    raise DockerException(
        f'Error while fetching server API version: {e}'
    ) from e
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The errors we are seeing here for podman are a bit weird. It appears that it built the images, but failed to load them?

jpbrodrick89 · 2025-05-02T19:20:41Z

Seems similar to what we observed with docker py and podman. The images were all fine and runnable from command line in our case. Would you like me to see if I can replicate your issue when back in office on Tues? (If you've not fixed it by then!)

heitorPB · 2025-05-02T19:32:51Z

The funny thing is: all tests pass on my Linux machine, the errors only appear in the CI. I can't replicate the failures locally. It would be great if you could run the tests and report whether it passes or not.

dionhaefner · 2025-05-05T09:11:12Z

Judging from this run it looks like every single image build fails. So this is unlikely to be an issue with resources running low.

I suggest to reproduce this on CI via a simple docker buildx build that does not use Tesseract code (potentially via tesseract build --generate-only, or even a dead simple dummy Dockerfile). If the problem reproduces, we know something's off with the setup. If it doesn't we'll keep digging to isolate the feature we use that podman chokes on.

As soon as this is isolated we'll have a much easier time to either experiment with the setup (e.g. upgrade podman) or create an upstream issue asking for help.

angela-ko · 2025-05-05T16:18:34Z

Heitor mentioned that it might be a podman versioning issue -- we reran the ubuntu 24 podman test (previous one got cancelled for some reason) and it seems to be passing.

dionhaefner · 2025-05-05T16:23:53Z

That was my suspicion too :) Can we nail down then which version we require? Just so we can add a blurb to the docs a la "make sure you grab at least Podman vfoo.bar".

(No need for this to be a tight bound, but even knowing "version X works while version Y doesn't" is useful.)

heitorPB · 2025-05-05T16:26:00Z

Podman 4.9 works on the CI, but version 3.x doesn't. I'm running with podman 5.4.1 and it also works :)

dionhaefner · 2025-05-05T16:32:35Z

.github/workflows/run_tests.yml

+      - name: Set up Podman
+        run: |
+          systemctl start --user podman.socket
+          systemctl status --user podman.socket
+
+          podman -v
+          cat /etc/os-release


Seeing this is the only difference from the Docker tests, could we please add this as an additional dimension of the test matrix above? That is, we use something like this:

strategy: matrix: os: [ubuntu-24.04] python-version: ["3.12"] arch: ["x64"] docker-engine: ["docker"] unit-tesseract: ${{ fromJson(needs.get-e2e-matrix.outputs.matrix) }} include: # Test on arm to ensure compatibility with Apple M1 chips # (OSX runners don't have access to Docker so we use Linux ARM runners instead) - os: "ubuntu-22.04" python-version: "3.12" arch: "arm" docker-engine: "docker" unit-tesseract: "base" # Test with podman as Docker engine - os: "ubuntu-22.04" python-version: "3.12" arch: "x64" docker-engine: "podman" unit-tesseract: "base"

The Podman test differs in two things:

starting the podman.socket

using the socket in the uv run ... command:
DOCKER_HOST=unix:///run/user/1001/podman/podman.sock \ uv run --no-sync pytest \ --always-run-endtoend \ --cov-report=term-missing:skip-covered \ --cov-report=xml:coverage.xml \ --cov=tesseract_core \ tests/endtoend_tests \ -k "not test_examples" ...

How to do both of that as a new matrix dimension?

You can do this: (untested)

- name: Set up Podman if: matrix.docker-engine == "podman" run: | systemctl start --user podman.socket systemctl status --user podman.socket podman -v cat /etc/os-release echo "DOCKER_HOST=unix:///run/user/1001/podman/podman.sock" >> $GITHUB_ENV

Handle empty strings and repo urls in assertions

79bd559

Handle more strings

4ab50b3

heitorPB changed the title ~~Handle empty strings and repo urls in assertions~~ fix: handle empty strings and repo urls in assertions Apr 30, 2025

heitorPB and others added 4 commits April 30, 2025 11:14

refactor logic to get an image

6e0eb83

Handle image_name with URL

265ed01

Change to any

cbf1484

Change image exists

e43a573

angela-ko marked this pull request as ready for review April 30, 2025 17:55

angela-ko requested review from dionhaefner, xalelax and apaleyes as code owners April 30, 2025 17:55

heitorPB and others added 3 commits April 30, 2025 15:53

test: skip tesseract_serve_ports with range on Podman

eec02a7

ci: run all E2E tests with Podman

18adba4

Add proper substring matching for image names

ee16afe

angela-ko changed the title ~~fix: handle empty strings and repo urls in assertions~~ fix: Podman compatibility and testing Apr 30, 2025

Podman typeguard

04fa690

angela-ko force-pushed the ako/podman_url branch from a9c8353 to 04fa690 Compare April 30, 2025 21:47

ci: display status of podman socket for debugging

479d9e1

Docker-py does not support partial string matching

507f6ec

ci(tests): test podman setup

5648db6

heitorPB added 3 commits May 5, 2025 11:22

Runs on multiple Ubuntus

6136e43

ci(test-podman): fix command to run tesseract

4bd8249

Fix the fixed command to build the images

73910ba

ci(tests): use Ubuntu24 for Podman tests

fc03af6

dionhaefner reviewed May 5, 2025

View reviewed changes

heitorPB added 2 commits May 6, 2025 11:47

ci(tests): refactor to use a matrix for Podman

a9eabc3

use curly braces?

3d4024d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Podman compatibility and testing #142

fix: Podman compatibility and testing #142

angela-ko commented Apr 29, 2025 •

edited

Loading

codecov bot commented Apr 29, 2025 •

edited

Loading

angela-ko commented Apr 30, 2025

angela-ko commented Apr 30, 2025

dionhaefner commented Apr 30, 2025

heitorPB commented Apr 30, 2025 •

edited

Loading

angela-ko commented Apr 30, 2025

heitorPB commented May 2, 2025

jpbrodrick89 commented May 2, 2025

heitorPB commented May 2, 2025

dionhaefner commented May 5, 2025 •

edited

Loading

angela-ko commented May 5, 2025

dionhaefner commented May 5, 2025

heitorPB commented May 5, 2025

dionhaefner May 5, 2025 •

edited

Loading

heitorPB May 5, 2025

dionhaefner May 6, 2025 •

edited

Loading

fix: Podman compatibility and testing #142

Are you sure you want to change the base?

fix: Podman compatibility and testing #142

Conversation

angela-ko commented Apr 29, 2025 • edited Loading

Description of changes

Testing done

License

codecov bot commented Apr 29, 2025 • edited Loading

Codecov Report

angela-ko commented Apr 30, 2025

angela-ko commented Apr 30, 2025

dionhaefner commented Apr 30, 2025

heitorPB commented Apr 30, 2025 • edited Loading

angela-ko commented Apr 30, 2025

heitorPB commented May 2, 2025

jpbrodrick89 commented May 2, 2025

heitorPB commented May 2, 2025

dionhaefner commented May 5, 2025 • edited Loading

angela-ko commented May 5, 2025

dionhaefner commented May 5, 2025

heitorPB commented May 5, 2025

dionhaefner May 5, 2025 • edited Loading

Choose a reason for hiding this comment

heitorPB May 5, 2025

Choose a reason for hiding this comment

dionhaefner May 6, 2025 • edited Loading

Choose a reason for hiding this comment

angela-ko commented Apr 29, 2025 •

edited

Loading

codecov bot commented Apr 29, 2025 •

edited

Loading

heitorPB commented Apr 30, 2025 •

edited

Loading

dionhaefner commented May 5, 2025 •

edited

Loading

dionhaefner May 5, 2025 •

edited

Loading

dionhaefner May 6, 2025 •

edited

Loading