`gpgsm` tests don't pass with `gpgsm<=2.4.3`

[dljsjr]

Description

Possibly related to #6241, but I wanted to start off with a new issue since this isn't specific to openSUSE.

But I'm basically seeing the same two tests failing on main on a fresh clone. Specifically:

$ cargo nextest run --workspace --status-level fail --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
────────────
 Nextest run ID d016bdd5-79e0-45b6-a00d-98ed8396e98c with nextest profile: default
    Starting 1980 tests across 12 binaries
        FAIL [   0.021s] jj-lib::runner test_gpg::gpgsm_signing_roundtrip_explicit_key
  stdout ───

    running 1 test
    test test_gpg::gpgsm_signing_roundtrip_explicit_key ... FAILED

    failures:

    failures:
        test_gpg::gpgsm_signing_roundtrip_explicit_key

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 455 filtered out; finished in 0.02s

  stderr ───

    thread 'test_gpg::gpgsm_signing_roundtrip_explicit_key' panicked at lib/tests/test_gpg.rs:362:69:
    called `Result::unwrap()` on an `Err` value: Backend(Command { exit_status: ExitStatus(unix_wait_status(256)), stderr: "gpgsm: WARNING: running with faked system time: 2025-03-20 13:25:10\ngpgsm: can't sign using 'someone@example.com': No public key" })
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.023s] jj-lib::runner test_gpg::gpgsm_signing_roundtrip
  stdout ───

    running 1 test
    test test_gpg::gpgsm_signing_roundtrip ... FAILED

    failures:

    failures:
        test_gpg::gpgsm_signing_roundtrip

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 455 filtered out; finished in 0.01s

  stderr ───

    thread 'test_gpg::gpgsm_signing_roundtrip' panicked at lib/tests/test_gpg.rs:335:31:
    called `Result::unwrap()` on an `Err` value: Backend(Command { exit_status: ExitStatus(unix_wait_status(256)), stderr: "gpgsm: WARNING: running with faked system time: 2025-03-20 13:25:10\ngpgsm: can't sign using 'someone@example.com': No public key" })
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  35.945s] 1980 tests run: 1978 passed, 2 failed, 0 skipped
        FAIL [   0.023s] jj-lib::runner test_gpg::gpgsm_signing_roundtrip
        FAIL [   0.021s] jj-lib::runner test_gpg::gpgsm_signing_roundtrip_explicit_key
error: test run failed

gpgsm is installed, the failure more seems oriented around this:

gpgsm: can't sign using 'someone@example.com': No public key"

Steps to Reproduce the Problem

1. Follow the development environment setup instructions in the Contributing docs: https://jj-vcs.github.io/jj/v0.28.2/contributing/#setting-up-a-development-environment
2. Invoke nextest using the commands from the docs: cargo nextest run --workspace (I added additional arguments in my repro above to downsample the output)

Expected Behavior

Tests should pass? Maybe not, since this isn't specifically a tagged release commit, but the failure itself (or its resolution) doesn't seem to be documented/tracked anywhere, so it feels like it's unexpected.

Actual Behavior

There are two failing tests.

Specifications

• Platform:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

$ uname -a
Linux <host-name> 5.15.0-138-generic #148-Ubuntu SMP Fri Mar 14 19:05:48 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

• Version:

$ jj log --no-graph -r main@origin -T 'commit_id'
489f693c47ac317c55ee1840ba9bc0878b14374f

[bryceberger]

Originally missed that gpgsm was installed.

Can successfully reproduce.

Dockerfile for environment:

from ubuntu:22.04

run apt-get update
run apt-get install -y --no-install-recommends ca-certificates
run update-ca-certificates
run apt-get install -y --no-install-recommends build-essential curl
run curl -Ssf https://sh.rustup.rs | sh -s -- -y

run apt-get install -y --no-install-recommends pkg-config git libssl-dev gpgsm

Command to test (I have all cargo artifacts shared at /tmp/cargo, otherwise unecessary):
podman run -v (pwd):/build -v /tmp/cargo:/build/target -v ~/.cargo/registry:/root/.cargo/registry --rm -it jj-tester sh -c '. /root/.cargo/env && cd build && cargo test -p jj-lib gpg'

With the same output:

failures:

---- test_gpg::gpgsm_signing_roundtrip_explicit_key stdout ----

thread 'test_gpg::gpgsm_signing_roundtrip_explicit_key' panicked at lib/tests/test_gpg.rs:362:69:
called `Result::unwrap()` on an `Err` value: Backend(Command { exit_status: ExitStatus(unix_wait_status(256)), stderr: "gpgsm: WARNING: running with faked system time: 2025-03-20 13:25:10\ngpgsm: can't sign using 'someone@example.com': No public key" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test_gpg::gpgsm_signing_roundtrip stdout ----

thread 'test_gpg::gpgsm_signing_roundtrip' panicked at lib/tests/test_gpg.rs:335:31:
called `Result::unwrap()` on an `Err` value: Backend(Command { exit_status: ExitStatus(unix_wait_status(256)), stderr: "gpgsm: WARNING: running with faked system time: 2025-03-20 13:25:10\ngpgsm: can't sign using 'someone@example.com': No public key" })


failures:
    test_gpg::gpgsm_signing_roundtrip
    test_gpg::gpgsm_signing_roundtrip_explicit_key

[bryceberger]

From some debugging, /tmp/jj-gpgsm-signing-test-Z1GSmD/pubring.kbx contains the following hex:

00000020010100004B42586600000000
681A7766681A77660000000000000000

Piping that into kbxutil gives the following:

BEGIN-RECORD: 0
Length: 32
Type:   Header
Version: 1
Flags:   0000
created-at: 1746564966
last-maint: 1746564966
END-RECORD

Which suggests the private key was never read. It should written, if my reading of the following is correct:

jj/lib/tests/test_gpg.rs

Line 149 in 489f693

.write_all(GPGSM_PRIVATE_KEY.as_bytes())

[bryceberger]

Inside GpgsmEnvironment::new(), the import command exits with the following stderr:

gpgsm: keybox '/tmp/jj-gpgsm-signing-test-dVZh4Y/pubring.kbx' created
gpgsm: error running '/usr/bin/gpg-agent': probably not installed
gpgsm: failed to start agent '/usr/bin/gpg-agent': Configuration error
gpgsm: can't connect to the agent: Configuration error
gpgsm: total number processed: 0

It exits with code 2, so the following test case succeeds.

Adding gpg-agent to the docker environment results in the following output instead:

gpgsm: keybox '/tmp/jj-gpgsm-signing-test-NsjmOq/pubring.kbx' created
gpgsm: ignoring gpg-agent inquiry 'PASSPHRASE'
gpgsm: encryptedData error at "pkcs5PBES2-params", offset 123
gpgsm: error at "bag.encryptedData", offset 49
gpgsm: error parsing or decrypting the PKCS#12 file
gpgsm: total number processed: 0

[bryceberger]

Using Ubuntu 24.04 as a base, with both gpgsm and gpg-agent installed, the tests work.

Ubuntu 22.04:

$ gpgsm --version
gpgsm (GnuPG) 2.2.27
libgcrypt 1.9.4
libksba 1.6.0-unknown
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Cipher: 3DES, AES128, AES192, AES256, SERPENT128, SERPENT192, SERPENT256, SEED, CAMELLIA128, CAMELLIA192, CAMELLIA256
Pubkey: RSA, ECC
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, WHIRLPOOL

Ubuntu 24.04:

$ gpgsm --version
gpgsm (GnuPG) 2.4.4
libgcrypt 1.10.3
libksba 1.6.6
Copyright (C) 2024 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Cipher: 3DES, AES128, AES192, AES256, SERPENT128, SERPENT192, SERPENT256, SEED, CAMELLIA128, CAMELLIA192, CAMELLIA256
Pubkey: RSA, ECC, ECC
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, WHIRLPOOL

[bryceberger]

Switching to nixpkgs, as the problem appears to be with gpgsm: the following fails:

$ nix shell github:nixos/nixpkgs/23.11#gnupg --command sh -c 'gpgsm --version && cargo test -p jj-lib gpg'
gpgsm (GnuPG) 2.4.1
...
gpgsm: keybox '/tmp/jj-gpgsm-signing-test-KoRwJ0/pubring.kbx' created
gpgsm: ignoring gpg-agent inquiry 'PASSPHRASE'
gpgsm: encryptedData error at "pkcs5PBES2-params", offset 123
gpgsm: error at "bag.encryptedData", offset 49
gpgsm: error parsing or decrypting the PKCS#12 file
gpgsm: total number processed: 0

[bryceberger]

$ nix shell github:nixos/nixpkgs/7457535be8d99adef3a445d4628e277e6fc7918a#gnupg --command sh -c 'gpgsm --version && cargo test -p jj-lib gpg'
gpgsm (GnuPG) 2.4.3
...
Supported algorithms:
Cipher: 3DES, AES128, AES192, AES256, SERPENT128, SERPENT192, SERPENT256, SEED, CAMELLIA128, CAMELLIA192, CAMELLIA256
Pubkey: RSA, ECC, ECC
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, WHIRLPOOL
...
gpgsm: keybox '/tmp/jj-gpgsm-signing-test-UfXhGx/pubring.kbx' created
gpgsm: ignoring gpg-agent inquiry 'PASSPHRASE'
gpgsm: DBG: cipher algo: 2a864886f70d0209
gpgsm: parse_bag_encrypted_data(pkcs5PBES2-params): offset 14.10 (tlv_expect_object_id): Success - Invalid cipher algorithm
gpgsm: p12_parse(bag.encryptedData): offset 14.10 (tlv_expect_object_id): Success - Invalid cipher algorithm
gpgsm: error parsing or decrypting the PKCS#12 file
gpgsm: total number processed: 0

[bryceberger]

Summary:

2.2.27 -> fail "error parsing"
2.4.1 -> fail "error parsing"
2.4.3 -> fail "invalid cipher algorithm"
2.4.4 -> suceess

[bryceberger]

Bisecting points to gpg/gnupg@24b3a5a. From the message alone, this makes sense.

[calebdw]

I received some pushback on using outdated algos when I PR'd this: #6078 (comment)

So I used a more up-to-date credential scheme supported by the gpgsm version in ci.