gdb can't catch any trace from sharun AppImages

[brunvonlope]

Description

Not sure if this is a sharun bug or something related to the overall AppImage design, but it's not possible to get a trace using gdb from host.

Reproduction

Let's prepare the environment:

1. Download this almost done sharun appimage (all plugins and extensions work): https://gitlab.gnome.org/GNOME/gimp/-/jobs/4944408/artifacts/browse/build/linux/appimage/_Output/
2. Ensure GDB is installed

To reproduce the bug:

3. Unzip the contents of this file (dockrc.zip) on $HOME/.config/GIMP/3.0/
4. Run gdb path_to_appimage or just run GIMP (it is able to auto call gdb as JIT debugger)
5. Go to the menu then Windows > Recently Closed Docks -> the second item on the list (GIMP will crash)
6. Check on terminal (if you opened GIMP with gdb) or on $HOME/.config/GIMP/3.0/CrashLog/*.txt for the stacktrace with the timestamp of the time GIMP crashed

Expected behavior: GIMP crashes can be caught by GDB since sharun appimages don't suffer from build-id mismatch of other appimages (https://gitlab.gnome.org/GNOME/gimp/-/issues/12929#note_2350842)

Actual behavior: none of the threads have minimally meaningful frames

[Samueru-sama]

Is it possible this is related to the appimage still having an AppRun that's an script instead of an elf?

[brunvonlope]

@Samueru-sama Unfortunately, no ☹️ I modified my AppImage locally to use .app and .env and the hollow traces persists.

[VHSgunzo]

@brunvonlope

Every binary you packaged with lib4bin was run through strip (-s) and all unnecessary debug symbols was stripped, maybe this is the reason for the empty trace.
https://gitlab.gnome.org/GNOME/gimp/-/blob/bruno/sharun-appimage/build/linux/appimage/3_dist-gimp-goappimage.sh#L364

Also note the executables in AppDir/usr/libexec they are left unwrapped in sharun

And the files AppDir/usr/lib/gimp/3.0/extensions/org.gimp.extension.goat-exercises/goat-exercise-vala and goat-exercise-c should be hard links to sharun

[brunvonlope]

Every binary you packaged with lib4bin was run through strip (-s) and all unnecessary debug symbols was stripped, maybe this is the reason for the empty trace.

Testing without that option, the traces are still empty. I don't have a link for the appimage yet but will post it soon.

Also note the executables in AppDir/usr/libexec they are left unwrapped in sharun

I don't know how to proceed with libexec/gimp-debug-tool* because GIMP doesn't search for it on PATH but lib4bin insists on putting every executable on PATH (usr/shared/bin<>usr/bin).

Just to note, gimp-debug-tool isn't needed to catch the traces, only to display a gui with them.

And the files AppDir/usr/lib/gimp/3.0/extensions/org.gimp.extension.goat-exercises/goat-exercise-vala and goat-exercise-c should be hard links to sharun

They already are.

[VHSgunzo]

I don't know how to proceed with libexec/gimp-debug-tool* because GIMP doesn't search for it on PATH but lib4bin insists on putting every executable on PATH (usr/shared/bin<>usr/bin).

Only ${SHARUN_DIR}/bin is included in the PATH and you need to put these executables in ${SHARUN_DIR}/shared/bin and create symlinks to ${SHARUN_DIR}/sharun -> ${SHARUN_DIR}/libexec/${EXE_NAME} instead

They already are.

so I had an older build.

[brunvonlope]

Only ${SHARUN_DIR}/bin is included in the PATH

What I mean with "path" is that if I put that binary (it does the same with the plug-ins) on lib4bin call it will wrongly put it on ${SHARUN_DIR}/bin (which is on PATH).

so I had an older build.

There is the latest build: https://gitlab.gnome.org/GNOME/gimp/-/jobs/4946237/artifacts/browse/build/linux/appimage/_Output/

It only runs calling AppRun after appimage--extract but it helps for triaging the stacktrace bug

[VHSgunzo]

What I mean with "path" is that if I put that binary (it does the same with the plug-ins) on lib4bin call it will wrongly put it on ${SHARUN_DIR}/bin (which is on PATH).

You can move them manually and create symlinks as I described above.
You already do this kind of thing for plugins and for extensions

[VHSgunzo]

It only runs calling AppRun after appimage--extract but it helps for triaging the stacktrace bug

I don't think it will be possible to debug with GDB applications wrapped with sharun, because sharun uses a hack to run applications with userland-execve by mapping the interpreter (such as ld-linux-x86-64.so.2) into memory, creating a stack for it (containing the auxiliary vector, arguments, and environment variables), and then jumping to the entry point with the new stack and passing as arguments the necessary executable file to this interpreter and paths with libraries.

If you compare the debug of an application not wrapped in sharun and the same application run through sharun, you will see that GDB simply cannot load the symbol table even if the application has not been stripped. Because GDB sees only the executable file of sharun itself.

But I'm not a hardcore GDB user.

[VHSgunzo]

[VHSgunzo]

Also, it seems a bit odd to me to supply debug symbols and thereby inflate the size of an application that should be portable and take up as little space as possible. It seems better to leave/debug with GDB for system (or debug) versions of the application.

[brunvonlope]

This is not related to debug symbols. GIMP, babl and GEGL binaries have debug symbols and the traces are empty.

The others are avaiable by installing debuginfod and export DEBUGINFOD_URLS="https://debuginfod.debian.net" before calling gdb. But, as said, if GIMP, babl and GEGL, which aren't stripped aren't showing, this is not about debug symbols

[VHSgunzo]

this is not about debug symbols

check out the screenshots above

[brunvonlope]

What the screenshots are saying is that gdb don't know the original binary that have the debug symbols so yeah this is related to debug symbols but they are there

[VHSgunzo]

There is the latest build: https://gitlab.gnome.org/GNOME/gimp/-/jobs/4946237/artifacts/browse/build/linux/appimage/_Output/

AppRun is not linked correctly in this build. If AppRun is not in ${SHARUN_DIR}, it is better to link it directly to ${SHARUN_DIR}/bin/org.gimp.GIMP.Stable rather than to ${SHARUN_DIR}/sharun and ${SHARUN_DIR}/.app has the wrong path, it should only contain the name of the executable from ${SHARUN_DIR}/bin. But in the case of linking directly to ${SHARUN_DIR}/bin/org.gimp.GIMP.Stable, this ${SHARUN_DIR}/.app file is not needed.

[Samueru-sama]

I want to add that the original idea of having the symlink and .app files were mine, since that's how I got it to work to get rid of the AppRun script.

I was under the wrong assuption that when sharun was hardlinked as the AppRun that it would allow launching ARGV0 but it seems this is not possible 😅

However I also discovered that you have symlinks in ${SHARUN_DIR}/bin like gimp that don't have a ${SHARUN_DIR}/shared/bin counterpart, those cannot be there, everything in ${SHARUN_DIR}/bin has to have its equivalent in ${SHARUN_DIR}/shared/bin

[brunvonlope]

But in the case of linking directly to ${SHARUN_DIR}/bin/org.gimp.GIMP.Stable, this ${SHARUN_DIR}/.app file is not needed.

@VHSgunzo Thanks, done it.

---

I was under the wrong assuption that when sharun was hardlinked as the AppRun that it would allow launching ARGV0 but it seems this is not possible

@Samueru-sama Even this not helping with ARGV0 your idea is very interesting. Runnung a binary sems to be less prone to error and faster than calling host sh 😁

[brunvonlope]

Only ${SHARUN_DIR}/bin is included in the PATH and you need to put these executables in ${SHARUN_DIR}/shared/bin and create symlinks to ${SHARUN_DIR}/sharun -> ${SHARUN_DIR}/libexec/${EXE_NAME} instead

I have done that with:

rm $USR_DIR/bin/gimp-debug-tool* && mv "$USR_DIR/libexec/gimp-debug-tool-$GIMP_APP_VERSION" $USR_DIR/shared/bin && ln -f $USR_DIR/sharun "$USR_DIR/libexec/gimp-debug-tool-$GIMP_APP_VERSION"

But I get the following error:

There is the appimage: https://gitlab.gnome.org/GNOME/gimp/-/jobs/4946575/artifacts/browse/build/linux/appimage/_Output/

[VHSgunzo]

ln -f $USR_DIR/sharun "$USR_DIR/libexec/gimp-debug-tool-$GIMP_APP_VERSION"

create relative symlinks

[Samueru-sama]

But I get the following error:

This is because when hardlinks are used, sharun uses SHARUN_DIR to determine where to find that location to find the bin and libs at shared

You have to run it like this

Or symlink to sharun instead of hardlink.

However note that every respective symlink/file has to have its equivalent in shared/bin, you cannot have libexec/gimp-debug-tool and shared only have shared/bin/gimp-debug-tool-3.0

[VHSgunzo]

However note that every respective symlink/file has to have its equivalent in shared/bin

I think I'll fix it in the next release though.

[brunvonlope]

Another thing that I noticed is that debuginfod can't download .debug symbols from sharun appimage but it can with the go-appimage one.

[VHSgunzo]

However note that every respective symlink/file has to have its equivalent in shared/bin

I think I'll fix it in the next release though.

@Samueru-sama @brunvonlope
fixed in v0.6.1

[VHSgunzo]

Another thing that I noticed is that debuginfod can't download .debug symbols from sharun appimage but it can with the go-appimage one.

The reason is the same, GDB/debuginfod doesn't understand how sharun runs dynamically linked binaries