Failed tests when compiling with openmp

[jalvesz]

Description

I tested building and running the tests, including OpenMP support, by including the flag:

cmake -B build -G Ninja -DBUILD_TESTING=on -DCMAKE_Fortran_FLAGS=-fopenmp -DCMAKE_MAXIMUM_RANK:String=4 -DCMAKE_BUILD_TYPE=Release -DCMAKE_Fortran_COMPILER=gfortran
cmake --build build
ctest --test-dir build/test

several of the tests failed:

86% tests passed, 11 tests failed out of 77

Label Time Summary:
quadruple_precision    =   0.17 sec*proc (2 tests)

Total Test time (real) =  12.68 sec

The following tests FAILED:
         12 - chaining_maps (SEGFAULT)
         13 - open_maps (SEGFAULT)
         14 - maps (SEGFAULT)
         15 - intrinsics (Failed)
         30 - linalg_pseudoinverse (Failed)
         38 - blas_lapack (Failed)
         43 - sorting (Exit code 0xc0000374
)
         47 - mean (Failed)
         59 - string_intrinsic (Failed)
         64 - string_to_number (Failed)
         69 - simps (Failed)

         64 - string_to_number (Failed)
         69 - simps (Failed)
         64 - string_to_number (Failed)
         69 - simps (Failed)
         64 - string_to_number (Failed)
         69 - simps (Failed)

I wonder if one of the CI jobs should include OpenMP in order to catch such behaviours early ?

Expected Behaviour

Should pass

Version of stdlib

master

Platform and Architecture

Windows / gfortran 14.2.0

Additional Information

No response

[jvdp1]

Strange that it failed on some procedures like blas_lapack or sorting. I agree that we should include OpenMP in at least one of the CI jobs.

[jalvesz]

I was looking at the intrinsics test and saw that they fail for sum and dot_product for xdp. I saw that there is a tolerance issue, for instance, if I print the tolerance and relative errors here

https://github.com/fortran-lang/stdlib/blob/60d0a769216322243e28a63b92ed7668d2df80d5/test/intrinsics/test_intrinsics.fypp#L213C1-L218C87

adding a print *, '${t}$ dot err:', tolerance, err(1:3)

I get without openmp:
real(xdp) dot err: 1.08420217248550443401E-0017 3.25260651745651330202E-0019 0.00000000000000000000 0.00000000000000000000

With openmp:
real(xdp) dot err: 1.08420217248550443401E-0017 2.22044604925031308085E-0016 0.00000000000000000000 5.55111512312578270212E-0016

For the latter, the errors seems to be funnily close to epsilon(0.d0) = 2.220446049250313E-016 ... I'm intrigued here, I wonder if the other tests might be suffering from something similar.

[perazz]

Yes, unfortunately I also noted this a while ago:

https://github.com/fortran-lang/fpm/blob/7535cab6efc89dd5a294f0d9643b5eebd6b237f0/src/fpm_meta.f90#L139-L142

I have never had time to dig into the issue, though.

I don't use openmp much, but I believe every time there is a static (save) variable somewhere, that must be declared THREADPRIVATE, otherwise all threads will write to it, causing unpredictable behavior.

[jalvesz]

On a different machine ( without the hash_functions tests #976 ) I got "only" the following fails when using openmp (here using GNU from msys2 instead of equation.com)

96% tests passed, 3 tests failed out of 73

Label Time Summary:
quadruple_precision    =   2.32 sec*proc (2 tests)

Total Test time (real) =  96.33 sec

The following tests FAILED:
         37 - sorting (SEGFAULT)
         60 - filesystem (Failed)
         63 - subprocess (Failed)

running: ctest --test-dir build/test --rerun-failed --output-on-failure

click to view log

1/3 Test #37: sorting ..........................***Exception: SegFault 10.68 sec
# Testing: sorting
  Starting char_ord_sorts ... (1/22)
  Starting string_ord_sorts ... (2/22)
  Starting bitset_large_ord_sorts ... (3/22)
  Starting bitset_64_ord_sorts ... (4/22)
  Starting int_radix_sorts ... (5/22)
  Starting real_radix_sorts ... (6/22)
  Starting int_sorts ... (7/22)
  Starting char_sorts ... (8/22)
  Starting string_sorts ... (9/22)
  Starting bitset_large_sorts ... (10/22)
  Starting bitset_64_sorts ... (11/22)
 ORD_SORT did not sort String Decrease.
 i =                     1
  Starting int_sort_indexes_default ... (12/22)
string_dummy(i-1:i) =
 ORD_SORT did not sort Bitset Random.
 i =                   235
bitset64_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000011110000110 0000000000000000000000000000000000000000000000000000000011101100
  Starting char_sort_indexes_default ... (13/22)
  Starting string_sort_indexes_default ... (14/22)
 reverse + work ORD_SORT did not sort Bitset Random.
 i =                     5
bitset64_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000111101000101 0000000000000000000000000000000000000000000000000000111101000111
  Starting bitset_large_sort_indexes_default ... (15/22)
 reverse + work ORD_SORT did not sort Bitset Decrease.
 i =                  2048
 SORT did not sort Bitset Decrease.
 i =                     1
bitsetl_dummy(i-1:i) = 00000000000000000000111111111111 00000000000000000000111111111110
bitset64_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000101000010001 0000000000000000000000000000000000000000000000000000000001100001
  Starting bitset_64_sort_indexes_default ... (16/22)
 RADIX_SORT did not sort Blocks.
 i =                    31
  Starting int_sort_indexes_low ... (17/22)
dummy(i-1:i)     83      0
  Starting char_sort_indexes_low ... (18/22)
 reverse ORD_SORT did not sort Bitset Random.
 i =                   537
bitset64_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000110111100111 0000000000000000000000000000000000000000000000000000110111100110
 reverse + work ORD_SORT did not sort Char. Decrease.
 i =                     1
       ... bitset_64_ord_sorts [FAILED]
  Message: Condition not fullfilled
char_dummy(i-1:i) =  pppp pppo
  Starting bitset_large_sort_indexes_low ... (20/22)
 SORT_INDEX did not sort Bitset Decrease.
 i =                     3
  Starting bitset_64_sort_indexes_low ... (21/22)
 reverse SORT did not sort Bitset Decrease.
 i =                     1
  Starting int_ord_sorts ... (22/22)
bitset64_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000101000110100 0000000000000000000000000000000000000000000000000000011011001111
bitsetl_dummy(i-1:i) = 0000000000000000000000000000000000000000000000000000111111111111 0000000000000000000000000000000000000000000000000000111111111110
       ... bitset_64_sort_indexes_default [FAILED]
  Message: Condition not fullfilled
  Starting string_sort_indexes_low ... (19/22)
       ... bitset_64_sorts [FAILED]
  Message: Condition not fullfilled
 ORD_SORT did not sort Blocks.
 i =                  2436
dummy(i-1:i)  46123  46124
 SORT_INDEX did not sort Blocks.
 i =                   256
a(index_low(i-1:i)   3805   3804
 SORT_INDEX did not sort Char. Decrease.
 i =                  4806
char_dummy(i-1:i) onkg dilo
       ... char_sort_indexes_default [FAILED]
  Message: Condition not fullfilled
 reverse RADIX_SORT did not sort Blocks.
 i =                   427
dummy(i-1:i)  65109  65108
       ... int_radix_sorts [FAILED]
  Message: Condition not fullfilled
       ... bitset_64_sort_indexes_low [PASSED]
 SORT_INDEX did not sort Blocks.
 i =                     2
a(index_default(i-  65534  65533
 reverse + work ORD_SORT did not sort Char. Decrease.
 i =                     1
char_dummy(i-1:i) =  afnn cfpi
       ... char_ord_sorts [FAILED]
  Message: Condition not fullfilled
 SORT did not sort Blocks.
 i =                  8437
dummy(i-1:i)   4709   4708
 SORT_INDEX did not sort String Decrease.
 i =                   229
string_dummy(i-1:
 SORT_INDEX did not sort Char. Decrease.
 i =                  7427
       ... string_sort_indexes_default [FAILED]
  Message: Condition not fullfilled
char_dummy(i-1:i) enme enif
       ... char_sort_indexes_low [FAILED]
  Message: Condition not fullfilled
 reverse + work ORD_SORT did not sort Blocks.
 i =                  2889
dummy(i-1:i)   8638  35054
       ... real_radix_sorts [PASSED]
 reverse + work ORD_SORT did not sort String Decrease.

    Start 60: filesystem
2/3 Test #60: filesystem .......................***Failed    0.65 sec
# Testing: filesystem
  Starting fs_is_directory_dir ... (1/2)
  Starting fs_is_directory_file ... (2/2)
       ... fs_is_directory_file [FAILED]
  Message: Cannot delete test file: File cannot be deleted
       ... fs_is_directory_dir [PASSED]
1 test(s) failed!
ERROR STOP

Error termination. Backtrace:
#0  0xd7e05dac in ???
#1  0xd7d819d1 in ???
#2  0xd7c7ed5b in ???
#3  0x14962cb5 in ???
#4  0x14962d01 in ???
#5  0x14961318 in __tmainCRTStartup
        at D:/M/B/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:259
#6  0x14961425 in mainCRTStartup
        at D:/M/B/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:179
#7  0xadde259c in ???
#8  0xaf12af37 in ???
#9  0xffffffff in ???

    Start 63: subprocess
3/3 Test #63: subprocess .......................   Passed    1.66 sec

33% tests passed, 2 tests failed out of 3

Total Test time (real) =  13.24 sec

The following tests FAILED:
         37 - sorting (SEGFAULT)
         60 - filesystem (Failed)

[perazz]

Regarding the filesystem tests, it would seem like it may be enough to ensure that the test file name is different from each thread.

[PierUgit]

I don't use openmp much, but I believe every time there is a static (save) variable somewhere, that must be declared THREADPRIVATE, otherwise all threads will write to it, causing unpredictable behavior.

As you are mentioning, the problem can arise only if the saved variable (which can be a module variable, which is saved by design) is written, there's no issue when reading the variable. But as a general rule, given the importance of multithreading in HPC nowadays, the thread-safety status of all stdlib routines should be mentioned: which ones are thread-safe, which ones are not.

[jalvesz]

Regarding the filesystem tests, it would seem like it may be enough to ensure that the test file name is different from each thread.

I would have said better to make it such that the deletion is executed by a single thread, like adding !$omp single where appropriate, no?