python abort with libmamba 2.0.5+

[opoplawski]

Troubleshooting docs

• My problem is not solved in the Troubleshooting docs

Anaconda default channels

• I do NOT use the Anaconda default channels (pkgs/* etc.)

How did you install Mamba?

Other (please describe)

Search tried in issue tracker

abort

Latest version of Mamba

• My problem is not solved with the latest version

Tried in Conda?

Not applicable

Describe your issue

I'm working on updating libmamba to 2.0.X in Fedora and running into a very strange issue. First off, I have had to set -DMAMBA_WARNING_AS_ERROR=OFF to work around #3791 . When I go to build conda 25.1.1, python aborts in one of the tests:

tests/core/test_solve.py::test_force_remove_1[libmamba] Fatal Python error: Aborted

Thread 0x00007fc1cd12e6c0 (most recent call first):
  File "/usr/lib64/python3.13/socket.py", line 295 in accept
  File "/usr/lib/python3.13/site-packages/pytest_rerunfailures.py", line 440 in run_server
  File "/usr/lib64/python3.13/threading.py", line 992 in run
  File "/usr/lib64/python3.13/threading.py", line 1041 in _bootstrap_inner
  File "/usr/lib64/python3.13/threading.py", line 1012 in _bootstrap

Current thread 0x00007fc2374bfbc0 (most recent call first):
  File "/usr/lib/python3.13/site-packages/conda_libmamba_solver/solver.py", line 891 in _conda_spec_to_libmamba_spec
  File "/usr/lib/python3.13/site-packages/conda_libmamba_solver/solver.py", line 409 in _specs_to_request_jobs
  File "/usr/lib/python3.13/site-packages/conda_libmamba_solver/solver.py", line 356 in _solve_attempt
  File "/usr/lib/python3.13/site-packages/conda_libmamba_solver/solver.py", line 307 in _solving_loop
...

So I added a print to that:

    def _conda_spec_to_libmamba_spec(self, spec: MatchSpec) -> LibmambaMatchSpec:
        print(f'_conda_spec_to_libmamba_spec: parsing {str(spec)}', file=sys.stderr)
        return LibmambaMatchSpec.parse(str(spec))

and I see:

_conda_spec_to_libmamba_spec: parsing numpy 1.7.1 py27_0

which seems pretty innocuous. If I then run in gdb I see:

#4  0x00007ffff440b0c2 in std::__glibcxx_assert_fail (file=file@entry=0x7fff87946368 "/usr/include/c++/15/string_view", line=line@entry=260,
    function=function@entry=0x7fff87946f80 "constexpr const std::basic_string_view<_CharT, _Traits>::value_type& std::basic_string_view<_CharT, _Traits>::operator[](size_type) const [with _CharT = char; _Traits = std::char_traits<char>; const_r"..., condition=condition@entry=0x7fff8793d827 "__pos < this->_M_len")
    at ../../../../../libstdc++-v3/src/c++11/assert_fail.cc:41
41          abort();
(gdb) up
#5  0x00007fff87615b75 in std::basic_string_view<char, std::char_traits<char> >::operator[](unsigned long) const [clone .part.0] [clone .lto_priv.0] [clone .lto_priv.0] (__pos=<optimized out>, this=<optimized out>, this=<optimized out>, __pos=<optimized out>) at /usr/include/c++/15/string_view:260
260             __glibcxx_assert(__pos < this->_M_len);
(gdb) up
#6  0x00007fff876198fe in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=<optimized out>, __pos=<optimized out>, this=<optimized out>,
    __pos=<optimized out>) at /usr/include/c++/15/bits/basic_string.h:410
410               __throw_length_error(__N(__s));
(gdb) up
#7  mamba::specs::MatchSpec::parse (
    str='\000' <repeats 112 times>, "\360\266\377\377\377\177\000\000\001\000\000\000\000\000\000\000*", '\000' <repeats 15 times>, "\020\267\377\377\377\177\000\000\001\000\000\000\000\000\000\000*", '\000' <repeats 39 times>...) at /usr/src/debug/libmamba-2.0.6-2.fc43.x86_64/libmamba/src/specs/match_spec.cpp:508
508             if (const auto idx = str.find('#'); idx != std::string::npos && str[idx - 1] == ' ')

at which point I seem to have garbage.

505         auto MatchSpec::parse(std::string_view str) -> expected_parse_t<MatchSpec>
506         {
507             // Remove comments, i.e. everything after ` #` (space included)
508             if (const auto idx = str.find('#'); idx != std::string::npos && str[idx - 1] == ' ')
509             {
510                 str = str.substr(0, idx);
511             }

(gdb) up
#8  0x00007fff87b2e57b in pybind11::detail::argument_loader<std::basic_string_view<char, std::char_traits<char> > >::call_impl<tl::expected<mamba::specs::MatchSpec, mamba::specs::ParseError>, tl::expected<mamba::specs::MatchSpec, mamba::specs::ParseError> (*&)(std::basic_string_view<char, std::char_traits<char> >), 0ul, pybind11::detail::void_type>(tl::expected<mamba::specs::MatchSpec, mamba::specs::ParseError> (*&)(std::basic_string_view<char, std::char_traits<char> >), std::integer_sequence<unsigned long, 0ul>, pybind11::detail::void_type&&) && (this=0x7fffffffb910, f=<optimized out>) at /usr/include/pybind11/cast.h:1631
1631            return std::forward<Func>(f)(cast_op<Args>(std::move(std::get<Is>(argcasters)))...);
(gdb) print *this
$5 = {static kwargs_pos = -1, static args_pos = -1, static arg_names = {text = "{str}"}, argcasters = std::tuple containing = {
    [0] = {<pybind11::detail::string_caster<std::basic_string_view<char, std::char_traits<char> >, true>> = {value = "numpy 1.7.1 py27_0"}, <No data fields>}}}

valgrind doesn't show any issues that I can see.

It doesn't appear to gcc 15 related as I can reproduce the same issue in F41 with gcc 14.

We compile with the following:

+ CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer '
+ export CFLAGS
+ CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer '
+ export CXXFLAGS
+ FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules '
+ export FFLAGS
+ FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules '
+ export FCFLAGS
+ VALAFLAGS=-g
+ export VALAFLAGS
+ RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes --cap-lints=warn'
+ export RUSTFLAGS
+ LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes '
+ export LDFLAGS
+ LT_SYS_LIBRARY_PATH=/usr/lib64:
+ export LT_SYS_LIBRARY_PATH
+ CC=gcc
+ export CC
+ CXX=g++
+ export CXX
+ /usr/bin/cmake -S . -B redhat-linux-build -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_INSTALL_FULL_SBINDIR:PATH=/usr/bin -DCMAKE_INSTALL_SBINDIR:PATH=bin -DINCLUDE_INSTALL_DIR:PATH=/usr/include -DLIB_INSTALL_DIR:PATH=/usr/lib64 -DSYSCONF_INSTALL_DIR:PATH=/etc -DSHARE_INSTALL_PREFIX:PATH=/usr/share -DLIB_SUFFIX=64 -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_STANDARD=17 -DBUILD_LIBMAMBA=ON -DBUILD_LIBMAMBAPY=ON -DBUILD_MAMBA=OFF -DBUILD_MICROMAMBA=OFF -DBUILD_EXE=OFF -DBUILD_SHARED=ON -DBUILD_STATIC=OFF -DENABLE_TESTS=ON -DMAMBA_WARNING_AS_ERROR=OFF

though the build is done in stages:

• libmamba
• libmambapy
• mamba

It looks like it's the -D_GLIBCXX_ASSERTIONS that is triggering the abort, but it does seem to be catching something very strange going on.

mamba info / micromamba info

Logs

environment.yml

~/.condarc

[opoplawski]

Also, I believe I saw this with earlier 2.0.X releases, but I don't have details.

[JohanMabille]

Hi, mamba 2.0.6 was marked as broken and the packages have been removed from conda-forge. Can you confirm you still have the issue with mamba 2.0.5?

[jjerphan]

This bug has been introduced here by #3512 which is part of 2.0.5

[opoplawski]

Hmm, I don't think that's quite it. It does seem like that code would attempt an out of range index lookup if parse() were passed "#". But that's not what's happening here.

I recompiled libmamba with -O0 and I think I have better debug info. abort is happening at split_version_and_build:459:

Thread 1 "python3" hit Breakpoint 1, mamba::specs::(anonymous namespace)::split_version_and_build (str=" 1.7.1 py27_0")
445             {
446                 str = util::strip(str);
447
448                 // Support faulty conda matchspecs such as `libblas=[build=*mkl]`, which is
449                 // the repr of `libblas=*=*mkl`
450                 str = util::rstrip(str, '=');
451
452                 auto pos = str.find_last_of(" =");
453                 if (pos == str.npos || pos == 0)
454                 {
455                     return { str, {} };
456                 }
457
458                 pos = str.find_last_of('=');
459                 const char d = str[pos - 1];

The string that was passed in was " 1.7.1 py27_0". After the strip() it's "1.7.1 py27_0". str.find_last_of(" =") returns 5 the position of the space. And then the final find_last_of('=') returns string::npos since str does not contain a '=' at this point.

That was introduced by c6f2f24 by @Hind-M

I'm not sure I can grok what mamba is trying to do at this point, so hopefully someone more familiar with the code can step in at this point.

[opoplawski]

Anyone?