Huge memory allocation in dav1d, reachable via libheif (GraphicsMagick oss-fuzz-513503494)

GraphicsMagick OSS-Fuzz testing has discovered huge memory allocations originating from dav1d. These problems do not happen if the AOM library is used instead. This is the reported stack trace: > > Time ran: 6.377075910568237 > > INFO: Running with entropic power schedule (0xFF, 100). > INFO: Seed: 147973573 > INFO: Loaded 1 modules (770462 inline 8-bit counters): 770462 [0x5c0641e7c988, 0x5c0641f38b26), > INFO: Loaded 1 PC tables (770462 PCs): 770462 [0x5c0641f38b28,0x5c0642afa508), > /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_graphicsmagick_364babd4f1406e0e2b68256230a27a3911dd0072/revisions/coder_HEIC_fuzzer: Running 1 inputs 100 time(s) each. > Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/oom-6260c2c81d6888865a26709a89c840b08bf7938b > 10:54:20 0:3.106327 2.970u 281 heif.c/ReadHEIFImageFrame/1264/CorruptImage: > An error has occurred reading from file () > ==281== ERROR: libFuzzer: out-of-memory (used: 2567Mb; limit: 2560Mb) > To change the out-of-memory limit use -rss_limit_mb=<N> > > Live Heap Allocations: 5542475024 bytes in 368 chunks; quarantined: 123299510 bytes in 216 chunks; 37682 other chunks; total chunks: 38266; showing top 95% (at most 8 unique contexts) > 2533359616 byte(s) (45%) in 1 allocation(s) > #0 0x5c063cb139c7 in ___interceptor_posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3 > #1 0x5c063e707bf3 in dav1d_alloc_aligned_internal /src/dav1d/src/mem.h:95:9 > #2 0x5c063e707bf3 in dav1d_decode_frame_init /src/dav1d/src/decode.c:2856:17 > #3 0x5c063e6f19bc in dav1d_worker_task /src/dav1d/src/thread_task.c:702:23 > #4 0x5c063cb107f6 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28 > > 1267916928 byte(s) (22%) in 1 allocation(s) > #0 0x5c063cb139c7 in ___interceptor_posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3 > #1 0x5c063e6d0870 in dav1d_alloc_aligned_internal /src/dav1d/src/mem.h:95:9 > #2 0x5c063e6d0870 in dav1d_mem_pool_pop /src/dav1d/src/mem.c:259:15 > #3 0x5c063e6e114d in dav1d_default_picture_alloc /src/dav1d/src/picture.c:70:26 > #4 0x5c063e6e2234 in picture_alloc /src/dav1d/src/picture.c:137:21 > #5 0x5c063e6e17b1 in dav1d_thread_picture_alloc /src/dav1d/src/picture.c:189:21 > #6 0x5c063e710549 in dav1d_submit_frame /src/dav1d/src/decode.c:3521:11 > #7 0x5c063e6dd1f9 in dav1d_parse_obus /src/dav1d/src/obu.c:1660:24 > #8 0x5c063e6cdd54 in gen_picture /src/dav1d/src/lib.c:420:31 > #9 0x5c063e6cdbfa in dav1d_send_data /src/dav1d/src/lib.c:451:15 > #10 0x5c063d5012e4 in push_pending_data_into_decoder(dav1d_decoder*) [libheif/libheif/plugins/decoder_dav1d.cc:174](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/plugins/decoder_dav1d.cc#L174):15 > #11 0x5c063d5010c2 in dav1d_push_data2(void*, void const*, unsigned long, unsigned long) [libheif/libheif/plugins/decoder_dav1d.cc:224](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/plugins/decoder_dav1d.cc#L224):10 > #12 0x5c063d475564 in Decoder::decode_sequence_frame_from_compressed_data(bool, heif_decoding_options const&, unsigned long, heif_security_limits const*) [libheif/libheif/codecs/decoder.cc:402](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/codecs/decoder.cc#L402):11 > #13 0x5c063d4775b8 in Decoder::decode_single_frame_from_compressed_data(heif_decoding_options const&, heif_security_limits const*) [libheif/libheif/codecs/decoder.cc:479](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/codecs/decoder.cc#L479):23 > #14 0x5c063d133dcf in ImageItem::decode_compressed_image(heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/image-items/image_item.cc:1001](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/image-items/image_item.cc#L1001):19 > #15 0x5c063d12d060 in ImageItem::decode_image(heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/image-items/image_item.cc:747](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/image-items/image_item.cc#L747):60 > #16 0x5c063d3b142a in HeifContext::decode_image(unsigned int, heif_colorspace, heif_chroma, heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/context.cc:1404](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/context.cc#L1404):34 > #17 0x5c063d0ffd8c in heif_decode_image [libheif/libheif/api/libheif/heif_decoding.cc:244](https://github.com/strukturag/libheif/blob/580ca121af7cc4c608473159a6cf6b977d99e9e0/libheif/api/libheif/heif_decoding.cc#L244):81 > #18 0x5c063ce1fe26 in ReadHEIFImageFrame /src/graphicsmagick/coders/heif.c:1246:17 > #19 0x5c063ce1c643 in ReadHEIFImage /src/graphicsmagick/coders/heif.c:2407:10 > #20 0x5c063cc27f26 in ReadImage /src/graphicsmagick/magick/constitute.c:1682:13 > #21 0x5c063cbd9234 in BlobToImage /src/graphicsmagick/magick/blob.c:785:13 > #22 0x5c063cb63316 in Magick::Image::read(Magick::Blob const&) /src/graphicsmagick/Magick++/lib/Image.cpp:1601:5 > #23 0x5c063cb57252 in LLVMFuzzerTestOneInput /src/graphicsmagick/fuzzing/coder_fuzzer.cc:24:11 > #24 0x5c063c9f45bd in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:619:13 > #25 0x5c063c9f5536 in fuzzer::Fuzzer::TryDetectingAMemoryLeak(unsigned char const*, unsigned long, bool) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:695:3 > #26 0x5c063c9df385 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:334:8 > #27 0x5c063c9e5200 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:865:9 > #28 0x5c063ca10d32 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 > #29 0x7d9528fe6082 in __libc_start_main /build/glibc-B3wQXB/glibc-2.31/csu/libc-start.c:308:16 > > 1266679808 byte(s) (22%) in 1 allocation(s) > #0 0x5c063cb12f14 in ___interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 > #1 0x5c063e708ed8 in dav1d_decode_frame_init /src/dav1d/src/decode.c:3009:33 > #2 0x5c063e6f19bc in dav1d_worker_task /src/dav1d/src/thread_task.c:702:23 > #3 0x5c063cb107f6 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28 > > 158334979 byte(s) (2%) in 1 allocation(s) > #0 0x5c063cb12f14 in ___interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 > #1 0x5c063e708e41 in dav1d_decode_frame_init /src/dav1d/src/decode.c:3002:23 > #2 0x5c063e6f19bc in dav1d_worker_task /src/dav1d/src/thread_task.c:702:23 > #3 0x5c063cb107f6 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28 > > 79244928 byte(s) (1%) in 1 allocation(s) > #0 0x5c063cb139c7 in ___interceptor_posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3 > #1 0x5c063e708b46 in dav1d_alloc_aligned_internal /src/dav1d/src/mem.h:95:9 > #2 0x5c063e708b46 in dav1d_decode_frame_init /src/dav1d/src/decode.c:2971:44 > #3 0x5c063e6f19bc in dav1d_worker_task /src/dav1d/src/thread_task.c:702:23 > #4 0x5c063cb107f6 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28 > > 79167488 byte(s) (1%) in 1 allocation(s) > #0 0x5c063cb139c7 in ___interceptor_posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3 > #1 0x5c063e707aa7 in dav1d_alloc_aligned_internal /src/dav1d/src/mem.h:95:9 > #2 0x5c063e707aa7 in dav1d_decode_frame_init /src/dav1d/src/decode.c:2843:17 > #3 0x5c063e6f19bc in dav1d_worker_task /src/dav1d/src/thread_task.c:702:23 > #4 0x5c063cb107f6 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28 > > SUMMARY: libFuzzer: out-of-memory These two test case files were provided: [clusterfuzz-testcase-coder_HEIC_fuzzer-6571634617286656.avif.gz](https://github.com/user-attachments/files/27801873/clusterfuzz-testcase-coder_HEIC_fuzzer-6571634617286656.avif.gz) [clusterfuzz-testcase-minimized-coder_HEIC_fuzzer-6571634617286656.avif.gz](https://github.com/user-attachments/files/27801879/clusterfuzz-testcase-minimized-coder_HEIC_fuzzer-6571634617286656.avif.gz)