segfault crash in knn::BinaryQuantizer_c::Quantize4Bit when threads>1 used in query (not thread safe code)
bug
### Bug Description:
intermittent segfault crash in knn::BinaryQuantizer_c::Quantize4Bit when query uses [threads>1] attached reproducible case, if crash doesn't happen restart manticore server and try again - just running the queries (don't reload the table again) - sometimes a few restarts are required. Looks like bug is caused by SpanResizeable_T<float> m_dVecMinusCentroid;
SpanResizeable_T<uint8_t> m_dQuantized; being defined as private members of BinaryQuantizer_c, which is reused between different threads. After I moved m_dVecMinusCentroid and m_dQuantized into the method Quantize4Bit - on the stack - the crash stopped. Not sure about performance repercussions of moving these to stack and if it's a good fix or not.
```
to reproduce, create and populate the table with
python test_manticore_crash.py --populate --data-file vector_data.tsv
then run
python test_manticore_crash.py --query --query-file bench_static_mrl_v1_d256_q1bit_c224o80.vec
to induce the crash, sometimes multiple runs are required.
```
seems the tickets
https://github.com/manticoresoftware/manticoresearch/issues/3854
https://github.com/manticoresoftware/manticoresearch/issues/4212
are the same bug.
(gdb) where
#0 0x00007fbcfbb4e1fd in knn::BinaryQuantizer_c::Quantize4Bit (this=0x7fb8dd000760, dVector=..., dCentroid=std::vector of length 256, capacity 256 = {...},
dResult=std::vector of length 0, capacity 0) at /var/lib/manticore/src/columnar/knn/quantizer.cpp:603
#1 0x00007fbcfba526f5 in knn::HNSWIndex_c::Search (this=0x7fb8dd0007e0, dResults=std::vector of length 0, capacity 0, dData=..., iResults=60, iEf=0,
dQuantized=std::vector of length 0, capacity 0, pDistanceComputations=0x0, pFilter=0x0, ePolicy=knn::HNSWTerminationPolicy_e::QUANTILE)
at /var/lib/manticore/src/columnar/knn/knn.cpp:362
#2 0x00007fbcfbb4a0a3 in knn::RowidIteratorKNN_c::RowidIteratorKNN_c (this=0x7fbcf000f060, tIndex=..., dData=..., iResults=<optimized out>, iEf=<optimized out>,
bCollectMetrics=<optimized out>, pFilter=0x0, ePolicy=knn::HNSWTerminationPolicy_e::QUANTILE) at /var/lib/manticore/src/columnar/knn/iterator.cpp:92
#3 0x00007fbcfbb4a174 in knn::CreateIterator (tIndex=..., dData=..., iResults=60, iEf=0, bCollectMetrics=<optimized out>, pFilter=0x0, ePolicy=knn::HNSWTerminationPolicy_e::QUANTILE)
at /var/lib/manticore/src/columnar/knn/iterator.cpp:143
#4 0x0000000001070b82 in CreateKNNIterator (pKNN=pKNN@entry=0x7fb8dc53d070, tQuery=..., tIndexSchema=..., tSorterSchema=..., pFilter=pFilter@entry=0x0,
ePolicy=ePolicy@entry=knn::HNSWTerminationPolicy_e::QUANTILE, pProfile=0x0, sError=...) at /var/lib/manticore/src/manticoresearch/src/knnmisc.cpp:794
#5 0x00000000010719fc in CreateKNNIterators (pKNN=0x7fb8dc53d070, tQuery=..., tIndexSchema=..., tSorterSchema=..., pFilter=pFilter@entry=0x0,
ePolicy=knn::HNSWTerminationPolicy_e::QUANTILE, pProfile=0x0, bError=@0x7fb9640f8f0f: false, sError=...) at /var/lib/manticore/src/manticoresearch/src/knnmisc.cpp:827
#6 0x0000000000d5a441 in CSphIndex_VLN::SpawnIterators (this=this@entry=0x7fb8dc53c7a0, tQuery=..., dFilters=..., dJsonSITransforms=..., tCtx=..., tFlx=..., tMaxSorterSchema=...,
dSorterSchemas=..., pModifiedMatchSchema=std::unique_ptr<ISphSchema> = {...}, tMeta=..., iCutoff=-1, iThreads=6, dModifiedFilters=..., bUseSICache=false, pRanker=0x0)
at /usr/include/c++/11/bits/unique_ptr.h:173
#7 0x0000000000d5f396 in CSphIndex_VLN::MultiScan (this=0x7fb8dc53c7a0, tResult=..., tQuery=..., dSorters=..., tArgs=..., tmMaxTimer=0)
at /var/lib/manticore/src/manticoresearch/src/sphinx.cpp:8878
#8 0x0000000000d70d6c in operator() (tMultiArgs=..., dLocalSorters=..., tQuery=..., tChunkResult=..., __closure=<optimized out>)
at /var/lib/manticore/src/manticoresearch/src/sphinx.cpp:11462
#9 operator() (__closure=0x7fb9640da670) at /var/lib/manticore/src/manticoresearch/src/sphinx.cpp:11322
#10 0x0000000001002426 in std::function<void()>::operator() (this=0x7fb964018f98) at /usr/include/c++/11/bits/std_function.h:590
#11 operator() (__closure=0x7fb964018f90) at /var/lib/manticore/src/manticoresearch/src/task_info.cpp:158
#12 std::__invoke_impl<void, myinfo::OwnMini(Threads::Handler)::<lambda()>&> (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#13 std::__invoke_r<void, myinfo::OwnMini(Threads::Handler)::<lambda()>&> (__fn=...) at /usr/include/c++/11/bits/invoke.h:111
#14 std::_Function_handler<void(), myinfo::OwnMini(Threads::Handler)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/11/bits/std_function.h:290
#15 0x0000000001ce513c in std::function<void()>::operator() (this=0x7fb9640da760) at /usr/include/c++/11/bits/std_function.h:590
#16 Threads::CoRoutine_c::WorkerLowest (tCtx=<optimized out>, this=0x7fb9640da758) at /var/lib/manticore/src/manticoresearch/src/coroutine.cpp:64
#17 Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::operator()(boost::context::detail::transfer_t) const (pT=..., __closure=0x0) at /var/lib/manticore/src/manticoresearch/src/coroutine.cpp:80
#18 Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::_FUN(boost::context::detail::transfer_t) () at /var/lib/manticore/src/manticoresearch/src/coroutine.cpp:81
#19 0x000000000210ef7f in make_fcontext ()
#20 0x0000000000000000 in ?? ()
```
### Manticore Search Version:
Manticore 25.0.0 ce3c27828@26032712 (columnar 13.0.0 e60b083@26032708) (secondary 13.0.0 e60b083@26032708) (knn 13.0.0 e60b083@26032708) (embeddings 1.1.1 e60b083@26032708)
### Operating System Version:
Almalinux 9
### Have you tried the latest development version?
Yes
### Internal Checklist:
<sup>To be completed by the assignee. Check off tasks that have been completed or are not applicable.</sup>
<details>
- [ ] Implementation completed
- [ ] Tests developed
- [ ] Documentation updated
- [ ] Documentation reviewed
</details>
1 条评论