[Bug]: During deleting the entities, it took about 12 minutes to recover from the delegator failure to search

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20241106-20534a3f-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server config

• qn: 4*8c32g
• deleteBufferRowCountProtection: [12m, 25m]
• l0SegmentsRowCountProtection: [25m 50m]
• levelZeroForwardPolicy: RemoteLoad
• streamingDeltaForwardPolicy: FilterByBF
• taskPrioritizer: level

test steps

1. Continuously run concurrent search, deleting 60 million of data in batches of 60,000 pks
2. 08:15:54 delete done
3. 08:16:30 kill-9 delegator qn-mcpj4
   

results

• qn oom during recovery

  • 08:19 qn-mcpj4 oom
  • 08:22 qn-tqvd5 & qn-sdqrp oom
  • 08:26 qn-gzlwz oom
    

• search recovery: 12min
  

• metrics of compact-opt-100m-3

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response