vector-index-tuning - implementation-playbook

向量索引调优实施手册本文件包含该技能引用的详细模式、检查清单和代码示例。核心概念1. 索引类型选择数据大小 推荐的索引 ──────────────────────────────────────── 10K 向量 → Flat精确搜索 10K - 1M → HNSW 1M - 100M → HNSW 量化 100M → IVF PQ 或 DiskANN2. HNSW 参数参数默认值效果M16每个节点的连接数↑ 召回率更好内存更多efConstruction100构建质量↑ 索引更好构建更慢efSearch50搜索质量↑ 召回率更好搜索更慢3. 量化类型全精度FP324 字节 × 维度 半精度FP162 字节 × 维度 INT8 标量 1 字节 × 维度 乘积量化 约 32-64 字节总计 二值化 维度/8 字节模板模板 1HNSW 参数调优importnumpyasnpfromtypingimportList,Tupleimporttimedefbenchmark_hnsw_parameters(vectors:np.ndarray,queries:np.ndarray,ground_truth:np.ndarray,m_values:List[int][8,16,32,64],ef_construction_values:List[int][64,128,256],ef_search_values:List[int][32,64,128,256])-List[dict]:Benchmark different HNSW configurations.importhnswlib results[]dimvectors.shape[1]nvectors.shape[0]forminm_values:foref_constructioninef_construction_values:# Build indexindexhnswlib.Index(spacecosine,dimdim)index.init_index(max_elementsn,Mm,ef_constructionef_construction)build_starttime.time()index.add_items(vectors)build_timetime.time()-build_start# Get memory usagememory_bytesindex.element_count*(dim*4# Vector storagem*2*4# Graph edges (approximate))foref_searchinef_search_values:index.set_ef(ef_search)# Measure searchsearch_starttime.time()labels,distancesindex.knn_query(queries,k10)search_timetime.time()-search_start# Calculate recallrecallcalculate_recall(labels,ground_truth,k10)results.append({M:m,ef_construction:ef_construction,ef_search:ef_search,build_time_s:build_time,search_time_ms:search_time*1000/len(queries),recall10:recall,memory_mb:memory_bytes/1024/1024})returnresultsdefcalculate_recall(predictions:np.ndarray,ground_truth:np.ndarray,k:int)-float:Calculate recallk.correct0forpred,truthinzip(predictions,ground_truth):correctlen(set(pred[:k])set(truth[:k]))returncorrect/(len(predictions)*k)defrecommend_hnsw_params(num_vectors:int,target_recall:float0.95,max_latency_ms:float10,available_memory_gb:float8)-dict:Recommend HNSW parameters based on requirements.# Base recommendationsifnum_vectors100_000:m16ef_construction100elifnum_vectors1_000_000:m32ef_construction200else:m48ef_construction256# Adjust ef_search based on recall targetiftarget_recall0.99:ef_search256eliftarget_recall0.95:ef_search128else:ef_search64return{M:m,ef_construction:ef_construction,ef_search:ef_search,notes:fEstimated for{num_vectors:,}vectors,{target_recall:.0%}recall}模板 2量化策略importnumpyasnpfromtypingimportOptionalclassVectorQuantizer:Quantization strategies for vector compression.staticmethoddefscalar_quantize_int8(vectors:np.ndarray,min_val:Optional[float]None,max_val:Optional[float]None)-Tuple[np.ndarray,dict]:Scalar quantization to INT8.ifmin_valisNone:min_valvectors.min()ifmax_valisNone:max_valvectors.max()# Scale to 0-255 rangescale255.0/(max_val-min_val)quantizednp.clip(np.round((vectors-min_val)*scale),0,255).astype(np.uint8)params{min_val:min_val,max_val:max_val,scale:scale}returnquantized,paramsstaticmethoddefdequantize_int8(quantized:np.ndarray,params:dict)-np.ndarray:Dequantize INT8 vectors.returnquantized.astype(np.float32)/params[scale]params[min_val]staticmethoddefproduct_quantize(vectors:np.ndarray,n_subvectors:int8,n_centroids:int256)-Tuple[np.ndarray,dict]:Product quantization for aggressive compression.fromsklearn.clusterimportKMeans n,dimvectors.shapeassertdim%n_subvectors0subvector_dimdim//n_subvectors codebooks[]codesnp.zeros((n,n_subvectors),dtypenp.uint8)foriinrange(n_subvectors):starti*subvector_dim end(i1)*subvector_dim subvectorsvectors[:,start:end]kmeansKMeans(n_clustersn_centroids,random_state42)codes[:,i]kmeans.fit_predict(subvectors)codebooks.append(kmeans.cluster_centers_)params{codebooks:codebooks,n_subvectors:n_subvectors,subvector_dim:subvector_dim}returncodes,paramsstaticmethoddefbinary_quantize(vectors:np.ndarray)-np.ndarray:Binary quantization (sign of each dimension).# Convert to binary: positive 1, negative 0binary(vectors0).astype(np.uint8)# Pack bits into bytesn,dimvectors.shape packed_dim(dim7)//8packednp.zeros((n,packed_dim),dtypenp.uint8)foriinrange(dim):byte_idxi//8bit_idxi%8packed[:,byte_idx]|(binary[:,i]bit_idx)returnpackeddefestimate_memory_usage(num_vectors:int,dimensions:int,quantization:strfp32,index_type:strhnsw,hnsw_m:int16)-dict:Estimate memory usage for different configurations.# Vector storagebytes_per_dimension{fp32:4,fp16:2,int8:1,pq:0.05,# Approximatebinary:0.125}vector_bytesnum_vectors*dimensions*bytes_per_dimension[quantization]# Index overheadifindex_typehnsw:# Each node has ~M*2 edges, each edge is 4 bytes (int32)index_bytesnum_vectors*hnsw_m*2*4elifindex_typeivf:# Inverted lists centroidsindex_bytesnum_vectors*865536*dimensions*4else:index_bytes0total_bytesvector_bytesindex_bytesreturn{vector_storage_mb:vector_bytes/1024/1024,index_overhead_mb:index_bytes/1024/1024,total_mb:total_bytes/1024/1024,total_gb:total_bytes/1024/1024/1024}模板 3Qdrant 索引配置fromqdrant_clientimportQdrantClientfromqdrant_client.httpimportmodelsdefcreate_optimized_collection(client:QdrantClient,collection_name:str,vector_size:int,num_vectors:int,optimize_for:strbalanced# recall, speed, memory)-None:Create collection with optimized settings.# HNSW configuration based on optimization targethnsw_configs{recall:models.HnswConfigDiff(m32,ef_construct256),speed:models.HnswConfigDiff(m16,ef_construct64),balanced:models.HnswConfigDiff(m16,ef_construct128),memory:models.HnswConfigDiff(m8,ef_construct64)}# Quantization configurationquantization_configs{recall:None,# No quantization for max recallspeed:models.ScalarQuantization(scalarmodels.ScalarQuantizationConfig(typemodels.ScalarType.INT8,quantile0.99,always_ramTrue)),balanced:models.ScalarQuantization(scalarmodels.ScalarQuantizationConfig(typemodels.ScalarType.INT8,quantile0.99,always_ramFalse)),memory:models.ProductQuantization(productmodels.ProductQuantizationConfig(compressionmodels.CompressionRatio.X16,always_ramFalse))}# Optimizer configurationoptimizer_configs{recall:models.OptimizersConfigDiff(indexing_threshold10000,memmap_threshold50000),speed:models.OptimizersConfigDiff(indexing_threshold5000,memmap_threshold20000),balanced:models.OptimizersConfigDiff(indexing_threshold20000,memmap_threshold50000),memory:models.OptimizersConfigDiff(indexing_threshold50000,memmap_threshold10000# Use disk sooner)}client.create_collection(collection_namecollection_name,vectors_configmodels.VectorParams(sizevector_size,distancemodels.Distance.COSINE),hnsw_confighnsw_configs[optimize_for],quantization_configquantization_configs[optimize_for],optimizers_configoptimizer_configs[optimize_for])deftune_search_parameters(client:QdrantClient,collection_name:str,target_recall:float0.95)-dict:Tune search parameters for target recall.# Search parameter recommendationsiftarget_recall0.99:search_paramsmodels.SearchParams(hnsw_ef256,exactFalse,quantizationmodels.QuantizationSearchParams(ignoreTrue,# Dont use quantization for searchrescoreTrue))eliftarget_recall0.95:search_paramsmodels.SearchParams(hnsw_ef128,exactFalse,quantizationmodels.QuantizationSearchParams(ignoreFalse,rescoreTrue,oversampling2.0))else:search_paramsmodels.SearchParams(hnsw_ef64,exactFalse,quantizationmodels.QuantizationSearchParams(ignoreFalse,rescoreFalse))returnsearch_params模板 4性能监控importtimefromdataclassesimportdataclassfromtypingimportListimportnumpyasnpdataclassclassSearchMetrics:latency_p50_ms:floatlatency_p95_ms:floatlatency_p99_ms:floatrecall:floatqps:floatclassVectorSearchMonitor:Monitor vector search performance.def__init__(self,ground_truth_fnNone):self.latencies[]self.recalls[]self.ground_truth_fnground_truth_fndefmeasure_search(self,search_fn,query_vectors:np.ndarray,k:int10,num_iterations:int100)-SearchMetrics:Benchmark search performance.latencies[]for_inrange(num_iterations):forqueryinquery_vectors:starttime.perf_counter()resultssearch_fn(query,kk)latency(time.perf_counter()-start)*1000latencies.append(latency)latenciesnp.array(latencies)total_queriesnum_iterations*len(query_vectors)total_timesum(latencies)/1000# secondsreturnSearchMetrics(latency_p50_msnp.percentile(latencies,50),latency_p95_msnp.percentile(latencies,95),latency_p99_msnp.percentile(latencies,99),recallself._calculate_recall(search_fn,query_vectors,k)ifself.ground_truth_fnelse0,qpstotal_queries/total_time)def_calculate_recall(self,search_fn,queries:np.ndarray,k:int)-float:Calculate recall against ground truth.ifnotself.ground_truth_fn:return0correct0total0forqueryinqueries:predictedset(search_fn(query,kk))actualset(self.ground_truth_fn(query,kk))correctlen(predictedactual)totalkreturncorrect/totaldefprofile_index_build(build_fn,vectors:np.ndarray,batch_sizes:List[int][1000,10000,50000])-dict:Profile index build performance.results{}forbatch_sizeinbatch_sizes:times[]foriinrange(0,len(vectors),batch_size):batchvectors[i:ibatch_size]starttime.perf_counter()build_fn(batch)times.append(time.perf_counter()-start)results[batch_size]{avg_batch_time_s:np.mean(times),vectors_per_second:batch_size/np.mean(times)}returnresults最佳实践该做的使用真实查询做基准测试- 合成数据可能不代表生产环境持续监控召回率- 数据漂移可能导致其退化从默认值开始- 仅在需要时调优使用量化- 可显著节省内存考虑分层存储- 热/冷数据分离不该做的不要过早过度优化- 先做性能分析不要忽视构建时间- 索引更新是有成本的不要忘记重建索引- 规划维护不要跳过预热- 冷索引很慢资源HNSW 论文Faiss WikiANN 基准