DuplicatesStrategy
Defined in: packages/core/src/ingestion/strategies/DuplicatesStrategy.ts:10
Handle duplicates by checking if documents already exist in the vector store. Documents that already exist (by ref_doc_id) are skipped. Note: This does NOT detect content changes - use UPSERTS strategy if you need to update changed documents.
Extends
Section titled “Extends”DuplicatesStrategy<
Options>(nodes,options?):BaseNode<Metadata>[] |Promise<BaseNode<Metadata>[]>
Defined in: packages/core/src/ingestion/strategies/DuplicatesStrategy.ts:10
Handle duplicates by checking if documents already exist in the vector store. Documents that already exist (by ref_doc_id) are skipped. Note: This does NOT detect content changes - use UPSERTS strategy if you need to update changed documents.
Type Parameters
Section titled “Type Parameters”Options
Section titled “Options”Options extends Record<string, unknown>
Parameters
Section titled “Parameters”options?
Section titled “options?”Options
Returns
Section titled “Returns”BaseNode<Metadata>[] | Promise<BaseNode<Metadata>[]>
Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new DuplicatesStrategy(
vectorStore):DuplicatesStrategy
Defined in: packages/core/src/ingestion/strategies/DuplicatesStrategy.ts:13
Parameters
Section titled “Parameters”vectorStore
Section titled “vectorStore”Returns
Section titled “Returns”DuplicatesStrategy
Overrides
Section titled “Overrides”RollbackableTransformComponent.constructor
Properties
Section titled “Properties”id:
string
Defined in: packages/core/src/schema/type.ts:22
Inherited from
Section titled “Inherited from”RollbackableTransformComponent.id
Methods
Section titled “Methods”rollback()
Section titled “rollback()”rollback(
vectorStore,nodes):Promise<void>
Defined in: packages/core/src/ingestion/strategies/rollback.ts:9
Remove all nodes for documents that exist in the vector store. Useful in case generating embeddings fails and we want to remove partially added docs.
Parameters
Section titled “Parameters”vectorStore
Section titled “vectorStore”Returns
Section titled “Returns”Promise<void>