TokenTextSplitterParams
TokenTextSplitterParams =
object
Defined in: packages/core/src/schema/params.ts:96
Parameters for the TokenTextSplitter
Properties
Section titled “Properties”chunkSize
Section titled “chunkSize”chunkSize:
number
Defined in: packages/core/src/schema/params.ts:98
The token chunk size for each chunk
chunkOverlap
Section titled “chunkOverlap”chunkOverlap:
number
Defined in: packages/core/src/schema/params.ts:100
The token overlap of each chunk when splitting
separator
Section titled “separator”separator:
string
Defined in: packages/core/src/schema/params.ts:102
Default separator for splitting
backupSeparators
Section titled “backupSeparators”backupSeparators:
string[]
Defined in: packages/core/src/schema/params.ts:104
Backup separators to try if primary fails