CHUNK STRUCTURE
HowDocsare Split.
Match the strategy to your document type.
StandardGeneral
Long ContextParent-child
TablesQ&A
Knowledge Pipeline
Craft Reliable Knowledge Pipelines for AI.
Knowledge Pipeline turns files, drives, online docs, and web content into a knowledge base that AI apps use with structure, metadata, and citations. Design, debug, and reuse the path from raw source to context in one visual canvas. Everything fully traceable.
One Pipeline, Source to Retrieval.
Each stage is a swappable node on the same canvas as Workflow Studio. Test the whole flow before any app connects.
Source
Files
Online Docs
Drives
Crawlers
Extract
Text
Tables
Images
Scans
Process
Chunk
Enrich
Clean
Code
Store
Vector
Full-Text
Metadata
Images
Retrieve
Semantic
Keyword
Hybrid
Rerank
Tune Retrieval At the Data Layer.
Each knowledge base carries its own chunking, indexing, retrieval, and refinement.
INDEX METHOD
HowChunksare stored.
Trade embedding cost for semantic depth.
EmbeddingsHigh Quality
KeywordsEconomical
RETRIEVAL METHOD
HowQueriesmatch.
Search by meaning, keywords, or both.
SemanticVector
KeywordFull-text
BothHybrid
REFINEMENT
HowContextis shaped.
Narrow, reorder, and attribute results before they reach the app.
ScopeMetadata Filters
OrderRerank
AttributionCitations
ImagesMultimodal
Observable.
Reusable.
A knowledge pipeline is not a one-off ETL script. It is a workspace asset your team can debug, share, and connect to every Dify app type.
Observable
See What Each Stage Does.
Test Run executes any stage. Variable Inspect surfaces intermediate values. Retrieval Testing simulates real queries before any app connects.
Reusable
One Base, Every App Type.
Publish once. Workflow, Chatflow, and Agent apps retrieve from it through the Knowledge Retrieval node, with citations enabled per app.
Build Your First
Knowledge Pipeline.
Start in Dify Cloud, or talk to us about private deployment.