We’re thrilled to welcome Bright Data as the newest partner in the Dify Marketplace! Known as the world’s leading web data infrastructure platform, Bright Data offers enterprise-grade solutions to access structured, real-time information from e-commerce sites, social media, and search engines. Its arrival enriches the Marketplace ecosystem, empowering enterprises to seamlessly bring external web data into the Knowledge Pipeline. Together, we enable builders ranging from individuals to enterprises to unlock richer knowledge sources and build more powerful, reliable agentic workflows.
Empower your knowledge pipeline with Extensions in Dify Marketplace
The Knowledge Pipeline is our newest RAG engineering workflow that makes the entire context-building path visible and controllable. Inheriting Dify Workflow's canvas, it turns fragmented, unstructured data — PDFs, PPTs, Excel, HTML, and more — into reliable, model-ready knowledge. Each step is a node, from source connection and document parsing to chunking and embedding, where builders can choose the right plugin for text, images, tables, or scans.
Since launch, the Dify Marketplace has been warmly embraced by builders worldwide. It has quickly grown into a thriving ecosystem, now hosting 500+ plugins — from Models and Tools to Agent Strategies, Extensions, and Data Sources.
Dify Marketplace empowering developers to build, customize, and scale innovative AI solutions with speed and flexibility. Backed by it, builders can assemble knowledge pipelines like building blocks, enrich content with models, apply rule-based cleaning with code, and create transparent, tunable flows that solve RAG's toughest pain points: fragmented sources, parsing loss, and black-box processing.
Among them, Bright Data web scraper joins as a powerful new data source plugin, enabling your workflow to seamlessly capture real-time, structured information from across the web and enrich your Knowledge Pipeline with fresh external knowledge.
Quick Start: Integrate Bright Data with Dify
It’s easy to get started — follow these quick steps to connect Bright Data with Dify and start using real-time web data in your Knowledge Pipeline.
Step 1: Install the Extension
Go to the Dify Marketplace and install the Bright Data Web Scraper plugin.
Step 2: Confirm Installation
After installation, make sure the plugin appears in your Installed Extensions list.
Step 3: Set Up Your Bright Data Account
Sign in to your Bright Data account, copy your API key, and configure your data collection settings.
Go to your Account Settings.
In the API key section, click the Add API key button (top right).
Set the user, permissions, and expiration date (or choose 'Unlimited'), then click Save.
Your API key will be shown only once—make sure to copy and save it securely.
*You can find more details in the Bright Data official documentation: How to generate an API key.
Step 4: Integrate Bright Data with Dify
In Dify’s Settings → Configuration, enter your Bright Data API key to authorize the connection between the two platforms.
Step 5: Build Your Knowledge Pipeline
You're all set!
Create a Knowledge Pipeline in Dify, add a Bright Data data source, and start collecting structured, real-time data to enrich your AI workflows.
About Bright Data
Bright Data is the global leader in web data collection, providing businesses and developers with powerful, reliable tools to access real-time, structured information from e-commerce sites, social media, search engines, and more. Its enterprise-grade scraping infrastructure and 72M+ proxy network enable customers to gather critical external data at scale while ensuring accuracy and compliance.
Trusted by over 20,000 organizations worldwide — including Fortune 500 companies, academic institutions, and startups — Bright Data powers data-driven decisions across industries from market research and e-commerce to finance and brand monitoring. For more information, visit: www.brightdata.com.
About Dify.AI
Dify.AI is revolutionizing AI-native application development by providing an open-source platform that simplifies the entire lifecycle of AI application creation, deployment, and management. With its extensible plugin ecosystem, Dify.AI enables developers and businesses to seamlessly integrate AI capabilities, customize workflows, and accelerate innovation. By lowering the barriers to AI adoption, Dify.AI empowers users to build intelligent applications with greater efficiency and flexibility.
