Hello everyone, I’m Yeuoly, a backend engineer at Dify, and the primary author of Dify’s plugin system. With the release of Dify v1.0.0, the plugin system has become a key feature. It separates horizontally scalable modules from Dify and operates them as independent runtimes. This mechanism brings several notable changes:

• Module Decoupling: Before pluginization, models and tools in Dify needed to be fully installed with Dify, and their code was tightly coupled with the main repository. After pluginization, these modules became independent plugin packages that can be installed, uninstalled, and run independently, greatly enhancing flexibility.

• Plugin Marketplace & Sharing Mechanism: We’ve introduced a plugin marketplace where users and the community can freely create and share plugins.

• Endpoint Plugin: A new Endpoint plugin enables more real-world use cases to be integrated into Dify’s internal ecosystem.

While these changes are obvious from a user experience perspective, for most users, the technical implementation of the plugin system remains a relatively obscure black box. This article will take you through the design philosophy, user value, and technical implementation details of the plugin system.

User Needs Analysis

Before designing the plugin system, we faced several key challenges:

1. Tightly Coupled Code: Adding new models and tools to Dify was cumbersome and introduced too many dependencies, leading to version management issues for tools and models.

2. Incomplete User Demand: Some requirements, such as integrating IM services, required wrapping an additional layer of services outside of Dify.

3. Fixed Custom Modules: For example, Dify’s PDF parser didn’t perform well, and customized modules like RAG couldn’t be easily adjusted.

To address these problems, we decided to implement a unified framework to decouple Dify’s tools and models, allowing them to be installed independently and selected as needed. Functions related to RAG, such as document parsers and OCR, were also pluginized to meet different use case needs. For scenarios that couldn’t be fully closed within Dify, the plugin system provided open interfaces to integrate with external systems, such as supporting outgoing webhooks for IM platforms.

These product requirements may seem simple, but the engineering implementation was quite challenging. In less than a week during the initial design phase, we encountered a series of issues:

• Multiple Workspace Design: Dify is a multi-workspace design, meaning that functionality can’t simply be implemented by mounting Python source code to the tools/models directory. This would also lead to dependency conflicts.

• Plugin Environment Consistency: We wanted plugins to behave consistently across different environments. While Docker could solve this, allocating a separate Docker container for each plugin would significantly increase deployment complexity.

• High Concurrency Cloud Service Load: Dify’s SaaS service has hundreds of thousands of users. If every 10 users had a custom plugin, Dify would face the runtime load of tens of thousands of plugins, putting significant pressure on cloud service costs.

• Plugin Development & Debugging: As developers, each time we modify code, we’d need to repack and reinstall the plugin, and logs would have to be sent to Dify’s backend, which significantly impacted the development experience.

• Long-Term Plugin Runtime: For instance, should plugins be allowed to run an HTTP server long-term to listen for IM platform webhook events?

Solutions

Debugging Experience

Before thinking about how to implement the plugin system, we first considered how to optimize the debugging experience, especially for developers. An ideal debugging experience should meet the following two requirements:

1. What You See Is What You Get: After modifying the code, no installation is needed; it should take effect directly in Dify.

2. Local Debugging: The plugin’s code should run locally so that we can debug it more easily with breakpoints.

We referenced well-established debuggers like GDB and used a separation of debugger and runtime: the debugger waits for the runtime to initiate a connection. Once the connection is established, the local plugin can create a long connection with Dify, and Dify treats it as an installed plugin, marking it for debugging. User requests are forwarded via this long connection to the local plugin, and the plugin’s response is sent back to Dify, achieving a smooth debugging experience.

However, this design faced an issue: long connections are stateful, but Dify’s current services are stateless. In a Kubernetes cluster, load balancing routes requests to different Dify pods. For example, Plugin 1 connects to Dify 1, and Plugin 2 connects to Dify 2. When a user requests Plugin 1, the request might be routed to Dify 2, making Plugin 1 inaccessible. To resolve this, we needed to implement a traffic forwarding mechanism.

Endpoint Plugin

We studied many existing IM and office collaboration software solutions and, based on current requirements, clarified the key issue to solve: how to make Dify receive webhook requests from these platforms and let plugins handle these HTTP requests. For example, using Dify’s App to process user messages.

To solve this, we designed a mechanism to generate random URLs and integrated these URLs with platforms like Discord. This approach avoids the need for each plugin to run a server long-term since Dify takes on the responsibility of forwarding HTTP requests. Dify uses the generated URL to receive webhook requests from platforms, and the plugin processes the forwarded requests.

After solving how to receive messages, the next challenge is how to process them. For example, if I’m developing a Discord bot and want Dify’s chatflow to reply to user messages, the code might look like this:

class Webhook:
    def _invoke(self, r: Request) -> Response:
        message = r.json()['message']
        response = invoke_app(app_id, message)
        return Response(response)

In this plugin, I need to invoke Dify’s app to handle the request, enabling the bot functionality. This introduces a key concept in Dify v1.0: Reverse Call.

Reverse Call

Reverse call is a critical concept in Dify’s plugin system, allowing plugins to call internal Dify services. For instance, plugins can call authenticated models, tools, or Dify’s apps. Reverse calls play a crucial role in several scenarios:

• LlamaIndex Implementation: LlamaIndex implements various agentic RAG strategies to summarize a retrieved list using LLMs. In Dify, it is used as a tool where users only need to configure model parameters and input lists, and the tool can be installed or uninstalled.

• Models as Tools: Previously, OCR, ASR, and TTS models could only be used as standalone models. Now, they can be used as tools, such as using Gemini as an OCR tool to simplify the operation process.

• OpenAI-Compatible API: Through the Endpoint plugin, Dify provides OpenAI-compatible formats, allowing plugins to call Dify’s app and return responses in a unified format, supporting different models like Claude or Gemini.

• Agents as Plugins: Reverse tool calling enables Agent pluginization, automating parameter reception, operation execution, result delivery, and custom Agent strategy implementation.

Implementation Details

The design of the plugin runtime was the first challenge we needed to solve. Ultimately, should the plugin runtime be a Docker container, a process, a virtual machine, or a serverless runtime? After evaluating Dify’s user base, we decided to implement four completely different runtimes:

• Local Deployment: Aimed at small teams and individual developers, with relatively low deployment demands, focusing on high availability without requiring large-scale use.

• SaaS Service: Designed for hundreds of thousands of users, Dify needs to consider high user load.

• Enterprise Version: Similar to SaaS, but the enterprise version requires higher controllability, privacy protection, and private deployment.

• Remote Debugging: Supports debugging mode and must consider providing runtime support for it.

Local Deployment

The local deployment version emphasizes “one-click deployment” and out-of-the-box usability. Users can run the entire Dify with a simple docker compose up -d command and install plugins without additional configuration. In this environment, the plugin runtime is designed as a subprocess managed by the parent process, which controls the lifecycle of the plugin, including installing dependencies. Communication between the two occurs through standard input-output pipes.

SaaS Service

Given the needs of large-scale users, the SaaS version was designed with a serverless architecture that can scale elastically based on usage, ensuring high concurrency, resource utilization, and availability. Ultimately, we chose AWS Lambda as the solution, as AWS, Dify’s partner, already supports Dify’s existing SaaS business, and Dify communicates with Lambda over the network, providing a suitable architecture.

Enterprise Version

Since we can’t force enterprise clients to choose Lambda as a solution, we designed a controllable and trusted runtime. This version offers high controllability and privacy protection, supporting private deployment within enterprises.

Remote Debugging

For debugging mode, Dify supports debugging plugins via TCP network long connections and resolves the stateful issue. We used a design similar to etcd and managed plugin connection states through Redis HashMap to ensure that plugin requests are forwarded to the correct pods. We set up two mechanisms for managing IPs:

• The cluster must maintain an IP pool, and pods will add their IPs to the pool. In production environments, a machine may have multiple IPs across different subnets, so pods use a voting mechanism to test reachability and mark usable IPs.

• The cluster requires a master node that periodically checks the health of the pods, cleaning up the status of exited nodes to ensure cluster stability.

Security

System Security

The security of the plugin system relies on cryptographic signatures rather than restrictive sandboxing. Sandboxing has strict limitations that prevent many dependency packages from being used due to potential system operations that haven’t been authorized, severely impacting the plugin experience. In contrast, plugins are already written code packages, and before installation, manual review can label them as “safe,” significantly reducing risks. We use a public-key cryptography-based signature strategy: if a plugin passes review, we sign it with a private key, marking it as “certified,” and users will see an “unsafe” warning if it doesn’t pass review. Unsigned plugins cannot be installed unless users manually change settings.

Privacy Policy

Plugins must declare their permissions, data storage, and other privacy policies, especially those involving sensitive data. For simple permission declarations, plugin developers must explicitly declare the functional permissions of the plugin, and Dify will directly reject unlisted permissions. For more complex privacy policies, developers must provide a detailed privacy strategy referenced in the manifest, and all plugins listed in the marketplace need to undergo privacy policy review.

Conclusion

This article briefly introduces the design and technical implementation of Dify’s plugin system. Through pluginization, Dify offers greater flexibility and customization, supports multiple use cases, and provides a better debugging and development experience for developers. We’ve open-sourced the related code, and we welcome everyone to participate and contribute new plugin ideas:

• dify-plugin-daemon

• dify-plugin-sdks

• dify

We look forward to progressing together with the community and exploring more innovative applications.