MCP Server Dataset Builder

MCP Server Dataset Builder

A comprehensive tool for building and maintaining a dataset of Model Context Protocol (MCP) servers. This tool automatically collects, categorizes, and updates information about MCP servers from multiple sources.

Overview

The MCP Server Dataset Builder is designed to:

1. Extract MCP server information from the awesome-mcp-servers repository
2. Search GitHub for additional MCP server repositories
3. Merge and deduplicate data from both sources
4. Generate a daily CSV file with comprehensive information about each server

Features

• Dual Data Sources: Combines data from curated lists and GitHub search
• Automatic Categorization: Assigns categories based on repository content
• Tech Stack Detection: Identifies programming languages and frameworks
• Emoji Tagging: Adds visual indicators for quick identification
• Daily Updates: Automatically runs to keep the dataset current
• Data Persistence: Maintains historical data while adding new entries

Dataset Structure

The generated CSV files contain the following fields:

| Field | Description | |-------|-------------| | name | Repository name | | description | Repository description | | html_url | URL to the repository | | stars | Number of GitHub stars | | forks | Number of GitHub forks | | keywords | Comma-separated list of keywords | | category | Primary category (e.g., framework, utility, client) | | techstack | Comma-separated list of technologies used | | emojis | Visual indicators for quick identification |

Usage

Automatic Daily Updates

The dataset is automatically updated daily via GitHub Actions. No manual intervention is required.

Manual Trigger

You can manually trigger the workflow from the GitHub Actions tab:

1. Go to the "Actions" tab in the repository
2. Select "Unified MCP Servers Extraction"
3. Click "Run workflow"
4. Optionally customize:
   • Keywords for GitHub search
   • Minimum stars and forks thresholds
   • Which extraction methods to run

Local Development

To run the scripts locally:

# Install dependencies
pip install -r requirements.txt

# Run README extraction
python extract_mcp_servers.py

# Run GitHub search
python daily.py

Environment Variables

The following environment variables can be used to customize the behavior:

| Variable | Description | Default | |----------|-------------|---------| | GITHUB_TOKEN | GitHub API token for authentication | - | | KEYWORDS_ENV | Comma-separated list of search keywords | MCP-related keywords | | MIN_STARS | Minimum number of stars for repositories | 10 | | MIN_FORKS | Minimum number of forks for repositories | 5 |

Data Sources

1. Awesome MCP Servers Repository

The tool extracts data from the awesome-mcp-servers repository, which contains a curated list of MCP servers organized by category.

2. GitHub Search

The tool searches GitHub for repositories matching MCP-related keywords, ensuring comprehensive coverage of the ecosystem.

Categorization System

Repositories are categorized based on their content and purpose:

• Framework: Core MCP server implementations
• Utility: Helper tools and utilities
• Client: Client libraries and applications
• Tutorial: Learning resources and examples
• Database: Database integrations
• API: API implementations
• Storage: Storage solutions
• AI: AI and LLM integrations
• Chat: Chat and messaging features
• Search: Search functionality

Tech Stack Detection

The tool identifies the following technologies:

• Languages: Python, TypeScript, Go, Rust, Java, C#
• Frameworks: FastAPI, Langchain, Spring
• Protocols: SSE, WebSocket, HTTP
• Deployment: Cloud, Local, Docker
• Platforms: iOS, Windows, Linux

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.