Market Insights Server

A real-time commodity tracking system using Apache Spark, OpenAI GPT, and the MCP protocol to generate actionable market insights.

Features

Real-time data collection from Reddit, News APIs, and Yahoo Finance
Scalable processing using Apache Spark (PySpark 3.5.0)
Natural language insights powered by GPT-4
Configurable for any commodity market: energy, metals, agriculture, and more
Built-in dynamic configuration generation and subreddit discovery
Ready for deployment with error handling, retries, and async collection

Requirements

Python 3.x
PySpark 3.5.0
Spark NLP 4.4.0
aiohttp
yfinance
openai
beautifulsoup4

Install dependencies:

pip install -r requirements.txt

Usage

python spark_market_insights_server.py --commodity "nickel"

Outputs:

Cleaned text data from Reddit and news sources
TF-IDF features
GPT-4-powered insight report
JSON export of insights

Architecture

Data Collection Layer
- Async scraping of Reddit and news articles
- Yahoo Finance for live price feeds
Processing Layer (Apache Spark)
- Tokenization → Stop words removal → TF-IDF vectorization
- Supports Spark NLP pipelines
AI Insight Layer
- Uses OpenAI GPT-4 to summarize and synthesize market narratives
Configuration Layer
- Automatically identifies relevant subreddits and keywords per commodity

🔍 Example Output

“Nickel prices rose sharply after Indonesia’s new export ban. Reddit sentiment is bullish, with posts anticipating supply constraints. Market data shows correlated uptick in EV-related equities like NIO and LI.”

🛠 Troubleshooting

| Problem | Fix | |--------|-----| | Spark stage stuck | Check memory settings, repartition input | | API returns 429 | Add backoff/retry logic, rotate API keys | | GPT returns empty | Use latest models, tune prompt |

Coming Soon

Youtube videos to be analyzed as well for the commodities
Public dashboard on Smithery.ai

Contributing

Have a new data source or insight model? PRs welcome!

License

MIT License