← Back to all projects

News

News project screenshot

A large-scale, fully automated news discovery and aggregation platform that collects articles daily from tens of thousands of global news sources. Features automatic categorization, historical archiving, intelligent deduplication, and a searchable Flask web interface for exploring global media output.

Visit SiteView Source

Technology

backendpythonflask
databasesqlitemysql
devopsdockerkubernetes
toolsbeautifulsoupfeedparserrequestsasync

📰 KNWS Global News Aggregation & Intelligence Platform

Long-Form Project Summary for KNWS Showcase

This project is a large-scale, fully automated news discovery, aggregation, and classification system that collects articles daily from tens of thousands of news sites worldwide, processes them into structured categories, stores historical archives, and provides a searchable web interface for exploring global media output in one place.

Designed as an internal news intelligence engine, it blends high-volume RSS ingestion, web crawling, caching, deduplication, categorisation, async processing, and a Flask front-end for rich browsing and search.

It functions as a private alternative to Google News + media intelligence platforms, operating entirely on KNWS infrastructure with no third-party dependencies.

🎯 Project Goals

  1. Build a scalable news ingestion engine capable of pulling headlines and stories from ~50,000 global news sources every day.
  2. Provide automatic categorisation into topics such as world news, politics, tech, business, sports, entertainment, local, etc.
  3. Maintain a historical searchable archive of all processed articles.
  4. Support fluent keyword search, date filtering, and source filtering.
  5. Deliver a streamlined Flask-based news portal for browsing, searching, and grouping stories.
  6. Use intelligent crawling logic for avoid duplicates, reduce noise, and handle broken feeds.
  7. Operate entirely self-hosted for privacy, reliability, and data longevity.
  8. Support future enhancements such as ML analytics, trend prediction, and mobile apps.

🧠 Core Features

🌍 1. Massive-Scale News Collection (50,000 sources)

The system processes a huge dataset of global sources:

  • ~50k live news sites
  • Thousands of direct RSS feeds
  • Custom crawlers for sites without RSS
  • Real-time fetching
  • Duplicate detection
  • Fault tolerance for unreachable sites
  • Automatic feed discovery

The ingestion system runs continuously or on schedule, depending on configuration.

⚙️ 2. Multi-Stage Ingestion Pipeline

The ingestion architecture includes:

RSS Sync Engine

  • Handles large-scale RSS polling
  • Rate-limited to avoid bans
  • Normalises feed data into a unified structure
  • Stores metadata and story content

Async Processing Layer

(Using asynchronous Python workers)

  • Fetching
  • Cleaning HTML content
  • Extracting titles, summaries, images
  • Extracting timestamps
  • Validating URLs
  • Categorising stories

Crawler Framework

For sites without RSS feeds:

  • Custom crawler classes per site
  • Pluggable architecture
  • HTML parsing with BeautifulSoup/lxml
  • Domain-specific rules
  • Automatic failover to generic scrapers

Designed for extensibility and easy addition of new sources.

🧬 3. Automatic Categorisation

Every article is sorted into one or more categories using:

  • Keyword heuristics
  • Site metadata patterns
  • Topic extraction
  • Title/summary analysis
  • Optional LLM-assisted tagging (offline-capable)

Categories include:

  • World
  • Politics
  • Tech
  • Sports
  • Science
  • Business
  • Entertainment
  • Local news
  • Breaking stories (detection based on velocity)

🕰️ 4. Historical Archive

The system maintains a long-term searchable record:

  • All articles stored to disk and database

  • Historical indexing

  • Search by:

    • Keyword
    • Date range
    • Category
    • Source
    • Domain

Ideal for research, media monitoring, or intelligence gathering.

🔍 5. Search Interface (Flask Web Portal)

A fully-featured news browsing website, including:

  • Paginated article lists
  • Category filters
  • Search box with substring matching
  • Story detail view
  • Thumbnail display
  • Archives browser
  • Fast loading via cached assets

Templates are served statically for maximum performance.

💾 6. Efficient Storage & Caching

The system uses:

  • Disk-based caching for downloaded content
  • Log directories for ingestion analytics
  • Lightweight storage for raw HTML
  • Normalised text storage for indexing
  • Optimised filesystem layout for fast read/write
  • Optional SQLite/MySQL indexing

Large-scale news processing requires efficient I/O and caching strategies, all of which are present.

🧰 Codebase Overview

The repository includes a robust backend:

Key Files

  • news.py — Main Flask web app
  • config.py — Site and DB configuration
  • sync_rss_feeds.py — Primary RSS sync engine
  • sync_rss_feeds_class.py — Class-based modular sync system
  • process_rss_async.py — Async ingestion worker
  • utils.py — Shared functions for parsing, time handling, caching
  • sites.csv — Database of 3,900+ RSS sources (base list, extended via auto-discovery)
  • crawlers/core — Base crawler architecture
  • crawlers/modules — Site-specific crawlers
  • downloaded/ — Cached content
  • logs/ — Ingestion logs and error tracking
  • templates/ and static/ — Full web UI

This modular design allows the system to scale horizontally and add new sources easily.

📈 Future / Planned Features

(Already sketched out in the README)

  • Real-time notifications (breaking news alerts)
  • Social media sentiment integration
  • ML-based trend prediction
  • Multi-language support
  • API Gateway + rate limiting
  • Redis caching for ultra-fast querying
  • Message-queue-based ingestion (e.g., RabbitMQ, Redis Streams)
  • Docker containerisation
  • Kubernetes orchestration
  • Mobile app integration
  • Monitoring dashboards
  • User accounts + authentication

These planned improvements position the platform for enterprise-scale media intelligence.

🛠 Technology Stack

Backend

  • Python
  • Flask
  • Async I/O
  • Feedparser / Requests
  • BeautifulSoup / HTML parsing
  • CSV-driven data ingestion
  • Custom crawler classes
  • Disk caching
  • Optional DB integration

Frontend

  • HTML templates
  • CSS / JS
  • Responsive layout
  • Fast, minimal UI

Infrastructure

  • Self-hosted
  • Cron or systemd timers for ingestion
  • Log-based monitoring
  • Future Docker/Kubernetes support

📊 Outcome & Impact

This platform demonstrates sophisticated engineering in:

  • Large-scale data ingestion
  • High-volume distributed crawling
  • Real-time classification
  • Text processing and content normalisation
  • Architecting scalable pipelines
  • Web UI development
  • Data storage and caching strategies
  • Designing for future microservices and distributed systems

It creates an internal news intelligence system far more powerful than typical RSS readers or news websites, and fully private within KNWS.