CoSearch

🏢 CoSearch -- UK Company Search Engine & Intelligence Platform

Long-Form Project Summary for KNWS Showcase

CoSearch is a high-performance, full-stack search platform designed to index, query, and visualise data for over 4 million UK companies, including their addresses, statuses, officers, registration details, SIC codes, and more. It is built as a real-time, big data lookup engine that combines asynchronous Python, optimised MySQL FULLTEXT search, and a sleek front-end to provide instant, relevance-based results for users.

The system is similar in ambition to Companies House search -- but privately hosted, significantly faster, more flexible, and capable of running advanced ranking algorithms, bulk datasets, and enhanced metadata not offered by the official public APIs.

Behind CoSearch sits the Companies House Intelligence system -- a comprehensive data pipeline, multi-source web scraper, CRM application, and Docker-based deployment platform that feeds, enriches, and maintains the 4M+ company database. This pipeline transforms raw public data into actionable business intelligence through automated scraping, AI-assisted enrichment, and a full customer relationship management layer.

---

🎯 Project Goals

1. Build a high-speed UK company search engine over millions of records.
2. Support rich search capabilities across name, address, number, postcode, officers, and other metadata.
3. Deliver instant, paginated, relevance-ranked results even on large datasets.
4. Provide a clean user interface for browsing company data.
5. Maintain a highly optimised backend pipeline capable of handling large ingestion volumes.
6. Ensure low-latency performance via async Python and database optimisations.
7. Enable private, offline-capable searching without relying on external APIs.
8. Build a comprehensive data pipeline for multi-source company intelligence gathering.
9. Provide a CRM system for managing entities, contacts, opportunities, and campaigns derived from company data.
10. Deploy the full stack via Docker for reproducible, scalable operation.

---

🧱 Core Features -- Search Engine

🔍 1. Massive Dataset: 4+ Million UK Companies

The backend stores:

• Company name
• Company number
• Address lines
• Postcode and locality
• SIC codes
• Status (active, dissolved, liquidation)
• Date of incorporation
• Officer information
• Filing history (optional extension)

The dataset is pre-indexed and kept in sync with Companies House bulk data.

---

⚡ 2. Ultra-Fast Search (Async Python + FULLTEXT Indexing)

Queries run through:

• MySQL FULLTEXT indexes
• Advanced query tuning
• Relevance scoring
• Result pagination
• Async connection pooling
• Memory-optimised lookups

This allows sub-100ms queries even across millions of records.

Performance tech includes:

• Async Python (asyncio, aiomysql)
• Connection pools for high concurrency
• Prepared statement caching
• Ranking algorithms for fuzzy matches
• Index alignment for instant lookup

---

🧠 3. Intelligent Ranking & Relevance Engine

The search engine uses:

• Exact match prioritisation
• Partial match fallback
• Address and postcode weightings
• Location-based scoring
• Result de-duplication
• Noise-word removal
• Smart tie-breaking rules

Examples:

• "Apple" -- Apple UK companies first
• "12345" -- direct match to a company number
• "London consulting ltd" -- weighted towards location + keywords

---

🏢 4. Detailed Company Profiles

Each company page shows:

• Name
• Status
• Registered office address
• Incorporation date
• SIC codes
• Officer data (directors / secretaries)
• Filing information (if included)
• Links to external sources
• Related companies with shared officers

View templates are optimised for readability and speed.

---

🔄 5. High-Performance Import Pipeline

The ingestion scripts allow:

• Bulk import of millions of company records
• Cleaning/normalisation of CSV datasets
• Removal of duplicates
• Automatic index optimisation
• Scheduled updates

The system can rebuild the entire dataset quickly with minimal downtime.

---

🖥️ 6. Front-End UI

Built for clarity and speed:

• Instant search feedback
• Clean list-style results
• Company cards with essential metadata
• Mobile-friendly layout
• Pagination for 100k+ results
• Fast transitions between company pages

Thin, optimised HTML/CSS/JS ensures near-zero load time.

---

🔬 Companies House Intelligence -- Data Pipeline & CRM

7. Multi-Source Web Scraping Engine

The backbone of the intelligence system is company_search.py -- a 222KB main script that orchestrates large-scale, multi-source data gathering:

• Google Maps scraper -- extracts business listings, reviews, contact details, and geographic coordinates for companies matching search criteria
• SearXNG integration -- queries the self-hosted SearXNG meta search engine (192.168.10.31:8888) for web presence, news mentions, and additional company data
• BT telephone directory PDF import -- parses British Telecom business directory PDFs to extract phone numbers, addresses, and business categories
• Companies House API -- direct queries against the official Companies House REST API for authoritative registration data
• Custom web scrapers -- site-specific scrapers for industry directories, trade registers, and business databases

The scraping engine manages:

• 50 concurrent threads for parallel data collection
• Database-managed proxy rotation -- proxies stored in MySQL with usage tracking, success rates, and automatic rotation
• Rate limiting -- Google Maps requests throttled to 60-second intervals, other sources to 30-second intervals to avoid bans
• Retry logic with exponential backoff for failed requests
• User-agent rotation and request header randomisation

---

8. Company Name Matching (FuzzyWuzzy)

A critical challenge when merging data from multiple sources is matching company names that appear differently across databases:

• FuzzyWuzzy string matching provides token-based and partial ratio scoring
• Configurable match thresholds for different confidence levels
• Handles variations like "Ltd" vs "Limited", missing punctuation, abbreviated names
• Batch matching mode for processing thousands of candidates against the master database
• Results are flagged for manual review when confidence falls below threshold

---

9. CRM Application

A full customer relationship management system built on top of the company intelligence data:

• Entities -- company records enriched with scraped data, contact details, and intelligence scores
• Contacts -- individual people linked to companies with roles, email, phone, and social profiles
• Opportunities -- sales pipeline tracking from lead through qualification to close
• Campaigns -- marketing campaign management with target lists, delivery tracking, and response metrics
• Credits system -- usage-based credit allocation for API access and data exports

The CRM transforms raw company data into a structured sales and business development platform.

---

10. Monitoring Dashboard

Real-time visibility into pipeline operations:

• Scraping job status -- running, completed, failed, queued
• Source-by-source success rates and data volumes
• Proxy health monitoring -- active count, failure rate, rotation schedule
• Database growth metrics -- new records per day, match rates
• Error logs and retry queues
• Rate limit status per source

---

11. Docker Deployment

The entire pipeline runs in Docker:

• Worker container -- runs the scraping engine, proxy management, and data processing jobs
• Web container -- serves the CRM interface and monitoring dashboard
• Redis -- job queuing, caching, and rate limit state
• MySQL -- primary data storage shared with the CoSearch frontend
• Docker Compose orchestration for single-command deployment
• Environment-based configuration for development and production

---

🛠️ Technology Stack

Search Engine Backend

• Python (async)
• asyncio, aiomysql
• MySQL or MariaDB for large dataset indexing
• FULLTEXT search + custom ranking
• Connection pooling
• Bulk data importers (CSV to MySQL pipeline)

Intelligence Pipeline

• Python 3.x with multi-threading (50 concurrent workers)
• Playwright for JavaScript-rendered page scraping
• FuzzyWuzzy for company name matching
• SearXNG API for meta-search queries
• FastAPI for internal pipeline APIs
• Redis for job queuing and caching
• Docker + Docker Compose for deployment

Frontend

• HTML5
• Lightweight CSS
• Async JS fetch calls
• Server-side rendering via Python templates

Infrastructure

• Docker deployment on KNWS Docker server (192.168.10.31)
• MySQL on the large database server (192.168.10.100) for 4M+ records
• SearXNG instance at 192.168.10.31:8888
• Proxy database with rotation and health tracking
• Systemd timers for scheduled scraping runs

---

📈 Outcome & Impact

CoSearch and the Companies House Intelligence pipeline together deliver:

• Lightning-fast search over millions of entities
• A reliable, private alternative to Companies House search
• Advanced ranking logic not available publicly
• Multi-source intelligence gathering from Google Maps, SearXNG, BT directories, and web scrapers
• Scalable scraping infrastructure with 50-thread concurrency, proxy rotation, and rate limiting
• Fuzzy matching for deduplicating and merging company records across sources
• Full CRM capability for converting company data into business development workflows
• Docker-based deployment for reproducible, scalable operation
• Support for officer-based relationships and deep metadata
• A platform suitable for:
  • Business intelligence
  • Fraud detection and due diligence
  • Research and statistical analysis
  • Sales prospecting and lead generation
  • Campaign management and outreach
  • Internal KNWS data exploration

This project demonstrates strong competency in:

• Big-data engineering
• Async Python and concurrency
• Large-scale MySQL indexing and optimisation
• Multi-source web scraping with proxy management
• Data pipeline architecture
• CRM system design
• Docker containerisation
• Search engine design (ranking, relevance, pagination)
• Clean frontend UX for dense datasets

---