#56 NLP Pipelines/Crawling
Google News Aggregator
Gather articles from thousands of publishers, group the ones about the same event, and show each reader a ranked, fresh front page.
What is being asked
Design a news aggregator: discover new articles quickly, remove duplicates, cluster articles into stories, classify them by topic and rank stories for each user.
Requirements
Functional
- Ingest articles from feeds and by crawling.
- Detect near-duplicates; cluster into stories; label by topic and place.
- Serve a ranked front page and topic pages.
Non-functional
- Breaking news should appear within minutes.
- Respect publishers' crawl rules and rate limits.
- Millions of articles a day.
Design
- A scheduler decides what to fetch and when: feeds of fast-moving publishers every minute, others less often, never exceeding a per-site rate.
- Fetchers download pages; a parser removes navigation and adverts and extracts title, body, author and time.
- Near-duplicates (the same agency story on fifty sites) are found with a similarity fingerprint such as SimHash.
- A language pipeline extracts named entities and topics, and turns the text into a vector.
- Clustering attaches each new article to an existing story when it is similar enough in content and time, or starts a new story.
- Stories are ranked by freshness, number and authority of sources, and the reader's interests.
Key decisions and trade-offs
- Feeds first, crawling second. Feeds and sitemaps announce new articles cheaply; blind crawling is slow and wasteful.
- Incremental clustering. Re-clustering everything for each article is too slow; assign to the nearest recent story and periodically merge or split stories offline.
- Freshness against quality. Publishing instantly risks showing an unverified first report; weighting by source authority tempers it.
- Diversity. Rank so that one story or one publisher does not fill the page.
Follow-up questions
- How do you find the original report among fifty copies?
- How would you personalise without trapping readers in one viewpoint?
- How do you handle an article that is updated after publication?
- How do you scale the crawl scheduler to millions of sources?