Hackeverse is an open source project by Wilson Lin, where he fetched, processed, and analyzed over 40 million Hacker News posts and comments using text embeddings and various data processing techniques. This allowed creating an interactive semantic map visualization, powerful search engine, virtual subcommunity views, sentiment analysis over time, and other analysis on the full Hacker News dataset.
You can try the hackerverse map out here:
Or read more in the blog post: Exploring Hacker News by mapping and analyzing 40 million posts and comments for fun.
Process:
Over 40 million Hacker News posts and comments fetched and processed into text embeddings
Built a powerful semantic search engine over the Hacker News content
Created an interactive semantic map visualization to explore the embeddings
Enabled viewing virtual subcommunities around topics of interest
Analyzed sentiment trends over time for topics like Rust
Tracked popularity of programming languages and topics
Opened sourced the data, code, and tools for others to explore further