Hackeverse is an open source project by Wilson Lin, where he fetched, processed, and analyzed over 40 million Hacker News posts and comments using text embeddings and various data processing techniques. This allowed creating an interactive semantic map visualization, powerful search engine, virtual subcommunity views, sentiment analysis over time, and other analysis on the full Hacker News dataset.

You can try the hackerverse map out here:

Or read more in the blog post: Exploring Hacker News by mapping and analyzing 40 million posts and comments for fun.

Process:

• Over 40 million Hacker News posts and comments fetched and processed into text embeddings

• Built a powerful semantic search engine over the Hacker News content

• Created an interactive semantic map visualization to explore the embeddings

• Enabled viewing virtual subcommunities around topics of interest

• Analyzed sentiment trends over time for topics like Rust

• Tracked popularity of programming languages and topics

• Opened sourced the data, code, and tools for others to explore further