Why Would OKCupid Write Their Own Web Server?

The Intricacies Behind Custom Web Server Development

When considering the intricacies and technical challenges that come with building a web service, one might wonder why a company like OKCupid would find it necessary to write their own web server. This article explores the reasons behind such a decision and the technical landscape that influenced it.

Introduction to Web Server Development

The roots of web server development trace back to the early days of the internet. Back in the 1990s, when video streaming and complex web applications were nascent, writing a small web server was a common practice. For instance, the author of this article once wrote a simple web server in Perl that was both lightweight and efficient. However, this was far from complete; it lacked the robustness and features of well-established servers like Apache, which in 1996 had around 80K lines of C code.

The Case of OKCupid

Technical Challenges and Performance

OKCupid faced unique and significant challenges when it launched in 2003. It was a time when the internet was still in its developmental phase, and resources were limited. Funding was hard to come by, and serving costs were much higher. Furthermore, the key technical components (like Apache, PHP, MySQL) used today were either unproven or simply did not exist. For example, Memcached, a key component for caching, was not available until mid-2003. Additionally, the team at OKCupid had no experience with Windows-based solutions, further solidifying their decision to go with a more custom approach.

Strategy and Prototyping

The founders of OKCupid saw rapid prototyping and quickly scaling to meet performance needs as crucial. They believed in launching a product that performed well enough initially and only investing in major architectural changes (or hardware) if the product became very successful. This strategy was previously employed in their first business venture, from 1999 to 2001, where they built features of The Spark while focusing on minimal investment in overengineering.

Unique Requirements

The primary difference that necessitated a custom web server for OKCupid was the unique requirements of the platform. Matching algorithms, caching mechanisms, and real-time data processing necessitated a more tailored and efficient solution. An event-driven model, fast and reliable in C, was developed to handle these needs effectively. Proxies, caching, general RPC servers, and a distributed cache were all part of this custom web server framework, which is what OKWS is—a web server framework that made everything work together seamlessly.

Modern Takeaways

Do You Need a Custom Web Server?

Given today's landscape, where excellent technologies are available and hosting costs are lower, writing a custom web server might not be the best approach for most startups. Unless you have a massive computational workload similar to what OkCupid faced, it is advisable to leverage proven, scalable, and easily manageable solutions.

Conclusion

The journey of OKCupid, from its founding to its current status, is a testament to the profound impact of technical decisions made at the inception of a product. While their custom web server development was driven by necessity and strategic foresight, it highlights the importance of adaptability and the role of performance in the success of any web service.