Why GoDaddy built its search engine from scratch
Although he believed he could buy a product off the shelf and layer some customizations on top of it as needed, he wanted to be sure because search is an integral part of GoDaddy's business. The response from his peers, many of whom had created comparative shopping engines, was unanimous: "God, no. You're going to have to do that yourself."
Many CIOs face buy-versus-build decisions as they seek to grow their businesses. Most commercial-off-the-shelf, or COTS, software products are safe because they've been vetted by other businesses and are typically ready to work out of the box. The tradeoff is that such products can’t be adapted to suit the unique needs of some businesses.
Custom software development is the preferred approach among online businesses. Online retailers as large as Amazon and as small as Zulily, build custom analytics, logistics and other tools to operate their businesses. Online marketplace Etsy rewrote its search engine with custom software, including a structured taxonomy. GoDaddy, as a retailer of domain names, had its own special search requirements.
When people type queries into a Google, Microsoft or Yahoo search box, they receive results from a finite corpus of documents indexed and ranked for relevancy by algorithms. But typical GoDaddy customers are entrepreneurs are looking for suitable domain names from which to hang their Web shingle. And GoDaddy needs to string together and render custom indexes at Google-esque speeds of about 200 milliseconds.
However, a major caveat is that while an apparel retailer may have tens or even hundreds of a shirt or pair of jeans in stock, GoDaddy has exactly one of each domain name. Once it's sold, it's gone. "We have to be able to tell you, 'sorry, that exact thing you wanted wasn't available, but here's a whole bunch of options that meet your needs,’ " McLaughlin says. That scenario happens a lot in a world where many of the most common, popular or trendy names are quickly seized.
[ Related: GoDaddy rebrands itself with an eye on diversity ]
"The fundamental problem that we're trying to solve is this [domain name] doesn't exist yet," says Charles Beadnall, the GoDaddy vice president of engineering who left Yahoo in 2013 to craft the search capabilities previously provided by a third-party. "We're pulling this string of names together and checking to see if they're available."
Beadnall says the engineering feat required GoDaddy to create search crawlers that can traverse hundreds of international registries, including in South Africa and Indonesia, generating tens of thousands of potential domain names in near real-time. The company also built machine-learning algorithms, in conjunction with open source Hadoop data processing software, to help surface the best domain names it can.
For example, a user searching for "apple" may also be interested in seeing synonyms, or similar fruit-oriented results, such as "orange." Domain searches incorporating numerals are also popular, prompting GoDaddy to plan for creative number patterns. "The whole point of building the [custom] search system is to get you to a name that you can buy and will want to buy," Beadnall says.
The biggest challenge is accounting for the various permutations of search models, which are based on such demographics as income level, language, countries, as well as the popularity of top-level domains. GoDaddy’s software must divine what’s going sell in what market, as well as what will be appropriate for a specific searcher.
[ Related: GoDaddy accounts vulnerable to social engineering and Photoshop ]
Beadnall, who helped launch the revamped search engine in April 2014, says the software is continuously upgraded, including anything from performance tuning to enabling speedier search to different ranking models. “It takes less than a minute to deploy it, and less than a minute to pull it back,” if something falls apart, he says.
The search challenge corresponds directly to GoDaddy's ambitions. Just a few years ago, the company restricted its service to the U.S. for such mainstay domain spaces as .com, .net and .edu. GoDaddy, which owns 300,000 domain names and has more than 61 million domains under management, is trying to address various name permutations associated with 350 different name spaces, including anything from .guru to .nyc, for 30 markets in 17 languages, with plans to launch in 11 markets in Asia next year.
"It went from a reasonably simple problem to complex problem really fast," McLaughlin says. “[Search] is a core competency we need to own."