Massive new multiplayer shooter Wardogs has taken over Steam – but it almost went catastrophically wrong for developer Bulkhead. “We started confident in the leadup to launch,” UI/UX Director Howard Philpott says with a smile. “Sales were through the roof, we just released our launch trailer, we got 245,000 concurrent players in the beta. So we were feeling great.” Then came the big moment to press the go button. “As soon as we hit it, shit hit the fan.”
Those initial hours of Wardogs were yet another example of a messy multiplayer game launch – but, to Bulkhead’s credit, they quickly became a showcase for how to handle issues effectively. Now the dust has settled, looking at its stats would lead you to believe that everything went to plan: a peak player count of over 365,000, sales passing 1.5 million, and an 83% Steam review average from 24,000 user ratings. The developer, which has been very up-front about its ups and downs over the past few months, has opened the doors to its studio once again to reflect on what went wrong.
Philpott, who has become somewhat of the frontman for Wardogs’ video presence, discusses how overactive anticheat, combined with the sheer number of players trying to all log in and play at the same time, nearly spelled doom for Bulkhead. “I know we said we had a zero-tolerance stance on anticheat, but that very, very quickly turned into banning our legitimate alpha testers.” This included notable FPS streamers Michael ‘Shroud’ Grzesiek and Jaryd ‘Summit1g’ Lazar, who were broadcasting the game on Twitch.
At the same time as all these erroneous bans were flying about, “hundreds and thousands of players” were flooding the Wardogs login queue with requests. Philpott remains his typically magnanimous self: “The reality is there is no excuse.” Still, the race was on to prove that these issues could be resolved before disappointed players started reaching for the refund button. “We knew in the moment: we either fix the queue now, or we’re fucked. This is our entire reputation on the line.”
Bulkhead believes the biggest problem was the sheer concentration of players attempting to get in at once. During beta, the firing range was opened six hours before the game proper, giving people time to filter in more gradually. The team shares footage from inside the studio during those crucial hours on launch day, where Production Director James Tatum-Dean can be seen pointing Philpott to a stats screen. 45,590 people are actively in the game. 168,000 are still waiting in the queue.
Because of the downtime between players attempting to log in and actually reaching the front of the queue, Bulkhead says that out of the 100 players per second that should have been let through the gate, just ten of them were. The other 90% were getting stuck, which caused the line to grow longer, and even more of those login ‘tickets’ to expire. “The decision was made that we had to deploy a hotfix,” Philpott recalls. That meant asking players to update, but it was the team’s best chance at saving the situation.
The solution was simple and smart: now, when you get to the front of the queue, you’re handed a fresh ticket from Valve’s login system instead of checking the old one you had when you first opened the game. Despite this, Bulkhead wasn’t taking any chances. “We had to control how quickly people were moving through, and we knew that this was a one-shot thing,” Tatum-Dean explains, “We wanted to make sure this was done properly and safely.”
The team manually let people through in sets of 10,000; a big number in its own right, but as Philpott remarks, “When you’re seeing 300,000 people waiting in the queue it’s excruciating.” The good news is that it worked, and eventually that queue number read a beautiful green zero. Players no longer needed to worry about waiting, and could start figuring out how to earn Wardogs money instead. “It was the best feeling,” Philpott laughs. “Everyone was in game and we could actually open a champagne bottle.” Now the team can focus on enjoying the game with the community for a while. “Monday, it’s back to work.”
Executive Producer Joe Brammer also admits Bulkhead did use a queue jump to get “about 11” streamers into the game as soon as possible. “That’s on me, that was my decision. It is a bit unfair,” he acknowledges. He explains that he believed the value of them “entertaining people and showing people the game” was worth it. “We were also able to get out announcements via them as well about restarting the servers, that kind of thing.”
The rest of the night still wasn’t quite flawless for the streamers, however, as Summit found out when one interaction with a cheater saw him hit with a ban by mistake. “You guys are worried about us having AI anti-cheat,” Brammer remarks. “We always have to have a human involved.” He points to one member of the team with a grin. “We’ve got this idiot here – he banned Summit instead of the cheater in Summit’s stream.” After such a rollercoaster release, perhaps a little human error is understandable, and Bulkhead’s willingness to be so open about such issues will serve it well.