Right this moment in “making videogames is difficult” information: Respawn’s journey to trace down a bug that precipitated months of audio points in Apex Legends. Grenades that do not explode, weapons that do not shoot, harm that has no supply, and months of agonizing investigation—all apparently attributable to a single line of code added in Apex Legends’ Season 16 replace.
As outlined in an intensive Reddit submit (opens in new tab) by Respawn group supervisor Amy Thiessen, the difficulty started firstly of Season 16 in February. The studio had began getting experiences of “disappearing nades” in Apex. Respawn quickly decided that grenades weren’t “disappearing” precisely, however they might generally fail to blow up regardless of damaging gamers.
“This had not occurred throughout our Season 16 playtesting, couldn’t be reproduced internally after preliminary experiences, and was very tough to pin down utilizing reside gameplay movies as the foundation trigger was not at all times proven within the participant’s POV,” the submit reads.
Respawn received a greater deal with on the issue after receiving comparable experiences about lacking gun sound FX and particle results. “After a preliminary investigation, the first suspect was discovered to be the system our servers use to dispatch ‘begin’/’cease’ instructions for numerous results (e.g. sure sounds, particle programs, physics impacts, bullet tracers, explosions).”
Dev Workforce Replace: Audio Replace from r/apexlegends
Basically, one thing was occurring throughout a match that might overload the server’s restrict for sound FX or particles, inflicting some sounds and FX to get dropped.
“From there, the idea was that one thing could also be flooding this engine limitation, requesting 1000’s of results each second!” the submit says. “However was this a systemic concern or may or not it’s a single entity appearing up? Each season replace contains 1000’s of adjustments to property, code, script, and ranges. Which meant discovering a needle in a haystack.”
Respawn turned to metrics to assist suss out the issue, however nothing within the telemetry indicated a transparent concern. This urged to Respawn that this bug was a novel scenario their programs had not beforehand seen.
“This left us with a fancy concern that we knew was impacting our group, however was laborious to breed regardless of detailed experiences, had minimal leads internally, and there have been no metrics to show definitively that this restrict was being hit in any respect.”
The place do you go from there? Respawn determined to check its principle of overloaded results by deliberately breaking Apex Legends servers. The workforce spun up a check construct and spawned 50 characters that each one fired weapons on the similar time and infinitely used skills to push the server results load over the sting. It labored—the workforce may lastly reproduce audio drops much like the bug experiences, however the way it was occurring to precise gamers was nonetheless a thriller.
“This gave us proof that FX would get dropped, however solely with utterly unrealistic check instances. Varied features of our server efficiency had been investigated, however nothing particular was discovered.”
Respawn stored an in depth eye on the problem as Season 16 raged on. The workforce ultimately observed that dropped audio experiences tended to come back from high-level play. This gave them the thought to deploy a server replace that allow Respawn observe new metrics in a smaller subset of matches, which immediately led to a breakthrough.
“Because the server replace was finalizing, we discovered it. A single line of code was recognized to be the foundation explanation for the problem. Season 16’s new weapon.”
That weapon is the Nemesis, Apex’s latest burst-fire vitality assault rifle. The Nemesis has a novel mechanic the place dealing harm will “cost” the gun and make it shoot sooner (as demonstrated (opens in new tab) by YouTuber Dazs above). This charging impact is represented visually on the gun by arcing electrical energy throughout the barrel. Respawn says {that a} line of code meant to inform this impact to “cease” whereas the gun wasn’t charged or holstered was truly repeating indefinitely for all gamers holding a Nemesis of their stock.
“Because of this each single participant with an uncharged Nemesis would create a ‘cease particle’ impact on the server each body, and this line of code was being referred to as even when the weapon was holstered.”
Funnily sufficient, this additionally explains why the audio drops had been occurring extra typically in high-level play. “14 shoppers with a Nemesis working at 180 fps could be sufficient to trigger FX to start being dropped.” For as soon as, it was the top-spec PC gamers who had a drawback.
Respawn says this additionally explains why its inside testing did not encounter the bug.
“The builds used for testing won’t have had sufficient holstered Nemesis in play, had a rarer correlation with lacking FX, or didn’t have sufficient shoppers at that fps—one thing for us to bear in mind and enhance on for future testing.”
A patch deployed final week lastly squashed the bug for good. And there you will have it—a meddlesome audio bug with a sophisticated root trigger that may, in the long term, assist Respawn catch comparable bugs earlier than they attain gamers. Respawn concluded the submit with an apart about testing, reminding gamers that “a minute of gamers enjoying Apex is the equal of 10 testers enjoying the sport for a yr!”