Hello everyone! I want to share with you the findings and results of our first AvA blitz phase test.
First, to recap the goals of this test:
- To validate that our AvA performance changes eliminated the database contention, queueing, and backups that we saw in old AvA.
- To validate that AvA server code performs well in a multi-server environment on production hardware.
- To expose any new edge-cases or bugs that only occur on production hardware.
- To test out the full AvA process, from start to finish.
Performance testing and results
To validate that our AvA performance changes eliminated the database contention, queueing, and backups that we saw in old AVA, we stressed the system quite well this week.
The hardest part of our revamp work was redoing the entire architecture of AvA. While it's not pretty, and it's not very visible, it took weeks and weeks of hard work to rebuild AvA from scratch. We're pleased to say that our work seems to have paid off in the first 24 hour blitz phase.
AvA code performed exceptionally well under load that was consistently 3x - 4x what we saw in previous phases. Some players had concerns that we wouldn’t have enough players to validate that AvA could handle production-level load; instead, we greatly exceeded the load we saw in previous AvA phases. This is partly due to keeping AvA attacks around 18 minutes, which let player send out far more attacks than in old AvA - even if it meant it kept the Red Priests of R’Hllor busy reviving Sworn Swords. And it is also partly due to the sheer enthusiasm of our players!
As you can see above, we regularly exceeded our average attacks per minute by a huge margin. There is a dip where the rogue app server threw the game around 3:40pm EDT on Monday, but aside from that time the AvA processes handled the load quite well.
We closely monitored server response time during the AvA test, and found that the server pool rarely strayed outside normal server response times. The chart below shows response times for evening EDT (usually our busiest time) and the band that we consider “normal” for any regular GoTA day:
That doesn’t mean there weren’t issues. We identified several database queries that were quite expensive under production load, and fixed them during Monday night. One of them dropped AvA garrison response time by quite a bit:
Every time we fix an issue like this, it means faster performance for you, and less load on the server.
Bug identification and fixing
With edge cases and bugs, we were able to identify and fix several bugs that came up during the test:
- VP not correctly scaling for some attacks. This was being investigated in Beta, but our blitz phase test allowed us to prove it was occurring. (Status: Underway)
- Battle Log not correctly reporting for players. Our engineering team saw this Monday night and fixed it that night. (Status: Fixed)
- Fixed several camp errors that cropped up.
- Found an issue with removing SS post-phase.
- Some players who didn’t receive their titles correctly due to misspelling their names in the file. (Status: Fixed)
- And a bunch more! We won’t put a full list in here, but we’re tracking the reported bugs.
All of them have been prioritized and we’re working on them alongside the updates to better Alliance tools.
Testing the AvA process
We were also able to test out the entire AvA process, including:
- Enabling a phase.
- Seeing players start camps.
- Seeing players attack camps.
- Seeing SS die, resurrect, and fight.
- Seeing spoils appear in the inventory.
- Seeing camps successfully upgrade up to level 20.
- Having AvA stop on-time.
- Assigning winner brackets.
- Delivering rewards.
- Having players get rewards.
Some fun stats about this blitz phase
- 1,150 Alliances participated, and built 4,500 camps.
- Players sent out 113,000 waves, resulting in more than 3,000,000 attacks.
- Kong’s Landing sent out the most attacks, sending out 157,000 attacks, with The Heroic Regiment of Bunnies a close second, at 142,000 attacks.
So what’s next?
The team continues to work on improvements to AvA throughout the week. The four improvements we are working on this week are:
- Adding an outgoing attack log to the game. The outgoing attack log is a lot more detailed than the current Battle Log - it aggregates each batch by sender, shows you the win/loss percentage, who they targeted, what camp they hit, the damage they did, and what they sent out.
- Adding in the ability to choose either Send Type or Send Group, so you can choose to send all Intrigue SS, for example, or just all Spy SS.
- Adding in the ability for Alliance Leaders and Officers to see their members’ VP for the current phase.
- Adding in Alliance chat to the new AvA screen.
We’re also working on other bugfixes and improvements to AvA. All of these changes will be pushed to the AvA beta server so you can check them out.
We will not have another AvA phase this week. We received enough feedback from this one to keep us busy this week with improvements, and we want the next one to have some improved tools for Alliances. The next phase will be a little bit longer and we’ll have a “closer to complete” AvA for you to test out.
Lastly, we reverted the changes for the Blitz phase (100 silver replacement cost, and 30 minute wound timer) this morning. We will keep the loss of SS level off for the time being, so players may continue to replace dead SS without a penalty.
Thanks for your participation!