State of the Game: Where we are at with lag

State of the game

Hi everyone, today we are here to take a deep dive into a very difficult topic. We have talked about this time and time again but today I am going to explain the state of the game, where we are at, where we may be headed and some of the possible solutions to try and correct our course.

I ask that you please go into this article as open minded as possible, knowing that the path before us is very difficult and a very delicate topic for many.

The problem: Lag.

It has been an issue since the inception of War of Nations, but today we are going to focus on one specific type of lag that we have seen repeatedly over the years. That is the battle system processing.

What is the battle system processing?

The battle system processing is the games servers handling of every single deployed army when it arrives at its destination. This system handles the mechanics of attacking, capturing, reinforcing, moving units between bases, etc.

We are going to zone in onto attacking, but this also applies for capturing.

When an attack lands at its destination, it gets added to what’s called an “event queue”, this queue is a list of attacks and the other events listed above. Each attack is synchronous, meaning that the attack before it must finish processing entirely before the next attack can be processed. The reasons for this should be quite obvious, but a quick explanation is that the attack before may affect the next attack due to units changing, etc.

When the attack gets to the top of the event queue it will be processed now. This means the server will go and grab the data it needs for the calculations in the battle, it will also grab anything else needed for this attack to process, once it has grabbed the data it will begin the algorithm to calculate all of the boosts that apply to the battle. Once these boosts are calculated it will proceed to the battle algorithm itself. This is the bread and butter of War of Nations, the part that creates all of the strategy that we see. This algorithm figures out each unit, its stats based on boosts, when it travels, when it fires, how much damage is taken/dealt, which unit moves next, how fast those units move. This is a very complicated algorithm and is where most time is taken in processing a battle. I could not even begin to go into how this algorithm works without seeing it myself, but needless to say it is very complicated and the reason that after 7 years players still try different unit mixes, find ways to beat certain unit mixes or opponents and still see’s questions from veteran players on what works best, how to become a better fighter. It is probably the biggest and most unique part of the game and the biggest reason that players find the games battling entertaining and try to learn more about it.

The time it takes for all of this to happen is significant, the data grabbing, and boost calculation is one portion, as well as the complex battle algorithm portion of this. However, it’s worth adding that making changes to this algorithm could potentially break it beyond understanding because of how complicated, old and specific it is.

The lag.

So, let’s throw a random number out there as it would be impossible to know the real number, lets for simplicities sake say it takes 1 second for the system to fully process an attack/battle. This is certainly larger than the real number, but I am simplifying this for explanations sake.

The cause of lag in this is quite simple, when the time it takes to process each attack in the event queue exceeds the time it takes for a new attack to be added to the event system, we meet what is called a “bottleneck” in the system. That is, the event queue is receiving new tasks faster than it can complete each task, so the list just keeps getting larger and larger, until it crashes, gets a reprieve or can catch up with itself (the number coming in decreases enough).

This is what is happening when we see a deployed attack land at its destination but just sit on top of the target instead of instantly disappearing and showing as an attack report. This is the same issue that iOS world 32 had for a long time before world merges happened, when they had stronghold events that had attacks sat on top of SH towers for several minutes - sometimes even up to an hour. The number of players and number of waves being sent was being increased all of the time and meant that the system could not keep up. This is known as a “technical limitation”. That means it is a limitation of the software in the game server, not a hardware limitation. Aka bigger, faster, better hardware servers does not noticeably affect the processing time of a battle.

Recently the WoN engineering team took a deep dive into this to figure out how to remove/reduce the lag enough to be unnoticeable in the game. Unfortunately, the result of that was not good. it would require significant changes, it would be a monumental task in terms of development time and investment, and would likely require rebuilding most of the base game along with the entire battle processing system and battle logic and writing it from scratch.

This would certainly mean the system would work differently and battling in WoN would change significantly. It would also be a massive time and cost investment to achieve. So massive that it would probably be a better investment to build an entirely new game. PopReach will continue looking into ways to improve this going forward and evaluate every possible avenue, but we need to plan for the worst.

What does this mean for War of Nations?

Over the years, the number of waves that each individual player sends has increased. There are several reasons for this. Many players have collected more and more commanders at the top level (130 & 140), allowing them to send more attacks with maximum commanders. We have also realised the ability to instant build (achieve 100%-unit training time reduction boost) was possible through a combination of boosts that were never designed correctly. Despite the number of players over time decreasing, the number of waves that a single

player sends has increased by a large amount over the same amount of time. Historically a player may only be able to send 10 waves at a time, due to commanders and number of units available. Today the number of waves a player can send at the same time is only limited by the number of commanders they have. There are some secondary limitations such as commander energy, resources to build instant units and how quickly the player can build units.

During highly intense events or hours of play, a single player could be instantly building enough units to be sending anywhere between 10-60 waves at the same time. Usually, the player gets limited by how quickly they can build the units to send their waves. Limits like the 1,000 unit per build queue, the client lag of building each unit, the resources that it costs exist but more often than not players find themselves comfortably sending 10-30 waves with ease. This can happen all event long, which historically was not possible due to unit building time & the limit of how many units the player had built for that event. The reason this is significant is that while the number of players in WoN has reduced, we have actually had an increase in stress on servers and an increase in the number of worlds that are exceeding the technical limit of the battle processing system.

Currently in the game, the easiest and most clear-cut case of this limitation is in the Towers of Strength event. In that event the “End of Round” win condition (where the player holding the tower at the end of each round wins) encourages players to concentrate and send a huge number of attacks all landing at the same time. This immediately creates a huge event queue to process these waves. The system can only handle so many of these attacks in the time available and so many of the attacks get returned to the players base without ever having landed or hitting their target. This is very frustrating for players, understandably. What we are seeing is as more players unlock the ability to instantly build units, they are able to send significantly more attacks at a time and for a much longer time than before.

This times 100 players create a huge number of additional waves flying around at any time during an event. The engineering deep dive showed that many worlds are far, far into the “red” in backing up their battle processing queues, meaning they are frequently causing a backlog and causing waves to sit and wait to process or be returned due to missing the available time to process on their target. As we continue into the future this is only likely to increase. In some worlds this is only seen during Tower of Strength end of round mechanic, but in other worlds this kind of lag is sometimes seen during stronghold events and at other times.

What can we do?

We have a difficult path ahead. Since we cannot fix the lag problem at its core in the battle system, we need to find ways to prevent worlds from reaching this technical limit and being unable to play the game in their normal ways.

In the grand scheme of the game’s development, the current increasing investment from PopReach is a positive sign for the game. With that investment comes longer term plans for the game, including eventually reaching a point of increasing new players visiting the game and staying to play. This means two things:

  1. We have to plan for and assume that eventually player numbers will increase, the landscape could be vastly different, but we will have more players.
  2. The number of waves being sent around will increase as player numbers increase, which means if we simply do nothing, we will realise this technical limit more and more in the game.

Over the past several years since instant build became popular in the game the community has consistently requested features to increase the ability to instantly build and further reduce the realised limits on how many waves/units can be sent by an individual in an event. Things such as removing commander energy, increasing unit queue sizes, unit build templates, increased waves per outpost, making it faster/cheaper to build the alliance base boost that activates instant build.

Unfortunately, the knock-on effects of all of this ends up in increasing number of waves over a sustained period during events as well as faster waves being sent out due to how much faster it is to build enough units to send out waves. All of these requests move us closer to the technical limit.

Going forward, we have very tough choices to make as a community. Because we cannot remove the technical limitations or reduce the processing time enough to make a different to the “red line” that causes this type of lag, we must instead make design choices to reduce the number of waves that a single player can send and/or the amount of time that a single player can spend sending waves, in order to reduce the number of waves that are flying around from the current average much closer to the numbers we saw pre-instant build.

In order to achieve this, we have 3 potential options, where option 3 can be many different things all of which achieve the same goal. I will expand into each option (If it needs expanding) to help to provide an understanding of the possible ways we could move forward with that option, in the hope of providing a larger picture understanding of what may be possible or could be looked into in order to achieve the fundamental goal of reducing the number of waves flying around by a single individual.

And that brings us on to

Possible solutions:

The potential options all have dramatic effects to the game as we play it, which is why this is such a tough conversation and problem to overcome. We have discussed option 2 many times in the past and it is almost always met with much anger from the community, but once again and with our new understanding of the limits of the game, we must consider and explore the option.

Option 1: Reduce the number of waves a player can send per outpost from 10 to something much less, probably something between 1-5, likely 3-4 being the realistic number.

This option is quite self-explanatory, essentially it reduces the maximum possible number of
waves a player could send (from 110 iOS, 100 android, based on 10 per outpost) to
something much smaller. This would significantly reduce the stress on the battle processing
because the number of waves each player can send would be reduced so when players try
to time lots of waves on a target they can only do so with far fewer waves, in TOS
specifically this would likely at least half the number of waves that build up to be processed
at the end of each round.

However, it also has other impacts on gameplay. Things like:

  • A player can only walk half of the units they could before.
  • The number of waves that can be in a capture/occupation is 5 so putting this number below 5 would require at least 2 outposts to be able to have a full occupation.
  • Farming becomes slower and takes much longer.

Probably some more things that I am missing.

Option 2: Replace instant build with balanced alternatives or limit it in other ways.

This one is a difficult one, of course, but is also one of the main reasons we are at the
technical limitation we are now facing.

Let me start by stating that there is unlikely to ever be a perfect replacement, in that instant
build itself is inherently unbalanced and game-breaking in terms of the limits it breaks that
lead to this issue. That being said, there are a lot of reasons that players love instant build.
In order to fully explore this option and understand or realise what could even replace it, we
need to understand exactly why players love instant build so much.

The two parts of instant build that create the problem we now face is:

  • It allows unlimited units, which translates to unlimited waves.
  • It allows players to be limited by only the number of hours they can play. This means that a player can have a seemingly unlimited number of units, which lets them play for 1, 6, 12 or 24 hours with as many units as they need. Previous to this unit build times were one of the important limits to a player’s ability to play for such long periods of time or exert massive impact into world events by the ability to play for huge amounts of time more than others.

Both of those two problems are two sides of the same coin and very similar in what they
actually do to the game and events. These are the issues that would need to be solved or
significantly reduced in order to achieve the design solutions needed to move us away from
the technical limitation.

That does not mean we would return to the state of the game before instant build.
There are many legitimate reasons that players love instant build.

Some of those that I know personally are:

  • Players can build units whenever they need, which means they do not need to walk units and worry about holding large numbers for events
  • It is far easier to build units, despite the pains of doing so 1k units at a time instant build prevents players having to spend a lot of hours queueing units in lots of factories across all of their outposts.
  • Units are instantly available instead of needing to hold large amounts and keep them safe via walking or putting into bubbles
  • Players spent a lot of time and/or money to achieve all of the boosts required to have instant build
  • Players can have far less factories per outpost (Just 1) in favour of other buildings

With this option we have two possible paths forward. We can either try to replace the best parts of instant build, minus the unlimited nature of it. Or we can impose limits upon the amount of units players can build in other ways.
Let’s explore the latter first as it’s far easier to envision. This could be done in several ways such as providing a minimum build time or significantly reducing the resources that are available in the economy, removing down to just the resources available via resource patches in game. While instant build itself may not necessarily be touched in this option, we would be imposing limits that would bring the number of waves a single individual can send in a single event down to an acceptable number.

The former, of course, requires expanding on and also going into with an open mind. There are a multitude of ways that we can achieve this and I look forward to hearing more out of the box ideas and ways that we may be able to incorporate some of the most important “must-haves” about instant build while bringing back an acceptable and reasonable unit limit.

Before continuing I must state that all of the below is theoretical and none of this official, endorsed or discussed at length with any PR employee - please take with a grain of salt, the idea is to formulate a feature or group of features that could replace instant build, improve the basic unit building experience in the game and make it easier for players while also reducing that unit build limit.

The below ideas are not going to address all of these issues, nor are they supposed to be a feature proposal, but rather they are intended to try to open players minds and help work towards a potential way that we could replace instant build with a grouping of features that provide players a more positive unit building experience while removing the unlimited instantly available units that allow players to send several magnitudes larger waves than in the past.

So let’s try to address some of the reasons players love insta and see if we can provide alternative features that would provide some or all of the love without the imbalanced, unlimitedness to it.

It is far easier to build units, despite the pains of doing so 1k units at a time instant build prevents players having to spend a lot of hours queueing units in lots of factories across all of their outposts.

Let’s begin with a simpler one - this has been discussed in the past and many players ask for it to be added on top of the current insta, but the original intention when I brainstormed it was as a replacement feature.
Unit build templates - Let’s allow players to queue a mix of units together, this is probably quite difficult to implement on the technical side, but would allow players to essentially queue 100 jeeps, tanks, etc everything that makes their normal wave mix into a single queue, with the time being the combination of all (obviously). This would allow players to queue units faster as well as not having to plan out their wave mix in factory queues which becomes difficult to manage. Players could click build on a factory and be presented with a number of unit build templates that they could build/edit. Essentially identical to the attack wave template system.
This would reduce the time spent planning builds and the time picking each unit and editing its quantity - instead players click train and the template they want to build and it would queue the units.

Auto-queues - A simple toggle switch on a factory, to remove the queues and have the factory repeatedly build the same unit/queue over and over again, until you run out of resources or become negative in fuel (possibly that could pause instead of toggle off?) this would continue building units from the factory without the need of the player to visit every factory and manually queue them. Do it once and it’s set for good. What if this was also coupled with unit building templates above? Players could have waves building 24/7 providing they had the resources and fuel. That would remove all of the pains that even instant build itself provides, but would have a much lower cap on the number of units a player could build per a single event.

Onto the next -

Units are instantly available instead of needing to hold large amounts and keep them safe via walking or putting into bubbles

Players can build units whenever they need, which means they do not need to walk units and worry about holding large numbers for events

I am grouping the above together as they are both addressed by the below idea.
These two are a bit more complicated, before instant build players would need to walk their units in order to keep them safe. One of the reasons that instant build is great is that it allows players to open the game and be playing within minutes, instead of waiting on units to return or relying on having a bubble/teammates bubble to put them in. However, is also the issue with instant build in the unlimited nature of this. The player has a seemingly unlimited supply of units that they can pull out of thin air and use.
How can we overcome that, with a feature that allows a player to have some units available at a moments notice but not an unlimited amount?

New building?
One idea is a Barracks-like building/feature. Players could put units into this barracks (maybe even waves, instead ?) and store a certain amount, this could be per building/base (and 1 per base), allowing the player to put units into the barracks would allow them to pull out of the base with a moments notice, similar to instant build. However, we would need to balance such a strong feature, perhaps the barracks units would not count as “live” units and would not be used in Global Domination event. Maybe we could force the users barracks to let out some units when the base is occupied down to a certain morale? Or maybe we can attack/destroy the barracks and stop it being able to take any new units/waves until repaired.

Players can have far less factories per outpost (Just 1) in favour of other buildings

This one is a tricky one, I am not sure personally how this could easily be fixed, there is somewhat a balance of buildings that was skewed heavily with instant build, things such as number of warehouses and refineries have increased while factories decreased. Pre-instant build players mostly had factories in order to build units. While I cannot brainstorm a good way to keep just 1 factory per base, maybe things above such as auto-queues already help with this. Maybe other ways to make factories better could be to allow a factory to build 2 queues at a time instead of 1, perhaps a VIP bonus at higher levels? This would at least half the number of factories players would need to achieve the same level of pre-instant unit queues.

Players spent a lot of time and/or money to achieve all of the boosts required to have instant build

This is perhaps the toughest point to address. Each individual player will have a different value on what they perceive as the “cost” of instant build. Some players got it through hard work and without any bundle purchases, some bought part of it and some bought all of it. Each of those players will have different ideas on what it is worth to them.

I am not going to try to answer this one definitively, I will only say that we must be open minded in how we move forward and try to find compromise on all sides to find an acceptable path. Some of the above could be offset by things like the VIP idea above, that would give players something for the VIP they have where they would no longer have instant build.

Now, lets combine these together to try and provide context:
Unit queue templates, along with Auto-queueing, unit storage in a “barracks”, double unit queues per factory.

With these combined features, we have a much easier unit queueing process, a way to hold more units (but not near-unlimited), less time spent queueing units and a safe place to hold a stockpile of units that can be pulled out at any time (but not near-unlimited units).

Option 3: Remove anything that encourages tons of waves being sent in a world

This is a stop-gap solution. It is a bit different and has been the go-to so far (as we saw in the last League system update) with the changes to ToS in order to remove the end of round mechanic, because that mechanic specifically encourages players to send even more waves than usual to land at the exact same time, which exacerbates the technical limitation 10-fold and creates lots of waves never being able to be processed getting returned as the server times them out.

This is the worst-case of the lag and where we more often see it, so removing this mechanic would at least buy us more time before reaching a point where lag in general gameplay from waves becomes too great because of too many players with instant build sending waves / playing at the same time.

It’s possible we have to branch out further than just ToS with this option and slowly discourage sending of a ton of waves in other areas of the game simply because every new player that has instant build and plays actively/consistently is another N number of waves that can be sent (where N is practically unlimited) through an event and X number at the same time.

So where do we go from here?

That is a difficult question. As players all we can do is share our feedback and opinions and hope that they are taken on board. If we wish to turn this into something positive we need to decide on an option and what that option may look like.

It is entirely possible that the lag becomes unmanageable even more quickly than we can imagine and PopReach take the hard decision to limit waves or instant build in some other way (like remove ability to buy more resources, creating a much lower limit on what a player can build in a 24 hour period). We do not want that to happen or for players to not have a chance to try to forge a better path towards the future that they can live and play with.

Please remember that PopReach is investing and spending more and more time and effort on War of Nations. We want that to continue into the future and increase further which means that we also need to plan for a realistic future.

Lastly, this “State of the game” post is not meant to be a solid way forward, but more a layout of the landscape and an attempt to broaden every individuals understanding of the game and hopefully open some minds and get more ideas and feedback. Please try to post your feedback in a positive way, no matter what your feedback is, so that we can have a healthy discussion.

ELI5 (Explain like I’m 5)

For those who do not understand the technical details above, maybe the following will help to explain in a non-technical way.

It doesn’t matter how big your brain is (servers), you can only speak at a certain speed, increasing the size of your brain won’t help you to speak faster than the fastest you already can. Maybe changing the way your brain thinks processes speech, but changing the way your brain thinks is a monumental task.
This is the same as lag. The server (brain) cannot calculate all the wave algorithms (speak) faster. Re-writing the entire game (changing the way the brain thinks) is too big of a task to conceivably invest in.

That is a crude example, but hopefully you get the gist.

Regards
Erielia

Well said - it is an extremely difficult subject but needs to be addressed. I hope a constructive discussion is started here because everyone needs to realise they might lose something they like to achieve a better gameplay in the game

I do like the idea of building to say templates instead of the same wave and building say 6k units at a time.
Also the idea if you can store them before using them but they don’t count as your units until you release them. If say 2 buildings in total with a max of 1 on each base( maybe one could be the hb). Maybe store 20 waves and there being auto build so they be ready for your start time. That sounds very exciting. All that in return for no instant build would be acceptable with one provision. On the bases with the storage and direct build into storage then with unit training on it could refill the storage at say a min time of say 15 mins a wave. So you have 40 waves to play with and both storage can refill at say 4 waves a hour so you will have some quicker waves returning.

I do like your thinking and I hope this can add to the discussion. I have not worked out numbers and maybe storage needs to be 10 waves max for lag but it the principal of the idea I want players to think about

Why not run boost wars and tower at the same time. It will cut down on waves hitting towers and maybe not be so farm since UV would be laying around the ported towers? Wipe out the idiot sitting with 500m at a tower with an alarm set for the last 30 seconds.

Why don’t you remove all bases from the 1400 area and change it into a special zone ( more details later).In tos all speed ups are disabled but on the outer part of the 1400 zone there would be a ring on the edge of the 1400 area which only allows one base per a player in that ring from league 1

The 1400 special zone is where all the tier 4 and tier 5 for league 1 teams would appear but outside tos it would have a special power which speeds up all builds 4 times in speed but the radiation in there causes your base to decay within 3 hours and the base only recovers with a full 24 hrs outside. If your base is over 3 hrs in there then it get destroyed.

Create a similar area on the map for league 2. That way most players will only be able to send 10 waves maximum from one base since others bases will be out of time.

When the event is over the outer ring starts decaying any bases left in it and after 24 hrs the base is destroyed if you don’t move it out. The ring is impossible to port or build on outside tos.

For the rest of the time any base can be built/ ported to 1400 area or the other area for tier 2 and it causes builds of units to be 4 times as fast but your base decays in 3 hrs and would be totally destroyed.

This decay can only be fixed by leaving the base outside the zones for 24 hrs constantly.

All porting to 1400 area is frozen 5 hrs before tos starts so no one can leave a base in it for the start of the event since it would be destroyed

Slightly complicated but would reduce waves to max of 10 per a player at any time and keep the same tos idea.

I like the idea of reducing the amount of waves sent per person, to maybe 2-3 at a time. I think this actually improves the gameplay to avoid somebody just spamming at an SH or tower

As for changing instant build, I think there should be a maximum built per hour, which doesn’t stop people playing whenever they like, but does stop them flooding the server with 100s of high uv waves

This restriction could also be removed during events like global domination, as there will be far fewer attacks going on throughout the world.