A Development Update on Replays

[Archimedes]

Hello Leaders!​

Thank you for the requests and your patience in waiting for updates on our Replay Reliability initiative. It is due time to inform you of where we are with it and where we are going. Replay consistency is of the utmost importance to us; we want to not only make it correct but keep it that way for the future.

The Problem​

We use a “deterministic client re-simulation” approach for replays in DomiNations. This means that an actual replay holds the bare necessities to set up the attack, then feeds user inputs to the simulation engine (example: A tank was dropped here and now). Your device works from those inputs to recreate the original attack (The tank will move here and attack this target). If your device does not simulate those inputs exactly as they originally played out, the results diverge from the original attack reality. The earlier in the simulation this happens, as well as the magnitude of the differentiation, can cause a dramatic difference in the outcomes of the replay (The ‘Butterfly Effect’ from chaos theory illustrates this phenomenon, albeit using discrepancies in input, not simulation, but the final outcomes are similar).

One initial question/suggestion we often see is “Why not just save the state of the original attack as it progresses into the replay?”. The answer is simply “for efficiency”. We are a global game, with many users playing on cellular connections, capped data, or paying for data usage. In addition, the smaller the file, the faster we can download it to your device and begin the playback. Storing vast amounts of data for each replay will make for an unbearable experience for many of our players.

DomiNations has a built-in replay validation mechanism. It knows when a replay differs, and in some cases, can tell us roughly where things went wrong. This was very useful in the early days of the game. We often, with this information, could deduce the cause of the error and fix it rapidly. After a decade of live operation, complexity has caught up to us, and this current system no longer suffices to pinpoint the exact causes of replay ‘drift’. The combination of variables and possibilities is just too great. It began to take multiple hours and sometimes days to try to find a single issue, and eventually, the complexity overcame our ability to keep up, and replay success rates began to drop despite scheduled efforts to find and fix them.

Our Current State​

Replay accuracy has dipped so low that they have become unreliable tools for attack assessment, and can do more harm than good when trying to view them for both offensive and defensive strategies. We recognized this some time ago and took multiple actions to strengthen the existing replay validation system. Sadly, after considerable investment, it became clear that salvaging the older method was not producing the efficiency and results that we needed.

We instead took a step back, a deep breath, and rewrote the entire mechanism. This work is finally considered complete enough that we have started using it to track down issues. We have a new internal toolchain and system that can tell us the exact frame in which a replay begins to diverge, why it is diverging, and often points us to the root cause. The fixes can still take time to implement, and ultimately must be internally tested and run through our Quality Assurance (QA) department before making it out to you. In addition, since an early divergence will ‘mask’ a later one, often one fix just leads to the next. It doesn’t make sense to release a replay fix to you if we know you’ll just hit the next one in line! We started from the early ages and worked our way through the game, fixing issues as we found them. We are happy to say that as of right now, a large portion of our internal replays run with no detectable issues found! The work isn’t done; there is still a large suite of test cases to try to identify anything and everything that we may have missed.

We are about to submit code into QA with an estimated 40+ bug fixes for replays across all ages and units in the game. The hope is that you will see these fixes in our next major release (Update 12.18) and begin to notice a difference in replay quality. We will continue searching and fixing additional issues as we find them. The goal is to have a marked improvement in replay quality every major release until we hit 100% accuracy.

The Future​

Where do we go from here? Our first step is to reach a point where we cannot find any more replay issues. We are hopeful that this goal is in sight, but our work will not be complete even then.

Our next step, already under development, is to package this replay validation framework into our server cloud, so that we can send replays to the system for immediate analysis. We hope to use this capability for a variety of services, including:

• Automated replay testing of all new code as it runs through our development and QA stages. We want to identify any new sources of replay failure before they ever make it out to you, much as we already do with other sections of the game code.
• Automated policing of attacks. If we know replays are correct, we can instantly validate key, prominent, and hopefully at some point ALL attacks, and take immediate action, even mid-war, if we find anyone trying to alter outcomes of battles.

We would like to thank our community both for your patience in this issue, as well as your vigilance in demanding that we improve.

 

[SolarRad75]

So, those cheaters with 14 Tupolev TTs are finally being banned... mid-battle? Amazing

 

[Harlems369th]

So, those cheaters with 14 Tupolev TTs are finally being banned... mid-battle? Amazing

The suggestion is potentially mid-war adjustments, not mid-battle. This is something we hope for the future, and not something that will be live in the 12.18 update.

 

[Nikolay]

Hello Leaders!​

Thank you for the requests and your patience in waiting for updates on our Replay Reliability initiative. It is due time to inform you of where we are with it and where we are going. Replay consistency is of the utmost importance to us; we want to not only make it correct but keep it that way for the future.

The Problem​

We use a “deterministic client re-simulation” approach for replays in DomiNations. This means that an actual replay holds the bare necessities to set up the attack, then feeds user inputs to the simulation engine (example: A tank was dropped here and now). Your device works from those inputs to recreate the original attack (The tank will move here and attack this target). If your device does not simulate those inputs exactly as they originally played out, the results diverge from the original attack reality. The earlier in the simulation this happens, as well as the magnitude of the differentiation, can cause a dramatic difference in the outcomes of the replay (The ‘Butterfly Effect’ from chaos theory illustrates this phenomenon, albeit using discrepancies in input, not simulation, but the final outcomes are similar).

One initial question/suggestion we often see is “Why not just save the state of the original attack as it progresses into the replay?”. The answer is simply “for efficiency”. We are a global game, with many users playing on cellular connections, capped data, or paying for data usage. In addition, the smaller the file, the faster we can download it to your device and begin the playback. Storing vast amounts of data for each replay will make for an unbearable experience for many of our players.

DomiNations has a built-in replay validation mechanism. It knows when a replay differs, and in some cases, can tell us roughly where things went wrong. This was very useful in the early days of the game. We often, with this information, could deduce the cause of the error and fix it rapidly. After a decade of live operation, complexity has caught up to us, and this current system no longer suffices to pinpoint the exact causes of replay ‘drift’. The combination of variables and possibilities is just too great. It began to take multiple hours and sometimes days to try to find a single issue, and eventually, the complexity overcame our ability to keep up, and replay success rates began to drop despite scheduled efforts to find and fix them.

Our Current State​

Replay accuracy has dipped so low that they have become unreliable tools for attack assessment, and can do more harm than good when trying to view them for both offensive and defensive strategies. We recognized this some time ago and took multiple actions to strengthen the existing replay validation system. Sadly, after considerable investment, it became clear that salvaging the older method was not producing the efficiency and results that we needed.

We instead took a step back, a deep breath, and rewrote the entire mechanism. This work is finally considered complete enough that we have started using it to track down issues. We have a new internal toolchain and system that can tell us the exact frame in which a replay begins to diverge, why it is diverging, and often points us to the root cause. The fixes can still take time to implement, and ultimately must be internally tested and run through our Quality Assurance (QA) department before making it out to you. In addition, since an early divergence will ‘mask’ a later one, often one fix just leads to the next. It doesn’t make sense to release a replay fix to you if we know you’ll just hit the next one in line! We started from the early ages and worked our way through the game, fixing issues as we found them. We are happy to say that as of right now, a large portion of our internal replays run with no detectable issues found! The work isn’t done; there is still a large suite of test cases to try to identify anything and everything that we may have missed.

We are about to submit code into QA with an estimated 40+ bug fixes for replays across all ages and units in the game. The hope is that you will see these fixes in our next major release (Update 12.18) and begin to notice a difference in replay quality. We will continue searching and fixing additional issues as we find them. The goal is to have a marked improvement in replay quality every major release until we hit 100% accuracy.

The Future​

Where do we go from here? Our first step is to reach a point where we cannot find any more replay issues. We are hopeful that this goal is in sight, but our work will not be complete even then.

Our next step, already under development, is to package this replay validation framework into our server cloud, so that we can send replays to the system for immediate analysis. We hope to use this capability for a variety of services, including:

• Automated replay testing of all new code as it runs through our development and QA stages. We want to identify any new sources of replay failure before they ever make it out to you, much as we already do with other sections of the game code.
• Automated policing of attacks. If we know replays are correct, we can instantly validate key, prominent, and hopefully at some point ALL attacks, and take immediate action, even mid-war, if we find anyone trying to alter outcomes of battles.

We would like to thank our community both for your patience in this issue, as well as your vigilance in demanding that we improve.

I have many questions for QA, which I have outlined in this post https://xen.bighugegames.com/index.php?threads/methods-of-dealing-with-critical-bugs.40452/
Please comment on them.
I am afraid that if QA works the same way as with all previous updates, we will get a new batch of bugs

 

[Nikolay]

Hello Leaders!​

Thank you for the requests and your patience in waiting for updates on our Replay Reliability initiative. It is due time to inform you of where we are with it and where we are going. Replay consistency is of the utmost importance to us; we want to not only make it correct but keep it that way for the future.

The Problem​

We use a “deterministic client re-simulation” approach for replays in DomiNations. This means that an actual replay holds the bare necessities to set up the attack, then feeds user inputs to the simulation engine (example: A tank was dropped here and now). Your device works from those inputs to recreate the original attack (The tank will move here and attack this target). If your device does not simulate those inputs exactly as they originally played out, the results diverge from the original attack reality. The earlier in the simulation this happens, as well as the magnitude of the differentiation, can cause a dramatic difference in the outcomes of the replay (The ‘Butterfly Effect’ from chaos theory illustrates this phenomenon, albeit using discrepancies in input, not simulation, but the final outcomes are similar).

One initial question/suggestion we often see is “Why not just save the state of the original attack as it progresses into the replay?”. The answer is simply “for efficiency”. We are a global game, with many users playing on cellular connections, capped data, or paying for data usage. In addition, the smaller the file, the faster we can download it to your device and begin the playback. Storing vast amounts of data for each replay will make for an unbearable experience for many of our players.

DomiNations has a built-in replay validation mechanism. It knows when a replay differs, and in some cases, can tell us roughly where things went wrong. This was very useful in the early days of the game. We often, with this information, could deduce the cause of the error and fix it rapidly. After a decade of live operation, complexity has caught up to us, and this current system no longer suffices to pinpoint the exact causes of replay ‘drift’. The combination of variables and possibilities is just too great. It began to take multiple hours and sometimes days to try to find a single issue, and eventually, the complexity overcame our ability to keep up, and replay success rates began to drop despite scheduled efforts to find and fix them.

Our Current State​

Replay accuracy has dipped so low that they have become unreliable tools for attack assessment, and can do more harm than good when trying to view them for both offensive and defensive strategies. We recognized this some time ago and took multiple actions to strengthen the existing replay validation system. Sadly, after considerable investment, it became clear that salvaging the older method was not producing the efficiency and results that we needed.

We instead took a step back, a deep breath, and rewrote the entire mechanism. This work is finally considered complete enough that we have started using it to track down issues. We have a new internal toolchain and system that can tell us the exact frame in which a replay begins to diverge, why it is diverging, and often points us to the root cause. The fixes can still take time to implement, and ultimately must be internally tested and run through our Quality Assurance (QA) department before making it out to you. In addition, since an early divergence will ‘mask’ a later one, often one fix just leads to the next. It doesn’t make sense to release a replay fix to you if we know you’ll just hit the next one in line! We started from the early ages and worked our way through the game, fixing issues as we found them. We are happy to say that as of right now, a large portion of our internal replays run with no detectable issues found! The work isn’t done; there is still a large suite of test cases to try to identify anything and everything that we may have missed.

We are about to submit code into QA with an estimated 40+ bug fixes for replays across all ages and units in the game. The hope is that you will see these fixes in our next major release (Update 12.18) and begin to notice a difference in replay quality. We will continue searching and fixing additional issues as we find them. The goal is to have a marked improvement in replay quality every major release until we hit 100% accuracy.

The Future​

Where do we go from here? Our first step is to reach a point where we cannot find any more replay issues. We are hopeful that this goal is in sight, but our work will not be complete even then.

Our next step, already under development, is to package this replay validation framework into our server cloud, so that we can send replays to the system for immediate analysis. We hope to use this capability for a variety of services, including:

• Automated replay testing of all new code as it runs through our development and QA stages. We want to identify any new sources of replay failure before they ever make it out to you, much as we already do with other sections of the game code.
• Automated policing of attacks. If we know replays are correct, we can instantly validate key, prominent, and hopefully at some point ALL attacks, and take immediate action, even mid-war, if we find anyone trying to alter outcomes of battles.

We would like to thank our community both for your patience in this issue, as well as your vigilance in demanding that we improve.

By the way, the replays of battles in the game Clash of Clans have an accurate reproduction without errors, why not use the same mechanism or are there technical nuances?

 

[Seek]

Hello Leaders!​

Thank you for the requests and your patience in waiting for updates on our Replay Reliability initiative. It is due time to inform you of where we are with it and where we are going. Replay consistency is of the utmost importance to us; we want to not only make it correct but keep it that way for the future.

The Problem​

We use a “deterministic client re-simulation” approach for replays in DomiNations. This means that an actual replay holds the bare necessities to set up the attack, then feeds user inputs to the simulation engine (example: A tank was dropped here and now). Your device works from those inputs to recreate the original attack (The tank will move here and attack this target). If your device does not simulate those inputs exactly as they originally played out, the results diverge from the original attack reality. The earlier in the simulation this happens, as well as the magnitude of the differentiation, can cause a dramatic difference in the outcomes of the replay (The ‘Butterfly Effect’ from chaos theory illustrates this phenomenon, albeit using discrepancies in input, not simulation, but the final outcomes are similar).

One initial question/suggestion we often see is “Why not just save the state of the original attack as it progresses into the replay?”. The answer is simply “for efficiency”. We are a global game, with many users playing on cellular connections, capped data, or paying for data usage. In addition, the smaller the file, the faster we can download it to your device and begin the playback. Storing vast amounts of data for each replay will make for an unbearable experience for many of our players.

DomiNations has a built-in replay validation mechanism. It knows when a replay differs, and in some cases, can tell us roughly where things went wrong. This was very useful in the early days of the game. We often, with this information, could deduce the cause of the error and fix it rapidly. After a decade of live operation, complexity has caught up to us, and this current system no longer suffices to pinpoint the exact causes of replay ‘drift’. The combination of variables and possibilities is just too great. It began to take multiple hours and sometimes days to try to find a single issue, and eventually, the complexity overcame our ability to keep up, and replay success rates began to drop despite scheduled efforts to find and fix them.

Our Current State​

Replay accuracy has dipped so low that they have become unreliable tools for attack assessment, and can do more harm than good when trying to view them for both offensive and defensive strategies. We recognized this some time ago and took multiple actions to strengthen the existing replay validation system. Sadly, after considerable investment, it became clear that salvaging the older method was not producing the efficiency and results that we needed.

We instead took a step back, a deep breath, and rewrote the entire mechanism. This work is finally considered complete enough that we have started using it to track down issues. We have a new internal toolchain and system that can tell us the exact frame in which a replay begins to diverge, why it is diverging, and often points us to the root cause. The fixes can still take time to implement, and ultimately must be internally tested and run through our Quality Assurance (QA) department before making it out to you. In addition, since an early divergence will ‘mask’ a later one, often one fix just leads to the next. It doesn’t make sense to release a replay fix to you if we know you’ll just hit the next one in line! We started from the early ages and worked our way through the game, fixing issues as we found them. We are happy to say that as of right now, a large portion of our internal replays run with no detectable issues found! The work isn’t done; there is still a large suite of test cases to try to identify anything and everything that we may have missed.

We are about to submit code into QA with an estimated 40+ bug fixes for replays across all ages and units in the game. The hope is that you will see these fixes in our next major release (Update 12.18) and begin to notice a difference in replay quality. We will continue searching and fixing additional issues as we find them. The goal is to have a marked improvement in replay quality every major release until we hit 100% accuracy.

The Future​

Where do we go from here? Our first step is to reach a point where we cannot find any more replay issues. We are hopeful that this goal is in sight, but our work will not be complete even then.

Our next step, already under development, is to package this replay validation framework into our server cloud, so that we can send replays to the system for immediate analysis. We hope to use this capability for a variety of services, including:

• Automated replay testing of all new code as it runs through our development and QA stages. We want to identify any new sources of replay failure before they ever make it out to you, much as we already do with other sections of the game code.
• Automated policing of attacks. If we know replays are correct, we can instantly validate key, prominent, and hopefully at some point ALL attacks, and take immediate action, even mid-war, if we find anyone trying to alter outcomes of battles.

We would like to thank our community both for your patience in this issue, as well as your vigilance in demanding that we improve.

How about taking baby step and most important to myself knowing excactly what was used battle. Not based on a simulation just like BHG tracks what we spend.

This would help big time with false claims of cheating and also help find cheaters. If war battle report is 100% accurate. Big time trust loss by BHG every battle report that shows too many air slots/ tactics being used.

Basically we want to know if someone hits us with 20 airslots and we report that player is busted. If BHG shows everything in War Battle Report and its accurate this can happen. Separate War Battle Report from problem prone sims. This way we have actual proof if battle is legit . Proof is a must

 

[AWD]

This post implies a tank may move in a slightly different direction or do a slightly different thing causing a replay that is different to the original battle. It doesn’t address the stupid situation when I use six fighters then watch the replay of my attack and it shows I used 14 fighters when I know I only used six

 

[I.D]

something new is done. what about the report battle or battle report? it was correct 2-3 years ago... showing the exact number of troops that was used to

 

[Seek]

See BHG what I been asking is what people want Proof of what is used in War Battles that is accurate is most important fix then comes replays. Focus on giving us 100% accurate War Battle Reports that show 100% what was used and 100% accuracy. Till this happens nobody will completely trust BHG or War Battles.

Never has their been a time a player ask a CS rep for actual proof a battle was legit and the CS can provide it. Big time problem when even the CS can’t tell you what was used. Hence why their is zero trust in war battles.

Remove War Battle Report from simulation and show accurately what was used in War Battles. Till this happens nobody is going trust BHG games. On the CS side could please inform your CS team telling players to trust the War Screen just is a slap in face to customers We Don’t Trust war screen war replays war battles reports. That is BHG fault for not providing Proof.

 

[nobodyknowsthetrouble]

The promised updates has the potential to destroy the game; but will probably be tested to the same high standard as, for example
- the detachments, which can be used in any size (up to 187 troops)
- the detachment update (12.16) that broke the library
- the library update (12.17), which deducted crowns and locked out players for days.
- the change in the web version that made ad based retrains impossible
(and all this is just the last 2 months; previously the situation was no different, but the list would be too long for the forum)

With these well-tested changes, nothing can go wrong. I'm already looking forward to reading about it when
- the first players get banned (for no reason) because the supposedly perfect replays misrepresent their attack
- the first alliances lose their reputation because they harbor cheaters
- above all: BHG remains silent about this for weeks again

 

[WileCoyote]

I am very happy to see this. Thanks for the explanation, I know debugging or rewriting code can be challenging.

 

[Archimedes]

By the way, the replays of battles in the game Clash of Clans have an accurate reproduction without errors, why not use the same mechanism or are there technical nuances?

I cannot discuss other games, but yes, there are technical differences/nuances involved!

 

[Archimedes]

something new is done. what about the report battle or battle report? it was correct 2-3 years ago... showing the exact number of troops that was used to

The post-replay report screen currently utilizes the replay results, so improvements in replay accuracy will directly affect the accuracy of the battle report that follows. We will monitor the level of improvement from the next release and discuss if shorter-term changes to the battle report specifically are warranted as we continue improving replays. The idea of showing troops brought in addition to / in lieu of troops lost will be discussed internally as well!

 

[I.D]

The post-replay report screen currently utilizes the replay results, so improvements in replay accuracy will directly affect the accuracy of the battle report that follows. We will monitor the level of improvement from the next release and discuss if shorter-term changes to the battle report specifically are warranted as we continue improving replays. The idea of showing troops brought in addition to / in lieu of troops lost will be discussed internally as well!

thanks. it sounds really interesting

 

[Lordmutis]

We need it urgently! There are several content creators on YouTube like myself (AMENIGAMES, DAVE METTAL, DOminations gamer 123, Arashi, Yulia, etc.) who strive to keep the DOminations community active and communicating in English and Spanish, and we put in time and effort. It would help us a lot to know that our war replays are accurate and real.

 

[Seek]

The post-replay report screen currently utilizes the replay results, so improvements in replay accuracy will directly affect the accuracy of the battle report that follows. We will monitor the level of improvement from the next release and discuss if shorter-term changes to the battle report specifically are warranted as we continue improving replays. The idea of showing troops brought in addition to / in lieu of troops lost will be discussed internally as well!

We want no disputed 100 accurate War Battle Report that are based on actual stuff used/ brought not based on simulated replays. That way there is no dispute when we see it. Proof not based on simulated replays. Proof a battle is legit.

No War Battle Report should be connected in anyway to a simulation it should be what is actually used/ brought That way when I see more air slots being used/ brought I know I turn player in player is banned and this will stop BS false accusations and we will never hear again from a CS they have no proof a battle is legit.

A simulation is just that a simulation not actual battle nor what was used/ brought in battle. No simulation is 100% and therefore can not count as proof a War Battle is legit. If BHG can track crowns I spend then this is not that hard to track what is used/ brought in War Battle. No matter what proof can not be based on simulation.

Thank you for actually replying. Now maybe we can get an actual War Battle report. ( Even I am sick of hearing me complain about this. )

 

[redprince]

By the way, the replays of battles in the game Clash of Clans have an accurate reproduction without errors, why not use the same mechanism or are there technical nuances?

I think the main difference is that in Clash of Clans you drop your troops and they go. There is no "Rally" option or anyhting like that. At least since I last played it like four years ago. One of the big reasons I quit Clash and focused more on Dominations is that there is more movement control with Rallys etc. Clash was very frustrating because you would release and watch, and your hands were tied.

 

[Seek]

The post-replay report screen currently utilizes the replay results, so improvements in replay accuracy will directly affect the accuracy of the battle report that follows. We will monitor the level of improvement from the next release and discuss if shorter-term changes to the battle report specifically are warranted as we continue improving replays. The idea of showing troops brought in addition to / in lieu of troops lost will be discussed internally as well!

Still waiting on those battle report fixes so i can stop complaining about them lol

 

[dadidi]

What have you fixed? It is already version 12.18, and the reply is still the same. After 10 years, what are you doing?