Co-Op Desync Update

Hey everyone,

Here comes another update from the team regarding Party battles!
As everyone is aware, this has been an ongoing issue that the team has spent some time investigating.

There are currently two issues:
Timing issues! - These issues occur when the server gets multiple instructions at the exact same moment, and due to the order they’re processed, things get confused.

Determinism! - a word I had not heard till today. Where data for random events is different in two different places when it should be the same.

Determinism is a tricky thing to fix and we are still investigating, but have at least narrowed down where to look. (We are very grateful for our big-brain server engineer for digging into this.)

We do have a fix in the works for the Timing issues however! We have tested this internally and it essentially involves trading the risk of the co-op battle failing for a small amount of latency in co-op battles. When we tested it, yes, it was noticeable at first, but we fairly quickly stopped noticing. When we turned the fix off to test the original state of co-op and started getting those beloved bugs again.
We are still determining whether or not to roll this fix out after 2.2 goes live.

There are a lot of moving parts when it comes to live co-op, which means finding the cause AND THEN the fix for it has taken some time. Appreciate everyone’s patience while the team have been looking into this further!

7 Likes

Thanks for the update - good to know it’s still being worked on as this is a fun multiplayer element

3 Likes

It is great to know that you are advancing towards making party play more stable. Thank you for your hard work!

2 Likes

Some more information as 2.2 is on the horizon.

Later this week in an attempt to further stabilise co-op battles, we will be introducing some intended lag. This will only affect Party battles. This will look like Gems taking a moment to lock into place after being moved, or spells taking a second to activate etc.

Going forward, we will look at ways to hide this lag and make the game feel more responsive. An example linked to the above would be the Gems would “appear” locked into their position before they are.

This has been a part of the ongoing testing and experiences of the team here in our Australian studio. We do expect the lag to be less notable in regions such as America where you are physically close to the servers. But as always, will keep an eye on feedback & further issues.

Again, we appreciate your patience as this has been an ongoing and major investigation on our side and will continue to monitor Party battles going forward.

6 Likes

Hi everyone,

The changes to Co-op Jeto outlined above have been released not long ago.

As mentioned, you will notice a slight pause when matching Gems in Party battles.
We found when testing this ourselves it was easy to get used to and became less noticeable.

We’d be interested to hear your experience with it once you’ve had a chance to play around with it.

First experience after the changes. Several battles and everything went smoothly, no crash at all. The pause is not too noticeable, I don’t find it bothersome at all. So thumbs up for the time being :slightly_smiling_face: I will come again after I do more parties to update on this.

Thanks for the hard work!

1 Like

After several days of doing parties not a single crash yet. Great work, thank you very much!

1 Like

Super LTTP on this one, but AIlile and I got around to giving the changes a whirl.

The delay in moving gems is significantly more noticeable when playing using AP Mode then with Timer mode. Perhaps this a result of AP Mode needing to check/report the state of the game board after every gem move, versus Timer mode only needing to resolve the state of the game board after the timer expired?

For most players, the delay shouldn’t be much of an added inconvenience. For more nimble players, the added sync lag is annoying because the lag time is generally greater than the time that such a player needs to move the next gem. This results in gem moves not being recognized by the game, while the move timer still is running, which is frustration central. Would not recommend using Timer mode in Co-Op play.

And… it wouldn’t be a co-op post from me without some crash codes, so… :stuck_out_tongue:

We had some consistent crashes during Co-Op runs. Strangely though, the crashes did not occur during the rounds themselves, but during resolution of the after-battle chests themselves.

A selection of some of the crash codes:

As far as we can ascertain:

  • These crashes appear to be attributable to the server processing resolution of the end-of-battle treasure chest.
  • It seems that one player can open the chest without issue, but then the other player crashes out (I opened the chest first, causing AIlile’s crashes in screenshots 2 and 3 when attempting to action the chest. For screenshot 1, I observed her opening the chest without issue and then I crashed out before being able to action the chest).
  • The crashed player drops group and is returned to the Tavern upon restart. The received chest is then actionable as normal per being placed in a blank chest slot of requiring immediate overflow resolution.

Possible attempts by server to incorrectly sync individual chests as if they were one chest (crash being cause by a player attempting to open a chest a server believes has already been opened)?

Group was private (may be a confounding issue or cause), there were no other players in the party. Dungeons run were Wyrmhelm and Pharos of Lothdyn, Difficulty 2.

I would like to add something to the change and the “look and feel” of it:

  • The lag is highly noticeable and very annoying :frowning:
  • It is frustrating to see the enemy (AI) move smoothly, while we bump against a stop sign every move
  • Playing with timer is horrible, as @Lyrian already mentioned, moves are not recognized, which leads to a double penalty: no smooth gameplay and wasting more seconds due to unrecognized moves
  • In about every 2nd game, gems are stuck on top of each other. This means that in the current turn, the player cannot continue playing (the move is not finished after the pause). It also means that in the next move, the old image of the previous gem is still in place, while the actual gem is also drawn in the same place. You need to figure out, which gem is the real one and will probably waste more time trying to match the previous (non-existing) gem color.
  • Sometimes, the gems are not exchanging places, so only one gem moves and gets stuck.

Until I tried playing Co-Op again, I really wanted to keep this to myself. But the way the devs are addressing this topic is really the wrong way of dealing with race-conditions or creating a deterministic chain of events based on remote pseudo-random generator. To be honest, this is a very clumsy way and it would be more helpful to remove Party Mode completely for the next 6-12 Month and re-do the whole client-/server code, preferably by re-using and/or licensing a working, useful network protocol implementation for online games.

Sometimes it is also occours in normal gameplay. I haven’t yet figured out if it is in Dungeon, Kingdom Defense or PVP, but definitely in solo play.
I will keep an eye on it, and post the data on the next occourence.