SurviveTheNight

Hello everyone!

The emergency maintenance will shortly be over and we are slowly allowing players to join again, starting off with players who got a rank.

I will go a little bit more into detail in that post, if you're not interested, you may skip to the conclusion section of this post!

⇒ WHAT HAPPENED?
As already said on discord and in our last patch, which you can read here our systems heavily rely on databases.
Databases store your data, they allow us to sync servers and add global features.

In order to gain access to that data our systems have to communicate with databases. They're grabbing a connection from our connection pool and then using that connection to transfer data. Usually it takes a couple milliseconds take long until our system gets a respond from our databases with the requested data after they sent one but in some cases these couple milliseconds turn into a couple 10s of milliseconds, sometimes in very rare cases even a couple 100s.

That lead into the servers having some lagspikes, as all of that data transfer was running on our main thread of the server, which you usually don't do.

That's when we decided to start working on the backend changes and at that time I also received several reports about lag. So we moved all of that data transfer to seperate threads, which don't impact the server's performance as much, they would still take their time to actually be done tho.

Yesterday - right after reopening the server from scheduled maintenance - we noticed something... odd with our item backend and profile backend, as they were saving items... weirdly if they were saving them at all and we decided to shut down our servers once again.

Why did it not save the items? That's a little bit more complicated, but the easiest way to say this, is: While the one server was saving the items to the database, the other server (where you connected to upon a lobby switch e.g.) was already loading the unfinished data from the old lobby.
That is caused by our way of storing items, which is probably done a bit poorly at the moment and we will look into that in Backend Change #2. Our current way of saving items takes up alot of resources and does take a couple 100 milliseconds.

⇒ THE DUPES
We managed to find one dupe immediately, where you could sometimes duplicate items upon server switches, which also lead us into enabling emergency maintenance.
After experimenting alot with that, we eventually came to the conclusion that this is very likely caused by the fact that admins play the game in creative mode.

Creative mode behaves a little bit.. weird when it comes to GUIs (so basically inventories). It can bypass some blocked functions aswell as technically "duplicating" items.

Why do we think this is caused by the creative mode? I rolled back our STN System from before the backend changes and the issue still persisted. Therefore this dupe technically must've been in this game forever. Also we weren't able to replicate this dupe in any way after about 500 tries, while the dupe in creative took us like 5 tries to actually do.

Right after changing our databases to prevent the wipes (next section of this thread) I found more dupes. You were able to basically copy your Bank Pages, Storage and much more over to other profiles, effectively duping all of your items in a way our systems would never detect.

Systems affected by this:

• Storage
• Amulet Bag
• Armorslots of the player's inventory
• Bank Pages
• Quest Progress
• Sewer Data
• and more...

Remember when I said that our database communication runs on a different thread now? Exactly that was the problem. Right when a player disconnects from the lobby, it doesn't have a static profile ID anymore and instead grabs the current profile from the database (which is always 100% up-to-date).

Due to the communication running on a different thread, the profile ID also got lost and the system grabbed the current profile ID from the database. So when switching profiles it grabbed the ID of the profile you're switching to and saving all the data of the previously played profile to the new active profile.

⇒ THE WIPES
We also managed to find a wipe. Due to the fact that items take a bit longer to actually being saved to databases and due to the fact that the server you're connecting to wasn't waiting on the server you came from, the new server started loading in unfinished data.

Therefore when, for example you were moving a piece of Growing Ore from slot 3 to slot 36 the server you came from started saving the inventory. Saving seems to be much slower than loading, so the server you connected to already started loading in your items.

It noticed that slot 3 was empty now (since you moved the Growing Ore to slot 36), so it left the slot empty and moved on. Eventually it overtook the saving process and when it got to slot 36, the new data wasn't actually available in the database just yet. This lead into the server also not spawning in your Growing Ore on slot 36 as it hasn't been a thing on that slot before you joined the previous server.

Temporary solution: We implemented a Checksum to all inventories. Servers will now check the inventory's checksums before starting to load in items, the servers now make sure all slots are 100% up-to-date.

Unfortunately this required us to put the saving process back on the main thread of the server effectively readding a small amount of the lobby lags.

This is not a permanent (and actually a very poorly designed) solution for now and will have to get a rework with our second part of the backend change.

⇒ CONCLUSION
I am very sorry for this being an issue at the moment, issues like that should just not happen at all.

These temporary fixes unfortunately do affect the server performance a tiny bit.

Is it save to play on STN again? Unfortunately I'm gonna have to say no. While we did alot of testing and didn't manage to lose any of our data, this might just have been lucky and there might still be some flaws to our systems. Unfortunately we are not able to figure them out by ourselves though and they might start occurring when more players log on and merge their data to the new system.

BUT! We create backups of all of your data very frequently right now and we will be refunding any lost items/data, so please don't be afraid to play on SurviveTheNight! Even if something is getting lost, we can restore all of that data.

Also please make sure to create a bugreport (preferrably on discord) once you find a dupe or a wipe. Make sure to explain it as good as you can, so we can start working on a fix as fast as possible.

Probably none of the current fixes are permanent, as they were made within 12h and they might impact the servers performance a little bit.

Aswell as that, to refund the lost time of the Player Rewards, we extended their duration by an extra 24h, they will now expire on Jan 18th, 2024 - 00:00 CET!

If you have any further questions about the current situation, please feel free to ask me on discord. You may also ofc ask GGJay, he will do his best to answer your questions regarding this :)

Other things changed/added/fixed:

• Fixed backend of the following systems:
  • Player Inventory
  • Player Storage
  • Player Amulet Bag
  • Bank
  • Sewers
  • Player Quest Progress
  • Profile Management
  • Playtime
  • Forge
  • Daily Quests
• Changed some parts of our Anti Dupe system
• Fixed new Rank Colors being outdated in death messages
• Extended duration of Player Rewards by 24h

That's it!

epic