HeroEngine Forums
Welcome, Guest. Please login or Register for HeroCloud Account.
Pages: [1] 2

Author Topic: Data Storage Plan Critique and Advice Wanted  (Read 4349 times)

Hiro_Protagonist

  • World Owners
  • ****
  • Posts: 30
    • View Profile
Data Storage Plan Critique and Advice Wanted
« on: Feb 07, 14, 07:42:00 PM »

I'm laying out my first big system right now for our game, and I'd like to run it past the experts and make sure I'm going down the right track.

In our game, skills and abilities have mutable data concerning their formulas- how much it costs to upgrade a skill, it's component percentages, all the formula secret sauce- which is supposed to be changed in real time as the game evolves. This data needs to be accessible by all the area servers, but not necessarily by the client- usually when one of these formulas is used it would be via an RPC call. Some of it does need to go to the client, so we really have two different sets of data to keep track of.

All the skills and abilities also have non-mutable data, such as their names and descriptions, and eventually things like animation parameters and whatever else we would want to store in a prototype.

My plan is to have a system area, SkillSystemArea, where all the instances of the skills are created and stored in the GOM, and processing for the changes can be done. I could put a custom GUI dashboard in the area to allow us to monitor what's going on and set the data by hand before the AI parts are complete. Inside the SkillSystemArea, I would just associate the Skill nodes with the area node for the main persistence.

My original plan was to use just server-to-server replication to make the skill data available to the other area servers, but this also seems like a great use of the Spec system, as it is a hybrid of mutable and non mutable data. However, the spec system seems very much geared just for client-server replication, and not server-server.

My questions are:
1. Is this the right tree to be barking up at all?

2. Should I use the Spec System by itself (assuming it can do server-server replication properly), or do I need a hybrid system using Specs and server data replicated via a side channel?

Hopefully this makes sense, thanks for any advice!
Logged

Thazager

  • General Accounts
  • *
  • Posts: 1160
  • Never stop learning
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #1 on: Feb 07, 14, 10:52:44 PM »

One way, though not using SpecOracle, would be making a class with the fields in it, then glomming that class onto the character.
Logged
Lead scripter for EO, Repop helper.
HSL Video tutorials:
https://community.heroengine.com/forums/index.php/topic,1719.msg36858.html#msg3685

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #2 on: Feb 07, 14, 11:13:50 PM »

use the spec system.

The system area is less for storing data, should be about processing data, so having non-mutable data consuming ram and space in a system area isn't a terrific idea. Will work but it's processor waste that's not needed.

Also you don't want to make calls to other areas for data / replication unless it's needed, and you want to try and keep that down, as it's a bit load costly, and you have to create proxy setups. Cause one area has to become the authority and the other areas have to use the proxy data.

The spec system is the perfect fit for what you need. The non-mutable data basically gets added to the player node, and then at player login is checked for any variances if there are any with the server versus client, it pulls the non-mutable data back down to the client, and stores it locally again. That removes a ton of load off the server, and bandwidth and areas. Think about this, you have a spell Ice ball. Any time someone opens their spell book it has to look up the name for that skill aka iceball. Cause it's non-mutable data stored on the client there is zero cost for a player to sit there and spam his spell book open and close. The client can do that all day long, and only his client will be affected. If you stored that non-mutable data on the server in an area, system area, or other means and made a call to it. A player opening a spell book of 30 spells makes you pull it every time he opens, causing a hit to performance and bandwidth. Take 4 million people doing that at one time, you go from 0 hits to the server with spec system Or 120 million remote calls to pull data. So logically you have to store that stuff on the client.

Now you could write a ton of scripts and make a bunch of fields and try to keep them replicated, but... once again, all you are doing is rebuilding the spec system, and most likely in a much less efficient way of doing it. Also you aren't taking use of the client side caching mechanisms or the spec system tools that make dealing with things easier. So eventually you end up just building the spec system anyways. So might as well save yourself the time and hassle.

Not to mention the amount of storage space you will save. Cause all the non-mutable data gets boiled down to just a simple ID when it comes time to instantiate that for the player. Instead of having each player having to add a few dozen fields for each skill/ability times every player. Starts to add up.
Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]

Tarra2012

  • General Accounts
  • *
  • Posts: 113
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #3 on: Feb 08, 14, 11:26:44 AM »

Another proposal, to make the setup easy:

Have the constant data in a prototype (GAMESKILLS)
Have the central formulas in a class (script serverside)
Have the mutable per player data (like skill-experience) at each character.

In many cases the usage of Prototypes (and their instanced objects called SYSTEMNODES) is more easy than the usage of SpecSystem.
I will show you how easy the access could be serverside.

constant data
$GAMESKILL[Skill1].name = Myname
$GAMESKILL[Skill1].range = 12
$GAMESKILL[Skill1].mana = 5
$GAMESKILL[Skill1].dmg = 10

mutable data
player.skilllist[skill1].experience = 1000
player.skilllist[skill1].level = 1

Formulas (ServerSide ScriptMethods)
untrusted remote trigger_skill(skillname)
  *do your calculations
  *e.g. (playerlevel + mainstat + $GAMESKILL[Skill1].dmg) * randomvalue
 

Dependent on how you communicate the Skillinfos, both data can be available on client.
This mean replicating a Prototype (server->client) and normal replication of a skilllist that belongs to each player.


>>
System nodes were adopted as the primary mechanism at the HSL script level enabling game-specific implementations of a licensee to extend/override the required functionality included in the Clean Engine. As with all system nodes, this is accomplished by GLOMming a game-specific class onto the prototype ($NAME) from which a singleton node ($NAME) is instantiated for a particular local GOM.

http://hewiki.heroengine.com/wiki/System_nodes



« Last Edit: Feb 08, 14, 12:14:00 PM by Tarra2012 »
Logged

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #4 on: Feb 08, 14, 08:38:09 PM »

problem with system areas is you have a copy / clone of each of them per area server. So that is a lot of data to duplicate on your server cluster. Also means more loaded into the area's memory. While you can make updates would have to do so for each area individually as each area has a copy of that data, which is unique to each area. So would have to have a system area change the system node data for each area.

I still vote for spec system, but I also see after re-reading you are wanting to change skill data realtime.

I hope by real time you mean like once a week, during down time, and that you include patch notes?

If you however mean real time as in the game looks to adjust those values as players play. I highly recommend against it.

Reason 1.) the changes should be internally beta tested. When you start having smart players that know every aspect about the system, making a single change can have a huge fall out. So lets say you increase the magic penetration values for fireball. What you didn't test was the mobs with weak fire penetrations ended up falling into a negative value, which your game formulas didn't account for and they insta die. Now the entire community sees this happening and super farms minions and your economy gets out of place, you have a ton of explaining to do about the bug, and people are royally upset that mages got a free few hours of super farming. Half the community will yell at you, others will yell at that half for being whimps and you have a forum battle to simmer down.
So always need to test, formula changes. That is a bit extreme, but you never want to assume, no bugs creep in.

Reason 2.) The players. Some like change others don't, all players hate un-announced change. Really don't want people to get to the point where the gear they worked on for months is instantlly nerfed with no explanation to why, just saying the system adjusted as needed will give you players leaving. It's why all games provide patch notes, that explain roughly what happened and why it was needed. Players will adapt especially to fix a bad formula or such. They won't take getting jerked around with no explanations.

Reason 3.) Any built in automated system you make will be exploited. So lets say your system looks to see if there is a particular damage type that is being used constantly, so it needs trimmed down or such, or it seems to crit too often. Lets use crits a ton, so your system looks to see that a skill should only be crit hitting about 25% but due to other factors it's averaging about 40%. So you make an adjustment, but the question is did it account that the average was higher cause a guild of crit geared players was running around throwing the balance, off and unless you went pure crit gear the average was still 25%, and while they were crit hitting they had low damage, cause they weren't geared to create damage. Gamers are vile calculating systematic beasts, they can and will figure any way to exploit any system they can. Giving them tools to real time nerf or buff skills, just going to cause headaches, in means that you will end up building such a complex system to manage the real time checking that your servers lose power to manage game play.

So to recap real time change would be a tremendious nightmare, system hog, that is liable to be exploited by the community.

So your back to more traditional maintenance patching of abilities which shifts you back to using the spec system as an ideal way to do it.

If you look at other options, you eventually end up adding on all the pieces to the custom system that the spec system already has in place. I leave you with this from the wiki.

Quote
Everything the Spec System does could be implemented in a different way by your development staff. Our general feeling at HeroEngine is that you could re-create the wheel, but why would you want to? As a company we have invested time and thought into the design of the Spec System, to solve a very complex problem in a manner particularly suited to take advantage of the HeroEngine development environment. You, as a licensee, can gain from this prior work, by using the resulting framework if you so choose.
Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]

FI-ScottZ

  • General Accounts
  • *
  • Posts: 1407
    • View Profile
    • Forever Interactive, Inc.
Re: Data Storage Plan Critique and Advice Wanted
« Reply #5 on: Feb 08, 14, 11:29:52 PM »

Have you looked over this page?
http://hewiki.heroengine.com/wiki/Data_storage_options

It details many different approaches and their pro's/con's, and will probably answer you questions.

I agree that the Spec System is ideal for data-driven design of NON-mutable data.  It comes with a built-in system for editing the specs which is customizable (forming the basis of most of the tools we have developed).  However, as stated, the specs themselves are intended to be immutable.  They can be changed in development, but being that they are underneath simply prototypes they should not be changed during a game on production servers.

For the mutable data, Spec-derived Nodes could be the way to go.  But I am curious as to what you are really meaning by storing formulas?  Naturally, only data can be stored in fields, so I guess you mean to store parameters that will be tweaked for use in functions.

Also you need to consider how the data will change, in particular will that be over continuous or discrete values?  For discrete, finite values, you might consider using the spec system where each spec represents a level with its own set of values.  Then you can "level up" a formula for a given user by simply having them use a different spec, and the values for those levels are very easy to tune as you test them out.

But for more continuous or infinite values you'll need to use truly mutable nodes.
« Last Edit: Feb 08, 14, 11:31:53 PM by ScottZarnke »
Logged
Scott Zarnke
Lead Programmer, Visions of Zosimos
CTO, Forever Interactive, Inc.

Hiro_Protagonist

  • World Owners
  • ****
  • Posts: 30
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #6 on: Feb 09, 14, 12:59:56 PM »

Thank you all for the feedback.

I have been mulling this over and reading the wiki more (I have about 30 tabs of stuff right now that I've read several times, including all the stuff about data storage, etc). The Spec system covers a great deal of what I want; but the problem is as keeperofstars noticed is that I really have two layers of storage and mutability. Allow me to better explain the nature of the design:

In our game, we define (and test) skills- most of this data in immutable or based on other values of the player (their level of the skill, their attributes, etc.) The mechanics of the skill are relatively constant. We do record how many times a skill has been used- this data is easily and correctly stored in the spec instance for the player.

When a player goes to upgrade their skill, the cost to do so is based on a formula with many parameters- the player's current skill level and number of times used, but also a bunch of floating point values with names like CurveFactor and the exponent base and all sorts of things that let us (and eventually the game) tweak things to achieve whatever balance it is we want. These values we do not need to keep on a per-player per skill basis- these are the values I want to keep isolated to the server cluster. These values are not discrete, and are indeed controlled by the game in real time based on what players are using and doing; as part of the greater economic control system*. These values are continuous and would not be well suited to a discrete or level system.

My proposed design is has become: Use the spec system for the majority of the skills- immutable data goes in the prototypes, and mutable data based on the Player's actions is stored on the player node via the spec instance. We will use a system area to hold the data about the skill costs definitions. Retrieving these values could be done either via RPC or replication- in the grand scheme of things it's really not that much data to be replicating, it changes slowly (every few minutes, perhaps, but not so slowly that it could all be put in prototypes) so replication delta traffic is pretty minimal. It seems a bit overkill to have to make an RPC call every time a player needs to update a skill, even if we do some kind of timing and caching, I think just globally distributing this data amongst the area servers seems the most efficient method. We have several systems that follow a similar pattern, and some of the other ones run calculations more often than this one, so I'd like to make sure I understand how to do this.



*keeperofstars: you raise some very valid points as to testing and players liking change and exploits and whatnot. We are doing this as part of the core game design- players will know that these costs are not fixed. If they actually come up with ways of exploiting the formulas (using skills they don't like to decrease the cost of skills they want, for instance), that is actually encouraged, especially in the context of our game. I could go on here, but all the points you raised were discussed and actually playtested in the paper prototypes. The actual effects of the abilities are not planned to change in realtime, just the costs, so player and gameplay backlash should not be an issue. I do appreciate your concern, though!
Logged

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #7 on: Feb 09, 14, 05:48:45 PM »

if you going to stick with keeping something like a cost curve or such. Then you should build a single node that is part of a system area. Not a system node, but spin up a system area, that holds these values.

From there make changes just to the system area's values. You will end up making a bunch of calls to this system area, as any time the values need to be used you would want to look to the values stored at that time in the system area.

It's going to be a hit but one you will just have to take, there is no point to cache the data to the client. If it's changing that frequently, cause you going to spend more time sending the data then just making a call to retrieve it. It's a fine line of balance here, but assuming you have cool downs, etc for skill use you probably would end up caching about as often as you are updating. Also it would have to be put to the area glom for checking as well.

By using the system area you can spin up more instances if needed to handle the load and manage things. It can listen to calls from all areas, so you don't have to replicate the data to each area. It would only be one source of authentication. You won't have to worry if an area is spun up or down, or spinning areas up or down. Don't have to worry about prototypes that are locked during game play, etc. If you build good marshalling between the client and the server, similar to what is done with the chat foundation system, you can minimize the hit.

So will mean your taking a bandwidth hit and a call back hit, but if it's a key aspect of your game it's a key aspect.

We all make those judegment calls, for example I'm taking a hit performance server side with advanced AI, but think it's worth the load.
Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]

JoshHalls

  • Founding Professional
  • *****
  • Posts: 335
    • View Profile
    • The Repopulation
Re: Data Storage Plan Critique and Advice Wanted
« Reply #8 on: Feb 11, 14, 10:23:09 AM »

If you are looking to have mutable data as distance you probably want to build it into a static formula.

So say you have a skill range of 1 to 100 you have something like this.

10 + x

x would be whatever you want to substitute in. it could be skill range/10 or could be a curve or well anything. Spec Oracles in general are going to be your friend and if you want to go in and say change the formula later you can either tweek the processing code or you can tweek the general formula (just kind of hard to do it in the fly in production).

MUD I use to run I abused that system to death. Could pretty much put all of the values in a single formula and then it would switch out the variable based on character data.

The more random bits of data you have floating around that you cannot easily define or track the harder it is going to be to go back and balance everything.
Logged
Co-Owner/Programmer - The Repopulation

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #9 on: Feb 11, 14, 10:48:48 AM »

Oh and meant to add to my previous post, would want to store the data in an arbitrary root node, then use the system area to make modifications / handle the requests for access to that data. Taking JoshHalls suggestion have the formulas stored in the spec and use value x that references the ARN for it's real time adjustments, will help some.

Completely forget any cache option, and build things with call backs, built in. I still have some concern with some players getting one value for costs while others have a different one. So going to need to make sure you validate all the time against the server. Thus the no cache client side option. 

Real time formula modifications is going to be a challenge to do. Less from a technical aspect, and more from just the logistics of it.

But let us know if you have any other questions with implementation.



Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]

Hiro_Protagonist

  • World Owners
  • ****
  • Posts: 30
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #10 on: Feb 16, 14, 08:07:42 PM »

Thanks for all the help guys, I think I can run with it from here. I have other questions of course but I'll post new threads for them as they are more specific. Thanks again!
Logged

Hiro_Protagonist

  • World Owners
  • ****
  • Posts: 30
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #11 on: Feb 27, 14, 12:21:41 AM »

...
From there make changes just to the system area's values. You will end up making a bunch of calls to this system area, as any time the values need to be used you would want to look to the values stored at that time in the system area.

It's going to be a hit but one you will just have to take, there is no point to cache the data to the client. If it's changing that frequently, cause you going to spend more time sending the data then just making a call to retrieve it. It's a fine line of balance here, but assuming you have cool downs, etc for skill use you probably would end up caching about as often as you are updating. Also it would have to be put to the area glom for checking as well.
...

I'm pretty sure I understand the design you are recommending, which is very similar to what I have in mind except you are recommending RPC instead of replication. This would indeed result in a bandwidth hit, especially without some kind of caching layer- that would not be too bad, I could simply set a timer to update the cache every five minutes or so. My question- simply for my own elucidation- is why not use replication to distribute the data in the system area? Obviously, having all other areas have a copy of this data increased the memory footprint, but it would seem to me to be very bandwidth conservative. Once the nodes are proxied, the replication system should only ever be sending out delta packets, which would only happen when the data is changed.

Thanks for sticking with me on this stuff!
Logged

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #12 on: Feb 27, 14, 08:01:47 AM »

It's a splitting hairs piece. The game play areas won't need to know or deal with the data period, the system area will be doing that.

The challenge with replication especially in a changing skill formula setup that can't have a cache mechanic, is volume versus us.

What I mean is lets say you have 1 million players, we will break them into warrior, mage, ranger, just for example. And we will assume each have their own skill set. Unique.

Even if you are doing pick any skill you want this example still applies i'm just simplifying it.

So that means 330,000 warriors, mages, rangers.

Well you make a change to the warrior's formulas, if you rely on replication, you will either have to parse out who needs to get that replication data or you end up replicating it to 770,000 users, that will never need that data. Start to scale this and start to scale it to 10's of millions and it adds up.

So you say well what about replication groups, well if you had the above example then yes the easy fix is to use replication and just keep the class types in a unique replication group. But if you are letting a player pick and choose the skills they need, I think it will be a harder task at hand. Still might be able to think up something, but will have to ponder on it some.

Caching can work some on client side, but need to talk about what skill data on client side means as well. client side skill data is purely for visual use. Nothing more, cause you can't trust the client to not be hacked, so the data that is on the client side is going to be a mute point of data. You will be checking their server side skill data for the damage / skill effects. So you could get away with a cache on the client data, but players might get upset their skill data never matches what the server data is.

So actually when you think about it anyway, client replication won't give you anything meaningful, beyond updating the data to the player, which won't provide anything meaningful cause it's never accurate. So only the static data of the skill will have any meaning to the player and the base spec will manage that.

And if I'm missing something thought wise, I'm sorry cause I'm tired, but I think my logic is sound. In the fact you can't trust the client, the area server won't exactly have a copy of the data cause it's in the system area, the players will make a call to the area server that they want to do action / skill A. The server will look up the players mutable skill data, make a call to the system area where it will apply the formulas and send back the data needed for the server area to apply the data / effect. Or could do all the look up and such in the system area, either way works, just depends where you want to put the loads. I would probably do a split up of it, where the server area calls the system area, just cause the server area wouldn't need to do player looks etc and would have the SAS to help out with the replication of the end result.

Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]

Hiro_Protagonist

  • World Owners
  • ****
  • Posts: 30
    • View Profile
Re: Data Storage Plan Critique and Advice Wanted
« Reply #13 on: Feb 27, 14, 09:18:36 PM »

Thanks for the well-thought out reply, keeper. I think the only thing that may not be clear is that all the data in the system area that I am talking about would only be replicated between the area servers- not to the client. So the trust issues are not a problem- any time the client would actually make use of this data it would be via RPC. I'm using a generic spec system for all the actual client-side data.

The condensed version at this point is I want to distribute mutable data between all the area servers in the cluster. I will have a single master system area that manages the data, all the others would simply read the data. In this case, it seems (to me) that replication is a viable (even preferable) option vs. RPC, although both would work.
Logged

keeperofstars

  • General Accounts
  • *
  • Posts: 998
    • View Profile
    • StarKeeper Online
Re: Data Storage Plan Critique and Advice Wanted
« Reply #14 on: Feb 28, 14, 09:09:55 AM »

The choice is where you want that work to be done then,

do you want to have the area servers managing the combat that requires those formulas, or have them look up the formulas.

Overall I think it's splitting hairs, but I wonder if the added work of the replication versus making the call balances out.

Once again think that you have to replicate to all the areas, this could take a bit of time, it's not going to be consistent time wise, on when it happens between the areas, so this could potentially cause problems with seamless linked areas where, the one area hasn't managed to update it's local cache of the formulas and the area next to it has, so players in that area are hitting harder. Depending on how fast the replication happens or doesn't happen, which for this would probably be one of those have to jump in queues to keep it fresh. You will also be burning resource power updating skills in areas where combat might not even be allowed, or in areas where the amount of fighting is minimal or not even happening. For example a more "travel" zone where players kind of spend time just playing around but rarely doing much combat. (think when you get to expansions and those newbie or starter areas are only visited by a handful of people if ever, just enough to keep them alive, but not enough to warrant the wasted energy to keep fully in sync.

But then again I'm not sure, it's a tough one, in regards to the servers needing to talk to the system area all the time for the validation of formulas, though in a crowded area with lots of fighting. Is it too taxing to make those calls or not, there wouldn't be a "bandwidth" issue cause they are both server side. Just not sure on that one though. Haven't built out a system like this to compare it against, and without knowing the super finites of that server to system area communication in ultra detail I can't say one way or the other.

I would personally lean to just having the area server poll the data from the system area.

Now just thought might go hybrid, option. In don't replicate, but have the areas setup to hold an Arbiterary Root Node, that is the formula data for the skills. Each area gets a copy of it, loaded in. then when a combat action takes place, have it check for "freshness" do this by having a last updated timestamp, and compare that to the current system time and see if it's greater than formula refresh time that the system area does.
So if (systemTime.now - lastRefresh is greater than 4 minutes. then repull the node and replace it.

That way you aren't updating the node in areas that aren't needing the data, you are still getting the new data when it's needed. And you can have some priority in nodes needing the data. IE the empty area won't be taking up replication time. Yet your area server can manage the combat data for the players. Without the need for every combat move to require a call to the system area

eh just another thought. There is always a billion ways to do the same task.
Logged
[img]http://screencast.com/t/x7btcSSyp3h0[\img]
Pages: [1] 2