Surface and Orbit Authority

Sim Speed Issues

Hi folks!

As you may have noticed, in the last couple of weeks the sim speed has been pretty steady between 0.7 and 0.8, with occasional excursions down to 0.5 when computationally-expensive NPCs spawn. I just wanted to tag that I’m aware of the issue and start a discussion.

It looks like the root cause is likely player grids, although large NPC grids (including station spawns) contribute when they’re present as well. I’d like to talk about how we want to address this before we start solving, because we have a couple of options:

  • We could decide that a 0.7 sim speed is perfectly acceptable and leave it alone.
  • We could delete some grids.
  • We could ask people to recycle some grids, and then delete grids if that doesn’t work.
  • We could look into a beefier server.
  • Probably other ideas I haven’t thought of.

What’re your thoughts?

I’ve deleted my old base, and all the associated named grids I could find, since I don’t plan on going back there. It improved the sim cpu from about 140% to 120% (sim speed is the reciprocal of that, so 140% is 0.71 and 120% is 0.83).

It was much improved, for a while, then dived back up to 140-150%. :frowning:

Beefier server doesn’t seem like a reasonable expectation unless we’re able to get funds flowing, we can’t ask you to keep ponying up more and more cash.

Could the PCU limits per player be lowered? For example: I’ve heard somewhere that a major drain on servers is resource processing, so there’s no need for each individual to have a massive refinery complex when the whole faction can share just one.

While it’s very cool to own a large fleet of ships that can do everything you need, it strikes me as being a little contrary to the idea behind multiplayer. If you can do everything, you have no need to interact with anyone else, and might as well be playing single player. Reducing individual PCU limits could encourage specialisation, cooperation and trade.

I realise this may be a bit of a radical idea as it doesn’t allow total freedom to do whatever players want, but total freedom (i.e. continuous expansion) is incompatible with the physical processing constraints of the server anyway, as we’re seeing with the slowdown.

I’m well aware that I am probably a major contributor to the problem, what with owning Port Sol, two LG ships, various SG ships and UCUs, etc. A lot of these are shared with the ORE faction, but there’s certainly room to downsize, so I’ll coordinate with ORE members and see what we actually need and what can be scrapped. And anyway, I have a new LG ship I want to make, but I’d feel guilty adding yet another one when I have so much I don’t really use!

I am in favor of researching guidelines or restrictions for player activity that would help. I know that ORE has a lot more refineries that we strictly need for example. Would it help to disable them when we are not using them? Does the server keep track of grids when no one is near by? Would it help to completely turn of Port Sol? Would it help to land ships with landing gear and turn them off instead of having them attached to connectors with active connections? Is it all TIM’s fault?

It would be nice there was a way to systematically go through the top 5 most likely culprits in some kind of test environment. For example, we could remove all mods, scripts, and NPC spawns and see if there was a dramatic improvement. I don’t know if it is possible to have a useful test with out having several ppl doing the stuff they normally do in the game for a couple of days.

This message includes a server admin note.

A couple of days ago, @SurprisingEdge mentioned in Discord that they’d used the Info tab to clean up their old base. That helped - it brought sim speed up to 0.8 pretty consistently, at least. Last night, I took a look around in the admin tools and noticed a U-shaped piece of debris, consisting of two extended pistons and a bit of armor holding them together, that was lying around the site, and deleted it.

The sim speed returned to 1.0 and has been between 0.9 and 1.0 ever since.

Sadly I didn’t note the PCU of that piece of debris, but given its small size and trivial construction, it can’t have been more than a few hundred points. However, something about it was causing an outsized amount of load on the server. This probably merits further investigation (per @David_s), but at least we have a smoking gun now, and at least things are improving.

I’d like to thank @SurprisingEdge for volunteering to bin such a substantial project!

Woofski’s point about everyone being an onni-specialist/total generalist, expanding endlessly into the void, is pretty spot on. We’ve had casual player interaction, but for the most part people are fulfilling their own needs rather than looking to each other, and the result is a lot of duplicated capability. (The flip side is that it also means everyone can more or less recover if someone gets griefed or has a bad NPC encounter and loses a critical capability.)

It turns out that I, personally, really enjoy tooling around in space with my refinery freighter, at least for now. It’s high capacity, and it’s already open to all and sundry. The only major issue with it is that nobody knows where it is, so it’s not really possible for anyone other than me to use it. I could easily change that - I could establish a regular route, or publish bookmarks on a regular basis, or turn the gain on the antenna up to the full 50 km radius, or otherwise advertise my position. The only thing really stopping me is mistrust that my shit won’t get wrecked - something I could probably learn to let go of.

(Not to say I don’t trust y’all. I do. The server regulars have been nothing but honourable towards one another that I’ve seen, and nothing but helpful. I just have issues stemming from a long history of Internet Grief.)

The LBC faction is particularly wonky because it’s three people each doing more or less their own thing, with no internal organization or structure. I’ve been spending all of my organizing energies here, rather than trying to run a faction. ORE seems a bit more unified, in general, although that may have slipped a bit now that y’all seem to have abandoned Port Sol for the time being and since Woof hasn’t been around as much.

I’m also hugely in favour of David’s suggestion to do actual research on this. Since I have access to the server and the data anyways, and since I care about this, I’m happy to pitch in on that front and run experiments, either on the server (via admin tooling, if necessary) or by taking snapshots and testing things offline. David’s list of candidates,

  • Mods & scripts
  • NPCs
  • Distant grids present/absent
  • Landing gears vs docking connectors
  • Is it TIM

is pretty solid. I’ll have a dig at some of them when I have time, possibly even this week.

NPC Encounters
There is any easy way to drastically limit the number of NPC spawns. I think it would make sense to temporarily reduce the stationary encounters on the server to 0 and limit the cargo encounters to 1 of each type while we work out some of the other performance issues. This would mean that there would only be 2 cargo encounters on the server at any given time. One in space and one on a planet/moon surface. This can all be accomplished by changing 4 settings on the main config file. I will write up a better description and proposal for discussion.

Pistons
This is very interesting.

I have at least three long strings of pistons that I have been using to tether inchworm style cranes. Two of them can be ground down w/out losing any real functionality.

Connected Ships
I know you can find anything on Reddit but I did just run across a post that suggested having 3+ ships connected to a static grid with a bunch of refineries running was linked with .7 sim speed. I am working out a process in my head to put Port Sol in hibernate mode. Something like this.

  • Turn off everything except one small reactor w/fuel.
  • Park all ships with landing gears on asteroid. Turn off everything except one small reactor w/fuel.

Testing
@derspiny, I am wondering if there is another indicator beside sim speed. It seems sim speed is like a late warning that basically says what percentage of the total calculations were accomplished during the that tick. If I understand this correctly, when it is at 1 that means everything was completed.

If sim speed is already at 1 and I put Port Sol in hibernate mode and then leave the area, is there a way to tell what kind of impact I may have had on the performance?

Drills
What about drills? I just ran into a few posts that indicated drills might be a problem including one that showed server rules that limited the number of drills a player could have to 6 each. I have one mining ship that easily has over 30. I also have a mining platform that might have that many. Again tho I wonder if a giant mining ship that is parked and turned off would have the same kind of impact that an active drilling platform would have. I don’t actually need this ship. If I thought it would make room for other things I would grind it down.

There is.

Some quick theory: Space Engineers runs its game simulation in discrete ticks, with an intended simulation rate of 60 ticks per second. The game’s simulation rate is capped: if tick N starts at 0 milliseconds, tick N+1 cannot start until at least 16.67 milliseconds have passed - the game will be idle, if necessary, rather than running ahead of the clock.

If a tick takes longer than 16.67 milliseconds to compute, then the game’s designers have a few options. Many games address this situation by dropping a tick - that is, extending the next tick to cover a larger time period in a single computation - to catch up. Another strategy is to stop simulating part of the game until the ticks have caught up with reality. Space Engineers does neither: it accepts the slowdown as-is, allowing the game to run slower than real time.

The “sim speed” metric is the ratio between the number of ticks actually executed per second and the base sim speed of 60 ticks per second. I don’t know the exact sampling period, but the bottom line is that the sim speed corresponds to the following simulation rates:

Sim speed Ticks per second Wall seconds per in-game second
1.0 60.0 1.0
0.9 54.0 1.11
0.8 48.0 1.25
0.7 42.0 1.43
0.6 36.0 1.67
0.5 30.0 2.0
0.4 24.0 2.5
0.3 18.0 3.3
0.2 12.0 5.0
0.1 6.0 10.0

The other metric the server exposes, which is closely correlated to this, is “server simulation thread CPU,” which is expressed as a percentage. It’s not as clear what this is measuring - my theory is that it’s the percentage of the 16.67 milliseconds per tick that’s actually being used to compute ticks - but by observation, any value below 100% corresponds to a sim speed of 1.0, and the sim speed drops predictably as values rise above 100%. When the sim speed drops to 0.75, for example, the server CPU percentage is around 130% to 135%. The relationship is pretty clear.

Measuring against this has the added bonus, relative to sim speed, of allowing us to see how much headroom we have. A sim speed of 1.0 could mean we’ve got lots of spare CPU power, or it could mean we’re just barely short of overtaxing the server. A CPU utilization number of 70%, however, is very easy to compare to a CPU utilization of 99%. If I were testing, this is one of the values I’d want to observe.

Thank you. This helps me quite a bit. It also allows me to do a bit of testing.

I think your description of how the game just slows down gives me a hint as to why things get ‘herky jerky’ when the sim speed drops to very low levels.

Yes, Server Sim CPU. That’s why I often talk about it. When sim speed is at 1, Server Sim CPU can be anywhere from 0% to 100%, and it’s only when it goes over 100% that sim speed will drop.

I was also hoping that inactive drills didn’t really cost much in performance. When I deleted the drill head grids only, server sim CPU didn’t really change, so maybe this is true. My single test is very anecdotal.

One thing I’m going to try to work toward, that I think may help a bit, is focusing on large grid for ships. The bang-for-pcu-buck with large grid is much better than small grid, though it does present different design challenges. Being resource-constrained, small grid makes sense, but once resources are less of a constraint… :man_shrugging:

I tried a couple of experiments with Port Sol in ‘hibernate mode’ yesterday. It does not seem that moving far away from the base had any appreciable impact on performance. I hung out for about 5 minutes about 40km from any known player grid. The Server Sim CPU Load hung between 85%-92%. There was no obvious change by me just eye balling the debug values. It is hard to draw any final conclusions from this one data point.

If I saw a 5% change and was able to reproduce it reliably by going back and forth then that would have been a positive result.

  • There was an effect below my detection threshold.
  • There was no effect on moving far away.
  • Removing Port Sol all together would be below my detection threshold.