Welcome Board Archive - The recent latency issues

( 5) From: Rufus         Title: The recent latency issues                
                         Posted On: Friday, June 08 2018, 07:45PM
---------------------------------------------------------------------------
The recent 'lag' problems

I'd rather not call them 'lag' ... the proper term is 'chunking.' 
It's actually the game stopping processing input/output while it 
works on something else. 

Legend runs on a pulse-based system where it does every main loop 
of the game every 250ms (4 times/second). When I added in the 'save 
people often' code, I determined this would be viable because 
we spent, on average, 216ms in 'sleep'. That means the mud was simply 
waiting for a set amount of time to pace (for those in the know, we 
don't use 'sleep()' because of its somewhat unpredictable nature, we 
have a more reliable algorithm).

When we added that code, our character list update -- which, just so 
you know, runs through 12,000+ players/mobs, 2500+ act updates, etc 
every quarter of a second -- was taking up no more than about 30ms 
on average. We had plenty of time and disk is fast now. Even your 
friendly local druid with a ton of herbs saves in a matter of a few 
miliseconds even though their pfiles are huge.

When someone would be removed from the game for being link-dead, it
takes a bigger hit, but even before this recent spat of chunking, 
it was rarely noticeable, taking < 45ms, and usually far, far less.

People started complaining about the lag on typing 'save'. I optimized 
some of that code but it was something else that had previously not 
really been a problem that suddenly became a problem, but there was 
an obvious potential slowdown of little value that I removed.

The chunking during the update cycle though is perplexing. 

We did change the compiler, but we'd not changed any code in the 
update or save paths in a while. One of the potential fixes (that 
didn't work) was reverting the compiler. 

I added a number of options to our internal profiler to help track 
this down. Thank you for your patience with the reboots! 

There are many places where we could do some optimization but the 
amount of work and amount of risk is substantial. Compare major 
alterations to this part of the code akin to open heart surgery. 

Personally, I run a virtual machine whose specs mirror, pretty closely,
the VPS we have the mud on. Even ramping up the pulse updates to 20/
second (50ms vs 250ms), the average cycle through the character list 
is 13ms. On the main mud it's 108ms. The peak on my testmud is 85, 
on the main mud right now, it's in excess of 6s.

Everything I've looked into points to environmental issues. We're 
going to explore down that road to see if we can't resolve this first 
as resolving this in code is a 4-6 month (full-time) project.

Thanks for sticking with us. Sorry about the issues!