Graal Forums - Playerworlds Server Lag

Graal Forums (https://forums.graalonline.com/forums/index.php)

- Bug Report (https://forums.graalonline.com/forums/forumdisplay.php?f=193)

- - Playerworlds Server Lag (https://forums.graalonline.com/forums/showthread.php?t=134267715)

Playerworlds Server Lag

I've been getting the occasional spike on Zodiac a lot recently, I've checked our logs and stats and am unable to pinpoint a cause. I suspect someone is up to shenanigans and slowing down the server machine.

Quote:

Originally Posted by fowlplay4 (Post 1711396)

Does Era share machines with any other playerworlds?

Quote:

Originally Posted by Stephen (Post 1711397)

Does Era share machines with any other playerworlds?

Pretty sure Era, UN, Zodiac, Zone, and possibly the Login server are all on the same server.

Quote:

Originally Posted by fowlplay4 (Post 1711396)

What kind of shenanigans are we talkin?

Quote:

Originally Posted by Dragon551 (Post 1711398)

Pretty sure Era, UN, Zodiac, Zone, and possibly the Login server are all on the same server.

If that's accurate then the lag may be originating from Era...

I'll check and see what's causing it if it's on Era. We may add you an RC since you're probably a lot better with script efficiency than I am.

Okay, so I've said /stats on RC a couple of times, and I can probably say with this data, that Era is the cause of the lag (I think??)..

PHP Code:


		
			
Server uptime: 1299 hrs 13 mins
Max supported players: 16382
Sleeptime (secs/min): 52386
Connection tries (last min): 6
Players-in (byte/l.m.): 5157586
Players-out (byte/l.m.): 13932311
RCs-in (byte/last min): 188
RCs-out (byte/last min): 373772
NPCServer-in (byte/l.m.): 4321752
NPCServer-out (byte/l.m.): 6190981
ServerList-in (byte/l.m.): 4091
ServerList-out (byte/l.m.): 38378
Player-avrg-in (bit/sec): 8674
Player-avrg-out (bit/sec): 18241
TSockets-in (byte/l.m.): 51088
TSockets-out (byte/l.m.): 318514
Levels: 268
Players: 73
RCs: 10
NPCServers: 1
ServerListers: 1
Memory (bytes): 29885912
Players connecting: 0
Script engine: 0 event objects, 69 destroyed objects, 1 static objects, 6 global variables, 3 script machines
NPC-Server statistics:
Server uptime: 1299 hrs 13 mins
Sleeptime (secs/min): 49056
Scriptime (last min): 58700
Connectiontries (last min): 0
NCs-in (byte/last min): 16
NCs-out (byte/last min): 0
GServer-in (byte/l.m.): 6167843
GServer-out (byte/l.m.): 4322143
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 3092
Active Levels: 375
NPCs: 15058
Active Objects: 2169
DB-NPCs: 9401
NPC-Placeholder: 4002
Projectiles: 19
Players: 83
NC count: 5
GServers: 1
Memory (bytes): 397242528
Top 10 of npcs taken most CPU time:
1. 1.429794232 % -KillStreaks (func_numbers)
2. 0.951229333 % GangSys/Bonus (func_numbers)
3. 0.710798501 % TopKills (in level control.nw at pos (30.5, 30)) (func_time)
4. 0.693522294 % PlayerControl (in level control.nw at pos (30.5, 30))
5. 0.406092802 % -Minimap (func_tween,func_easing)
6. 0.348140001 % DB_AmmoMart (in level era_ammomartrestock.nw at pos (28, 26.5))
7. 0.328036149 % -BulletSystem (hpfunctions)
8. 0.321574211 % Control-NPC (tradefunctions,hpfunctions,partyfunctions,mail_functions)
9. 0.30520002 % DB_PyroTech (in level era_pyrotechxinside06.nw at pos (27, 17))
10. 0.267698764 % LagBot (in level control.nw at pos (30.5, 30))
Most used classes:
1. bounds: 2848 times
2. housefunctions: 1627 times
3. house_object: 999 times
4. christmaspresent: 989 times
5. xmaslights: 974 times
6. block: 954 times
7. tree: 844 times
8. func_sql: 708 times
9. func_time: 687 times
10. trash: 673 times
Script engine: 3 event objects, 2510 destroyed objects, 255 static objects, 10724 global variables, 7 script machines

Notice the 5-6% jump
EDIT: I've checked stats a couple more times, these results don't seem to appear again..

PHP Code:


		
			
GServer statistics:
Server uptime: 1299 hrs 14 mins
Max supported players: 16382
Sleeptime (secs/min): 34055
Connection tries (last min): 6
Players-in (byte/l.m.): 4424975
Players-out (byte/l.m.): 8194827
RCs-in (byte/last min): 95
RCs-out (byte/last min): 192430
NPCServer-in (byte/l.m.): 2910074
NPCServer-out (byte/l.m.): 5256280
ServerList-in (byte/l.m.): 3979
ServerList-out (byte/l.m.): 18004
Player-avrg-in (bit/sec): 8674
Player-avrg-out (bit/sec): 18242
TSockets-in (byte/l.m.): 55681
TSockets-out (byte/l.m.): 146919
Levels: 251
Players: 71
RCs: 10
NPCServers: 1
ServerListers: 1
Memory (bytes): 29665792
Players connecting: 0
Script engine: 0 event objects, 72 destroyed objects, 1 static objects, 6 global variables, 3 script machines
NPC-Server statistics:
Server uptime: 1299 hrs 14 mins
Sleeptime (secs/min): 37968
Scriptime (last min): 45300
Connectiontries (last min): 1
NCs-in (byte/last min): 57
NCs-out (byte/last min): 12359
GServer-in (byte/l.m.): 4807242
GServer-out (byte/l.m.): 2970851
TSockets-in (byte/l.m.): 0
TSockets-out (byte/l.m.): 0
Levels: 3094
Active Levels: 373
NPCs: 14899
Active Objects: 2156
DB-NPCs: 9392
NPC-Placeholder: 3866
Projectiles: 13
Players: 81
NC count: 5
GServers: 1
Memory (bytes): 399990016
Top 10 of npcs taken most CPU time:
1. 6.772886514 % Account-System
2. 5.362913211 % -Polling (func_sql,func_time)
3. 5.266596873 % npcs[3811] (in level era_present_aa-00.nw at pos (52, 1)) (cloneplayer,func_offlineplayers)
4. 3.863579432 % npcs[2154] (in level era_present_01-00.nw at pos (44, 46)) (cloneplayer,func_offlineplayers)
5. 2.589734792 % npcs[4178] (in level era_present_aa-aa.nw at pos (47, 61)) (cloneplayer,func_offlineplayers)
6. 1.426584323 % npcs[3810] (in level era_present_aa-00.nw at pos (42, 0)) (cloneplayer,func_offlineplayers)
7. 1.153976917 % -KillStreaks (func_numbers)
8. 0.921966234 % DB_GunRack
9. 0.802496274 % GangSystem (func_sql)
10. 0.788161754 % Sales_Event (in level era_sales-test.nw at pos (32, 32)) (func_sql,func_time)
Most used classes:
1. bounds: 2848 times
2. housefunctions: 1628 times
3. house_object: 999 times
4. christmaspresent: 989 times
5. xmaslights: 974 times
6. block: 953 times
7. tree: 844 times
8. func_sql: 707 times
9. func_time: 687 times
10. trash: 686 times
Script engine: 3 event objects, 2543 destroyed objects, 327 static objects, 10787 global variables, 7 script machines

Are y'all getting a lot of lag on Era? I know the login server has been acting funny for me lately. Not loading anything when I login every once in awhile.

With just a quick look at Graal Stats it looks like Zodiac, UN, Era, Zone and Login are all on the same machine. Possibly more but those appear to be the main ones.

I've fixed a wrong use of "new TServerPlayer" on Unholy Nation, this object needs to be destroyed by script. Just call .destroy() on it (real players will not be destroyed).

I've been receiving complaints about this happening again.

Quote:

Originally Posted by fowlplay4 (Post 1712428)

I've been receiving complaints about this happening again.

There are huge logs on Era. For some reason -- I'm told because of their file structure -- whenever someone opens the file browser it lags the whole server for several seconds, sometimes a minute.

This may be the root of the whole problem.

Quote:

Originally Posted by Stephen (Post 1712429)

There are huge logs on Era. For some reason -- I'm told because of their file structure -- whenever someone opens the file browser it lags the whole server for several seconds, sometimes a minute.

This is usually caused by some staff member giving themselves more folder rights than they need (like */*), or some new scripter who didn't know better making a bunch of new log folders rather than reusing the existing logging infrastructure. The server freezes up when file browser is opened if there are a large number of folders.

I re-did Era's file logging infrastructure a while back to avoid creating too many folders. Make sure nobody has access to logs outside of what they really need (don't give out logs/*/*/* etc. because there are old logs still there that Stefan never cleared despite me asking multiple times, make sure nobody has the accounts folder or levels/accounts/*..., etc).

As Stefan said a while back, Era really does need to cut down on logs (there are typically many gigabytes of logs stored). Unfortunately, he was never willing to actually help us by implementing log rotation or similar.

Yes, it seems to be anyone who has access to logs on Era. I have removed the logs rights from my rights maybe a month ago, my filebrowser loads almost instantly. I have done the same with Zach's rights and a few other people too.

But for the PR's... They kinda need to open filebrowser for log checking.

Does it still lagg with client-rc I wonder?

lets remove logs and go back to simpler times on graal (:

time to clear the logs maybe?

Quote:

Originally Posted by dbug (Post 1712441)

time to clear the logs maybe?

Not really sure if it has anything to do with the amount of logs, they we're cleared not too long ago.

What Era doesn't tell u is its there own fault for lag
if u go on there hit f2 and go to the script logs its so long with just serverside scripts errors u cant fit them into 3 pm's
cause its too long if you go on Zodiac i couldn't find one which this is a ++ UN has a few but really that big so stuff keeps breaking on Era like the strafe key keeps breaking the mining system lags bad the events some race events u wont see no one move then when the server gets on track u c a winner i think Era should be taken down and be re scripted or something cause its really unplayable i would put it in here but i keep getting access deny

Quote:

Originally Posted by Rave_J (Post 1712590)

if u go on there hit f2 and go to the script logs its so long with just serverside scripts errors u cant fit them into 3 pm's

The F2 log window does not even show any serverside scripting errors, these always go straight to RC.

so when u get something like this example
Script: Function Cancel_Rank.destroy not found in function destroy_Windows of tim_shop_control in script of Weapon -Tim/Mall

thats not a error u are saying knowing tim mall exist
or this

Script: Function HUD_Box_options.destroy not found in function registerBox in script of Weapon -HUD
??? how u figure it's showing up as errors

None of the errors you quoted are serious, and I seriously doubt they will cause any noticeable issues. The scripts are simply trying to destroy GUI elements that don't yet exist. Not best practice, but not really dangerous.

As Crow said, any serverside errors are going to log to RC (or log to npclog.txt).

even if there like 50 something of them ???

Quote:

Originally Posted by bluemelon (Post 1712443)

not really sure if it has anything to do with the amount of logs, they we're cleared not too long ago.

ace reporter zeus reporting in

Quote:

Originally Posted by Rave_J (Post 1712607)

even if there like 50 something of them ???

Yes. The function calls that produce these errors probably only happen once or not very often, anyway. They don't have any negative impact whatsoever.

I'm still getting player's complaining about this on the daily @ Zodiac.

Quote:

Originally Posted by fowlplay4 (Post 1714493)

I'm still getting player's complaining about this on the daily @ Zodiac.

This kind of stuff is the main reason why I no longer spend time developing for Graal. Issues like this are just so ridiculously common, and getting any kind of response from the administration is practically impossible. There were days where the servers were completely unplayable.

Simple fixes like this would go a long way toward restoring developers' confidence in Graal.

Quote:

Originally Posted by cbk1994 (Post 1714522)

Simple fixes like this would go a long way toward restoring developers' confidence in Graal.

Stefan doesn't care about this. He just wants people that are "happy developing" aka people that don't ask questions, just do **** and keep doing it with a smile on their faces. If I ever make anything for this game, I do it for the players. Even that's a rare occurance nowadays from me.

Quote:

Originally Posted by Crono (Post 1714523)

If I ever make anything for this game, I do it for the players. Even that's a rare occurance noawadays from me.

Couldn't agree more.

i have so many people in south east asia, australia, brazil, netherlands that experiance massive lag even before that so called ip mask or source blah blah was found, RIGHT after it was moved back to texas. who was the guy to suggest stefan to move it back to texas because it was such a good server /sarcasm then they wonder why playerbase dropped :confused:

anyways this was during the server lag

http://i45.tinypic.com/2ev68lk.jpg
http://i46.tinypic.com/2dkz9rd.jpg
http://i45.tinypic.com/2s0ke82.jpg
http://i45.tinypic.com/244709y.jpg
http://i47.tinypic.com/11hvvrq.jpg
http://i47.tinypic.com/2yvj50z.jpg
http://i47.tinypic.com/2w3185g.jpg

*pst* Good excuse to jail someone he hates *pst*
http://i45.tinypic.com/2dj1kyf.jpg

Quote:

Originally Posted by Stefan (Post 1716476)

What?

He/she lives in South East Asia and is complaining about terrible pings, and in turn wants you to move the server.

Obligatory whynotboth.jpg

We could have a dedicated playerworld clones for our friends on the other side of the world. So that they can all play together. All eight of them.

Looks like playerworlds got split up and put on their own VMs/machines.

Server Update
15 hours ago
Saturday / April 20, 2013 - 08:30 GMT
Changed host of Atlantis, Bomy Island, Editor Dev, Graal Kingdoms, Graal The Adventure, Login 2, N-Pulse, Ol' West, Skills Debug, Valikorlia

Server Update
15 hours ago
Saturday / April 20, 2013 - 08:20 GMT
Changed host of Era, Kingdoms iPhone, Login, Unholy Nation, Zodiac, Zodiac iPhone, Zone, Zone Debug

That split of the server was the worst idea ever. Mostly every playerworld is more laggy than before. Host them back on the old one!!

Actually nothing has really been changed, the servers are still at the same place x-x

Quote:

Originally Posted by Stefan (Post 1716951)

Actually nothing has really been changed, the servers are still at the same place x-x

Should move them back to Europe.

Well the lag on Era has steadily been increasing since last year. It got really bad about a month or two back (yeah I think in february correct me if i'm wrong). A lot of the international players from Brazil and Asia don't play anymore because of the hard lag there so I understand why they'd be pissed since most of them pay egregious amounts for gold monthly. If the filebrowser is what's making stuff lag on era and in turn makes the server lag which in turn makes zodiac lag then why not do what Chris is suggesting and rotate logs?

Quote:

Originally Posted by Crow (Post 1716955)

Should move them back to Europe.

this!

Quote:

Originally Posted by Crow (Post 1716955)

Should move them back to Europe.

But I like having 10ms ping.

The "high MS means really laggy" comparison doesn't really hold true on era. It's about the strength of the connection not necessarily the distance away you are from the server. Hence why brazilians lag while Europeans don't as much. Somewhat equidistant from the server, yet the Europeans can have 1st world internet while brazil is stuck with 3rd world dial up or something like that.

Quote:

Originally Posted by -HateD- (Post 1717081)

The "high MS means really laggy" comparison doesn't really hold true on era.

It always does. You know the ms in ping measurement stands for milliseconds, right? Milliseconds it takes for a server to answer a client. The longer it takes for the answer packet to arrive at the client, the "laggier" it gets, naturally.