Rampage AA + big pulls = zone issues?

[AndMetal]

I know this has been an issue on Storm Haven for Melees doing AE on anything more than a few mobs at once & I'm not sure if anyone has experienced similar issues.

Basically, if you pull too many mobs at once, then use the AA Ability Rampage (attacks everything with a range of 30), it will either cause one to go Linkdead or crash the zone.

During my testing, I used the new #aggrozone command in Sanctus Seru (probably the most populated zone in the PEQ DB, 782 NPCs after a #repop force). If I used a weapon (with augs, so 1275 damage + 635 cold damage + 1270 human bane damage, 500% strikethrough) with no procs, the zone would hang for a moment (after jumping to about 20% utilization of the process on a 3.2 GHz Intel processor) and a bunch of mobs would die. In addition, the bandwidth utilization on the client was between 9,000 & 10,000 bytes/sec. However, if I put on an Earthshaker without any augments, etc, the zone process shoots up to 100% utilization, the client lags out, and once the client disconnects, the zone utilization goes down to 0 (but doesn't crash).

It seems like the zone server is working so hard on crunching all the data that it doesn't go back to let the client know it's still there.

After thinking about it, I think a temporary fix would be to send some sort of arbitrary packet (maybe like an HP Update) to all the clients in the zone every couple of iterations through EntityList::AEAttack in zone/effects.cpp, that way the client knows the zone is still there & doesn't lag out completely.

Does anyone have any similar experiences, or maybe some ideas on this?

[Derision]

A bit off-topic, but #cast 6801 (AE range 10000) is also a good way to pull the entire zone:

http://lucy.allakhazam.com/spell.htm...01&source=Test

[KLS]

I don't think expecting AEs to flow smoothly across 100+ npcs in a single zone is reasonable =p Can look and see if we can improve it though.

[AndMetal]

Originally Posted by KLS

I don't think expecting AEs to flow smoothly across 100+ npcs in a single zone is reasonable =p

Agreed. That's why, instead of trying to stop a speeding train, I'm thinking at least do something so everything down the line know it's coming.

The big thing I'm not sure about is, should we do this just for AE rampage (an exception), or are there any other circumstances where we would need to tell the client the zone is still alive? And if so, what's the best way to implement it without using much more bandwidth?

If memory serves me, Live implemented something back around Luclin or PoP (I forget which, couldn't find it browsing through the patch logs) where only so many (10-30?) mobs would attack at once, the rest would sit & twiddle their thumbs a safe distance away, then when one dies, another would get closer to take its place. It was odd, because you could pull a massive train, but when some started hitting you, the rest scattered like roaches. Then, if you got far enough that they weren't attacking you, then would all catch up (was a pita to get them all grouped together). I'm not sure if it's still the same way on Live, but maybe this is what we're looking at in the long run.

[ChaosSlayer]

I don't remeber where that was, but it was somewhere, that there was a max cap on how many mobs can be agroed at player at once.
this was capped at like 20 or 30 or so. Specificly for the purpose to prevent massive chain-agro trains and massive lag.

Of course aoe farmers would hate that, but if helps to improve zone stability- something like this could be put into rules =)

PS. related question of zone lag - is it posible to server side set MAX viewable distance for a zone? and fog density?

[trevius]

The odd thing is that he was able to test rampage with a normal weapon and not cause any large CPU spike or "crash", but using a procing weapon, it caused the problem right away and CPU went to 100%. Now, in his tests, he was using an AE procing weapon, which I can understand would cause a huge spike in CPU over a normal non-proc weapon when rampaging a huge pull. But, the rampage crashes on my server seem to happen with just normal procs, so maybe it is the proc process that is driving CPU Utilization up. I may have to do some more in-depth testing myself if I get time. If it is the proc process, maybe we can figure out what is causing it to be so much more CPU intensive than normal melee hits, because I don't think the formulas are any more complex.

[paaco]

With a wep like he was using that has to be a massive amount of number crunching hitting all of those mobs. I don't know if it's much faster at crunching numbers but if anyone wants to test anything like this on my server just as a comparison I'll give you whatever items, skills, lvl you need. It would be awesome to fix rampage and AoE so they aren't crashing zones so bad. That is a nasty problem on Storm Haven, I like to train myself in Delve late at night and rampage for aa's. But every now and then I crash the zone. ( sorry! ) That is only using my 1.5 and the cog blaster wep in offhand.

my server runs 32 bit windows
Phenom 9850 Quad Core
4 gigs of ram
on a 10mbit down 2mbit up cable connection.

If anyone wants to test on it just gimme the word.

[AndMetal]

Originally Posted by paaco

Phenom 9850 Quad Core

The processing power (2.5GHz x 4 cores) would be awesome, except the zones themselves don't support multithreading (at least on my Dual Xenon 3.2GHz machine). Otherwise, I'd want to see what happens

I think the main issue w/ the procs is that the spell code had a lot more that needs to be checked (resists, focuses, etc) vs just the melee code. Add it on top of each other, and things go crazy.

I mean, yeah, my example in SSeru is worse case scenario, but it's still dangerous for stability to use on more than a few mobs at once with a proccing weapon.

One thing that may be worth testing is the exact impact on a single proccing weapon vs an AE proccing weapon.

[KLS]

Spell packets are also considerably larger than melee packets and there's more of them. Knowing procs cause it is a good place to start at least.

[steve]

There seems to be already some form of limiting how many mobs come to agro on a player.

I use the spell Innoruuk's Wrath (http://lucy.allakhazam.com/spell.htm...19&source=Live) to pull NPCs in a zone and only so many come running to attack me. Once I Destroy (http://lucy.allakhazam.com/spell.htm...48&source=Live) them, more come running.

[So_1337]

Is it possible they were initially aggro'd and just took that long to get to you? I found that even using the new #aggrozone command that it would take a considerable amount of time for all the enemies in a zone to get to me.

[jax0rz]

From my testing it does seem a lot more stable if you are using a pure melee weapon instead of a proc weapon. My idea for a fix would be if you are calling function Attack via rampage, call TryWeaponProc once and save the calculation's it does for each hand if you are dual wielding. Then just apply that saved proc info on a flat rate based on the players dex until rampage is over. That way you are still proccing, but not spiking the CPU so badly doing the same calculation over and over.

I tried to code this, but I'm not the best coder and it seemed like a lot of work that would be best left to someone more expert with the current eqemu code.

[Loccochris]

If it is indeed the spell packets over actual proc calc then I wonder if it is possible to check everytime an aoe proc goes off how many mobs are in the area and if its > 10 just don't send the spell packet. Will basically add 1 extra comparison everytime a proc goes off (if proc type = aoe) and then if it is an aoe its going to add more overhead by counting in an area around the player. Depending on where the call to send initial spell packet is it might be easier to just clone PBAOE spells and have a version identical to the other just without a spell effect and if it is a pbaoe proc to switch which version of spell gets cast.

[trevius]

While the large amount of traffic being generated by AEing is definitely a concern, I don't think that is the actual cause of this issue. I think the problem is the server calculating it all at once. I think it gets so involved in the process of calculating the AE spells and hits and updates that it waits to long in between sending a keep alive to the client to keep them from crashing. At least that is my understanding of what AndMetal found.

Though, your idea is probably a good one to use anyway, because it would at least cut down on some unneeded bandwidth usage during AE.

[Loccochris]

While the large amount of traffic being generated by AEing is definitely a concern, I don't think that is the actual cause of this issue. I think the problem is the server calculating it all at once. I think it gets so involved in the process of calculating the AE spells and hits and updates that it waits to long in between sending a keep alive to the client to keep them from crashing. At least that is my understanding of what AndMetal found.

Though, your idea is probably a good one to use anyway, because it would at least cut down on some unneeded bandwidth usage during AE.

Hmm, I would think that the communication from server to client is threaded separately so it should have a chance at processor/nic card. There might be an issue where if they are in the same thread the calcs would prevent position updates (plus rest of packets needed to keep you in game) from happening. Would require threading specific parts of the packet lib that ONLY deal with keeping a player connected to its own dedicated thread.

If that is the problem threading it would probably also end up fixing all the LD's from zoning or multi boxing as a nice side effect. Of course finding the source and fixing it is not going to be easy, have to dig pretty deep in packet libs and pray the coder wrote good comments lol.