SoF Augment Code

[trevius]

Since the EQEmu forums have been down frequently lately, I started a discussion on this on the PEQ forums here:

http://www.projecteq.net/phpBB2/viewtopic.php?t=6999

But, while the EQEmu forums are up, this is probably the best place to discuss the SoF Augment work.

With much help from Leere and a few others in that thread, I now have augments actually working on my test server for the most part. The only thing left to do is getting augmented items that are stored within bags to work properly. Unfortunately, I cannot put any of this new code on the SVN until it is fully functional, as it causes client crashes if an item with augments is stored inside of a bag. Once it is fully functional though, I will get it on the SVN ASAP so people can start testing it out.

Basically, the main issue right now is that Items in bag slots are considered sub items, and augments inside an item are also considered sub items. So, augments inside items that are inside a bag would be considered sub items inside sub items. The sub item stuff is working fine, but sub items inside sub items is not.

Here is the basics of how augments are loaded in SoF:
1. If an item has augments in it, the last 4 bytes (int32) in the that items part of the packet will be the total count of augments in the item.
2. The next 4 bytes after that are a spacer between the item and it's augments.
3. That spacer is an int32 that reports the number of the aug slot that the following augment is supposed to go in.
4. This is not the same as the aug slot type, it is a number from 0 to 4 (5 is the max number of aug slots).
5. After that, is sent, the actual augment item serialization for that aug slot is sent.
6. This continues until all of the reported aug slots are accounted for. If everything doesn't go in exactly as it needs to, it will throw off the entire rest of the packet and almost always cause a client crash while attempting to load all of the items.

Right now, all of that is working on my test server, but the sub items within sub items isn't being handled correctly yet, so it breaks the serialization.

OK, onto the actual code I have set for this so far. I am sure that the code I have for this could be adjusted to work better, but for the purpose of getting augments actually working, this seems to work fairly well (other than the sub items issue of course):

NOTE: This code is for TESTING ONLY! Do not try to add this to a production server as it WILL cause crashes. I am only submitting this here so other people can help finish the code for augment handling.

In SoF_structs.h change the following line at the end of the item structure here:

	uint8 padding[8]; //some of this is null term for augs + end of item, shouldn't do this this way but for now

To this:

	uint32 evolve_string; // Some String, but being evolution related is just a guess
	uint32 augment_count;

Then, in SoF.cpp, replace the entire OP_CharInventory encode here:

ENCODE(OP_CharInventory) {
	//consume the packet
	EQApplicationPacket *in = *p;
	*p = NULL;

	if(in->size == 0) {
		in->size = 4;
		in->pBuffer = new uchar[in->size];
		*((uint32 *) in->pBuffer) = 0;
		dest->FastQueuePacket(&in, ack_req);
		return;
	}
	
	//store away the emu struct
	unsigned char *__emu_buffer = in->pBuffer;

	int itemcount = in->size / sizeof(InternalSerializedItem_Struct);
	if(itemcount == 0 || (in->size % sizeof(InternalSerializedItem_Struct)) != 0) {
		_log(NET__STRUCTS, "Wrong size on outbound %s: Got %d, expected multiple of %d", opcodes->EmuToName(in->GetOpcode()), in->size, sizeof(InternalSerializedItem_Struct));
		delete in;
		return;
	}
	InternalSerializedItem_Struct *eq = (InternalSerializedItem_Struct *) in->pBuffer;
	
	in->size = 4;
	in->pBuffer = new uchar[in->size];
	*((uint32 *) in->pBuffer) = 0;
	dest->FastQueuePacket(&in, ack_req);

	//EQApplicationPacket * outapp = new EQApplicationPacket((const EmuOpcode)0x78Cd);

	int r;
	char* serialized = NULL;
	uint32 length = 0;
	for(r = 0; r < itemcount; r++, eq++) 
	{
		length = 0;
		serialized = NULL;
        serialized = SerializeItem((const ItemInst*)eq->inst,eq->slot_id,&length,0);
		if(serialized)
		{
			EQApplicationPacket * outapp = new EQApplicationPacket(OP_ItemPacket, length+4);
			uint32 * type = (uint32*)outapp->pBuffer;
			*type = ItemPacketTrade;
			memcpy(outapp->pBuffer+4, serialized, length);

			_log(NET__ERROR, "Sending item to client");
			_hex(NET__ERROR, outapp->pBuffer, outapp->size);

			dest->FastQueuePacket(&outapp);
			delete[] serialized;
			serialized = NULL;
			if((const ItemInst*)eq->inst,eq->slot_id >= 22 && (const ItemInst*)eq->inst,eq->slot_id <= 30)
			{
				for(int x = 0; x < 10; ++x)
				{
					const ItemInst* subitem = ((const ItemInst*)eq->inst)->GetItem(x);
					if(subitem)
					{
						uint32 sub_length;
						serialized = NULL;
						serialized = SerializeItem(subitem, (((eq->slot_id+3)*10)+x+1), &sub_length, 0);
						if(serialized)
						{
							EQApplicationPacket * suboutapp = new EQApplicationPacket(OP_ItemPacket, sub_length+4);
							uint32 * subtype = (uint32*)suboutapp->pBuffer;
							*subtype = ItemPacketTrade;
							memcpy(suboutapp->pBuffer+4, serialized, sub_length);
							_log(NET__ERROR, "Sending sub item to client");
							_hex(NET__ERROR, suboutapp->pBuffer, suboutapp->size);
							dest->FastQueuePacket(&suboutapp);
							delete[] serialized;
							serialized = NULL;
						}
					}
				}
			}
			else if((const ItemInst*)eq->inst,eq->slot_id >= 2000 && (const ItemInst*)eq->inst,eq->slot_id <= 2023)
			{
				for(int x = 0; x < 10; ++x)
				{
					const ItemInst* subitem = ((const ItemInst*)eq->inst)->GetItem(x);
					if(subitem)
					{
						uint32 sub_length;
						serialized = NULL;
						serialized = SerializeItem(subitem, (((eq->slot_id-2000)*10)+2030+x+1), &sub_length, 0);
						if(serialized)
						{
							EQApplicationPacket * suboutapp = new EQApplicationPacket(OP_ItemPacket, sub_length+4);
							uint32 * subtype = (uint32*)suboutapp->pBuffer;
							*subtype = ItemPacketTrade;
							memcpy(suboutapp->pBuffer+4, serialized, sub_length);
							_log(NET__ERROR, "Sending sub item to client");
							_hex(NET__ERROR, suboutapp->pBuffer, suboutapp->size);
							dest->FastQueuePacket(&suboutapp);
							delete[] serialized;
							serialized = NULL;
						}
					}
				}
			}
			else if((const ItemInst*)eq->inst,eq->slot_id >= 2500 && (const ItemInst*)eq->inst,eq->slot_id <= 2501)
			{
				for(int x = 0; x < 10; ++x)
				{
					const ItemInst* subitem = ((const ItemInst*)eq->inst)->GetItem(x);
					if(subitem)
					{
						uint32 sub_length;
						serialized = NULL;
						serialized = SerializeItem(subitem, (((eq->slot_id-2500)*10)+2530+x+1), &sub_length, 0);
						if(serialized)
						{
							EQApplicationPacket * suboutapp = new EQApplicationPacket(OP_ItemPacket, sub_length+4);
							uint32 * subtype = (uint32*)suboutapp->pBuffer;
							*subtype = ItemPacketTrade;
							memcpy(suboutapp->pBuffer+4, serialized, sub_length);
							_log(NET__ERROR, "Sending sub item to client");
							_hex(NET__ERROR, suboutapp->pBuffer, suboutapp->size);
							dest->FastQueuePacket(&suboutapp);
							delete[] serialized;
							serialized = NULL;
						}
					}
				}
			}
		}
	}

	//Proper way below crashing
	//Workaround above
	//Goal: get the item struct good enough that we don't need the workaround.
	/*uchar *data = NULL;
	uchar *dataptr = NULL;
	uchar *tempdata = NULL;

	//do the transform...
	int r;

	data = new uchar[4];
	uint32 *item_opcode;
	item_opcode = (uint32*)data;
	*item_opcode = 0x69;//0x35;


	uint32 total_length = 4;
	uint32 length = 0;

	char* serialized = NULL;
	for(r = 0; r < itemcount; r++, eq++) 
	{
		length = 0;
		serialized = NULL;
        serialized = SerializeItem((const ItemInst*)eq->inst,eq->slot_id,&length,0);
		if(serialized)
		{
			tempdata = data;
			data = NULL;
			data = new uchar[total_length+length];
			memcpy(data, tempdata, total_length);
			memcpy(data+total_length, serialized, length);
			
			total_length += length;
			delete[] tempdata;
			tempdata = NULL;
			delete[] serialized;
			serialized = NULL;

			if((const ItemInst*)eq->inst,eq->slot_id >= 22 && (const ItemInst*)eq->inst,eq->slot_id < 30)
			{
				for(int x = 0; x < 10; ++x)
				{
					serialized = NULL;
					uint32 sub_length = 0;
					const ItemInst* subitem = ((const ItemInst*)eq->inst)->GetItem(x);
					if(subitem)
					{
						serialized = SerializeItem(subitem, (((eq->slot_id+3)*10)+x+1), &sub_length, 0);
						if(serialized)
						{
							tempdata = data;
							data = NULL;
							data = new uchar[total_length+sub_length];
							memcpy(data, tempdata, total_length);
							memcpy(data+total_length, serialized, sub_length);
							total_length += length;
							delete[] tempdata;
							tempdata = NULL;
							delete[] serialized;
							serialized = NULL;
						}
					}
				}
			}

		}
		else
		{
			_log(NET__ERROR, "Serialization failed on item slot %d during OP_CharInventory.  Item skipped.",eq->slot_id);
		}
	}

	in->size = total_length;
	in->pBuffer = new unsigned char[in->size];
	memcpy(in->pBuffer, data, in->size);

	delete[] __emu_buffer;

	_log(NET__ERROR, "Sending inventory to client");
	_hex(NET__ERROR, in->pBuffer, in->size);

	dest->FastQueuePacket(&in, ack_req);*/
}

With this:

ENCODE(OP_CharInventory) {
	//consume the packet
	EQApplicationPacket *in = *p;
	*p = NULL;

	if(in->size == 0) {
		in->size = 4;
		in->pBuffer = new uchar[in->size];
		*((uint32 *) in->pBuffer) = 0;
		dest->FastQueuePacket(&in, ack_req);
		return;
	}
	
	//store away the emu struct
	unsigned char *__emu_buffer = in->pBuffer;

	int itemcount = in->size / sizeof(InternalSerializedItem_Struct);
	if(itemcount == 0 || (in->size % sizeof(InternalSerializedItem_Struct)) != 0) {
		_log(NET__STRUCTS, "Wrong size on outbound %s: Got %d, expected multiple of %d", opcodes->EmuToName(in->GetOpcode()), in->size, sizeof(InternalSerializedItem_Struct));
		delete in;
		return;
	}
	InternalSerializedItem_Struct *eq = (InternalSerializedItem_Struct *) in->pBuffer;

	uchar *data = NULL;
	//uchar *dataptr = NULL;
	uchar *tempdata = NULL;

	data = new uchar[4];
	uint32 *item_opcode;
	item_opcode = (uint32*)data;
	*item_opcode = 500; //should replace this with Total Item Count including Sub Items
	
	//do the transform...
	int r;
	uint32 total_length = 4;
	uint32 length = 0;

	char* serialized = NULL;
	for(r = 0; r < itemcount; r++, eq++) 
	{
		length = 0;
		serialized = NULL;
        serialized = SerializeItem((const ItemInst*)eq->inst,eq->slot_id,&length,0);
		if(serialized)
		{
			bool normal_item = false;
			const Item_Struct *item = ((const ItemInst*)eq->inst)->GetItem();
			uint8 itemclass = item->ItemClass;
			if(itemclass == 0)
				normal_item = true;

			tempdata = data;
			data = NULL;
			data = new uchar[total_length+length];
			memcpy(data, tempdata, total_length);
			memcpy(data+total_length, serialized, length);
			total_length += length;

			delete[] tempdata;
			tempdata = NULL;
			delete[] serialized;
			serialized = NULL;

			for(int x = 0; x < 10; ++x)
			{
				serialized = NULL;
				uint32 sub_length = 0;
				const ItemInst* subitem = ((const ItemInst*)eq->inst)->GetItem(x);
				if(subitem)
				{
					if((const ItemInst*)eq->inst,eq->slot_id >= 22 && (const ItemInst*)eq->inst,eq->slot_id < 30)
						serialized = SerializeItem(subitem, (((eq->slot_id+3)*10)+x+1), &sub_length, 0);
					else if((const ItemInst*)eq->inst,eq->slot_id >= 2000 && (const ItemInst*)eq->inst,eq->slot_id <= 2023)
						serialized = SerializeItem(subitem, (((eq->slot_id-2000)*10)+2030+x+1), &sub_length, 0);
					else if((const ItemInst*)eq->inst,eq->slot_id >= 2500 && (const ItemInst*)eq->inst,eq->slot_id <= 2501)
						serialized = SerializeItem(subitem, (((eq->slot_id-2500)*10)+2530+x+1), &sub_length, 0);
					else
						serialized = SerializeItem(subitem, eq->slot_id, &sub_length, 0);

					if(serialized)
					{
						tempdata = data;
						data = NULL;
						if(normal_item)
						{
							data = new uchar[total_length+sub_length+4];
							memcpy(data, tempdata, total_length);
							*((uint32*)(data+total_length)) = x;
							total_length += 4;
							memcpy(data+total_length, serialized, sub_length);
							total_length += sub_length;
							delete[] tempdata;
							tempdata = NULL;
							delete[] serialized;
							serialized = NULL; 
						}
						else
						{
							data = new uchar[total_length+sub_length];
							memcpy(data, tempdata, total_length);
							memcpy(data+total_length, serialized, sub_length);
							total_length += sub_length;
							delete[] tempdata;
							tempdata = NULL;
							delete[] serialized;
							serialized = NULL; 
							/*
							for(int k = 0; k < MAX_AUGMENT_SLOTS; ++k)
							{
								serialized = NULL;
								uint32 sub_sub_length = 0;
								const ItemInst* subsubitem = (((const ItemInst*)eq->inst)eq->inst)->GetItem(k);
								if(subsubitem)
								{
									serialized = SerializeItem(subsubitem, eq->slot_id, &sub_sub_length, 0);
									if(serialized)
									{
										tempdata = data;
										data = NULL;
										data = new uchar[total_length+sub_sub_length+4];
										memcpy(data, tempdata, total_length);
										*((uint32*)(data+total_length)) = k;
										total_length += 4;
										memcpy(data+total_length, serialized, sub_sub_length);
										total_length += sub_sub_length;
										delete[] tempdata;
										tempdata = NULL;
										delete[] serialized;
										serialized = NULL; 
									}
								}
							}
							*/
						}
					}
				}
			}
		}
		else
		{
			_log(NET__ERROR, "Serialization failed on item slot %d during OP_CharInventory.  Item skipped.",eq->slot_id);
		}
	}

	in->size = total_length;
	in->pBuffer = new unsigned char[in->size];
	memcpy(in->pBuffer, data, in->size);

	delete[] __emu_buffer;

	_log(NET__ERROR, "Sending inventory to client");
	_hex(NET__ERROR, in->pBuffer, in->size);

	dest->FastQueuePacket(&in, ack_req);
}

Last, after the following line from the Item Serialization in SoF.cpp here:

	itbs.unknown15 = 0xffffffff;

Add the following code:

	uint8 aug_total = 0;
	if(hdr.ItemClass == 0)
	{
		for(int x = 0; x < 5; ++x)
		{
			const ItemInst* subitem = ((const ItemInst*)inst)->GetItem(x);
			if(subitem)
			{
				aug_total++;
			}
		}
	}

	itbs.augment_count = aug_total;

I think that is it. This should let you load a character that has augmented items as long as none of the augmented items are stored inside of a bag.

[trevius]

Also, for augments to be 100% functional, we would need to update the OP_ItemPacket encode to handle them. This would make it so that newly augmented items would show the augments without having to zone. It should also make it so that augments will show up in itemlinks.

This is not functioning for handling subitems yet, and I am not sure why. It causes a crash (zone crash I think) when I try to click a link of an item that is augmented. So, I have commented out the subitem part until that is working fully. But, as far as I can tell, this way works just as well as the way the item packet is currently set to be handled. Just need to figure out how to get subitems reporting propely.

In SoF.cpp replace the entire OP_ItemPacket code here:

ENCODE(OP_ItemLinkResponse) {  ENCODE_FORWARD(OP_ItemPacket); }
ENCODE(OP_ItemPacket) {
	//consume the packet
	EQApplicationPacket *in = *p;
	*p = NULL;
	
	unsigned char *__emu_buffer = in->pBuffer;
	ItemPacket_Struct *old_item_pkt=(ItemPacket_Struct *)__emu_buffer;
	InternalSerializedItem_Struct *int_struct=(InternalSerializedItem_Struct *)(old_item_pkt->SerializedItem);

	uint32 length;
	char *serialized=SerializeItem((ItemInst *)int_struct->inst,int_struct->slot_id,&length,0);

	if (!serialized) {
		_log(NET__STRUCTS, "Serialization failed on item slot %d.",int_struct->slot_id);
		delete in;
		return;
	}
	in->size = length+4;
	in->pBuffer = new unsigned char[in->size];
	ItemPacket_Struct *new_item_pkt=(ItemPacket_Struct *)in->pBuffer;
	new_item_pkt->PacketType=old_item_pkt->PacketType;
	memcpy(new_item_pkt->SerializedItem,serialized,length);

	delete[] __emu_buffer;
	safe_delete_array(serialized);
	dest->FastQueuePacket(&in, ack_req);
}

With this:

ENCODE(OP_ItemLinkResponse) {  ENCODE_FORWARD(OP_ItemPacket); }
ENCODE(OP_ItemPacket) {
	//consume the packet
	EQApplicationPacket *in = *p;
	*p = NULL;
	
	unsigned char *__emu_buffer = in->pBuffer;
	ItemPacket_Struct *old_item_pkt=(ItemPacket_Struct *)__emu_buffer;
	InternalSerializedItem_Struct *int_struct=(InternalSerializedItem_Struct *)(old_item_pkt->SerializedItem);

	uchar *data = NULL;
	uchar *tempdata = NULL;

	int r;
	char* serialized = NULL;
	uint32 total_length = 0;
	uint32 length = 0;

	//serialized = SerializeItem((ItemInst *)int_struct->inst,int_struct->slot_id,&length,0);
	serialized = SerializeItem((const ItemInst*)int_struct->inst,int_struct->slot_id,&length,0);

	if(serialized)
	{
		bool normal_item = false;
		const Item_Struct *item = ((const ItemInst*)int_struct->inst)->GetItem();
		uint8 itemclass = item->ItemClass;
		if(itemclass == 0)
			normal_item = true;

		tempdata = data;
		data = NULL;
		data = new uchar[total_length+length];
		memcpy(data, tempdata, total_length);
		memcpy(data+total_length, serialized, length);
		total_length += length;

		delete[] tempdata;
		tempdata = NULL;
		delete[] serialized;
		serialized = NULL;

		uint32 sub_length;
		/*
		if(normal_item)
		{
			for(int x = 0; x < MAX_AUGMENT_SLOTS; ++x)
			{
				const ItemInst* subitem = ((const ItemInst*)int_struct->inst)->GetItem(x);
				if(subitem)
				{
					sub_length = 0;
					serialized = NULL;
					serialized = SerializeItem(subitem, int_struct->slot_id, &sub_length, 0);

					if(serialized)
					{
						data = new uchar[total_length+sub_length+4];
						memcpy(data, tempdata, total_length);
						*((uint32*)(data+total_length)) = x;
						total_length += 4;
						memcpy(data+total_length, serialized, sub_length);
						total_length += sub_length;
						delete[] tempdata;
						tempdata = NULL;
						delete[] serialized;
						serialized = NULL; 
					}
				}
			}
		}
		/*/
	}
	else
	{
		_log(NET__ERROR, "Serialization failed on item slot %d.",int_struct->slot_id);
		delete in;
		return;
	}

	in->size = total_length;
	in->pBuffer = new unsigned char[in->size];
	ItemPacket_Struct *new_item_pkt=(ItemPacket_Struct *)in->pBuffer;
	new_item_pkt->PacketType=old_item_pkt->PacketType;
	memcpy(new_item_pkt->SerializedItem,data,in->size);

	delete[] __emu_buffer;
	_log(NET__ERROR, "Sending item to client");
	_hex(NET__ERROR, in->pBuffer, in->size);
	safe_delete_array(serialized);

	dest->FastQueuePacket(&in, ack_req);


}

Note that this code change shouldn't cause any crashes, so it should be ok to actually replace this now. But, I would like to get it working properly for augments before updating the SVN with it. As it is now, there is no noticeable difference from the current code to the new code, so no real reason to make this change yet.

[Derision]

I think I got this working. Will do a bit more testing/cleanup and post it later.

It seems aug_count is really subitem count, so applies to bags as well. I also handled the subitem serialization from recursively within SerializeItem.

[trevius]

That is awesome news, Derision! Yeah, it did apply to bags as well. That is why I had it check that the "hdr.ItemClass == 0", which would mean that it only counts subitems on normal items, since bags and books would be 1 and 2 respectively.

LOL, with your help lately, SoF should be fully functional in no time! Maybe I should start manually pulling out the new AA tables info from Live. It is going to really suck to do manually, but at least it can be done that way.

I have most of the itemlink issues worked out for itemlinks from commands like #peekinv and #searchitem and such. Just trying to finish that up and will have 1 more thing done for SoF. Then, itemlinks will almost be finished for SoF. Though, for them to be 100% caught up with Titanium, the itempacket will also have to handle subitems similar to the client inventory encode.

I can't wait to see augs working smoothly