Brainstorming: Partial DB Standard

Damilis · #1 11-06-2006, 06:24 AM

Greets all. I just wanted to throw down a brainstorm that I have many many months ago and see what people think the Pros/Cons are.

Since EQEmu is an open-sourced application, users can do pretty much whatever they want with any bit or byte they want. This has lead to the segregation of efforts in many different facets of the project, namely NPCIDs, Items and Quests. These three categories represent the largest portion of the database and thus the largest point of segregation. Granted, people have migrated towards consolidating DB's and we have never really had more than 3 or 4 mainstream DB distros out at any given time... this is a good thing, sort of.
I know I have spoken on these boards before about centralization of data and efforts pertaining to the EQEmu effort and have been met with very good insight from others. Here is a refined version of my thoughts:

Standardize certain portions of the database. For instance: 'Reserve' the first 250,000 NPCID's for EQLive-like mobs. Any custom created content *should* start at 250,001. Enact a similar reservation for ITEMID's, NPC types, factions etc. This should allow for the seamless creation of a Npc, Item and Quest set that would work 'right out of the box.' Also, people can offer up their custom DB mods to the public and be able to state:
"This add-on uses NPCID 300000-310000, ITEMID 300000-310000, blah blah blah"
This will allow serverops to see where/if this mod can fit in to their db. Additionally, as long as the db mod is totally self related or related to itself and the 'base' DB, then a serverop can just use a script and bulk change the NPCIDs/ITEMIDs/etc into a range that will fit on his/her server.

All the numbers I used in my example are trivial as its the concept that I would like to discuss. To answer the obvious: Yes I know this is potentially a HUGE effort and NO it won't mandatory, just standardized.

Aight, open floor! Questions, comments, concerns, ideas?

Rhodan · #2 11-06-2006, 07:00 AM

I think reserving IDs is the only sensible thing to do. Years ago I ran a UO emulator and thats exactly what they did. The standard world had reserved ID ranges - if you used those for custom things they would be overwritten during an update.

Some people even went so far as to agree between themselves which ranges individuals would use - this let certain people work on specific aspects of a world and combine their efforts when wanted.

ANd this is a timely discussion. I was just wanting to make a custom item and was coming here to ask if there were any kind of designated ID ranges

Damilis · #3 11-06-2006, 07:08 AM

Quote:

Originally Posted by Rhodan

Some people even went so far as to agree between themselves which ranges individuals would use - this let certain people work on specific aspects of a world and combine their efforts when wanted.

I had several add-on thoughts that I didn't post, one of which being a system to allocate the non-reserved (aka 300k+) ID's for a person in 1k, 5k or 10k blocks. That way even the 'registered' db mods won't overlap.... but that would require a little more of the Devs copious amounts of free time

so I opted out on posting it initally.

John Adams · #4 11-08-2006, 12:04 PM

Pretty sure there is already a generally accepted standard for NPC Ids, and Item IDs. 13th Floor items I believe use the same itemID you find in the live game - or it's an amazing coincidence that all my magelo gear IDs match items in EQEmu's Items table. And NPCs standard numbering is zoneID * 1000, with the first NPC ID for zone 123 being 123000. etc. Fairly sure people have tried to keep mob-specific IDs to this scheme too, though some mobs (like a rat) can spawn anywhere.

Bottom line is, your first statement was accurate indeed - open source, anyone can and will do what they want. But those who feed BACK to the official PEQ project need to follow their guidelines, or it won't become part of an official release. And very few offer their changes because many tweak for customization anyway, and again, the core project is "Live-like"...

Guess I am saying, I think we're fairly standard already.

but always a good idea to remind the new folk.

Damilis · #5 11-08-2006, 02:03 PM

So CaveDudes and Angelox's DBs are based on PEQ?

**GeorgeS** · #6 11-08-2006, 02:20 PM

Quote:

Originally Posted by Damilis

So CaveDudes and Angelox's DBs are based on PEQ?

That's correct

Damilis · #7 11-08-2006, 02:27 PM

Well thats certainly good news to hear. Is this 'standard' documented anywhere, or perhaps, rules for contributing to PEQ?

bushman77 · #8 11-09-2006, 04:05 AM

you can always go checkout <link>http://www.projecteq.net/</link> as they are the ones working on this database.

John Adams · #9 11-09-2006, 05:09 AM

The ProjectEQ forums have a place to post SQL updates that are then considered for inclusion. I am not sure they come here to get them.

**cavedude** · #10 11-09-2006, 05:28 AM

Quote:

Originally Posted by GeorgeS

That's correct

No it isn't, not starting anything here but my database is my own. I started it from scratch.

As for a standard, that is something DRP is working on. I have already begun stripping my database apart and re-organizing all IDs to fit a standard. Duplicate mobs, loot, and faction entries will all be removed as well. The problem is there are essentionally 3 different databases out there (and yes, they are all quite different and have their own strengths/weaknesses) so the other two databases would need to go through the same perging mine is to fit a standard. Any database built from EQBuilder follow the zonenumberid*1000 standard but as Sesmar and I were discussing on our forum, after the zone is initially built, new entries either get tacked on to the end of the db, or somewhere in the middle causing a mess. Our db editing tools will need to be updated to follow the standard. Again, with 3 different databases and multiple people making cool editors how will that work? While I'll continue work on my own database, I am all for a standard and I'll work with whomever I need to to create and follow one.

Damilis · #11 11-09-2006, 06:07 AM

Well, my issue is with linking NPCs with zones is that many NPCs can exists in many different zones... so what then? I am thinking that simply blocking off the first two or three hundred thousand NPCID's and just having the NPCs there, in no particular order. If you attempt to order in any way, then you introduce ambiguity to your standardizing system. Just make it so that all non-custom NPCs are ID <300,000 and standardized from db variant to db variant.

**cavedude** · #12 11-09-2006, 08:32 AM

We thought about that, too. I like having the NPC ID being based on the zone ID because then you can look at its entry and say hey, this guy is in Overthere. It makes world building MUCH easier. Part of the purge I am doing to my database involves deleting duplicate identical mobs and updating their spawn entries to the single NPC entry. Those mobs that are "universal" and are found in multiple zones I feel should have their own block of reserved IDs, possibly in the 800-900k range (or some other obscure number that will probably not get a zone assigned to it any time soon by SOE) That way, when world builders look at a tree snake for example, they can automatically tell by its ID that hey, this guy is a roamer.

John Adams · #13 11-09-2006, 10:49 AM

That's pretty much exactly how I picture it, Cavedude. One question though - regarding memory utilization. I [assume] with items, the higher the item ID, the more memory it takes to load them all in... I assume this because of the 100k cap. Now, do NPCs have this same memory issue when a zone boots or is made static? Say you have 900 NPCs in zone 123, so NPC id 123000 - 123900. Then add 500 roamers (totally exaggerated populations, i agree) starting at 800000-800500. What will happen to that zones memory consumption? Will the zone allocate RAM for all the empty (but possible) spaces between 123900 and 800000?

I'm having a brain fart, so forgive me if I am totally on crack at the moment. I love publicly humiliating myself. Chicks dig it, too.

Rhodan · #14 11-14-2006, 08:06 AM

I'd like to know too.

If the database has two items, #1 and #2,426,333,000, will that use enough ram for 2,426,333,000 items or will it use enough for two items?

The whole idea of cutomized/allocated ID numbers flies out the window if the former is the case.

eq4me · #15 11-14-2006, 10:25 AM

Quote:

Originally Posted by Rhodan

I'd like to know too.

If the database has two items, #1 and #2,426,333,000, will that use enough ram for 2,426,333,000 items or will it use enough for two items?

It certainly appears so, at least with some tables that are loaded into shared memory.

Quote:

Originally Posted by Rhodan

The whole idea of cutomized/allocated ID numbers flies out the window if the former is the case.

I bet there is no way for the routine that is reading the tables to determine what is actual data and what is just the unused default value defined for the table. Someone with time an skill has to look into the source and find out how exactly tables are read into shared memory and change it so that not needed entries(the ones with the default value)are not using up memory. I know I am not explaining it right but I dont have much time right now. I think something in the line of double pointers. so eg. item[ID] does not hold the data but a pointer to it. So item[1001] to item[10001] could all point to the same data structure.