Minimizing Spawner Lag

========================================
Sorry for long PM; hopefully it will help you make your map!

Contents:
* Introduction
* How to reduce spawner rendering/GPU lag
* How spawners act on CPU/serverside
* How to make them less CPU-intensive
* A sample way you can implement your system
========================================

Hi, I saw your SimCity Problems video, and I thought I'd help you address the first problem - spawner lag. I've been working quite a bit on how to minimize spawner lag, by studying spawner mechanics and their code. I've done much of this work as I'm working on creating a regenerating dungeon with semi-randomized features, but the research applies equally well to your project.

I think what you want is possible, albeit difficult. You can actually cut out on spawner lag drastically if you consider certain mechanics.

========================================

First, rendering:

* Outside a certain distance (the same distance where entities stop rendering), spawners will not render their spinning entity, and spawnercarts will not render at all. This drastically cuts down on rendering lag when the system is in "resting" state, even if the spawners remain in loaded chunks.

* SethBling's filter has a slight issue you may want to ask him (or someone else) to fix: the spawnerblock will render a spawnerminecart within a spawnerminecart within a spawnerminecart, etc., recursively until the deepest spawnerminecart. This is an issue with spawners rendering - they really do recursively render what a spawnerminecart spawns. The fix is easy: have the topmost/first spawnerminecart render a customdisplaytile of any block other than a spawner. I personally use Air (ID 0) to minimize polygons. Either way, do this, and the spawnerblock will merely render a spinning minecart - no recursion tornado!

* If spawners/minecarts are outside the player's view frustum, they do not render at all. This means you should put the spawners somewhere the player will not usually be looking.

* Particles only render when a player is within RequiredPlayerRange, which brings me to my next section of advice...

========================================

CPU Processing of spawners:

* A spawner is at its least disruptive if no player is within its RequiredPlayerRange. In this state, a spawner merely checks all players in the world, sees how far they are, and aborts when it realizes none are nearby. Any time you intend the system (or any part of it!) to be resting, make sure nobody is within their range.

* If a player is within the range, it will decrement its Delay unless it hits zero. Pretty innocent on its own, but when it hits zero, the CPU intensive stuff begins.

* When Delay is 0, the spawner will make SpawnCount attempts of the following:
** Construct an entity with *base* properties, *not* including the contents of the SpawnData tag. For a SpawnerMinecart, this is basically as simple as it can be; all the complicated data for what it spawns will not yet be created.
** Calculate an Axis Aligned Bounding Box of size spawnrange*2+1 x 9 x spawnrange*2+1 (vertical size could be 8 instead of 9... My experiments and the code don't quite agree here, but it could be an entity-specific quirk).
** Check all Chunk Sections overlapped by this box. Chunk Sections are 1/16 vertical sections of chunks, and entities are stored within these at runtime (when saved to disk, they're all mushed together, but at runtime they're in separate sections to speed up calculations like these). Check all entities within these chunk sections - if an entity has the same ID as the EntityID of the entity we are spawning, check whether its bounding box intersects the box we created in the previous step. Note that entities in the void are stored in the lowest chunk section, and entities above sky limit are stored in the highest chunk section.
** If the number of matching entities is greater than or equal to MaxNearbyEntities, don't spawn, and instead REFRESH THE SPAWNER.
** Otherwise, generate a set of coordinates within SpawnRange x 2 x SpawnRange (floating point horizontal coords, but integer y coord) around the spawner. Move the entity to these coordinates.
** Check if this location conflicts with the entity's spawn conditions. Hostile mobs need this place to be dark, passives need it to be above grass, etc., and most living mobs require air or another non-colliding block. Spawnerminecarts, however, have no such restrictions, and will always pass this test.
** If the previous check succeeded, and it has SpawnData, read that SpawnData and apply it to the entity. Finalize its creation in the world: it exists now. REFRESH THE SPAWNER (note that the spawner is not refreshed until the SpawnCount attempts are performed, but in your case, SpawnCount is 1).
** Otherwise, abort (entity is not actually spawned), and continue with the next SpawnCount attempt. Do not refresh the spawner (well, if SpawnCount is greater than 1, and at least one attempt succeeds, a refresh will happen).
** Once SpawnCount attempts have been made, we are done. If at least one entity was spawned, or we hit a MaxNearbyEntities limit, REFRESH THE SPAWNER.

* I marked "REFRESH THE SPAWNER" in all caps, as it's a function the spawner occasionally does. This function creates a new Delay (randomized using MinSpawnDelay and MaxSpawnDelay), and more importantly, it selects a random SpawnPotential (based on weights) and sets SpawnData to its Properties and EntityID to its Type. This is where you have to watch out: when this data is re-initialized, so is the spinning entity in the spawnercage. The server will send all clients the full data (except SpawnPotentials of the topmost spawnercart, but it still sends the rest afaik) of the entity (this can get ugly on a network if it happens too often). That doesn't mean much for singleplayer, of course, but the game also has to reconstruct the spinning entity in the cage.

* Note the following: SethBling's structure spawners use MaxNearbyEntities to prevent the spawnercart from respawning until it is needed. This means that, every time a player is within its range, and Delay ticks to 0, it has to reconstruct the spawnercart entity in the cage, *whether or not* anything was actually spawned!

========================================

How to actually optimize these damned things:

* In your case, you should probably keep the city in perpetually loaded chunks, and keep the spawners normally unloaded. If you're going to have 25600 spawners, there's no way to escape the fact that the game must tick these and check whether a player is nearby EVERY GAMETICK, regardless of what Delay is. You can drastically reduce the lag by only loading the spawners when they are needed. Don't have all of them in one space, either, or using them will also lag - maybe split them up into quadrants or something, or maybe even dedicate a region to each grid cell of your city. Have about 11 chunks of void surrounding each region, so they do not get loaded when not used.

* Keep Delay decently large: the Refresh step is surprisingly costly, and will constantly happen if you make Min/MaxSpawnDelay small. It's better to give the player a loading screen than crash their Minecraft. You can avoid the Refresh step altogether by using light or air-controlled spawners (requires a living entity with 0HP riding the topmost spawnercart), but you'll probably find it far easier to just keep using SethBling/TrazLander's method.

* If possible, make it so the player is never within RequiredPlayerRange for a spawner that does not need to operate. This, too, will let you avoid that costly Refresh step... In fact, it'll make the system as damned efficient as it can possibly be, at the cost of juggling the player around a little.

========================================

Now, as for wiring... Since you essentially have to minimize the amount of spawners loaded at once, you'll need to use wireless redstone. My method of choice would be scoreboard objectives: When a player selects, say, "create coal at 10, 3", you set their score in 3 objectives.

* Two objectives represent the cell they are modifying - you can decode this to choose where to teleport them. If each cell's 400 spawners are in an independent area (11+ chunks away from all other spawners), all you have to do is send them to the corresponding area, right on a pressureplate/tripwire.

* Now, another objective would represent the structure they are creating, or that they wish to destroy a structure at that coordinate (let's say 1-100 represent "create structure with this ID", and 0 represents "delete whatever is here").

* That tripwire/plate they were teleported onto will tell the system to decode this score: teleport them onto a structurespawner-activating-pressureplate for the desired structure, if we are creating a structure. Otherwise, if we're deleting the structure at that coordinate, look up *another* objective, which stores the ID of the current structure at this cell, and use that ID to decide which structure-deleter-pressureplate to teleport the player onto.

* Finally, wrap things up by updating the objective which store's that cell's ID, and teleport the player back where they came from.

========================================

I hope this can help, and I wish you luck with this project. I definitely think it's possible, but I don't think it will be easy.

Regards,
Deretythe/WolfieMario

P.S. If you respond, I may not be able to get back to you for a while, as I'm leaving today for 3 weeks. That's actually part of the reason why I made this message so damned long - I wanted to make sure I could get as much info across as I thought would help you.