The Editor's layer visibility calculation code is absolutely, utterly broken. That much is not even a question.
If you want proof - take any of the stock Drakan maps (Ruined Village works well for this purpose), recalculate layer visibility, and observe the results.
"Total disaster" is quite an understatement.
Now, since the way the Editor stores layer data in memory is fairly well understood, it is entirely feasible to write some completely new code for calculating layer visibility, without even bothering with any of the existing broken mess (since about the only reusable part of it is the code that displays the progress bars!).
Now the good question is, how to calculate the layer visibility?
One possibility is to iterate (on a 3D grid - say, with a pitch of 1lu in each direction) over all the points of the "traversable volume" of each layer... except that this is insane, because a typical singleplayer map is 200+ layers, each of which can be quite large.
A much more reasonable approach is to calculate the XYZ extents of the bounding box for each of the layers, then iterate over all the layers with the following algorithm:
- Calculate the XYZ extents of the traversable volume of the current layer, up to the flight ceiling OR the highest vertex in the map, (whichever is higher),
- Enlarge said working volume by the fog radius (plus the camera follow distance... plus another fudge factor for Rynn's jump height, etc.?), in the X, Y and Z directions independently, on all of the sides - thus "growing" our rectangular cuboid box, while preserving its sharp corners,
- Check for 3D intersections between it, and all of the other layers' bounding boxes,
- Any layers whose bounding boxes are found to collide with the "enlarged bounding box" of the current layer are deemed visible from the current layer.
- Fairly simple to implement,
- Very fast in its execution,
- Conservative: for any given fog distance and flight ceiling, it will never result in any layer not being visible when it should have been,
- Unaffected by the vertical ordering ("stacking") of the layers,
- Unaffected by the actual shape of the layers' collision mesh.
- Conservative suboptimal: a lot of layers that are actually not visible (due to eg. occlusion by intervening geometry) invariably also get included in the visibility set,
- Unaffected by the vertical ordering ("stacking") of the layers: see "Conservative", above,
- Unaffected by the actual shape of the layers' collision mesh: also see "Conservative", above,
- Because of the shape of the enlarged bounding area (rectangular cuboid, instead of one with rounded edges and corners), the "trigger zone" has some volumes which result in a layer visibility "hit" despite exceeding the worst-case "nearest viewing distance" by as much as >1.4 times - leading to even more of the non-visible layers being included in the visibility set.
Any other ideas, anyone?