@Walter_heen After experimenting with this, I was able to find a solution. I noticed that when the layers are set to 3D, the 3D layers need to have the same stacking order (with respect to the camera view) as the layers in the Element panel. So if layer A is behind layer B in the 3D scene, layer A needs to be behind layer B in the Element panel. If not, you get that ugly black alpha edge I show here.
Is this always the case? Is there a workaround to have the alpha edge look correct, regardless of layer order?