When courts have analyzed cases in terms of detriment to the child, the focus has been on whether it would be detrimental for the child to remain with the parent who has actual custody, not whether it would be detrimental to the child if actual custody were changed. See, e.g., Grumney, 641 So.2d at 907.
Informative article, just what I was looking for.