Does replacing windows translate to removing existing lights and their embedded bucks in the panel walls ? Only caution would be to look out for stray electrical wiring. After removing exising bucks cutting the RO for the new bucks can be easily done with a combination of a Dremel plunge bit for the OSB or other panel enclosures and any convenient recip to remove residual foam. inletting the foam to house the new bucks behind the panel material can be done with any number of short stroke recips or even a utility knife. Do this carefully so the flat side of the new bucks is in contact with the foam.
Tedious but effective.
Work safe, have fun.