Even if the bore holes are "off" you can pare only what needs removing, leaving a section of the bore hole on one side of the mortice.

I was asked a while back what I used for bit sizes, if they were under the mortice size, I found my measuring device and the 1-1/2" bit is right on, not undersized. I like it that way.