Before drilling, bang a nail on the surface that you are going to drill to leave a small dent. Then drill into the dent, and this will keep the drill from wandering.
You can drill a large diameter hole much faster if you first drill a small diameter hole.