I would think you could use a pitch gauge like with a normal tap. Probabaly a good idea to call whoevers AN fittings you plan to use. Earls, russell, etc.
most come in an AN thread, like #6 or #8. If you do that, you need a chamfer at the end for the o-ring to seal against. I have also seen them done in pipe thread, with sealer on the threads.
#6 an should be a 9/16 thread with 18 threads per inch. Like Jody said you need a champher if you are going to use an o-ring. I think you can also buy a cutter that will properly machine an o-ring receiver. I did mine in a lathe. I did a counterbore with a light champher just to start the o-ring.