Preheat oven to 325 degrees.
Zest the limes and then juice them.
Process the graham crackers in a food processor or blender, then melt the butter and have it ready.
Whisk the egg yolks and lime zest together in a bowl until tinted light green. This takes about 2 minutes.
Beat in milk, then juice, and set aside at room temperature till it thickens.
Mix the graham cracker crumbs and sugar in another bowl.
Add butter and stir with a fork until well blended.
Pour this mixture into a 9-inch pie pan and press over the bottom and up the sides of the pan. Try using the bottom of a measuring cup to make sure the graham cracker mixture forms a firm crust on the bottom of the pie pan.
Bake on the center rack for about 15 minutes until the crust is lightly brown; remove, and let cool to room temperature.
Pour the lime filling into the crust, spread evenly, and then bake for 15 minutes until the center sets, but still wiggles when shaken.
Remove from oven and cool to room temperature.
Refrigerate for at least three hours until well chilled.