Place the yams and ginger in a pot, cover with cold water, and boil until the yams are very tender. Drain well and thoroughly mash. Whip in the milk, butter, nutmeg, salt and pepper. Transfer to a wide serving dish; sprinkle with the almonds and serve.
Note: To toast almonds, place in a single layer in baking pan and cook in a 300 degrees F oven until lightly browned, about 12 to 15 minutes.