Preheat your oven to 350°F (175°C). Grease a standard 9x5 inch loaf pan generously with butter or baking spray. For easy removal, you can also line the bottom with parchment paper, letting the ends hang over the sides to create "handles."
In a large mixing bowl, whisk together the all-purpose flour, baking powder, baking soda, fine sea salt, ground cinnamon, and ground nutmeg. Set aside.
In a separate large mixing bowl, using an electric mixer (stand or hand), cream the softened unsalted butter and granulated sugar together until light and fluffy, about 2-3 minutes. This is a critical step for texture!
Beat in the large eggs one at a time, mixing well after each addition until fully incorporated. Scrape down the sides of the bowl as needed. Stir in the vanilla extract.
Add about half of the dry ingredients to the wet ingredients and mix on low speed just until combined. Add half of the buttermilk and mix until just incorporated. Repeat with the remaining dry ingredients and buttermilk, mixing until the batter is just combined and no dry streaks remain. Do not overmix! Overmixing develops gluten and can lead to a tough loaf.
Gently fold in the diced apples using a rubber spatula.
In a small bowl, whisk together the 1/4 cup all-purpose flour, packed light brown sugar, and 1 tablespoon ground cinnamon. Pour the melted unsalted butter over the top and use a fork to mix it into the dry ingredients until the mixture resembles coarse crumbs.
Pour about half of the loaf batter into the prepared loaf pan and spread it evenly. Sprinkle about half of the cinnamon swirl mixture evenly over the batter. Dollop the remaining loaf batter over the cinnamon layer, trying to cover the swirl mixture. Gently spread it out, being careful not to disturb the swirl layer too much. Sprinkle the remaining cinnamon swirl mixture evenly over the top layer of batter.
Use a knife or an offset spatula to gently swirl the cinnamon mixture into the batter. Insert the knife into the batter about 1 inch deep and make serpentine motions or figure-eights through the batter a few times. Don't over-swirl, or you'll lose the distinct layers!
Place the loaf pan in the preheated oven. Bake for 50-60 minutes, or until a wooden skewer or toothpick inserted into the center of the loaf comes out clean or with just a few moist crumbs attached. The top should be golden brown and the loaf should spring back slightly when gently pressed. If the top browns too quickly, you can loosely tent it with foil.
Let the loaf cool in the pan on a wire rack for 15-20 minutes before attempting to remove it. Then, carefully invert the loaf onto the wire rack to cool completely before slicing. Cooling completely ensures the texture is perfect and the swirl doesn't collapse.
While the loaf cools, whisk together the powdered sugar, vanilla extract, and 1-2 tablespoons of milk or cream until smooth and pourable. Once the loaf is completely cool, drizzle the glaze over the top. Let the glaze set before slicing.