Paper #5: Why Classification Networks Shrink the Image
Classification networks shrink their feature maps on purpose: it lets later layers see wide areas cheaply. For one label that costs nothing. For a label at every pixel, it throws away the detail the answer needs.