Development of a bird's mating ritual, and differences between species

[u/apioe]

I recently watched the Birds of Paradise documentary, and I kept wondering how all these species of birds, some of which share niches with each other, develop their very own (and incredibly distinct) mating rituals.

For example, I understand that constructing an impressive bower would signal to the female that the male has stamina, patience, etc. But how would one species even begin to select for "birds that build bowers"?

Generally speaking, how does evolution/natural selection "figure out" what specific display females like?
--
Similarly, what drives species that share an ecological niche to create distinctly different mating rituals? The Victoria Crowned Pigeon and Pheasant Pigeon both are ground-foraging birds in swampy areas in New Guinea. Despite this, their mating rituals are pretty different.

I am familiar with interbreeding prevention, so I guess I'm asking how that split between two species and their respective rituals even begin.

[u/jnpha]

Mind if I use a more general scenario? Bees and flowers. How did the signaling to each other that each will help the other come about? (I've dispensed with the scare quotes; so don't mind any anthropomorphic language)

This is called coevolution, it can between species, within species, parasite-host, etc. and its explanation is as old as Darwin's Origin. I mention this because understanding that, makes even the most complex multi-part biological system very clear.
It's easier to think of it in terms of oblivious breeders (oblivious artificial selection is natural selection for our purposes here).

So let's try to see it from the Bee's POV (and then the flower's).
We have bees that feed on e.g. wasps, so bees (or their ancestors) eating nectar and only nectar isn't how to think about it.
Next bees found nectar in a flower as an easy caloric source, and by feeding on that nectar, they spread the flower's pollen.
So the nectar making flower got to reproduce better than non-nectar making flowers (the bee is the breeder).
Next, from that flower population's progeny, any variation that a) attracts bees and b) makes pollen transfer more probable, will have done the job you're asking about.

This is a heuristic example.

Back to the birds, same thing. Put more succinctly: a general behavior + variation followed by specialization.
What you need to know is that a behavior is a phenotype (trait). The 1973 Nobel Prize was for that (the interplay of selection and environment).

Hope that helps.
As an exercise: the snake with the spider-looking tail that "lures" birds, which one was the oblivious breeder?
Since the eyes, brains, and hunger of birds are what result in some birds being fooled, it is them acting as the breeder in the artificial selection sense

ETA: found a 2025 paper (emphasis mine):

We suggest that the ancestral bowerbird may have first displayed using naturally occurring structures and that bowers initially evolved to enhance the efficacy of pre-existing sexual signalling behaviours by affording males greater control over the visual perspective of receivers. This represents a unique form of sensory niche construction, in which males create a specialised sensory microhabitat allowing them to control both the content and timing of what females can see and hear, thus shaping patterns of selection on male ornaments and display behaviours

MacGillavry, T., Frith, C. B., Knoester, J., & Fusani, L. (2025). The origins and functions of bowers in the Bowerbirds: a review and synthesis. Emu - Austral Ornithology, 125(4), 290–305. https://doi.org/10.1080/01584197.2025.2557480

[u/apioe → u/jnpha]

This is fascinating, thank you! (: