Talks on forming a coalition government in Germany have collapsed, leaving Angela Merkel facing her biggest challenge in 12 years as chancellor.