What happens if a swap fails?

So, let's say a swap doesn't go as planned. What kind of implications does that have? Are there any penalties or consequences? How do the parties involved handle the situation? And what about the funds or assets that were supposed to be exchanged, what happens to those? I'm sure there's a protocol in place to deal with these kinds of situations, but I'd like to hear the specifics.