Why are prime numbers used as keys?

Could you elaborate on the use of prime numbers as keys in cryptography? Are they inherently more secure than other numbers? How do they contribute to the encryption process and ensure the confidentiality of data? Additionally, are there any specific challenges or limitations associated with using prime numbers as keys that practitioners should be aware of? Understanding the intricacies behind this choice is crucial for those working in the field of cryptography and security.