What is tokenization in machine learning?

Could you elaborate on the concept of tokenization in the realm of machine learning? As a key component in natural language processing, I'm curious to understand how it transforms text data into a format that machines can comprehend. Specifically, I'd like to know about the various techniques involved, like word tokenization, sentence tokenization, and how they facilitate further analysis, such as in sentiment analysis or text classification tasks. Additionally, I'm interested in any real-world applications where tokenization plays a pivotal role in improving the performance of machine learning models.