Probability is the branch of mathematics concerned with describing the relative likelihood of things. Probabilistic methods have proven fruitful in characterizing patterns in natural language. Here are some starting points for teaching yourself the math.

• There are many good probability textbooks, but my favorite is Introduction to Probability Theory and Statistical Inference.
• The two standard textbooks on probability and natural language are Foundations of Statistical Natural Language Processing and Speech and Language Processing.
• Elements of Information Theory is an excellent overview of that useful field.

• The "Probability Tutorial Using Dice" explains the basic concepts of probability using a familiar real world example.
• The "Maximum Mutual Information Criterion Tutorial" explains how to use mutual information in feature reduction. A sample script shows how this works.

• John Goldsmith has an excellent tutorial on probability for linguists.
• Adam Berger has a great collection of links about maximum entropy methods.