#abbreviations

A regex tokenizer is meant to keep abbreviations such as "U.S.A." as one token:

import re
t = "The U.S.A. sent 2.5 tons."
print(len(re.findall(r"\w+|(?:[A-Z]\.)+|[^\w\s]", t)))

What does it print?