#chinese

Question 1: Single choice

Chinese and Japanese are written without spaces between words. Is splitting the text on whitespace a good word-level tokenizer for these languages?

No: a whole sentence would become one token; use a word segmenter or character or subword units1
Yes: in these languages every character is a word, so the split already gives words2
Yes, as long as the text is lowercased and punctuation is set off by spaces first3
No, because word-level tokenization only works for languages with a Latin alphabet4