nlp - Special characters and spaces in BertTokenizer

Question

I have a problem, I am using BertTokenizer to encode and decode some text, but when de text has special characters, the tokenizer add some white spaces. I need the encoded sentence to train a transformers model and then decode similar sentences to see the predictions. Ex:

from transformers import BertTokenizer

pretrained_model_name = "bert-base-uncased"

tokenizer = BertTokenizer.from_pretrained(pretrained_model_name)

text = "didn't not_empty. (na?ve) t-shirt"

print(text)
print(tokenizer.decode(tokenizer.encode(text, add_special_tokens=False)))

And response:

1 >>> didn't not_empty. (na?ve) t-shirt
2 >>> didn't not _ empty. ( naive ) t - shirt

How can i fix it? thanks

Answer 1

等待大神答复