I looked into my code that the view is being refreshed multiple times based on the multiple calls from the view model class. I clearly don't have an idea how the object state is working in the Combine and SwiftUI, this is the root caused of this issue.
Thank you @pawello2222 for posting me to the right direction.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…