command line - Why does `zsh` insert a `~` when I press the delete key?

I have set my zsh to use vim style for editing command lines.

However, when I use the Delete key, it inserts a ~ character.

Have I misconfigured my zsh?

How do I turn this off?

Do I need to go back to a different command editing mode for zsh?

I'm a bit new to zsh, and perhaps answers to this question could help other newcomers with the initial configuration.