Unable to navigate markdown files

When poly-markdown-mode is enabled, I'm unable to navigate through files containing source code blocks. I'm using evil-mode, and opening a new line, or entering insert mode makes the cursor to go all the way to a code block instead of where the cursor was initially (or going to the new line if pressing `o`)