feat(MDX): add optional hard breaks option

[eaglethrost]

🎫 Resolve CX-2095

Summary

• Adds an optional hardBreaks flag to the MDX compile() function that enables remark-breaks, converting soft line breaks (\n) into hard breaks (<br>)
• This would allow text like hello \n there to be split now
• Disabled by default to preserve existing behavior, different app components can opt in as needed

Context

The legacy rendering engine parsed newline characters as line breaks. Some customers want this behaviour in refactored, such as in API descriptions in the API explorer (see attached linear ticket), hence we should have an option in the MDX rendering to allow that. See also this linear comment: https://linear.app/readme-io/issue/CX-2095/line-break-characters-in-descriptions-arent-being-processed-correctly#comment-a6580c9e

If we agree to do this, we would need to also update the readme side to use the new flag for UI areas that need this behaviour.

[erunion]

this lgtm but would love a second approval from @rafegoldberg or @dannobytes

[rafegoldberg]

@eaglethrost will this option make it match Xish?

[rafegoldberg]

Not a huge fan of adding any extra flags or optional behavior unless it's absolutely one thousand percent necessary.

[eaglethrost]

@eaglethrost will this option make it match Xish?

Yes it will, and legacy as well

Not a huge fan of adding any extra flags or optional behavior unless it's absolutely one thousand percent necessary.

Yeah honestly I think we're doing this just because a customer wants the behaviour. I don't think we should make it the default as it will affects thousands of documents unexpectedly, hence why I made it an option. I think this is more of a business decision, on the technical side it works but just not the cleanest. Deferring decision to you @rafegoldberg 🙏

So if we were to go with this and address the ticket, we would need a monorepo side change to set on this new flag for RDMD components in the API Explorer Description Field only, to keep the blast radius low. Now MDX (left) would match legacy / Xish (right)