Explore the option to create Monarch-compatible renderings from input graphics

[gvzdv]

Use the image-text-image model to create renderings that can be displayed on a Monarch (or another pin array tablet).

Basic approach: use the original graphic and prompting.
Advanced approach: fine-tune (LoRA/QLoRA?) a model based on the pre-generated renderings.

[gvzdv]

Initial experimentation results

I used the gemini-2.5-flash-image-preview model (aka nano-banana).

Regular graphic/photo:

Create a rendering of this graphic that can be displayed on a pin array tablet. 
The tablet has only two settings of pins: raised and lowered. 
Therefore, the rendering should be black and white with black color used for the details you need to display, and white color used for background/empty space. 
The tablet has a low resolution, so your focus should be on major details.

Multistage diagram:

Create a rendering of this graphic that can be displayed on a pin array tablet. 
The tablet has only two settings of pins: raised and lowered. 
Therefore, the rendering should be black and white with black color used for the details you need to display, and white color used for background/empty space. 
The tablet has a low resolution, so your focus should be on major details.

In the diagram, make sure to show the shapes of the phases and the links/dependencies between them.
Omit background, unnecessary elements, and labels.

Political map:

Create a rendering of this graphic that can be displayed on a pin array tablet.
The tablet has only two settings of pins: raised and lowered.
Therefore, the rendering should be black and white, with black color used for the details you need to display (raised pins), and white color used for background/empty space (lowered pins).
The tablet has a low resolution, so your focus should be on major details.
In the map, show with black lines outlines of the continent, as well as borders between countries. Fill with black color the following country: Peru.
Omit background, unnecessary elements, and labels.

Attempt 1:

Attempt 2:

Fix the following identified issues:
1. Labels present in the graphic.
2. Wrong country filled with color. Country to fill from the original request: Peru.

Attempt 3:

Fix the following identified issues:
1. No country filled with color. Country to fill from the original request: Peru.

Thoughts and observations

1. The quality depends on the graphic type the model is working with (a multistage diagram worked well; a political map did not).
2. There is room for improvement if we keep experimenting with prompts, hyperparameters, and models.
3. The graphics are still hi-res; we need to check how they will be displayed on the actual tablet.
4. For this kind of task, we need to introduce a feedback loop, with a vision model checking the output and either approving it or returning comments to the image generation model (an agent!). It will increase the latency, but given the mistakes we've already observed, we cannot provide the results to the users without prior verification, either by a human or by a machine.

@jeffbl please take a look at these findings when you have a chance. Can this approach be beneficial for the IMAGE stack, or is it worthy of a research paper?

[jeffbl]

Research paper: uncertain.
IMAGE: Probably. If nothing else, the segmentations are super nice for the diagram(s), and would have material that could be used for zoom in/out experiences, I expect.

Remember that the IMAGE Monarch client takes SVG and does its own rasterization, so could probably try some of these to see how they pan out. Text, etc. obviously useless, and would need to be converted to audio labels. (I see you're prompting for no labels)

If specify the display only has 96x40 pixels, does that make it tone down the detail?