OpenStreetMap is blocking use of their tile servers whenever Referrer is not included

[petervanwylen]

Describe the bug
OpenStreetMap is blocking use of their tile servers whenever Referrer is not included. This is a change in the last few days. See https://wiki.openstreetmap.org/wiki/Blocked_tiles.

To Reproduce

Use folium.Map in any way with the default OSM tiles as the map.

Remaining to be understood
Can you send a Referrer header when running in an embedded way say in a Jupyter Notebook? How could their policy be satisfied when it's running locally or just being viewed in a notebook / potentially even a VS Code Notebook that doesn't have a URL (not Jupyter in a browser but Jupyter in an app).

[hansthen]

Hi Peter,

I was unable to reproduce the issue. Do you also have this issue when you open our documentation? https://python-visualization.github.io/folium/latest/index.html should contain some maps that use the OpenStreetMap base tiles.

[petervanwylen]

Hi Hans,

It's only happening for me in two situations. Also this only started a few days ago when they implemented the policy change at OSM.

Situation 1:

m = folium.Map(location=[lat,lng], zoom_start=11)
m.save("smoothed_overlay.html")

Then open up the html file in a browser such that the browser is reading the local filesystem directly / no hosting / not localhost in the location bar. In this case most all browsers will not send referrer information.

Situation 2:
If I create a map in VS Code's notebook app and then run display(m) it will also not have any referrer information. I think it might work in Jupyter Notebook (although I don't ever use that) because Jupyter runs in the browser and may will then send a localhost-like referrer.

-Peter

[hansthen]

Ah I see. That's annoying. We cannot add these headers from Folium, they are added by the browser (or any other html renderer). The best advice I can give for now is to use a different tile provider. Maybe you can host your own?

[petervanwylen]

Yeah I figured as much. Mostly wanted to give you a heads up in case there's another open tile provider you want to switch to by default for the purpose of ensuring broad compatibility out of the box instead of each developer having to figure out an API key if they happen to be working in an environment like VS Code Jupyter that requires it.

[breezy-codes]

I'm facing the same issue currently

[stanislav-a]

I am also facing the issue

[Julianiolo]

Ah I see. That's annoying. We cannot add these headers from Folium, they are added by the browser (or any other html renderer). The best advice I can give for now is to use a different tile provider. Maybe you can host your own?

Looking at https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/referrerPolicy, it looks like you can set a referrerpolicy attribute on the <img> element?

There is also this PR Leaflet/Leaflet#9883, it seems like it may solve this?

[hansthen]

@Julianiolo Not sure if the upstream PR will solve this particular issue. That PR is for sites where on site level the referrer header is blocked or otherwise not setup correctly for OSM. What people have reported in this thread is issues with OSM when there is no site where the map is served from (the map is rendered from a file or from inside VSCode). In that case there really is no valid referrer value, whatever the referrerPolicy is that is set on the image.

I am afraid we are out of luck. OSM requires a referrer header which should give the URL where the original map is served from (so they can track who is using OSM). For local files or use within VSCode there is no such URL, so the browser does not send it.

[reory]

I have successfully reproduced the 403r Access Blocked error in a local file:// environment and confirmed a working fix. While the browser defaults to a strict referrer policy for local files, leaflets TileLayer already supports a referrerPolicy option that satisfies OSM's new requirements.

A fix is by injecting "referrerPolicy: "no-referrer-when-downgrade"into theL.tileLayer` options, the browser is instructed to send the local file path as the referrer header. I verified that this immediately clears the 403 block on local browsers.

Proposed JS fix
Folium could resolve this by adding a referrerPolicy argument to the TileLayer and Map classes to pass this string to the leaflet template.

// Verified working JS fix:
var tile_layer = L.tileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png", { "referrerPolicy": "no-referrer-when-downgrade", // ... other options }).addTo(map);

I have a minimal reproduction of the html with the fix, if any one wants to inspect it further.

[pprianto]

i tried @reory solution and it worked when i set referer policy as strict origin
to generate a folium map in VSCode and save it to HTML

this works for my case in generating a small geographical scope (county-level)
haven't really tested for much wider coverage

import folium as fl

map = fl.Map(location=[_x_, _y_],
                tiles=None,
                zoom_start=7)
fl.TileLayer(
    tiles="https://tile.openstreetmap.org/{z}/{x}/{y}.png",
    attr="&copy; <a href='https://www.openstreetmap.org/copyright'>OpenStreetMap</a> contributors",
    referrer_policy="strict-origin",
).add_to(map)

map.save("test_layer.html")

[tonny62]

I'd tried @pprianto , however it doesn't work for my case.
So I have a quick fix for anyone working with Folium on a web app. Just add a js script to find the map object, inject the header, and redraw.

<script>
  // fix OSM map referrer policy
  var _osmFixInterval = setInterval(function() {
    var iframes = document.querySelectorAll('iframe');
    for (var i = 0; i < iframes.length; i++) {
      try {
        var win = iframes[i].contentWindow;
        var mapKey = Object.keys(win).find(function(key) {
          try {
            var val = win[key];
            return val && typeof val === 'object' && typeof val.eachLayer === 'function';
          } catch(e) { return false; }
        });
        if (mapKey) {
          clearInterval(_osmFixInterval);
          win[mapKey].eachLayer(function(layer) {
            if (layer instanceof win.L.TileLayer) {
              layer.options.referrerPolicy = 'strict-origin';
              layer.redraw();
            }
          });
          break;
        }
      } catch(e) {}
    }
  }, 300);
</script>

[reory]

@hansthen Is this bug now fixed please? Do you need any assistance to fix this?

[estefano-souza]

Is there any planned update for the folium.TileLayer function to solve this issue?

[kaushikbh99]

I am also facing the issue.

[neezr]

It looks like there is no way to provide a valid HTTP Referer from within a Jupyter Notebook and the OSM usage guidelines explicitly forbid using strict-origin as a policy to avoid sending a Referer.

A somewhat drastic fix for this could be to change the default tile provider to something else like CartoDB Positron. I like the design of CartoDB, but what is your opinion on this?