-
Notifications
You must be signed in to change notification settings - Fork 650
Layergroup Workflow
this document explains how CartoDB tiles are served using Multilayer API
(also called layergroup
API) https://github.com/CartoDB/Windshaft/wiki/Multilayer-API.
To explain it an example scenario is provided where two clients want to fetch tiles for the same layergroup, that's it, they want to render a map of a table called european_countries where the countries are rendered in white;
The layergroup as explained in Multilayer API
document should be like:
{
"version": "1.0.1",
"layers": [{
"type": "cartodb",
"options": {
"cartocss_version": "2.1.1",
"cartocss": "#layer { polygon-fill: #FFF; }",
"sql": "select * from european_countries"
}
}]
}
We take into account that is the first time that map is requested and all the caches are empty.
NOTE: the image is a fireworks png.
-
Client 1
builds the json document for the layergroup (described before) using cartodb.js. A POST to the tiler is done. Important things here:-
the url for the tiler is provided either by the vizjson or the user using the configuration parameters in cartodb.js. See a a basic example of how fetch that layergroup: https://github.com/CartoDB/cartodb.js/blob/develop/examples/modestmaps.html#L34
-
CORS is used for that POST, if CORS is not avalable, JSONP is used with a cache_buster (the current timestamp in the client)
-
2.- windshaft
reads the json document and goes to the SQL API
to get two things:
- which tables are involved in the queries inside layergroup document
- what is the last time (
last_updated
) those tables where modified
With the last_updated
and a hash(SQL + CartoCSS)
a layergroup_id
is generated:
{ layergroup_id: 8d6096d7ee7352bbae40074ca8dee352:12345678}
first part 8d6096d7ee7352bbae40074ca8dee352
is the hash, second part, after the ':' is the
``last_update`
The tiler code extracting "last_updated" is here: http://github.com/CartoDB/Windshaft-cartodb/blob/1.5.0/lib/cartodb/server_options.js#L295
NOTE: actually windshaft should request SQL API
though varnish
but is disabled due a bug.
Here's the code extracting https://github.com/CartoDB/Windshaft-cartodb/blob/1.5.0/lib/cartodb/server_options.js#L96
3.- windshaft
returns a json with what we call layergroup_id
and the last_updated
(not
used currently but useful for debugging pourposes)
4.- client 1
builds the tiles url using the CDN url provided in vizjson or by the user. If there
is no CDN url, default CDN url is used (https://github.com/CartoDB/cartodb.js/blob/develop/src/cartodb.js#L18). If no_cdn
option is used with cartodb.js the url generated is direct to the tiler like:
http://user.cartodb.com/tiles/layergroup/:z/:x/:y.png
and request the urls. There is a cache miss since the tiles are not cached:
- browser cache
- CDN
- varnish
5.- windshaft
fetch the data for that layergroup using the layergroup_id
(from redis
right
now), calls mapnik
with some XML (generated from SQL and CartoCSS in the layergroup), mapnik
fetch the data from postgres and returns the png and windshaft
serves the tile.
important things:
-
windshaft
set headers to cache the tile forever, there are the interesting headers:Cache-Control: public,max-age=31536000 Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
-
the tile is cached in
Varnish
and theCDN
-
important note: between
CDN
andvarnish
there is a nginx proxy (doc link needed) to route request from CDN to the correct user server
6.- client 2
request the layergroup (see 1)
7.- see 2
8.- see 3. Notice the layergroup is the same since the neither SQL or CartoCSS have changed. We supose the tables weren't changed (not inserts, updates, deletes, privacy changes, schema changes...)
9.- client 2
builds the url for the tiles, the same urls than client 1
fetches them
10.- the tiles are returned by CDN
because they were previously cached in 5
11.- the client 1
does an insert in the table (european_countries) so the last_updated
for that table is changed
12.- the same than 1
13.- same than 2 but in this case last_update
changes since the table was updated in 11
14.- same than 3 but the layergroup_id
is different, the last_updated
part has changed.
Notice the first part (the hash) didn't because the layergroup
is the same
and at this point the workflow returns to 4