Urbanization_v5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0f2016b3",
   "metadata": {},
   "source": [
    "# Spatiotemporal Trends in Urbanization: Cameroon\n",
    "*Using yearly estimates (2000-2015) of population, built-area, and economic indicators to track city-by-city growth and change over time.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85fd14a1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "903293ba",
   "metadata": {},
   "source": [
    "### Research questions \n",
    "\n",
    "#### 1. How has the size of Settlement X changed over time? \n",
    "\n",
    "- Population size \n",
    "\n",
    "- Geographical extents \n",
    "\n",
    "- Population density \n",
    "\n",
    "#### 2. In what year did Settlement X become a new urban class?  \n",
    "\n",
    "- From semi-dense to high-density city \n",
    "\n",
    "- Small settlement area to built-up area \n",
    "\n",
    "- When a hamlet area or small settlement area first appeared\n",
    "\n",
    "#### 3. Is there a discernable pattern between the spatio-temporal distribution of economic density and population density? \n",
    "\n",
    "#### 4. How much of urban space attributable to City X is outside of the administrative limits of the city? \n",
    "\n",
    "- When did this fragment(s) appear? \n",
    "\n",
    "- Which district/municipality/authority has purview over the fragment(s)? \n",
    "\n",
    "#### 5. For the questions above, how does the answer change based on different understandings of urban limits? \n",
    "\n",
    "- Scenario A: where \"city\" is delimited by an official administrative boundary \n",
    "\n",
    "- Scenario B: where \"city\" includes all contiguous (and near-contiguous) built up area \n",
    "\n",
    "#### 6. Subnational and inter-national comparisons. Examples: \n",
    "\n",
    "- Compare the rates (pop, build-up, economic…) of the fastest growing settlement of each ADM1 region. \n",
    "\n",
    "- Which African metropoles experience the most vs. the least fragmentation? Is there a confluence between amount of urban fragmentation and rate of densification? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e55d1a9",
   "metadata": {},
   "source": [
    "### Datasets\n",
    "1. Most up-to-date administrative boundaries: **ADM3.**\n",
    "2. Built-up area, yearly: **World Settlement Footprint Evolution.** Resolution: 30m.\n",
    "3. Settlement types: **GRID3 settlement extents.** Captured between 2009-2019.\n",
    "4. Population, yearly: **WorldPop.** UN-adjusted, unconstrained. Resolution: 100m.\n",
    "5. Nighttime lights, yearly: **Harmonization of DMSP and VIIRS.** Resolution: 1km.\n",
    "6. City names: **UCDB, Africapolis, and GeoNames.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cc5dafb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85a61167",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9276b447",
   "metadata": {},
   "source": [
    "## 1. PREPARE WORKSPACE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0190f8",
   "metadata": {},
   "source": [
    "### 1.1 Off-script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9093fd6",
   "metadata": {},
   "source": [
    "##### Off-script: Create folders in working directory.\n",
    "> *ADM\n",
    "<br>Buildup\n",
    "<br>PlaceName\n",
    "<br>Population\n",
    "<br>Settlement\n",
    "<br>NTL*\n",
    "\n",
    "##### Off-script: Download datasets (as shapefile, GeoJSON, or tif where possible) and place or extract into corresponding folder:\n",
    "- ADM: *Sourced internally.*\n",
    "- Buildup: https://download.geoservice.dlr.de/WSF_EVO/files/\n",
    "- PlaceName: \n",
    "    - GeoNames: (file: cities500.zip) https://download.geonames.org/export/dump/\n",
    "    - Africapolis: https://africapolis.org/en/data\n",
    "    - Urban Centres Database: https://ghsl.jrc.ec.europa.eu/ghs_stat_ucdb2015mt_r2019a.php\n",
    "- Population: https://hub.worldpop.org/geodata/listing?id=69\n",
    "- Settlement: https://data.grid3.org/datasets/GRID3::grid3-cameroon-settlement-extents-version-01-01-/explore\n",
    "- NTL: https://figshare.com/articles/dataset/Harmonization_of_DMSP_and_VIIRS_nighttime_light_data_from_1992-2018_at_the_global_scale/9828827/2\n",
    "\n",
    "##### Other off-script:\n",
    "- Convert GeoNames from .txt file to shape (delimiter = tab, header rows = 0) and rename fields.\n",
    "- If necessary, mosaic WSFE rasters that cover the area of interest to create a single file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2a2548",
   "metadata": {},
   "source": [
    "### 1.2 Load all packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c42795f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: Most but not all of these packages were used in final form. \n",
    "\n",
    "import os, sys, glob, re, time\n",
    "from os.path import exists\n",
    "from functools import reduce\n",
    "\n",
    "import geopandas as gpd \n",
    "import pandas as pd\n",
    "from shapely.geometry import Point, LineString, Polygon, shape, MultiPoint\n",
    "from shapely.ops import cascaded_union\n",
    "from shapely.validation import make_valid, explain_validity\n",
    "import shapely.wkt\n",
    "import scipy\n",
    "\n",
    "#from xrspatial import zonal_stats \n",
    "#import xarray as xr \n",
    "import numpy as np \n",
    "import fiona, rioxarray\n",
    "import rasterio\n",
    "from rasterio.plot import show\n",
    "from rasterio import features\n",
    "from rasterio.features import shapes\n",
    "from rasterio import mask\n",
    "from osgeo import gdal, osr, ogr, gdal_array\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be5f9d7",
   "metadata": {},
   "source": [
    "### 1.3 Set workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a20dfb25",
   "metadata": {},
   "outputs": [],
   "source": [
    "ProjectFolder = os.getcwd()\n",
    "ResultsFolder = os.path.join(ProjectFolder, 'Results')\n",
    "print(ProjectFolder)\n",
    "print(ResultsFolder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5537c1ce",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c08550",
   "metadata": {},
   "source": [
    "## 2. PREPARE BUILDUP, SETTLEMENT, AND ADMIN DATASETS\n",
    "Projection for all datasets: Africa Albers Equal Area Conic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d194cf",
   "metadata": {},
   "source": [
    "### 2.1 WSFE: Check contents and change NoData value as necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1fa8859",
   "metadata": {},
   "source": [
    "##### Off-script: Run this block in QGIS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a191c49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # OPEN QGIS FOR THIS PORTION. CODE DOCUMENTED HERE.\n",
    "# Change NoData value to zero, as this won't interfere with a possible value of 99999 in GRID3 and ADM.\n",
    "# Then make sure there are no values above 2015 (such as 99999) or below 1985 in the dataset by reclassifying them as NoData.\n",
    "# Was having trouble with rasterio & gdal here, so moved to QGIS.\n",
    "\n",
    "# processing.run(\"native:reclassifybytable\", {'INPUT_RASTER':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Buildup/WSFE_CMN.tif','RASTER_BAND':1,'TABLE':['2016','','0','','1984','0'],'NO_DATA':0,'RANGE_BOUNDARIES':0,'NODATA_FOR_MISSING':False,'DATA_TYPE':5,'OUTPUT':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Buildup/WSFE.tif'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e591ea5f",
   "metadata": {},
   "source": [
    "### 2.2 Prepare raster locations for GRID3 and Admin areas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7946e25",
   "metadata": {},
   "outputs": [],
   "source": [
    "ADM_vec = gpd.read_file(glob.glob('ADM/*.shp')[0])[['geometry']].to_crs(\"ESRI:102022\") # This glob() function pulls the first file ([0]) in the ADM folder which ended in '.shp'\n",
    "GRID3_vec = gpd.read_file(glob.glob('Settlement/*.shp')[0])[['type','geometry']].to_crs(\"ESRI:102022\")\n",
    "ADM_vec['ADM_ID'] = range(0,len(ADM_vec))\n",
    "GRID3_vec['G3_ID'] = range(0,len(GRID3_vec))\n",
    "ADM_vec.to_file(driver='GPKG', filename=r'ADM/ADM_warp.gpkg', layer='ADM')\n",
    "GRID3_vec.to_file(driver='GPKG', filename=r'Settlement/Settlement_warp.gpkg', layer='GRID3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d44280f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ADM_vec = gpd.read_file(r'ADM/ADM_warp.gpkg', layer='ADM')\n",
    "GRID3_vec = gpd.read_file(r'Settlement/Settlement_warp.gpkg', layer='GRID3')\n",
    "\n",
    "print(ADM_vec.info(), \"\\n\\n\", \n",
    "      ADM_vec.sample(5),\n",
    "      ADM_vec.crs, \"\\n\\n\", \n",
    "      len(str(ADM_vec['ADM_ID'].max()))) # We need to know how many digits need to be allocated to each dataset in the \"join\" serial.\n",
    "print(GRID3_vec.info(), \"\\n\\n\",\n",
    "      GRID3_vec.sample(5),\n",
    "      GRID3_vec.crs, \"\\n\\n\", \n",
    "      len(str(GRID3_vec['G3_ID'].max())))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b44be4f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440aa2a2",
   "metadata": {},
   "source": [
    "## 3. WSFE AND ADM; GRID3 AND ADM\n",
    "RASTERIZE: Bring ADM and GRID3 into raster space.\n",
    "\n",
    "RASTER MATH: \"Join\" ADM ID onto GRID3 and onto WSFE by creating unique concatenation string.\n",
    "\n",
    "VECTORIZE: Bring joined data into vector space.\n",
    "\n",
    "VECTOR MATH: Split unique ID from raster math step into separate columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca195d70",
   "metadata": {},
   "source": [
    "### 3.1 Reproject WSFE to project CRS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71ebd2eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "WSFE_in = glob.glob('Buildup/*.tif')[0]\n",
    "WSFE_warp = './Buildup/WSFE_warp.tif'\n",
    "ProjCRS = gdal.WarpOptions(dstSRS='ESRI:102022')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0058146f",
   "metadata": {},
   "outputs": [],
   "source": [
    "Warp = gdal.Warp(WSFE_warp, # Where to store the warped raster\n",
    "                 WSFE_in, # Which raster to warp\n",
    "                 format='GTiff', \n",
    "                 options=ProjCRS) # Reproject to Africa Albers Equal Area Conic\n",
    "Warp = None\n",
    "print('Reprojected dataset. %s' % time.ctime())\n",
    "\n",
    "try:  \n",
    "    os.remove(os.path.join(ProjectFolder, WSFE_in))\n",
    "except OSError:\n",
    "    pass\n",
    "print('Removed (or skipped if error) intermediate file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d266b5c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "WSFE = rasterio.open(os.path.join(ProjectFolder, \"Buildup\", os.listdir('Buildup/')[0]))\n",
    "print(WSFE) # WSFE values are all 4 digits long (1985-2015)\n",
    "print(dir(WSFE))\n",
    "print(WSFE.crs)\n",
    "print(WSFE.dtypes)\n",
    "NoDataValue = WSFE.nodatavals\n",
    "print(NoDataValue)\n",
    "print(WSFE.read(1).min(), WSFE.read(1).mean(), np.median(WSFE.read(1)), WSFE.read(1).max())\n",
    "\n",
    "# If NoDataValue != 0, change to 0. (See step 2.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9663bd1",
   "metadata": {},
   "source": [
    "### 3.2 Rasterize admin areas and GRID3 using WSFE specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af504c4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy and update the metadata from WSFE for the output\n",
    "meta = WSFE.meta.copy()\n",
    "meta.update(compress='lzw')\n",
    "WSFE.meta\n",
    "\n",
    "ADM_out = './ADM/ADM_rasterized.tif'\n",
    "GRID3_out = './Settlement/GRID3_rasterized.tif'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f778fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Rasterizing dataset. %s\" % time.ctime())\n",
    "with rasterio.open(ADM_out, 'w+', **meta) as out:\n",
    "    out_arr = out.read(1)\n",
    "\n",
    "    # this is where we create a generator of geom, value pairs to use in rasterizing\n",
    "    shapes = ((geom,value) for geom, value in zip(ADM_vec.geometry, ADM_vec.ADM_ID))\n",
    "\n",
    "    burned = features.rasterize(shapes=shapes, fill=0, out=out_arr, transform=out.transform)\n",
    "    out.write_band(1, burned)\n",
    "out = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a368d2e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Rasterizing dataset. %s\" % time.ctime())\n",
    "with rasterio.open(GRID3_out, 'w+', **meta) as out:\n",
    "    out_arr = out.read(1)\n",
    "\n",
    "    # this is where we create a generator of geom, value pairs to use in rasterizing\n",
    "    shapes = ((geom,value) for geom, value in zip(GRID3_vec.geometry, GRID3_vec.G3_ID))\n",
    "\n",
    "    burned = features.rasterize(shapes=shapes, fill=0, out=out_arr, transform=out.transform)\n",
    "    out.write_band(1, burned)\n",
    "out = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91fbc740",
   "metadata": {},
   "source": [
    "*Validation: Check the dimensions, type, and basic stats of the three datasets. All should be the same dimension and NoData value.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a4a7d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "RastersList = [gdal.Open(r\"ADM/ADM_rasterized.tif\"), \n",
    "               gdal.Open(r\"Settlement/GRID3_rasterized.tif\"),\n",
    "               gdal.Open(os.path.join(ProjectFolder, \"Buildup\", os.listdir('Buildup/')[0]))]\n",
    "\n",
    "for item in RastersList:\n",
    "    print(gdal.GetDataTypeName(item.GetRasterBand(1).DataType), \n",
    "          item.GetRasterBand(1).GetNoDataValue(),\n",
    "         \"\\n\\n\")\n",
    "\n",
    "RastersList = None\n",
    "\n",
    "RastersList = [rasterio.open(r\"ADM/ADM_rasterized.tif\"), \n",
    "               rasterio.open(r\"Settlement/GRID3_rasterized.tif\"), \n",
    "               rasterio.open(os.path.join(ProjectFolder, \"Buildup\", os.listdir('Buildup/')[0]))]\n",
    "\n",
    "for item in RastersList:\n",
    "    print(item.name, \"\\nBands= \", item.count, \"\\nWxH= \", item.width, \"x\", item.height, \"\\n\\n\")\n",
    "\n",
    "stats = []\n",
    "for item in RastersList:\n",
    "    band = item.read(1)\n",
    "    stats.append({\n",
    "        'raster': item.name,\n",
    "        'min': band.min(),\n",
    "        'mean': band.mean(),\n",
    "        'median': np.median(band),\n",
    "        'max': band.max()})\n",
    "\n",
    "# Show stats for each channel\n",
    "print(\"\\n\", stats)\n",
    "\n",
    "RastersList = None\n",
    "band = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406ceaa3",
   "metadata": {},
   "source": [
    "### 3.2 Raster math to \"join\" admin to GRID3 and to WSFE.\n",
    "Processing is more rapid when \"joining,\" i.e. creating serial codes out of two datasets, in raster rather than vector space.\n",
    "Here, we are concatenating the ID fields of the two datasets to create a serial number that we can then split in vector space later to create two ID fields."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e46a2406",
   "metadata": {},
   "source": [
    "*Adding together the values to create join IDs. This is in effect a concatenation of their ID strings, by way of summation. The number of zeros in the calc multiplication corresponds with number of digits of the maximum value in the \"B\" dataset. (e.g. Chad ADM codes go up 4 digits, so it's calc=(A*10000)+B).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e831cd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # OPEN TERMINAL FOR THIS PORTION. CODE DOCUMENTED HERE.\n",
    "\n",
    "# Gdal_calc.py # To see info.\n",
    "\n",
    "# gdal_calc.py -A C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\Settlement\\GRID3_rasterized.tif -B  C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\ADM\\ADM_rasterized.tif --outfile=C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\Settlement\\GRID3_ADM.tif --overwrite --calc=\"(A*1000)+B\"\n",
    "# gdal_calc.py -A C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\Buildup\\WSFE_warp.tif -B  C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\ADM\\ADM_rasterized.tif --outfile=C:\\Users\\grace\\GIS\\povertyequity\\urban_growth\\Cameroon\\Buildup\\WSFE_ADM.tif --overwrite --calc=\"(A*1000)+B\"\n",
    "\n",
    "# # END TERMINAL-ONLY ASPECT. RETURN HERE FOR NEXT STEPS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b8df5020",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Buildup/WSFE_ADM.tif \n",
      "Bands=  1 \n",
      "WxH=  26387 x 43889 \n",
      "\n",
      "\n",
      "Settlement/GRID3_ADM.tif \n",
      "Bands=  1 \n",
      "WxH=  26387 x 43889 \n",
      "\n",
      "\n",
      "\n",
      " [{'raster': 'Buildup/WSFE_ADM.tif', 'min': 1985001, 'mean': 4285775835.831103, 'median': 4294967293.0, 'max': 4294967293}, {'raster': 'Settlement/GRID3_ADM.tif', 'min': 1323, 'mean': 4243096252.2932878, 'median': 4294967293.0, 'max': 4294967293}]\n"
     ]
    }
   ],
   "source": [
    "# Validation: check the basic statistics of the resulting datasets.\n",
    "RastersList = [rasterio.open(r\"Buildup/WSFE_ADM.tif\"), \n",
    "               rasterio.open(r\"Settlement/GRID3_ADM.tif\")]\n",
    "for item in RastersList:\n",
    "    print(item.name, \"\\nBands= \", item.count, \"\\nWxH= \", item.width, \"x\", item.height, \"\\n\\n\")\n",
    "    \n",
    "stats = []\n",
    "for item in RastersList:\n",
    "    band = item.read(1)\n",
    "    stats.append({\n",
    "        'raster': item.name,\n",
    "        'min': band.min(),\n",
    "        'mean': band.mean(),\n",
    "        'median': np.median(band),\n",
    "        'max': band.max()})\n",
    "\n",
    "# Show stats for each channel\n",
    "print(\"\\n\", stats)\n",
    "\n",
    "RastersList = None\n",
    "band = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ea3639",
   "metadata": {},
   "source": [
    "### 3.3 Vectorize \"joined\" layers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b5d9d4",
   "metadata": {},
   "source": [
    "##### Off-script: Run this block in QGIS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85dfc892",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPEN QGIS FOR THIS PORTION. CODE DOCUMENTED HERE.\n",
    "\n",
    "# Due to dtype errors with both gdal and rasterio here, I decided to run the raster to polygon function in QGIS instead.\n",
    "# It is possible to run QGIS functions within a Jupyter Notebook, but I ran it within the GUI. Arc or R are other options.\n",
    "# Command line code here.\n",
    "\n",
    "# processing.run(\"gdal:polygonize\", {'INPUT':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Settlement/GRID3_ADM.tif','BAND':1,'FIELD':'gridcode','EIGHT_CONNECTEDNESS':False,'EXTRA':'','OUTPUT':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Settlement/GRID3_ADM.shp'})\n",
    "# processing.run(\"gdal:polygonize\", {'INPUT':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Buildup/WSFE_ADM.tif','BAND':1,'FIELD':'gridcode','EIGHT_CONNECTEDNESS':False,'EXTRA':'','OUTPUT':'C:/Users/grace/GIS/povertyequity/urban_growth/Cameroon/Buildup/WSFE_ADM.shp'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03937b47",
   "metadata": {},
   "source": [
    "### 3.4 Vector math to split raster strings into admin area, GRID3, and WSFE year assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb900b13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 208406 entries, 0 to 208405\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   gridcode  208406 non-null  int64   \n",
      " 1   geometry  208406 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 3.2 MB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 538320 entries, 0 to 538319\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   gridcode  538320 non-null  int64   \n",
      " 1   geometry  538320 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 8.2 MB\n",
      "None \n",
      "\n",
      "          gridcode                                           geometry\n",
      "138125   53824305  POLYGON ((-1552758.662 645500.557, -1552728.05...\n",
      "121221   60928157  POLYGON ((-1549820.627 698354.567, -1549759.41...\n",
      "12979   180559232  POLYGON ((-1157134.000 1262854.984, -1157042.1...\n",
      "96208    98680191  POLYGON ((-1478053.022 757176.459, -1477961.20...\n",
      "207277    2217081  POLYGON ((-1426055.939 281582.180, -1426025.33...\n",
      "181091   32417106  POLYGON ((-1338435.191 475308.807, -1338312.77...\n",
      "168471  196275173  POLYGON ((-1665383.304 518706.020, -1665291.49...\n",
      "52642   140710206  POLYGON ((-1052558.346 1021691.347, -1052466.5...\n",
      "26320   174266251  POLYGON ((-1003713.528 1198187.628, -1003652.3...\n",
      "75946   128405209  POLYGON ((-1181831.849 819273.035, -1181740.03... \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " None \n",
      "\n",
      "         gridcode                                           geometry\n",
      "78669    1998195  POLYGON ((-1355787.955 823067.996, -1355757.35...\n",
      "239604   1998139  POLYGON ((-1492988.029 647734.687, -1492957.42...\n",
      "25594    1990241  POLYGON ((-1117531.747 1244216.830, -1117501.1...\n",
      "354021   1998122  POLYGON ((-1594533.835 537925.660, -1594503.23...\n",
      "228254   1995283  POLYGON ((-1549698.209 651713.275, -1549667.60...\n",
      "259034   2007282  POLYGON ((-1558145.058 642195.268, -1558114.45...\n",
      "159720   2007161  POLYGON ((-1575528.426 678492.232, -1575436.61...\n",
      "310692   2001290  POLYGON ((-1528030.207 603296.920, -1527999.60...\n",
      "185233   2011269  POLYGON ((-1536660.683 667933.672, -1536630.07...\n",
      "316257   2011171  POLYGON ((-1578833.715 597267.829, -1578803.11... \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]]\n"
     ]
    }
   ],
   "source": [
    "# Load newly created vectorized datasets.\n",
    "GRID3_ADM = gpd.read_file(r\"Settlement/GRID3_ADM.shp\")\n",
    "WSFE_ADM = gpd.read_file(r\"Buildup/WSFE_ADM.shp\")\n",
    "print(GRID3_ADM.info(), \"\\n\\n\", GRID3_ADM.sample(10), \"\\n\\n\", GRID3_ADM.crs, \"\\n\\n\", \n",
    "      WSFE_ADM.info(), \"\\n\\n\", WSFE_ADM.sample(10), \"\\n\\n\", WSFE_ADM.crs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3085bb9f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201820297 2015328\n"
     ]
    }
   ],
   "source": [
    "print(GRID3_ADM['gridcode'].max(), WSFE_ADM['gridcode'].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8f41b97c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         gridcode                                           geometry  \\\n",
      "177325   20280274  POLYGON ((-1428228.860 487917.870, -1428137.04...   \n",
      "180658   20038276  POLYGON ((-1422628.232 476686.011, -1422597.62...   \n",
      "31273   161716226  POLYGON ((-1177914.470 1183925.921, -1177822.6...   \n",
      "5626    191969256  POLYGON ((-1045396.888 1301998.168, -1045274.4...   \n",
      "66687   119296197  POLYGON ((-1288121.356 855631.208, -1288060.14...   \n",
      "127234   56247156  POLYGON ((-1542291.915 686541.221, -1542077.68...   \n",
      "39970   157432226  POLYGON ((-1183453.889 1149863.087, -1183392.6...   \n",
      "168502   22520271  POLYGON ((-1405244.863 518889.647, -1405183.65...   \n",
      "127353   59517156  POLYGON ((-1531825.168 678369.814, -1531733.35...   \n",
      "156129   44406072  POLYGON ((-1081938.688 566785.725, -1081846.87...   \n",
      "\n",
      "       gridstring  Sett_ID  ADM_ID  \n",
      "177325  020280274    20280     274  \n",
      "180658  020038276    20038     276  \n",
      "31273   161716226   161716     226  \n",
      "5626    191969256   191969     256  \n",
      "66687   119296197   119296     197  \n",
      "127234  056247156    56247     156  \n",
      "39970   157432226   157432     226  \n",
      "168502  022520271    22520     271  \n",
      "127353  059517156    59517     156  \n",
      "156129  044406072    44406      72           gridcode                                           geometry  \\\n",
      "78136    1999195  POLYGON ((-1353951.684 833565.347, -1353921.07...   \n",
      "164784   1988156  POLYGON ((-1532804.513 676472.333, -1532773.90...   \n",
      "187595   1998269  POLYGON ((-1539598.717 666984.932, -1539568.11...   \n",
      "203400   1999325  POLYGON ((-1474105.039 660864.027, -1474074.43...   \n",
      "180701   1998269  POLYGON ((-1539843.553 669739.339, -1539812.94...   \n",
      "341092   2015129  POLYGON ((-1399001.541 546831.576, -1398970.93...   \n",
      "321940   2005285  POLYGON ((-1502383.618 585974.761, -1502322.40...   \n",
      "27566    1998241  POLYGON ((-1121051.268 1239595.547, -1121020.6...   \n",
      "269506   2001282  POLYGON ((-1554656.142 638430.912, -1554625.53...   \n",
      "203619   1999268  POLYGON ((-1543026.424 660772.213, -1542995.81...   \n",
      "\n",
      "       gridstring  year  ADM_ID  \n",
      "78136     1999195  1999     195  \n",
      "164784    1988156  1988     156  \n",
      "187595    1998269  1998     269  \n",
      "203400    1999325  1999     325  \n",
      "180701    1998269  1998     269  \n",
      "341092    2015129  2015     129  \n",
      "321940    2005285  2005     285  \n",
      "27566     1998241  1998     241  \n",
      "269506    2001282  2001     282  \n",
      "203619    1999268  1999     268  \n"
     ]
    }
   ],
   "source": [
    "# Split serial back into separate dataset fields.\n",
    "# For Burkina: WSFE and ADM: 4+3=7 digits. GRID3 and ADM: 6+3=9 digits.\n",
    "GRID3_ADM['gridstring'] = GRID3_ADM['gridcode'].astype(str).str.zfill(9)\n",
    "WSFE_ADM['gridstring'] = WSFE_ADM['gridcode'].astype(str).str.zfill(7)\n",
    "\n",
    "GRID3_ADM['Sett_ID'] = GRID3_ADM['gridstring'].str[:-3].astype(int) # Remove the last 4 digits to get the GRID3 portion.\n",
    "GRID3_ADM['ADM_ID'] = GRID3_ADM['gridstring'].str[-3:].astype(int) # Keep only the last 4 digits to get the ADM portion.\n",
    "WSFE_ADM['year'] = WSFE_ADM['gridstring'].str[:-3].astype(int)\n",
    "WSFE_ADM['ADM_ID'] = WSFE_ADM['gridstring'].str[-3:].astype(int)\n",
    "\n",
    "print(GRID3_ADM.sample(10), WSFE_ADM.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3127bbf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 204257 entries, 0 to 204256\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count   Dtype   \n",
      "---  ------      --------------   -----   \n",
      " 0   Sett_ID     204257 non-null  int64   \n",
      " 1   ADM_ID      204257 non-null  int64   \n",
      " 2   geometry    204257 non-null  geometry\n",
      " 3   gridcode    204257 non-null  int64   \n",
      " 4   gridstring  204257 non-null  object  \n",
      "dtypes: geometry(1), int64(3), object(1)\n",
      "memory usage: 7.8+ MB\n",
      "None    Sett_ID  ADM_ID                                           geometry  \\\n",
      "0        1     323  MULTIPOLYGON (((-1572100.720 271452.083, -1572...   \n",
      "1        2     323  POLYGON ((-1572896.437 272094.778, -1572804.62...   \n",
      "2        3     323  POLYGON ((-1571274.398 273778.027, -1571213.18...   \n",
      "3        4     323  POLYGON ((-1575038.754 274298.304, -1574977.54...   \n",
      "4        5     323  POLYGON ((-1573416.714 277205.733, -1573233.08...   \n",
      "\n",
      "   gridcode gridstring  \n",
      "0      1323  000001323  \n",
      "1      2323  000002323  \n",
      "2      3323  000003323  \n",
      "3      4323  000004323  \n",
      "4      5323  000005323  \n"
     ]
    }
   ],
   "source": [
    "# Dissolve any features that have the same G3 and ADM values so that we have a single unique feature per settlement.\n",
    "# Note: we do NOT want to dissolve the WSFE features. Distinct features for noncontiguous builtup areas of the same year is necessary to separate them in the Near tool step.\n",
    "GRID3_ADM = GRID3_ADM.dissolve(by=['Sett_ID', 'ADM_ID'], as_index=False)\n",
    "print(GRID3_ADM.info(), GRID3_ADM.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f172cebe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: WSFE (538320, 5) and GRID3 (204257, 5)\n",
      "\n",
      "After: WSFE (538320, 5) and GRID3 (204257, 5)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Remove features where year, settlement, or admin area = 0.\n",
    "# This was supposed to be resolved earlier with the gdal_calc NoDataValue parameter.\n",
    "\n",
    "print(\"Before: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))\n",
    "WSFE_ADM = WSFE_ADM.loc[(WSFE_ADM[\"year\"] != 0) & (WSFE_ADM[\"ADM_ID\"] != 0)] # Since we change the datatype to integer, no need to include all digits. Otherwise, it would need to be: != '0000'\n",
    "GRID3_ADM = GRID3_ADM.loc[(GRID3_ADM[\"Sett_ID\"] != 0) & (GRID3_ADM[\"ADM_ID\"] != 0)]\n",
    "print(\"After: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9c6f651d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The Bounded_ID is our new unique settlement identifier for subsequent matching steps.\n",
    "GRID3_ADM['Bounded_ID'] = GRID3_ADM.index\n",
    "WSFE_ADM['WSFE_ID'] = WSFE_ADM.index\n",
    "GRID3_ADM = GRID3_ADM[['Sett_ID', 'Bounded_ID', 'ADM_ID', 'geometry']]\n",
    "WSFE_ADM = WSFE_ADM[['year', 'ADM_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abd7f5e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "013f0a8e",
   "metadata": {},
   "source": [
    "## 4. UNIQUE SETTLEMENTS FROM WSFE AND GRID3: TWO VERSIONS\n",
    "\n",
    "Note that there are 2 versions here, so that we can create a fragmentation index:\n",
    "1. **Boundless, aka boundary-agnostic settlements**: Unique settlements are linked to GRID3 settlement IDs. Administrative areas do not influence the extents of the settlement.\n",
    "2. **Bounded, aka politically-defined settlements**: Settlements in the Boundless dataset which spread across more than one administrative area are split into separate settlements in the Bounded dataset. The largest polygon after the split is considered the \"principal\" settlement, and polygons in other admin areas are considered \"fragments.\" By dividing the fragment area(s) of the Bounded settlement by the area of the Boundless settlement, we can acquire a fragmentation index for each locality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32df6624",
   "metadata": {},
   "source": [
    "### 4.1 BOUNDED SETTLEMENTS: Near Join by ADM group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3b111165",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# The sharding step below doesn't work if any ADM group contains features from only one of the two datasets.\n",
    "WSFE_u = sorted(WSFE_ADM.ADM_ID.unique().tolist())\n",
    "GRID3_u = sorted(GRID3_ADM.ADM_ID.unique().tolist())\n",
    "\n",
    "not_matching = list(set(GRID3_u).symmetric_difference(set(WSFE_u)))\n",
    "print(not_matching) # Validate: If there are many ADM_IDs in this list, investigate why GRID3 or WSFE is missing in so many areas.\n",
    "\n",
    "# Take only the features that share an ADM with at least one GRID3 feature.\n",
    "WSFE_matching = WSFE_ADM[~WSFE_ADM[\"ADM_ID\"].isin(not_matching)] \n",
    "GRID3_matching = GRID3_ADM[~GRID3_ADM[\"ADM_ID\"].isin(not_matching)]\n",
    "\n",
    "WSFE_u = sorted(WSFE_matching.ADM_ID.unique().tolist())\n",
    "GRID3_u = sorted(GRID3_matching.ADM_ID.unique().tolist())\n",
    "\n",
    "not_matching = list(set(GRID3_u).symmetric_difference(set(WSFE_u)))\n",
    "print(not_matching) # This should now be empty.\n",
    "\n",
    "del WSFE_u, GRID3_u, not_matching, WSFE_ADM, GRID3_ADM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b0979d24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "MultiIndex: 540184 entries, (0, 104264) to (325, 3696)\n",
      "Data columns (total 7 columns):\n",
      " #   Column        Non-Null Count   Dtype   \n",
      "---  ------        --------------   -----   \n",
      " 0   year          540184 non-null  int32   \n",
      " 1   ADM_ID_left   540184 non-null  int32   \n",
      " 2   geometry      540184 non-null  geometry\n",
      " 3   index_right   536223 non-null  float64 \n",
      " 4   Sett_ID       536223 non-null  float64 \n",
      " 5   Bounded_ID    536223 non-null  float64 \n",
      " 6   ADM_ID_right  536223 non-null  float64 \n",
      "dtypes: float64(4), geometry(1), int32(2)\n",
      "memory usage: 48.1 MB\n",
      "None\n",
      "            year  ADM_ID_left  \\\n",
      "53  373202  2011           55   \n",
      "265 220317  1999          268   \n",
      "11  445552  1997           12   \n",
      "116 372497  2010          118   \n",
      "141 166566  1998          143   \n",
      "35  500371  2007           36   \n",
      "207 78724   2013          209   \n",
      "5   227170  1985            6   \n",
      "222 33397   2015          225   \n",
      "193 80322   1999          195   \n",
      "\n",
      "                                                     geometry  index_right  \\\n",
      "53  373202  POLYGON ((-1197593.178 500404.516, -1197562.57...      37769.0   \n",
      "265 220317  POLYGON ((-1538741.790 654222.846, -1538711.18...      57885.0   \n",
      "11  445552  POLYGON ((-1395910.484 459761.710, -1395879.87...      19821.0   \n",
      "116 372497  POLYGON ((-1615314.306 501965.346, -1615283.70...      13686.0   \n",
      "141 166566  POLYGON ((-1479552.644 675278.757, -1479522.03...      70786.0   \n",
      "35  500371  POLYGON ((-1475910.706 427963.611, -1475880.10...      15232.0   \n",
      "207 78724   POLYGON ((-1139811.840 822058.047, -1139781.23...     128706.0   \n",
      "5   227170  POLYGON ((-1523592.552 652202.947, -1523500.73...      57900.0   \n",
      "222 33397   POLYGON ((-1150156.168 1194453.876, -1150125.5...     159468.0   \n",
      "193 80322   POLYGON ((-1371671.702 795585.135, -1371641.09...     106895.0   \n",
      "\n",
      "             Sett_ID  Bounded_ID  ADM_ID_right  \n",
      "53  373202   36524.0     37769.0          55.0  \n",
      "265 220317   56244.0     57885.0         268.0  \n",
      "11  445552   19120.0     19821.0          12.0  \n",
      "116 372497   13085.0     13686.0         118.0  \n",
      "141 166566   68806.0     70786.0         143.0  \n",
      "35  500371   14617.0     15232.0          36.0  \n",
      "207 78724   126110.0    128706.0         209.0  \n",
      "5   227170   56252.0     57900.0           6.0  \n",
      "222 33397   156483.0    159468.0         225.0  \n",
      "193 80322   104513.0    106895.0         195.0  \n"
     ]
    }
   ],
   "source": [
    "# Shard the dataframe whose variables we want to join into a dict\n",
    "shards = {k:d for k, d in GRID3_matching.groupby('ADM_ID', as_index=False)}\n",
    "\n",
    "# Take the dataframe whose geometry we want to retain.\n",
    "# Group by ADM, then sjoin_nearest among the smaller dataframe's matching ADM shard\n",
    "Bounded = WSFE_matching.groupby('ADM_ID', as_index=False).apply(\n",
    "    lambda d: gpd.sjoin_nearest(\n",
    "    d, shards[d['ADM_ID'].values[0]], \n",
    "        how='left', \n",
    "        max_distance=500))\n",
    "\n",
    "print(Bounded.info())\n",
    "print(Bounded.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7d22854d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 83356 entries, 0 to 83355\n",
      "Data columns (total 7 columns):\n",
      " #   Column        Non-Null Count  Dtype   \n",
      "---  ------        --------------  -----   \n",
      " 0   year          83356 non-null  int64   \n",
      " 1   Bounded_ID    83356 non-null  float64 \n",
      " 2   geometry      83356 non-null  geometry\n",
      " 3   ADM_ID_left   83356 non-null  int32   \n",
      " 4   index_right   83356 non-null  float64 \n",
      " 5   Sett_ID       83356 non-null  float64 \n",
      " 6   ADM_ID_right  83356 non-null  float64 \n",
      "dtypes: float64(4), geometry(1), int32(1), int64(1)\n",
      "memory usage: 4.1 MB\n",
      "None        year  Bounded_ID                                           geometry  \\\n",
      "55826  2009     15299.0  POLYGON ((-1487601.633 455232.240, -1487571.02...   \n",
      "78164  2014    186411.0  MULTIPOLYGON (((-1120745.222 1230506.004, -112...   \n",
      "32907  2001    199988.0  POLYGON ((-1415313.751 423372.933, -1415283.14...   \n",
      "69264  2012    148906.0  POLYGON ((-1179995.578 1090673.941, -1179964.9...   \n",
      "26920  2000     13606.0  MULTIPOLYGON (((-1577884.975 579058.138, -1577...   \n",
      "20740  1998     51526.0  MULTIPOLYGON (((-1540363.830 606694.022, -1540...   \n",
      "19738  1997    149249.0  POLYGON ((-1167447.724 1118493.452, -1167417.1...   \n",
      "77658  2014    114084.0  MULTIPOLYGON (((-1246897.064 777803.907, -1246...   \n",
      "46045  2006     26587.0  POLYGON ((-1385076.483 451590.302, -1385045.87...   \n",
      "55020  2008    202245.0  MULTIPOLYGON (((-1570601.098 658782.919, -1570...   \n",
      "\n",
      "       ADM_ID_left  index_right   Sett_ID  ADM_ID_right  \n",
      "55826           32      15299.0   14681.0          32.0  \n",
      "78164          240     186411.0  183018.0         240.0  \n",
      "32907           85     199988.0  197676.0          85.0  \n",
      "69264            2     148906.0  146057.0           2.0  \n",
      "26920          124      13606.0   13007.0         124.0  \n",
      "20740          292      51526.0   50050.0         292.0  \n",
      "19738          222     149249.0  146394.0         222.0  \n",
      "77658          198     114084.0  111667.0         198.0  \n",
      "46045          265      26587.0   25668.0         265.0  \n",
      "55020          301     202245.0  199872.0         301.0  \n"
     ]
    }
   ],
   "source": [
    "# Now we can dissolve with the WSFE years, now that we can group them by their administratively split ID.\n",
    "Bounded = Bounded.dissolve(by=['year', 'Bounded_ID'], as_index=False)\n",
    "print(Bounded.info(), Bounded.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "02684aba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up and save to file.\n",
    "Bounded = Bounded.rename(\n",
    "    columns={\"ADM_ID_left\": \"ADM_ID\"})[['ADM_ID', 'year', 'Bounded_ID', 'Sett_ID', 'geometry']]\n",
    "Bounded.to_file(\n",
    "    driver='GPKG', filename=r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Bounded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "75e43acf",
   "metadata": {},
   "outputs": [],
   "source": [
    "del WSFE_matching, GRID3_matching, shards"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea6eadab",
   "metadata": {},
   "source": [
    "### 4.2 BOUNDLESS SETTLEMENTS: Simple near join."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f2638d9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 80363 entries, 0 to 80362\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   year        80363 non-null  int64   \n",
      " 1   Sett_ID     80363 non-null  float64 \n",
      " 2   geometry    80363 non-null  geometry\n",
      " 3   ADM_ID      80363 non-null  int32   \n",
      " 4   Bounded_ID  80363 non-null  float64 \n",
      "dtypes: float64(2), geometry(1), int32(1), int64(1)\n",
      "memory usage: 2.8 MB\n",
      "None        year   Sett_ID                                           geometry  \\\n",
      "14755  1993   97680.0  POLYGON ((-1480348.361 758278.222, -1480317.75...   \n",
      "69479  2013   51742.0  MULTIPOLYGON (((-1554778.560 648652.823, -1554...   \n",
      "65305  2012   51567.0  MULTIPOLYGON (((-1563194.804 621047.543, -1563...   \n",
      "76471  2015    4407.0  MULTIPOLYGON (((-1329896.529 363235.046, -1329...   \n",
      "70609  2013  146186.0  POLYGON ((-1178159.307 1100865.247, -1178128.7...   \n",
      "11961  1988   46452.0  POLYGON ((-1664954.841 676900.797, -1664924.23...   \n",
      "1605   1985    4378.0  MULTIPOLYGON (((-1351411.508 395033.145, -1351...   \n",
      "70594  2013  146062.0  MULTIPOLYGON (((-1180944.318 1091653.285, -118...   \n",
      "39518  2004  126350.0  POLYGON ((-1170722.408 888316.838, -1170691.80...   \n",
      "74211  2014   68893.0  POLYGON ((-1446101.901 644215.167, -1446071.29...   \n",
      "\n",
      "       ADM_ID  Bounded_ID  \n",
      "14755     190    100004.0  \n",
      "69479     305     53301.0  \n",
      "65305     281     53097.0  \n",
      "76471      74      4619.0  \n",
      "70609     222    149040.0  \n",
      "11961     181     47856.0  \n",
      "1605       79      4581.0  \n",
      "70594       2    148911.0  \n",
      "39518     207    128947.0  \n",
      "74211     138     70875.0  \n"
     ]
    }
   ],
   "source": [
    "# Fragments of any bounded settlement will be combined into a single \"boundless\" settlement in this version.\n",
    "# It is based on their \"Sett_ID\", which is a direct loan from the GRID3 settlement features.\n",
    "Boundless = Bounded.dissolve(by=['year', 'Sett_ID'], as_index=False)\n",
    "print(Boundless.info(), Boundless.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b76e2a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up and save to file.\n",
    "Boundless.to_file(driver='GPKG', filename=r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Boundless')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbba12eb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c027eba",
   "metadata": {},
   "source": [
    "## 5. CUMULATIVE ANNUALIZED SETTLEMENT EXTENTS\n",
    "DISSOLVE BY YEAR SETS: Create separate feature layers of each cumulative year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d70322",
   "metadata": {},
   "source": [
    "### 5.1 Define study years for each for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "60c3aff1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015] \n",
      "\n",
      " [1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015]\n",
      "\n",
      "\n",
      " [2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999]\n"
     ]
    }
   ],
   "source": [
    "# Boundless = gpd.read_file(r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Boundless')\n",
    "\n",
    "def CreateList(r1, r2):\n",
    "    return [item for item in range(r1, r2+1)]\n",
    "\n",
    "CuStart, CuEnd = Boundless['year'].min(), Boundless['year'].max()\n",
    "StudyStart, StudyEnd = 1999, Boundless['year'].max()\n",
    "\n",
    "AllCuYears = CreateList(CuStart, CuEnd) # All years in the WSFE dataset\n",
    "AllStudyYears = CreateList(StudyStart, StudyEnd) # All years for which there will be growth stats in the present study.\n",
    "print(AllCuYears, '\\n\\n', AllStudyYears)\n",
    "\n",
    "ReversedStudyYears = []\n",
    "for i in AllStudyYears:\n",
    "    ReversedStudyYears.insert(0,i)\n",
    "ReversedStudyYears.remove(StudyEnd)\n",
    "print('\\n\\n', ReversedStudyYears)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15de5331",
   "metadata": {},
   "source": [
    "### 5.2 Starting with main Boundless dataset, create a cumulative area feature layer for each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b3c1be53",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1999. Thu Dec  8 16:03:29 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:03:29 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:03:52 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Thu Dec  8 16:03:59 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:03:59 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:04:24 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Thu Dec  8 16:04:31 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:04:31 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:04:58 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Thu Dec  8 16:05:06 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:05:06 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:05:34 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Thu Dec  8 16:05:41 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:05:41 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:06:11 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Thu Dec  8 16:06:18 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:06:18 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:06:49 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Thu Dec  8 16:06:56 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:06:56 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:07:29 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Thu Dec  8 16:07:36 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:07:36 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:08:09 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Thu Dec  8 16:08:16 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:08:16 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:08:51 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Thu Dec  8 16:08:58 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:08:58 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:09:34 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Thu Dec  8 16:09:41 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:09:41 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:10:18 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Thu Dec  8 16:10:25 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:10:25 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:11:03 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Thu Dec  8 16:11:10 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:11:10 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:11:49 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Thu Dec  8 16:11:56 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:11:56 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:12:37 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Thu Dec  8 16:12:43 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:12:43 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:13:23 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Thu Dec  8 16:13:29 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:13:29 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:14:10 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Thu Dec  8 16:14:17 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Dec  8 16:14:17 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\1564490967.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:14:59 2022\n",
      "\n",
      "Done with all years in set. Thu Dec  8 16:15:06 2022\n"
     ]
    }
   ],
   "source": [
    "# For each year in the growth stats study, we are taking features from all years prior to and including that year, \n",
    "# dissolving those features, and exporting as its own file.\n",
    "\n",
    "for item in AllStudyYears:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    CuYearSet = Boundless[Boundless['year'].between(\n",
    "        CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    CuYearDissolve = CuYearSet.dissolve(by='Sett_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\"}, # Though ADM_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    CuYearName = ''.join(['Cu', str(item), '_Boundless'])\n",
    "    CuYearDissolve.to_file(driver='GPKG', filename=r'Results/CumulativeSettlements.gpkg', layer=CuYearName)\n",
    "    del CuYearSet, CuYearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "307c4746",
   "metadata": {},
   "source": [
    "##### Join area information from each cumulative layer onto the latest year dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b33dc9c9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2014. Thu Dec  8 16:15:07 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:08 2022\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2015) via table join. Thu Dec  8 16:15:08 2022\n",
      "\n",
      "Loading cumulative layer for year 2013. Thu Dec  8 16:15:08 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:10 2022\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2015) via table join. Thu Dec  8 16:15:10 2022\n",
      "\n",
      "Loading cumulative layer for year 2012. Thu Dec  8 16:15:10 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:11 2022\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2015) via table join. Thu Dec  8 16:15:11 2022\n",
      "\n",
      "Loading cumulative layer for year 2011. Thu Dec  8 16:15:11 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:12 2022\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2015) via table join. Thu Dec  8 16:15:12 2022\n",
      "\n",
      "Loading cumulative layer for year 2010. Thu Dec  8 16:15:12 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:14 2022\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2015) via table join. Thu Dec  8 16:15:14 2022\n",
      "\n",
      "Loading cumulative layer for year 2009. Thu Dec  8 16:15:14 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:15 2022\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2015) via table join. Thu Dec  8 16:15:15 2022\n",
      "\n",
      "Loading cumulative layer for year 2008. Thu Dec  8 16:15:15 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:17 2022\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2015) via table join. Thu Dec  8 16:15:17 2022\n",
      "\n",
      "Loading cumulative layer for year 2007. Thu Dec  8 16:15:17 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:18 2022\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2015) via table join. Thu Dec  8 16:15:18 2022\n",
      "\n",
      "Loading cumulative layer for year 2006. Thu Dec  8 16:15:18 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:19 2022\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2015) via table join. Thu Dec  8 16:15:19 2022\n",
      "\n",
      "Loading cumulative layer for year 2005. Thu Dec  8 16:15:19 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:21 2022\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2015) via table join. Thu Dec  8 16:15:21 2022\n",
      "\n",
      "Loading cumulative layer for year 2004. Thu Dec  8 16:15:21 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:22 2022\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2015) via table join. Thu Dec  8 16:15:22 2022\n",
      "\n",
      "Loading cumulative layer for year 2003. Thu Dec  8 16:15:22 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:24 2022\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2015) via table join. Thu Dec  8 16:15:24 2022\n",
      "\n",
      "Loading cumulative layer for year 2002. Thu Dec  8 16:15:24 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:25 2022\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2015) via table join. Thu Dec  8 16:15:25 2022\n",
      "\n",
      "Loading cumulative layer for year 2001. Thu Dec  8 16:15:25 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:27 2022\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2015) via table join. Thu Dec  8 16:15:27 2022\n",
      "\n",
      "Loading cumulative layer for year 2000. Thu Dec  8 16:15:27 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:28 2022\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2015) via table join. Thu Dec  8 16:15:28 2022\n",
      "\n",
      "Loading cumulative layer for year 1999. Thu Dec  8 16:15:28 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:15:29 2022\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2015) via table join. Thu Dec  8 16:15:29 2022\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Thu Dec  8 16:15:29 2022\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 13621\n",
      "Data columns (total 20 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   13622 non-null  float64\n",
      " 1   year      13622 non-null  int64  \n",
      " 2   ADM_ID    13622 non-null  int64  \n",
      " 3   Area2015  13622 non-null  float64\n",
      " 4   Area2014  13514 non-null  float64\n",
      " 5   Area2013  13375 non-null  float64\n",
      " 6   Area2012  13253 non-null  float64\n",
      " 7   Area2011  13133 non-null  float64\n",
      " 8   Area2010  13033 non-null  float64\n",
      " 9   Area2009  12931 non-null  float64\n",
      " 10  Area2008  12865 non-null  float64\n",
      " 11  Area2007  12759 non-null  float64\n",
      " 12  Area2006  12664 non-null  float64\n",
      " 13  Area2005  12578 non-null  float64\n",
      " 14  Area2004  12494 non-null  float64\n",
      " 15  Area2003  12429 non-null  float64\n",
      " 16  Area2002  12355 non-null  float64\n",
      " 17  Area2001  12275 non-null  float64\n",
      " 18  Area2000  12183 non-null  float64\n",
      " 19  Area1999  12039 non-null  float64\n",
      "dtypes: float64(18), int64(2)\n",
      "memory usage: 2.2 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# The latest year in the study contains all settlements. Merge all other years' areas onto this dataset.\n",
    "SettAreas = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=\n",
    "                          ''.join(['Cu', str(StudyEnd), '_Boundless'])) \n",
    "SettAreas['Area2015'] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry') # We have settlement IDs, so no need to join spatially!\n",
    "\n",
    "\n",
    "for item in ReversedStudyYears:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=''.join(['Cu', str(item), '_Boundless']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['Area', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Sett_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, StudyEnd, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Sett_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(ResultsFolder, 'Areas%sto%s.csv' % (StudyStart, StudyEnd)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1d0879f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a1cafcd",
   "metadata": {},
   "source": [
    "### 5.3 Repeat for Bounded dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ad1dcf6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1999. Thu Dec  8 16:15:30 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:15:30 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:15:50 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Thu Dec  8 16:15:57 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:15:57 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:16:19 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Thu Dec  8 16:16:27 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:16:27 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:16:51 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Thu Dec  8 16:16:58 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:16:58 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:17:23 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Thu Dec  8 16:17:30 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:17:30 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:17:57 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Thu Dec  8 16:18:05 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:18:05 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:18:35 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Thu Dec  8 16:18:42 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:18:42 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:19:13 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Thu Dec  8 16:19:21 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:19:21 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:19:53 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Thu Dec  8 16:20:00 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:20:00 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:20:33 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Thu Dec  8 16:20:41 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:20:41 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:21:15 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Thu Dec  8 16:21:22 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:21:22 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:21:58 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Thu Dec  8 16:22:05 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:22:05 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:22:42 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Thu Dec  8 16:22:49 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:22:49 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:23:27 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Thu Dec  8 16:23:35 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:23:35 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:24:14 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Thu Dec  8 16:24:21 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:24:21 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:25:02 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Thu Dec  8 16:25:09 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:25:09 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:25:51 2022\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Thu Dec  8 16:25:58 2022\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Dec  8 16:25:58 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28364\\2505613866.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Dec  8 16:26:40 2022\n",
      "\n",
      "Done with all years in set. Thu Dec  8 16:26:48 2022\n"
     ]
    }
   ],
   "source": [
    "# Bounded = gpd.read_file(r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Bounded')\n",
    "\n",
    "for item in AllStudyYears:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    CuYearSet = Bounded[Bounded['year'].between(CuStart, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    CuYearDissolve = CuYearSet.dissolve(by='Bounded_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\", \"Sett_ID\":\"min\"}, # Though ADM_ID and Sett_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    CuYearName = ''.join(['Cu', str(item), '_Bounded'])\n",
    "    CuYearDissolve.to_file(driver='GPKG', filename=r'Results/CumulativeSettlements.gpkg', layer=CuYearName)\n",
    "    del CuYearSet, CuYearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "49eaeaeb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2014. Thu Dec  8 16:26:49 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:51 2022\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2015) via table join. Thu Dec  8 16:26:51 2022\n",
      "\n",
      "Loading cumulative layer for year 2013. Thu Dec  8 16:26:51 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:52 2022\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2015) via table join. Thu Dec  8 16:26:52 2022\n",
      "\n",
      "Loading cumulative layer for year 2012. Thu Dec  8 16:26:52 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:53 2022\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2015) via table join. Thu Dec  8 16:26:53 2022\n",
      "\n",
      "Loading cumulative layer for year 2011. Thu Dec  8 16:26:53 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:55 2022\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2015) via table join. Thu Dec  8 16:26:55 2022\n",
      "\n",
      "Loading cumulative layer for year 2010. Thu Dec  8 16:26:55 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:56 2022\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2015) via table join. Thu Dec  8 16:26:56 2022\n",
      "\n",
      "Loading cumulative layer for year 2009. Thu Dec  8 16:26:56 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:58 2022\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2015) via table join. Thu Dec  8 16:26:58 2022\n",
      "\n",
      "Loading cumulative layer for year 2008. Thu Dec  8 16:26:58 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:26:59 2022\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2015) via table join. Thu Dec  8 16:26:59 2022\n",
      "\n",
      "Loading cumulative layer for year 2007. Thu Dec  8 16:26:59 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:01 2022\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2015) via table join. Thu Dec  8 16:27:01 2022\n",
      "\n",
      "Loading cumulative layer for year 2006. Thu Dec  8 16:27:01 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:02 2022\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2015) via table join. Thu Dec  8 16:27:03 2022\n",
      "\n",
      "Loading cumulative layer for year 2005. Thu Dec  8 16:27:03 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:04 2022\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2015) via table join. Thu Dec  8 16:27:04 2022\n",
      "\n",
      "Loading cumulative layer for year 2004. Thu Dec  8 16:27:04 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:06 2022\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2015) via table join. Thu Dec  8 16:27:06 2022\n",
      "\n",
      "Loading cumulative layer for year 2003. Thu Dec  8 16:27:06 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:07 2022\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2015) via table join. Thu Dec  8 16:27:07 2022\n",
      "\n",
      "Loading cumulative layer for year 2002. Thu Dec  8 16:27:07 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:09 2022\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2015) via table join. Thu Dec  8 16:27:09 2022\n",
      "\n",
      "Loading cumulative layer for year 2001. Thu Dec  8 16:27:09 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:10 2022\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2015) via table join. Thu Dec  8 16:27:10 2022\n",
      "\n",
      "Loading cumulative layer for year 2000. Thu Dec  8 16:27:10 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:12 2022\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2015) via table join. Thu Dec  8 16:27:12 2022\n",
      "\n",
      "Loading cumulative layer for year 1999. Thu Dec  8 16:27:12 2022\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Thu Dec  8 16:27:13 2022\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2015) via table join. Thu Dec  8 16:27:13 2022\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Thu Dec  8 16:27:13 2022\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 14269 entries, 0 to 14268\n",
      "Data columns (total 21 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Bounded_ID  14269 non-null  float64\n",
      " 1   year        14269 non-null  int64  \n",
      " 2   ADM_ID      14269 non-null  int64  \n",
      " 3   Sett_ID     14269 non-null  float64\n",
      " 4   Area2015    14269 non-null  float64\n",
      " 5   Area2014    14159 non-null  float64\n",
      " 6   Area2013    14016 non-null  float64\n",
      " 7   Area2012    13888 non-null  float64\n",
      " 8   Area2011    13763 non-null  float64\n",
      " 9   Area2010    13658 non-null  float64\n",
      " 10  Area2009    13551 non-null  float64\n",
      " 11  Area2008    13483 non-null  float64\n",
      " 12  Area2007    13373 non-null  float64\n",
      " 13  Area2006    13277 non-null  float64\n",
      " 14  Area2005    13187 non-null  float64\n",
      " 15  Area2004    13101 non-null  float64\n",
      " 16  Area2003    13033 non-null  float64\n",
      " 17  Area2002    12955 non-null  float64\n",
      " 18  Area2001    12872 non-null  float64\n",
      " 19  Area2000    12778 non-null  float64\n",
      " 20  Area1999    12617 non-null  float64\n",
      "dtypes: float64(19), int64(2)\n",
      "memory usage: 2.4 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "SettAreas = gpd.read_file(r'Results/CumulativeSettlements.gpkg', \n",
    "                          layer=''.join(['Cu', str(StudyEnd), '_Bounded']))\n",
    "SettAreas['Area2015'] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry')\n",
    "\n",
    "\n",
    "for item in ReversedStudyYears:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=''.join(['Cu', str(item), '_Bounded']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['Area', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Bounded_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, StudyEnd, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Bounded_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(ResultsFolder, 'Areas%sto%s_%s.csv' % (StudyStart, StudyEnd, 'Bounded')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e7574cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "447ecaf8",
   "metadata": {},
   "source": [
    "### 5.4 One settlement geofile to rule them all. ...and in the Sett_ID bind them.\n",
    "The annualized values can be stored as distinct non-spatial dataframes. Their Sett_IDs will be used to join onto this geoversion with place names for the summary stats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d12c8f85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13622 entries, 0 to 13621\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   13622 non-null  float64 \n",
      " 1   ADM_ID    13622 non-null  int64   \n",
      " 2   geometry  13622 non-null  geometry\n",
      "dtypes: float64(1), geometry(1), int64(1)\n",
      "memory usage: 319.4 KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "Settlements = gpd.read_file(r'Results/CumulativeSettlements.gpkg', \n",
    "                           layer=''.join(['Cu', str(StudyEnd), '_Boundless']))[['Sett_ID', 'ADM_ID', 'geometry']]\n",
    "print(Settlements.info())\n",
    "Settlements.to_file(driver='GPKG', \n",
    "                       filename=r'Results/SETTLEMENTS.gpkg', \n",
    "                       layer='SETTLEMENTS')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb8856d7",
   "metadata": {},
   "source": [
    "### 5.5 Buffer the area of the Boundless dataset's latest year to mask raster data in later sections.\n",
    "The Bounded dataset would also be fine for our purposes here. The buffer is dissolved to a single feature to be used for its total extents, which are identical between Bounded & Boundless datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e8bb2910",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Thu Dec  8 16:27:22 2022\n",
      "Finished buffer layer creation. Thu Dec  8 17:36:33 2022\n",
      "Saved to file. Thu Dec  8 17:36:38 2022\n"
     ]
    }
   ],
   "source": [
    "# Create buffer layer(s) to use as maximum distance for Near joins.\n",
    "\n",
    "# Population buffer: 2km\n",
    "Distance = 2000\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS')\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "BufferFileName1 = ''.join(['Buff', str(Distance), 'm_', str(StudyEnd)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=r'Results/Catchment.gpkg', layer=BufferFileName1)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f1556d01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Thu Dec  8 17:36:38 2022\n",
      "Finished buffer layer creation. Thu Dec  8 17:38:43 2022\n",
      "Saved to file. Thu Dec  8 17:38:49 2022\n"
     ]
    }
   ],
   "source": [
    "# NTL buffer: 250m\n",
    "Distance = 250\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS')\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "BufferFileName2 = ''.join(['Buff', str(Distance), 'm_', str(StudyEnd)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=r'Results/Catchment.gpkg', layer=BufferFileName2)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbef5b7f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e33ce84a",
   "metadata": {},
   "source": [
    "## 6. PLACE NAMES\n",
    "Join urban place names from UCDB, Africapolis, and GeoNames onto the settlement vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b456bf69",
   "metadata": {},
   "source": [
    "### 6.1 Load placename datasets, filter, and project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "612f815f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13622 entries, 0 to 13621\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   13622 non-null  float64 \n",
      " 1   ADM_ID    13622 non-null  int64   \n",
      " 2   geometry  13622 non-null  geometry\n",
      " 3   Area2015  13622 non-null  float64 \n",
      "dtypes: float64(2), geometry(1), int64(1)\n",
      "memory usage: 425.8 KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13135 entries, 0 to 13134\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   UCDB_Name  13135 non-null  object  \n",
      " 1   geometry   13135 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 205.4+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 7720 entries, 0 to 7719\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   Afpl_Name  7720 non-null   object  \n",
      " 1   geometry   7720 non-null   geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 120.8+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 199390 entries, 0 to 199389\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   GeoName   199390 non-null  object  \n",
      " 1   geometry  199390 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 3.0+ MB\n",
      "None None None None\n"
     ]
    }
   ],
   "source": [
    "# If restarting here:\n",
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS')\n",
    "Settlements['Area2015'] = Settlements['geometry'].area / 10**6\n",
    "\n",
    "# Load, pull name field, rename, and reproject to match the catchments CRS.\n",
    "UCDB = gpd.read_file('PlaceName/GHS_STAT_UCDB2015MT_GLOBE_R2019A_V1_2.gpkg', \n",
    "                     layer=0)[['UC_NM_MN', 'geometry']].rename(\n",
    "    columns={\"UC_NM_MN\": \"UCDB_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "Africapolis = gpd.read_file('PlaceName/AFRICAPOLIS2020.shp')[['agglosName', 'geometry']].rename(\n",
    "    columns={\"agglosName\": \"Afpl_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "GeoNames = gpd.read_file('PlaceName/GeoNames.gpkg', \n",
    "                         layer=0)[['GeoName', 'geometry']].to_crs(\"ESRI:102022\")\n",
    "\n",
    "print(Settlements.info(), UCDB.info(), Africapolis.info(), GeoNames.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e934567",
   "metadata": {},
   "source": [
    "### 6.2 Join placenames onto settlements geodataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "106f2719",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We wrap it in pd.DataFrame() since the sjoin() is the last time we need the geometry.\n",
    "\n",
    "GeoNames = pd.DataFrame(gpd.sjoin(Settlements, GeoNames, \n",
    "                             how='left', predicate='contains', # Name file is point type, so we can do contain.\n",
    "                             lsuffix=\"G3\", rsuffix=\"GN\")).drop(columns='geometry')\n",
    "Africapolis = pd.DataFrame(gpd.sjoin(Settlements, Africapolis, \n",
    "                             how='left', predicate='intersects', # Name file is polygon type.\n",
    "                             lsuffix=\"G3\", rsuffix=\"Af\")).drop(columns='geometry')\n",
    "UCDB = pd.DataFrame(gpd.sjoin(Settlements, UCDB, \n",
    "                             how='left', predicate='intersects', # Name file is polygon type.\n",
    "                             lsuffix=\"G3\", rsuffix=\"UC\")).drop(columns='geometry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "2e2f0ee0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13628 entries, 0 to 13621\n",
      "Data columns (total 5 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   13628 non-null  float64\n",
      " 1   ADM_ID    13628 non-null  int64  \n",
      " 2   Area2015  13628 non-null  float64\n",
      " 3   index_GN  119 non-null    float64\n",
      " 4   GeoName   119 non-null    object \n",
      "dtypes: float64(3), int64(1), object(1)\n",
      "memory usage: 638.8+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13629 entries, 0 to 13621\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    13629 non-null  float64\n",
      " 1   ADM_ID     13629 non-null  int64  \n",
      " 2   Area2015   13629 non-null  float64\n",
      " 3   index_Af   1433 non-null   float64\n",
      " 4   Afpl_Name  1433 non-null   object \n",
      "dtypes: float64(3), int64(1), object(1)\n",
      "memory usage: 638.9+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13623 entries, 0 to 13621\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    13623 non-null  float64\n",
      " 1   ADM_ID     13623 non-null  int64  \n",
      " 2   Area2015   13623 non-null  float64\n",
      " 3   index_UC   196 non-null    float64\n",
      " 4   UCDB_Name  196 non-null    object \n",
      "dtypes: float64(3), int64(1), object(1)\n",
      "memory usage: 638.6+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(GeoNames.info())\n",
    "print(Africapolis.info())\n",
    "print(UCDB.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fe7a96a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "alldatasets = [Africapolis[['Sett_ID', 'Afpl_Name', 'Area2015']], \n",
    "               GeoNames[['Sett_ID', 'GeoName']],\n",
    "               UCDB[['Sett_ID', 'UCDB_Name']]]\n",
    "\n",
    "SettlementsNamed = reduce(lambda left,right: pd.merge(left,right,on=['Sett_ID'],\n",
    "                                            how='left'), alldatasets).fillna('UNK')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3af429c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13637 entries, 0 to 13636\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    13637 non-null  float64\n",
      " 1   Afpl_Name  13637 non-null  object \n",
      " 2   Area2015   13637 non-null  float64\n",
      " 3   GeoName    13637 non-null  object \n",
      " 4   UCDB_Name  13637 non-null  object \n",
      "dtypes: float64(2), object(3)\n",
      "memory usage: 639.2+ KB\n",
      "None\n",
      "        Sett_ID  Afpl_Name  Area2015 GeoName UCDB_Name\n",
      "8830    64802.0        UNK  0.000937     UNK       UNK\n",
      "12632  194607.0        UNK  0.003747     UNK       UNK\n",
      "5139    24082.0        UNK  0.016859     UNK       UNK\n",
      "7885    54849.0  Bafoussam  0.010303     UNK       UNK\n",
      "2592     8155.0        UNK  0.020606     UNK       UNK\n",
      "1236     3443.0        UNK  0.019669     UNK       UNK\n",
      "5122    23965.0        UNK  0.000937     UNK       UNK\n",
      "8277    58848.0        UNK  0.004683     UNK       UNK\n",
      "10772  104569.0        UNK  0.032782     UNK       UNK\n",
      "10434   82497.0        UNK  0.051515     UNK       UNK\n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info())\n",
    "print(SettlementsNamed.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "8f92fab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "del UCDB, Africapolis, GeoNames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2fc7cd",
   "metadata": {},
   "source": [
    "### 6.3 Reduce to single name column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "94b606e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 13636\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    13622 non-null  float64\n",
      " 1   Afpl_Name  13622 non-null  object \n",
      " 2   Area2015   13622 non-null  float64\n",
      " 3   GeoName    13622 non-null  object \n",
      " 4   UCDB_Name  13622 non-null  object \n",
      "dtypes: float64(2), object(3)\n",
      "memory usage: 638.5+ KB\n"
     ]
    }
   ],
   "source": [
    "# The left joins should have prevented duplication of rows, but sometimes they creep in.\n",
    "# Dataset should have same number of rows as the original Settlements file.\n",
    "SettlementsNamed.drop_duplicates(subset=['Sett_ID'], inplace=True, keep='first')\n",
    "SettlementsNamed.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b754859a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>Area2015</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>SettName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>583</th>\n",
       "      <td>1643.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.034656</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6070</th>\n",
       "      <td>34197.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.000937</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7748</th>\n",
       "      <td>53857.0</td>\n",
       "      <td>Bafoussam</td>\n",
       "      <td>0.000937</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>Bafoussam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9341</th>\n",
       "      <td>68812.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.085234</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7385</th>\n",
       "      <td>50636.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.091790</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11967</th>\n",
       "      <td>172307.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.001873</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10551</th>\n",
       "      <td>89322.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.014050</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11822</th>\n",
       "      <td>163514.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.014986</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4725</th>\n",
       "      <td>22476.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.087107</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9239</th>\n",
       "      <td>68708.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.117080</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13327</th>\n",
       "      <td>198812.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.003747</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1773</th>\n",
       "      <td>4711.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.002810</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1519</th>\n",
       "      <td>4178.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.029972</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10329</th>\n",
       "      <td>79897.0</td>\n",
       "      <td>Batibo</td>\n",
       "      <td>0.073058</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>Batibo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9506</th>\n",
       "      <td>69021.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.078677</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5406</th>\n",
       "      <td>24915.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.021543</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7862</th>\n",
       "      <td>54741.0</td>\n",
       "      <td>Bafoussam</td>\n",
       "      <td>0.005620</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>Bafoussam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10527</th>\n",
       "      <td>89196.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.039339</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1462</th>\n",
       "      <td>4104.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.081487</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2288</th>\n",
       "      <td>7122.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>0.003747</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Sett_ID  Afpl_Name  Area2015 GeoName UCDB_Name   SettName\n",
       "583      1643.0        UNK  0.034656     UNK       UNK        UNK\n",
       "6070    34197.0        UNK  0.000937     UNK       UNK        UNK\n",
       "7748    53857.0  Bafoussam  0.000937     UNK       UNK  Bafoussam\n",
       "9341    68812.0        UNK  0.085234     UNK       UNK        UNK\n",
       "7385    50636.0        UNK  0.091790     UNK       UNK        UNK\n",
       "11967  172307.0        UNK  0.001873     UNK       UNK        UNK\n",
       "10551   89322.0        UNK  0.014050     UNK       UNK        UNK\n",
       "11822  163514.0        UNK  0.014986     UNK       UNK        UNK\n",
       "4725    22476.0        UNK  0.087107     UNK       UNK        UNK\n",
       "9239    68708.0        UNK  0.117080     UNK       UNK        UNK\n",
       "13327  198812.0        UNK  0.003747     UNK       UNK        UNK\n",
       "1773     4711.0        UNK  0.002810     UNK       UNK        UNK\n",
       "1519     4178.0        UNK  0.029972     UNK       UNK        UNK\n",
       "10329   79897.0     Batibo  0.073058     UNK       UNK     Batibo\n",
       "9506    69021.0        UNK  0.078677     UNK       UNK        UNK\n",
       "5406    24915.0        UNK  0.021543     UNK       UNK        UNK\n",
       "7862    54741.0  Bafoussam  0.005620     UNK       UNK  Bafoussam\n",
       "10527   89196.0        UNK  0.039339     UNK       UNK        UNK\n",
       "1462     4104.0        UNK  0.081487     UNK       UNK        UNK\n",
       "2288     7122.0        UNK  0.003747     UNK       UNK        UNK"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a single name column where non-named settlements are \"UNK\" but all others use one of the three name sources.\n",
    "SettlementsNamed['SettName'] = \"UNK\"\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['Afpl_Name'] != \"UNK\", \n",
    "    'SettName'] = SettlementsNamed['Afpl_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    (SettlementsNamed['SettName'] == \"UNK\") & (SettlementsNamed['UCDB_Name'] != \"UNK\"), \n",
    "    'SettName'] = SettlementsNamed['UCDB_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    (SettlementsNamed['SettName'] == \"UNK\") & (SettlementsNamed['GeoName'] != \"UNK\"), \n",
    "    'SettName'] = SettlementsNamed['GeoName']\n",
    "\n",
    "SettlementsNamed.sample(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c748e23",
   "metadata": {},
   "source": [
    "### 6.4 Make sure place name is unique by stripping smaller localities of duplicated names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "bf4024d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        Sett_ID   Afpl_Name    Area2015    GeoName  UCDB_Name    SettName\n",
      "6109    34755.0  Akonolinga    3.784013        UNK        UNK  Akonolinga\n",
      "633      1753.0       Ambam    0.112396        UNK        UNK       Ambam\n",
      "8291    59171.0       Awing    5.009134        UNK        UNK       Awing\n",
      "6027    33965.0        Ayos    0.103967        UNK        UNK        Ayos\n",
      "10542   89229.0     Babungo    0.001873        UNK        UNK     Babungo\n",
      "...         ...         ...         ...        ...        ...         ...\n",
      "12059  175759.0      Tourou    0.033719        UNK        UNK      Tourou\n",
      "10444   85444.0         Wum    0.027162        UNK        UNK         Wum\n",
      "11990  174679.0      Yagoua    0.003747        UNK        UNK      Yagoua\n",
      "13509  199780.0     Yaounde  101.741235    Yaoundé    Yaounde     Yaounde\n",
      "2381     7387.0   Yokadouma    1.451787  Yokadouma  Yokadouma   Yokadouma\n",
      "\n",
      "[100 rows x 6 columns]\n"
     ]
    }
   ],
   "source": [
    "Dupes = SettlementsNamed[ \n",
    "    (SettlementsNamed['SettName'] != 'UNK') & \n",
    "    (SettlementsNamed.duplicated('SettName')) ]\n",
    "\n",
    "Largest = Dupes.loc[Dupes.groupby([\"SettName\"])[\"Area2015\"].idxmax()]\n",
    "print(Largest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "73867e4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1493"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False] # Count number of non-UNK settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "82bd26e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "SettlementsNamed.loc[~SettlementsNamed.Sett_ID.isin(Largest.Sett_ID), 'SettName'] = 'UNK'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "28bddc71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False] # Count number of non-UNK settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "885dd231",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 13636\n",
      "Data columns (total 6 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    13622 non-null  float64\n",
      " 1   Afpl_Name  13622 non-null  object \n",
      " 2   Area2015   13622 non-null  float64\n",
      " 3   GeoName    13622 non-null  object \n",
      " 4   UCDB_Name  13622 non-null  object \n",
      " 5   SettName   13622 non-null  object \n",
      "dtypes: float64(2), object(4)\n",
      "memory usage: 1.2+ MB\n",
      "None         Sett_ID  Afpl_Name  Area2015 GeoName UCDB_Name SettName\n",
      "3085    12352.0        UNK  0.018733     UNK       UNK      UNK\n",
      "947      2665.0        UNK  0.117080     UNK       UNK      UNK\n",
      "7718    53572.0  Bafoussam  0.000937     UNK       UNK      UNK\n",
      "12638  194918.0        UNK  0.034656     UNK       UNK      UNK\n",
      "3795    17065.0        UNK  0.005620     UNK       UNK      UNK\n",
      "11449  145485.0        UNK  0.320330     UNK       UNK      UNK\n",
      "4694    22044.0        UNK  0.103967     UNK       UNK      UNK\n",
      "2207     6551.0        UNK  0.002810     UNK       UNK      UNK\n",
      "76         89.0        UNK  0.049642     UNK       UNK      UNK\n",
      "6810    46457.0        UNK  0.532946     UNK       UNK      UNK\n",
      "4467    20302.0        UNK  0.002810     UNK       UNK      UNK\n",
      "981      2699.0        UNK  0.075868     UNK       UNK      UNK\n",
      "6630    45202.0        UNK  0.047768     UNK       UNK      UNK\n",
      "1531     4192.0        UNK  0.037465     UNK       UNK      UNK\n",
      "12189  188072.0        UNK  0.015923     UNK       UNK      UNK\n",
      "934      2651.0        UNK  0.157355     UNK       UNK      UNK\n",
      "13207  198312.0        UNK  0.010303     UNK       UNK      UNK\n",
      "7623    52717.0  Bafoussam  0.002810     UNK       UNK      UNK\n",
      "4786    22704.0        UNK  0.078677     UNK       UNK      UNK\n",
      "11351  136295.0        UNK  0.079614     UNK       UNK      UNK\n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info(), SettlementsNamed.sample(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "0d202603",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop extra columns and save to file.\n",
    "SettlementsNamed = SettlementsNamed[['Sett_ID', 'SettName']]\n",
    "SettlementsNamed.to_csv(r'Results/PlaceNames.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "db1e4aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettlementsNamed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c1c3d6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8d7b6d",
   "metadata": {},
   "source": [
    "## 7. CREATE FRAGMENTATION INDEX\n",
    "We are determining what percentage of a settlement's area lies outside of its administrative zone each year.\n",
    "The index is a range of 0 to 100, i.e. the percent of the settlement area which is fragmented.\n",
    "\n",
    "For each Sett_ID:\n",
    "((Area of Boundless settlement - Area of largest Bounded settlement feature) / Area of Boundless settlement) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa7537f",
   "metadata": {},
   "source": [
    "### 7.1 Load boundless and bounded cumulative settlements and clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "7b9bdf01",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. Thu Dec  8 17:39:08 2022\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 13622 entries, 0 to 13621\n",
      "Data columns (total 21 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  13622 non-null  int64  \n",
      " 1   Sett_ID     13622 non-null  float64\n",
      " 2   year        13622 non-null  int64  \n",
      " 3   ADM_ID      13622 non-null  int64  \n",
      " 4   Area2015    13622 non-null  float64\n",
      " 5   Area2014    13514 non-null  float64\n",
      " 6   Area2013    13375 non-null  float64\n",
      " 7   Area2012    13253 non-null  float64\n",
      " 8   Area2011    13133 non-null  float64\n",
      " 9   Area2010    13033 non-null  float64\n",
      " 10  Area2009    12931 non-null  float64\n",
      " 11  Area2008    12865 non-null  float64\n",
      " 12  Area2007    12759 non-null  float64\n",
      " 13  Area2006    12664 non-null  float64\n",
      " 14  Area2005    12578 non-null  float64\n",
      " 15  Area2004    12494 non-null  float64\n",
      " 16  Area2003    12429 non-null  float64\n",
      " 17  Area2002    12355 non-null  float64\n",
      " 18  Area2001    12275 non-null  float64\n",
      " 19  Area2000    12183 non-null  float64\n",
      " 20  Area1999    12039 non-null  float64\n",
      "dtypes: float64(18), int64(3)\n",
      "memory usage: 2.2 MB\n",
      "None\n",
      "Loaded Bounded dataset, which will factor into the fragmentation calculation. Thu Dec  8 17:39:08 2022\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 14269 entries, 0 to 14268\n",
      "Data columns (total 22 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  14269 non-null  int64  \n",
      " 1   Bounded_ID  14269 non-null  float64\n",
      " 2   year        14269 non-null  int64  \n",
      " 3   ADM_ID      14269 non-null  int64  \n",
      " 4   Sett_ID     14269 non-null  float64\n",
      " 5   Area2015    14269 non-null  float64\n",
      " 6   Area2014    14159 non-null  float64\n",
      " 7   Area2013    14016 non-null  float64\n",
      " 8   Area2012    13888 non-null  float64\n",
      " 9   Area2011    13763 non-null  float64\n",
      " 10  Area2010    13658 non-null  float64\n",
      " 11  Area2009    13551 non-null  float64\n",
      " 12  Area2008    13483 non-null  float64\n",
      " 13  Area2007    13373 non-null  float64\n",
      " 14  Area2006    13277 non-null  float64\n",
      " 15  Area2005    13187 non-null  float64\n",
      " 16  Area2004    13101 non-null  float64\n",
      " 17  Area2003    13033 non-null  float64\n",
      " 18  Area2002    12955 non-null  float64\n",
      " 19  Area2001    12872 non-null  float64\n",
      " 20  Area2000    12778 non-null  float64\n",
      " 21  Area1999    12617 non-null  float64\n",
      "dtypes: float64(19), int64(3)\n",
      "memory usage: 2.4 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "BoundlessAreas = pd.read_csv(os.path.join(ResultsFolder, ('Areas%sto%s.csv' % (StudyStart, StudyEnd))))\n",
    "print('Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. %s' \n",
    "      % time.ctime())\n",
    "print(BoundlessAreas.info())\n",
    "\n",
    "BoundedAreas = pd.read_csv(os.path.join(ResultsFolder, ('Areas%sto%s_%s.csv' % (StudyStart, StudyEnd, 'Bounded'))))\n",
    "print('Loaded Bounded dataset, which will factor into the fragmentation calculation. %s' % time.ctime())\n",
    "print(BoundedAreas.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "261495a2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 14268\n",
      "Data columns (total 22 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  13622 non-null  int64  \n",
      " 1   Bounded_ID  13622 non-null  float64\n",
      " 2   year        13622 non-null  int64  \n",
      " 3   ADM_ID      13622 non-null  int64  \n",
      " 4   Sett_ID     13622 non-null  float64\n",
      " 5   Area2015    13622 non-null  float64\n",
      " 6   Area2014    13514 non-null  float64\n",
      " 7   Area2013    13375 non-null  float64\n",
      " 8   Area2012    13251 non-null  float64\n",
      " 9   Area2011    13131 non-null  float64\n",
      " 10  Area2010    13031 non-null  float64\n",
      " 11  Area2009    12929 non-null  float64\n",
      " 12  Area2008    12863 non-null  float64\n",
      " 13  Area2007    12756 non-null  float64\n",
      " 14  Area2006    12662 non-null  float64\n",
      " 15  Area2005    12576 non-null  float64\n",
      " 16  Area2004    12492 non-null  float64\n",
      " 17  Area2003    12426 non-null  float64\n",
      " 18  Area2002    12352 non-null  float64\n",
      " 19  Area2001    12271 non-null  float64\n",
      " 20  Area2000    12179 non-null  float64\n",
      " 21  Area1999    12032 non-null  float64\n",
      "dtypes: float64(19), int64(3)\n",
      "memory usage: 2.4 MB\n",
      "None\n",
      "Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. Thu Dec  8 17:39:09 2022\n",
      "Renamed columns to avoid duplication during merge, and dropped unnecessary columns. Thu Dec  8 17:39:09 2022\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 13621\n",
      "Data columns (total 40 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   Unnamed: 0_x  13622 non-null  int64  \n",
      " 1   Sett_ID       13622 non-null  float64\n",
      " 2   year          13622 non-null  int64  \n",
      " 3   ADM_ID        13622 non-null  int64  \n",
      " 4   Area2015      13622 non-null  float64\n",
      " 5   Area2014      13514 non-null  float64\n",
      " 6   Area2013      13375 non-null  float64\n",
      " 7   Area2012      13253 non-null  float64\n",
      " 8   Area2011      13133 non-null  float64\n",
      " 9   Area2010      13033 non-null  float64\n",
      " 10  Area2009      12931 non-null  float64\n",
      " 11  Area2008      12865 non-null  float64\n",
      " 12  Area2007      12759 non-null  float64\n",
      " 13  Area2006      12664 non-null  float64\n",
      " 14  Area2005      12578 non-null  float64\n",
      " 15  Area2004      12494 non-null  float64\n",
      " 16  Area2003      12429 non-null  float64\n",
      " 17  Area2002      12355 non-null  float64\n",
      " 18  Area2001      12275 non-null  float64\n",
      " 19  Area2000      12183 non-null  float64\n",
      " 20  Area1999      12039 non-null  float64\n",
      " 21  Unnamed: 0_y  13622 non-null  int64  \n",
      " 22  Bounded_ID    13622 non-null  float64\n",
      " 23  Largest2015   13622 non-null  float64\n",
      " 24  Largest2014   13514 non-null  float64\n",
      " 25  Largest2013   13375 non-null  float64\n",
      " 26  Largest2012   13251 non-null  float64\n",
      " 27  Largest2011   13131 non-null  float64\n",
      " 28  Largest2010   13031 non-null  float64\n",
      " 29  Largest2009   12929 non-null  float64\n",
      " 30  Largest2008   12863 non-null  float64\n",
      " 31  Largest2007   12756 non-null  float64\n",
      " 32  Largest2006   12662 non-null  float64\n",
      " 33  Largest2005   12576 non-null  float64\n",
      " 34  Largest2004   12492 non-null  float64\n",
      " 35  Largest2003   12426 non-null  float64\n",
      " 36  Largest2002   12352 non-null  float64\n",
      " 37  Largest2001   12271 non-null  float64\n",
      " 38  Largest2000   12179 non-null  float64\n",
      " 39  Largest1999   12032 non-null  float64\n",
      "dtypes: float64(36), int64(4)\n",
      "memory usage: 4.3 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "LargestFragments = BoundedAreas.loc[BoundedAreas.groupby([\"Sett_ID\"])[\"Area2015\"].idxmax()] \n",
    "print(LargestFragments.info())\n",
    "print(\"Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. %s\" % time.ctime())\n",
    "LargestFragments.columns = LargestFragments.columns.str.replace('Area', 'Largest')\n",
    "LargestFragments = LargestFragments.drop(columns=['year', 'ADM_ID'])\n",
    "print(\"Renamed columns to avoid duplication during merge, and dropped unnecessary columns. %s\" % time.ctime())\n",
    "FragIndices = BoundlessAreas.merge(LargestFragments, how='left', on='Sett_ID')\n",
    "print(FragIndices.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "d7a828bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "del BoundlessAreas, BoundedAreas, LargestFragments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "964d4c7b",
   "metadata": {},
   "source": [
    "### 7.2 Merge and run fragmentation calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b17bcaeb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created names for Year 1999's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 1999. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2000's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2000. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2001's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2001. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2002's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2002. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2003's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2003. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2004's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2004. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2005's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2005. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2006's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2006. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2007's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2007. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2008's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2008. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2009's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2009. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2010's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2010. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2011's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2011. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2012's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2012. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2013's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2013. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2014's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2014. Thu Dec  8 17:39:09 2022\n",
      "Created names for Year 2015's variables and temporary objects. Thu Dec  8 17:39:09 2022\n",
      "Calculated fragmentation index for year 2015. Thu Dec  8 17:39:09 2022\n",
      "Completed fragmentation index calculations for all years. Thu Dec  8 17:39:09 2022\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13622 entries, 0 to 13621\n",
      "Data columns (total 23 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   Unnamed: 0_x  13622 non-null  int64  \n",
      " 1   Sett_ID       13622 non-null  float64\n",
      " 2   year          13622 non-null  int64  \n",
      " 3   ADM_ID        13622 non-null  int64  \n",
      " 4   Unnamed: 0_y  13622 non-null  int64  \n",
      " 5   Bounded_ID    13622 non-null  float64\n",
      " 6   Frag1999      13622 non-null  int32  \n",
      " 7   Frag2000      13622 non-null  int32  \n",
      " 8   Frag2001      13622 non-null  int32  \n",
      " 9   Frag2002      13622 non-null  int32  \n",
      " 10  Frag2003      13622 non-null  int32  \n",
      " 11  Frag2004      13622 non-null  int32  \n",
      " 12  Frag2005      13622 non-null  int32  \n",
      " 13  Frag2006      13622 non-null  int32  \n",
      " 14  Frag2007      13622 non-null  int32  \n",
      " 15  Frag2008      13622 non-null  int32  \n",
      " 16  Frag2009      13622 non-null  int32  \n",
      " 17  Frag2010      13622 non-null  int32  \n",
      " 18  Frag2011      13622 non-null  int32  \n",
      " 19  Frag2012      13622 non-null  int32  \n",
      " 20  Frag2013      13622 non-null  int32  \n",
      " 21  Frag2014      13622 non-null  int32  \n",
      " 22  Frag2015      13622 non-null  int32  \n",
      "dtypes: float64(2), int32(17), int64(4)\n",
      "memory usage: 1.6 MB\n",
      "None\n",
      "       Unnamed: 0_x   Sett_ID  year  ADM_ID  Unnamed: 0_y  Bounded_ID  \\\n",
      "2314           2314    7229.0  2010      46          2463      7580.0   \n",
      "11996         11996  174895.0  2015     229         12593    178124.0   \n",
      "4218           4218   19432.0  2009     278          4474     20197.0   \n",
      "10097         10097   74984.0  2014      67         10582     77013.0   \n",
      "6154           6154   35737.0  1985      94          6463     36909.0   \n",
      "\n",
      "       Frag1999  Frag2000  Frag2001  Frag2002  ...  Frag2006  Frag2007  \\\n",
      "2314          0         0         0         0  ...         0         0   \n",
      "11996         0         0         0         0  ...         0         0   \n",
      "4218          0         0         0         0  ...         0         0   \n",
      "10097         0         0         0         0  ...         0         0   \n",
      "6154          0         0         0         0  ...         0         0   \n",
      "\n",
      "       Frag2008  Frag2009  Frag2010  Frag2011  Frag2012  Frag2013  Frag2014  \\\n",
      "2314          0         0         0         0         0         0         0   \n",
      "11996         0         0         0         0         0         0         0   \n",
      "4218          0         0         0         0         0         0         0   \n",
      "10097         0         0         0         0         0         0         0   \n",
      "6154          0         0         0         0         0         0         0   \n",
      "\n",
      "       Frag2015  \n",
      "2314          0  \n",
      "11996         0  \n",
      "4218          0  \n",
      "10097         0  \n",
      "6154          0  \n",
      "\n",
      "[5 rows x 23 columns]\n"
     ]
    }
   ],
   "source": [
    "for item in AllStudyYears:\n",
    "    YY = str(item) # 4-digit year\n",
    "    AreaYY = ''.join([\"Area\", YY]) # The Boundless area variable name\n",
    "    LargestYY = ''.join(['Largest', YY]) # The Bounded largest area variable name\n",
    "    FragYY = ''.join([\"Frag\", YY]) # Name for the fragmentation index variable\n",
    "    print(\"Created names for Year %s's variables and temporary objects. %s\" % (item, time.ctime()))\n",
    "    \n",
    "    FragIndices[FragYY] = ((FragIndices[AreaYY] - FragIndices[LargestYY]) / FragIndices[AreaYY]) * 100\n",
    "    FragIndices[FragYY] = (FragIndices[FragYY].fillna(0).replace([np.inf, -np.inf], 0)).astype('int')\n",
    "    print(\"Calculated fragmentation index for year %s. %s\" % (item, time.ctime()))\n",
    "\n",
    "# Remove unnecessary columns.\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('Largest')]\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('Area')]\n",
    "\n",
    "print('Completed fragmentation index calculations for all years. %s' % time.ctime())\n",
    "print(FragIndices.info())\n",
    "print(FragIndices.sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "5972dd80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Thu Dec  8 17:39:09 2022\n"
     ]
    }
   ],
   "source": [
    "FragIndices = FragIndices.drop(columns=['Unnamed: 0_x', 'Unnamed: 0_y', 'year', 'ADM_ID'])\n",
    "FragIndices.to_csv(os.path.join(ResultsFolder, 'FragIndex%sto%s.csv' % (StudyStart, StudyEnd)))\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "df90dce8",
   "metadata": {},
   "outputs": [],
   "source": [
    "del FragIndices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec36f039",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585db18c",
   "metadata": {},
   "source": [
    "## 8. PREPARE YEARLY DATASETS: POPULATION\n",
    "Can use this as a template for other annualized rasters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d8bd172",
   "metadata": {},
   "source": [
    "### 8.1 Reproject and reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "189136b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cmr_ppp_2000_UNadj.tif',\n",
       " 'cmr_ppp_2001_UNadj.tif',\n",
       " 'cmr_ppp_2002_UNadj.tif',\n",
       " 'cmr_ppp_2003_UNadj.tif',\n",
       " 'cmr_ppp_2004_UNadj.tif',\n",
       " 'cmr_ppp_2005_UNadj.tif',\n",
       " 'cmr_ppp_2006_UNadj.tif',\n",
       " 'cmr_ppp_2007_UNadj.tif',\n",
       " 'cmr_ppp_2008_UNadj.tif',\n",
       " 'cmr_ppp_2009_UNadj.tif',\n",
       " 'cmr_ppp_2010_UNadj.tif',\n",
       " 'cmr_ppp_2011_UNadj.tif',\n",
       " 'cmr_ppp_2012_UNadj.tif',\n",
       " 'cmr_ppp_2013_UNadj.tif',\n",
       " 'cmr_ppp_2014_UNadj.tif',\n",
       " 'cmr_ppp_2015_UNadj.tif']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ProjCRS = gdal.WarpOptions(dstSRS='ESRI:102022')\n",
    "AnnualizedSourceFiles = [i for i in os.listdir('Population/') if i.endswith('.tif')]\n",
    "\n",
    "with fiona.open(r'Results/Catchment.gpkg', mode=\"r\", layer=\"Buff2000m_2015\") as shapefile:\n",
    "    MaskGeom = [feature[\"geometry\"] for feature in shapefile] # Identify the bounding areas of the mask.\n",
    "# Mask_out = './LatestYearBuffer.tif'\n",
    "AnnualizedSourceFiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "36d9e3e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Warp() for year 2000. Thu Dec  8 17:39:15 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2000. Thu Dec  8 17:39:19 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:39:20 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:39:20 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2001. Thu Dec  8 17:39:26 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2001. Thu Dec  8 17:39:30 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:39:31 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:39:31 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2002. Thu Dec  8 17:39:37 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2002. Thu Dec  8 17:39:41 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:39:42 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:39:42 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2003. Thu Dec  8 17:39:47 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2003. Thu Dec  8 17:39:52 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:39:54 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:39:54 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2004. Thu Dec  8 17:39:59 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2004. Thu Dec  8 17:40:03 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:40:05 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:40:05 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2005. Thu Dec  8 17:40:10 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2005. Thu Dec  8 17:40:14 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:40:16 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:40:16 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2006. Thu Dec  8 17:40:21 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2006. Thu Dec  8 17:40:25 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:40:27 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:40:27 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2007. Thu Dec  8 17:40:32 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2007. Thu Dec  8 17:40:37 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:40:38 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:40:38 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2008. Thu Dec  8 17:40:44 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2008. Thu Dec  8 17:40:48 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:40:49 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:40:49 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2009. Thu Dec  8 17:40:55 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2009. Thu Dec  8 17:40:59 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:00 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:01 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2010. Thu Dec  8 17:41:06 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2010. Thu Dec  8 17:41:10 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:12 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:12 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2011. Thu Dec  8 17:41:17 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2011. Thu Dec  8 17:41:21 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:23 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:23 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2012. Thu Dec  8 17:41:28 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2012. Thu Dec  8 17:41:32 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:34 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:34 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2013. Thu Dec  8 17:41:39 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2013. Thu Dec  8 17:41:43 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:45 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:45 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2014. Thu Dec  8 17:41:51 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2014. Thu Dec  8 17:41:55 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:41:56 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:41:57 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2015. Thu Dec  8 17:42:02 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2015. Thu Dec  8 17:42:06 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 17:42:08 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 17:42:08 2022 \n",
      "\n",
      "\n",
      " \n",
      " Finished all years in list. Thu Dec  8 17:42:08 2022\n"
     ]
    }
   ],
   "source": [
    "# This codeblock changes each annual population raster's projection (gdal.Warp()), \n",
    "# then masks it to within a specified distance of the settlements (rasterio.mask.mask()).\n",
    "\n",
    "for YearFile in AnnualizedSourceFiles:\n",
    "    InputRasterName = os.path.join(ProjectFolder, \"Population\", YearFile)\n",
    "    Year = str(re.sub(r'[^0-9]', '', YearFile))\n",
    "    InputRasterObject = gdal.Open(InputRasterName)\n",
    "    TempOutputName = \"Temp_\" + Year + \"_albers.tif\"\n",
    "    TempOutputPath = os.path.join(ProjectFolder, \"Population\", TempOutputName)\n",
    "    if exists(TempOutputPath):\n",
    "        pass\n",
    "    else:\n",
    "        # Reproject to same CRS as settlements.\n",
    "        Warp = gdal.Warp(TempOutputPath, # Where to store the warped raster\n",
    "                     InputRasterObject, # Which raster to warp\n",
    "                     format='GTiff', \n",
    "                     options=ProjCRS) # Reproject to Africa Albers Equal Area Conic\n",
    "        print('Finished gdal.Warp() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "        \n",
    "        Warp = None # Close the files\n",
    "        InputRasterObject = None\n",
    "\n",
    "        # Reclassify as nodata if outside settlement buffer zones.\n",
    "        with rasterio.open(TempOutputPath) as InputRasterObject:\n",
    "            MaskedOutputRaster, OutTransform = rasterio.mask.mask(\n",
    "                InputRasterObject, MaskGeom, crop=True) # Anything outside the mask is reclassed to the raster's NoData value.\n",
    "            OutMetaData = InputRasterObject.meta.copy()\n",
    "        print('Finished rasterio.mask.mask() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "        OutMetaData.update({\"driver\": \"GTiff\",\n",
    "                         \"height\": MaskedOutputRaster.shape[1],\n",
    "                         \"width\": MaskedOutputRaster.shape[2],\n",
    "                         \"transform\": OutTransform})\n",
    "        FinalOutputPath = os.path.join(ProjectFolder, \"Population\", ''.join(['Masked_', Year, '.tif'])) # ''.join([r'Population/', 'Masked_', Year, '.tif']\n",
    "        with rasterio.open(FinalOutputPath, \"w\", **OutMetaData) as dest:\n",
    "            dest.write(MaskedOutputRaster)\n",
    "        print('Written to file. %s \\n' % time.ctime())\n",
    "    InputRasterObject = None\n",
    "    \n",
    "    try:  # Finally, remove the intermediate file from disk\n",
    "        os.remove(TempOutputPath)\n",
    "    except OSError:\n",
    "        pass\n",
    "    print('Removed intermediate file. %s \\n' % time.ctime())\n",
    "\n",
    "print('\\n \\n Finished all years in list. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "aa0f89e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cmr_ppp_2000_UNadj.tif', 'cmr_ppp_2001_UNadj.tif', 'cmr_ppp_2002_UNadj.tif', 'cmr_ppp_2003_UNadj.tif', 'cmr_ppp_2004_UNadj.tif', 'cmr_ppp_2005_UNadj.tif', 'cmr_ppp_2006_UNadj.tif', 'cmr_ppp_2007_UNadj.tif', 'cmr_ppp_2008_UNadj.tif', 'cmr_ppp_2009_UNadj.tif', 'cmr_ppp_2010_UNadj.tif', 'cmr_ppp_2011_UNadj.tif', 'cmr_ppp_2012_UNadj.tif', 'cmr_ppp_2013_UNadj.tif', 'cmr_ppp_2014_UNadj.tif', 'cmr_ppp_2015_UNadj.tif', 'Masked_2000.tif', 'Masked_2001.tif', 'Masked_2002.tif', 'Masked_2003.tif', 'Masked_2004.tif', 'Masked_2005.tif', 'Masked_2006.tif', 'Masked_2007.tif', 'Masked_2008.tif', 'Masked_2009.tif', 'Masked_2010.tif', 'Masked_2011.tif', 'Masked_2012.tif', 'Masked_2013.tif', 'Masked_2014.tif', 'Masked_2015.tif']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('Population/'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "1ba45368",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnnualizedSourceFiles = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f624d801",
   "metadata": {},
   "source": [
    "### 8.2 Raster values summarized by settlement.\n",
    "1. Convert each annualized raster to .xyz, \n",
    "2. then bring them to vector space and assign their Sett_ID,\n",
    "3. and finally, aggregate the value as appropriate to the settlement level and save table to file.\n",
    "\n",
    "XYZ is similar to .csv. Raster cell centers are stored as x and y, and their value is stored as z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "fddceeab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Masked_2000.tif',\n",
       " 'Masked_2001.tif',\n",
       " 'Masked_2002.tif',\n",
       " 'Masked_2003.tif',\n",
       " 'Masked_2004.tif',\n",
       " 'Masked_2005.tif',\n",
       " 'Masked_2006.tif',\n",
       " 'Masked_2007.tif',\n",
       " 'Masked_2008.tif',\n",
       " 'Masked_2009.tif',\n",
       " 'Masked_2010.tif',\n",
       " 'Masked_2011.tif',\n",
       " 'Masked_2012.tif',\n",
       " 'Masked_2013.tif',\n",
       " 'Masked_2014.tif',\n",
       " 'Masked_2015.tif']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NoDataVal = -99999 \n",
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS')\n",
    "AllSummaries = pd.DataFrame(Settlements).drop(columns='geometry')\n",
    "\n",
    "AnnualizedMaskedFiles = [i for i in os.listdir('Population/') if i.startswith('Masked') and i.endswith('.tif')]\n",
    "AnnualizedMaskedFiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "109f3c71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data for year 2000. Thu Dec  8 17:42:09 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2000. Thu Dec  8 17:44:53 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2000. Thu Dec  8 17:48:01 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2000. Thu Dec  8 17:48:05 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2000. Thu Dec  8 17:59:44 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "77166079  -1.485701e+06  6.520920e+05  1.317379   \n",
      "90963241  -1.263179e+06  4.969494e+05  0.113568   \n",
      "105159018 -1.432571e+06  3.372770e+05  0.624236   \n",
      "76041317  -1.520429e+06  6.647375e+05  2.629990   \n",
      "94520722  -1.291773e+06  4.569369e+05  0.312197   \n",
      "95215909  -1.411244e+06  4.491043e+05  0.967864   \n",
      "74338324  -1.484569e+06  6.838944e+05  1.102470   \n",
      "39510721  -1.160128e+06  1.075620e+06  0.160976   \n",
      "28334650  -1.090200e+06  1.201320e+06  0.736261   \n",
      "115023008 -1.001777e+06  2.263934e+05  0.221321   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "77166079    POINT (-1485701.032 652092.014)         9219   68701.0     139  \n",
      "90963241    POINT (-1263178.899 496949.356)         6387   37535.0      57  \n",
      "105159018   POINT (-1432571.278 337276.986)         1130    3056.0       7  \n",
      "76041317    POINT (-1520428.829 664737.461)         8161   57223.0     267  \n",
      "94520722    POINT (-1291772.710 456936.895)         6204   36322.0     308  \n",
      "95215909    POINT (-1411243.881 449104.267)        13453  199536.0     298  \n",
      "74338324    POINT (-1484568.604 683894.371)         9359   68843.0     143  \n",
      "39510721   POINT (-1160127.937 1075620.144)        11534  145923.0     223  \n",
      "28334650   POINT (-1090200.498 1201319.670)        11809  163514.0     234  \n",
      "115023008   POINT (-1001776.733 226393.396)         2206    6577.0      44  \n",
      "\n",
      "Exported as table, year 2000. Thu Dec  8 18:00:44 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 18:00:44 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2000. Thu Dec  8 18:00:44 2022 \n",
      "\n",
      "        Sett_ID   PopSum2000\n",
      "8414    59715.0   121.526185\n",
      "11469  145611.0  1757.385497\n",
      "1623     4314.0   267.741163\n",
      "6653    45234.0  2072.476529\n",
      "5858    30402.0    66.819575\n",
      "5480    25470.0    66.294123\n",
      "9343    68829.0   816.380802\n",
      "4035    19203.0   350.201914\n",
      "12644  195351.0   132.478757\n",
      "12874  196744.0   136.110305\n",
      "\n",
      "Merged year 2000 onto latest year settlement feature layer. Thu Dec  8 18:00:44 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000\n",
      "12431  191547.0     264   545.493629\n",
      "8285    59178.0     161  4763.589504\n",
      "11225  134094.0      13   327.241838\n",
      "199       567.0      16    30.139665\n",
      "6101    34749.0     106   170.395730\n",
      "4435    20167.0     275   324.649488\n",
      "13518  199872.0     301   819.557976\n",
      "11335  136289.0     204   206.616861\n",
      "12777  196237.0     169   138.506296\n",
      "2284     7124.0      46    32.731790\n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2001. Thu Dec  8 18:00:44 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2001. Thu Dec  8 18:03:27 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2001. Thu Dec  8 18:06:37 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2001. Thu Dec  8 18:06:40 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2001. Thu Dec  8 18:17:55 2022 \n",
      "\n",
      "                     X             Y          Z  \\\n",
      "64809962 -1.124645e+06  7.910976e+05   0.308623   \n",
      "79037413 -1.472395e+06  6.310477e+05   0.181356   \n",
      "57316104 -1.190232e+06  8.753691e+05   1.662401   \n",
      "76074637 -1.543455e+06  6.643600e+05   2.297127   \n",
      "62372613 -1.498252e+06  8.184646e+05   0.149611   \n",
      "95106851 -1.408885e+06  4.503311e+05  23.339411   \n",
      "55554218 -1.169093e+06  8.951866e+05   0.180026   \n",
      "37043746 -1.162110e+06  1.103365e+06   0.514391   \n",
      "60127281 -1.172113e+06  8.437555e+05   0.382265   \n",
      "76008328 -1.466167e+06  6.651149e+05   0.715018   \n",
      "\n",
      "                                  geometry  index_right   Sett_ID  ADM_ID  \n",
      "64809962   POINT (-1124645.188 791097.570)        11025  123619.0     201  \n",
      "79037413   POINT (-1472395.001 631047.724)         9607   69906.0     138  \n",
      "57316104   POINT (-1190231.652 875369.099)        11069  126046.0      13  \n",
      "76074637   POINT (-1543454.868 664359.985)         7970   56244.0     136  \n",
      "62372613   POINT (-1498252.111 818464.584)        10735  104196.0     189  \n",
      "95106851   POINT (-1408884.656 450331.064)        13018  197714.0     298  \n",
      "55554218   POINT (-1169092.993 895186.591)        11172  126372.0     207  \n",
      "37043746  POINT (-1162109.686 1103364.634)        11447  145496.0     223  \n",
      "60127281   POINT (-1172112.802 843755.479)        11096  126135.0     209  \n",
      "76008328   POINT (-1466166.646 665114.938)         9897   72888.0     138  \n",
      "\n",
      "Exported as table, year 2001. Thu Dec  8 18:18:54 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 18:18:54 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2001. Thu Dec  8 18:18:55 2022 \n",
      "\n",
      "        Sett_ID   PopSum2001\n",
      "4350    19728.0   128.139289\n",
      "7435    51594.0  2502.088702\n",
      "2196     6540.0    37.133329\n",
      "8068    56409.0    68.109647\n",
      "11841  169147.0  2457.501657\n",
      "8076    56448.0    35.719724\n",
      "7053    48428.0   471.808913\n",
      "8654    63109.0   125.550800\n",
      "7225    49207.0   820.220001\n",
      "9871    72681.0   211.729093\n",
      "\n",
      "Merged year 2001 onto latest year settlement feature layer. Thu Dec  8 18:18:55 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001\n",
      "3396    14338.0       3   501.497587   535.219876\n",
      "3603    15498.0      36   173.869468   193.408900\n",
      "1087     2837.0       7    95.641119    90.871917\n",
      "209       595.0      15    77.878742    90.816763\n",
      "10433   85446.0     311  1626.764590  2043.566337\n",
      "6732    45636.0     168   317.133588   323.102167\n",
      "9740    71360.0     325   131.901816   133.261005\n",
      "12166  188020.0     260  1219.511286  1206.203268\n",
      "6184    36279.0      49    63.527896    90.805113\n",
      "6792    46446.0     182  1909.999703  2086.699413\n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2002. Thu Dec  8 18:18:55 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2002. Thu Dec  8 18:21:32 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2002. Thu Dec  8 18:24:35 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2002. Thu Dec  8 18:24:38 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2002. Thu Dec  8 18:35:52 2022 \n",
      "\n",
      "                      X              Y         Z  \\\n",
      "87079189  -1.170603e+06  540642.209328  1.546167   \n",
      "58424421  -1.123701e+06  862912.388891  0.130771   \n",
      "74698032  -1.588846e+06  679836.503576  0.337615   \n",
      "100080967 -1.573747e+06  394370.239102  0.251695   \n",
      "97053201  -1.443046e+06  428437.452812  0.201525   \n",
      "76352393  -1.462958e+06  661245.808005  0.303006   \n",
      "91896567  -1.081519e+06  486474.395838  0.650062   \n",
      "81686979  -1.660189e+06  601227.115705  0.043912   \n",
      "70804633  -1.586487e+06  723623.725630  0.238328   \n",
      "73893097  -1.532131e+06  688895.928828  2.172502   \n",
      "\n",
      "                                  geometry  index_right   Sett_ID  ADM_ID  \n",
      "87079189   POINT (-1170602.898 540642.209)        10039   74882.0       8  \n",
      "58424421   POINT (-1123701.498 862912.389)        11213  133215.0     208  \n",
      "74698032   POINT (-1588846.363 679836.504)         6848   46509.0     183  \n",
      "100080967  POINT (-1573747.321 394370.239)         2760    9280.0       5  \n",
      "97053201   POINT (-1443046.238 428437.453)         3695   16246.0      38  \n",
      "76352393   POINT (-1462958.100 661245.808)         9442   68954.0     138  \n",
      "91896567   POINT (-1081518.549 486474.396)         6493   38279.0      60  \n",
      "81686979   POINT (-1660189.337 601227.116)        12850  196685.0     176  \n",
      "70804633   POINT (-1586487.138 723623.726)         6995   47354.0     184  \n",
      "73893097   POINT (-1532130.587 688895.929)         8278   59171.0     156  \n",
      "\n",
      "Exported as table, year 2002. Thu Dec  8 18:36:51 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 18:36:52 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2002. Thu Dec  8 18:36:52 2022 \n",
      "\n",
      "       Sett_ID   PopSum2002\n",
      "6587   39429.0   185.791478\n",
      "629     1749.0    73.667490\n",
      "9602   69267.0   521.577697\n",
      "5531   25897.0    60.102322\n",
      "5973   33886.0  4181.340435\n",
      "8374   59305.0   902.495055\n",
      "8436   60011.0   263.293943\n",
      "374     1103.0    81.836516\n",
      "10693  97836.0   320.612894\n",
      "3246   12999.0   309.265099\n",
      "\n",
      "Merged year 2002 onto latest year settlement feature layer. Thu Dec  8 18:36:52 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002\n",
      "849      2523.0      81    25.293187    29.490692    29.058309\n",
      "12630  195330.0     133  9127.201566  9007.615002  8881.259893\n",
      "8756    63250.0     110   269.400513   167.126549   155.904924\n",
      "2968    10413.0       5   143.869134   147.577606   157.643123\n",
      "2620     8661.0     166  2267.132714  2635.750967  2053.542976\n",
      "6999    48360.0     170  2163.914238  2084.263906  2096.766501\n",
      "3781    17006.0      37    21.435755    19.553956    20.466707\n",
      "7318    49982.0     292  4977.069508  5166.475820  5247.290093\n",
      "6434    38125.0      50    53.470106    69.670617    56.453284\n",
      "10018   74103.0     143   905.522737   918.656867   971.009265\n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2003. Thu Dec  8 18:36:52 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2003. Thu Dec  8 18:39:28 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2003. Thu Dec  8 18:42:29 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2003. Thu Dec  8 18:42:32 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2003. Thu Dec  8 18:53:31 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "101197847 -1.490986e+06  3.818192e+05  0.363024   \n",
      "105888592 -1.474377e+06  3.290669e+05  0.120589   \n",
      "8288803   -1.066420e+06  1.426767e+06  0.253015   \n",
      "91901549  -1.403223e+06  4.863800e+05  0.558720   \n",
      "99596418  -1.372836e+06  3.998436e+05  0.288774   \n",
      "46047639  -1.129081e+06  1.002107e+06  0.568933   \n",
      "66551849  -1.449369e+06  7.714688e+05  0.263148   \n",
      "38394281  -1.201367e+06  1.088171e+06  0.947264   \n",
      "89207317  -1.471263e+06  5.166725e+05  0.137083   \n",
      "67401773  -1.219769e+06  7.619375e+05  2.227901   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "101197847   POINT (-1490985.697 381819.160)          500    1386.0      17  \n",
      "105888592   POINT (-1474376.751 329066.882)          882    2592.0       7  \n",
      "8288803    POINT (-1066419.507 1426767.242)        12358  191338.0     263  \n",
      "91901549    POINT (-1403222.515 486380.027)         4190   19398.0     278  \n",
      "99596418    POINT (-1372835.693 399843.642)         4832   22765.0      91  \n",
      "46047639   POINT (-1129080.532 1002106.683)        11271  136037.0     205  \n",
      "66551849    POINT (-1449368.962 771468.815)        10693   97828.0     192  \n",
      "38394281   POINT (-1201367.196 1088171.223)        11552  146029.0       2  \n",
      "89207317    POINT (-1471262.573 516672.480)         2751    9251.0     107  \n",
      "67401773    POINT (-1219769.153 761937.545)        10843  111590.0     199  \n",
      "\n",
      "Exported as table, year 2003. Thu Dec  8 18:54:29 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 18:54:29 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2003. Thu Dec  8 18:54:30 2022 \n",
      "\n",
      "        Sett_ID  PopSum2003\n",
      "5826    28589.0  166.593444\n",
      "569      1581.0   35.049651\n",
      "12836  196618.0  121.654788\n",
      "6516    38333.0  272.258817\n",
      "9638    70783.0  140.836964\n",
      "10186   79583.0  850.750270\n",
      "12745  196200.0  808.831335\n",
      "2777     9332.0  255.850038\n",
      "3524    14771.0   93.108426\n",
      "8396    59579.0   57.705552\n",
      "\n",
      "Merged year 2003 onto latest year settlement feature layer. Thu Dec  8 18:54:30 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000  PopSum2001  PopSum2002  PopSum2003\n",
      "8598    63017.0     109  323.800037  338.980391  287.807350  320.309347\n",
      "3086    12524.0       9  290.882315  587.538302  524.019159  454.346220\n",
      "12583  193474.0     257  132.355197  120.648894  129.080596  131.864770\n",
      "8382    59322.0     270  713.595121  712.042171  757.403898  773.072367\n",
      "13123  198029.0     299   52.453613   61.023949   59.966129   53.377455\n",
      "854      2543.0      81   29.660429   38.888087   36.891486   43.538847\n",
      "9494    69022.0     140   96.627849   98.175277  112.690862  106.465172\n",
      "13562  201054.0     301  876.118406  746.860951  892.139806  879.421463\n",
      "3109    12551.0       9   50.429366  113.623792  103.403890   85.691244\n",
      "12796  196262.0     176  171.780676  212.634877  230.657275  230.532195\n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2004. Thu Dec  8 18:54:30 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2004. Thu Dec  8 18:57:03 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2004. Thu Dec  8 19:00:02 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2004. Thu Dec  8 19:00:05 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2004. Thu Dec  8 19:11:03 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "93898668  -1.397466e+06  4.639202e+05  4.278266   \n",
      "102172686 -1.351036e+06  3.708724e+05  0.332803   \n",
      "92665836  -1.336598e+06  4.777924e+05  0.138476   \n",
      "79867583  -1.523260e+06  6.217052e+05  1.950034   \n",
      "102489809 -1.514767e+06  3.672863e+05  0.144886   \n",
      "93049360  -1.568935e+06  4.734515e+05  0.260550   \n",
      "104623202 -1.318574e+06  3.433166e+05  0.052906   \n",
      "72576118  -1.493628e+06  7.037119e+05  1.623829   \n",
      "30649437  -1.196743e+06  1.175274e+06  3.983381   \n",
      "104169776 -1.348017e+06  3.484125e+05  0.716349   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "93898668    POINT (-1397466.005 463920.202)         3964   19120.0      12  \n",
      "102172686   POINT (-1351036.451 370872.355)         1619    4309.0      78  \n",
      "92665836    POINT (-1336597.992 477792.447)         5873   32210.0     106  \n",
      "79867583    POINT (-1523259.899 621705.192)         7161   49133.0       6  \n",
      "102489809   POINT (-1514766.688 367286.332)          268     823.0      16  \n",
      "93049360    POINT (-1568934.502 473451.472)         2985   10742.0      25  \n",
      "104623202   POINT (-1318573.510 343316.603)         1753    4654.0      78  \n",
      "72576118    POINT (-1493628.029 703711.864)        10515   89197.0     147  \n",
      "30649437   POINT (-1196743.114 1175273.822)        11724  156016.0     226  \n",
      "104169776   POINT (-1348016.642 348412.530)         1507    4169.0      74  \n",
      "\n",
      "Exported as table, year 2004. Thu Dec  8 19:12:01 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 19:12:02 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2004. Thu Dec  8 19:12:02 2022 \n",
      "\n",
      "        Sett_ID   PopSum2004\n",
      "10230   79655.0  1536.874172\n",
      "7294    49779.0    43.591407\n",
      "2906    10066.0    96.360612\n",
      "2052     6301.0   442.357613\n",
      "4421    20066.0    74.480588\n",
      "2649     9097.0   715.537527\n",
      "13514  199877.0  2667.558399\n",
      "5608    26615.0   438.088020\n",
      "6128    35320.0   105.192831\n",
      "13254  198668.0    69.209477\n",
      "\n",
      "Merged year 2004 onto latest year settlement feature layer. Thu Dec  8 19:12:02 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "10838  110961.0     195    33.250088    30.790648    29.911818    35.593071   \n",
      "12308  191228.0     257  1249.055731  1555.155458  1540.735645  1595.386936   \n",
      "5625    26972.0     103   305.125740   279.066570   272.448814   313.236331   \n",
      "13193  198318.0     298    38.359763    31.916904    32.171362    31.869795   \n",
      "11962  173651.0     255    46.570383    51.755437    67.239058    71.310573   \n",
      "15         18.0      14   165.165347   217.280761   197.804733   230.124528   \n",
      "13407  199233.0     298   220.951249   226.618409   228.628710   245.326510   \n",
      "8843    65054.0     110    19.326454    14.181748    13.539865    13.131552   \n",
      "267       813.0      15    60.822264    53.905943    47.169600    57.135367   \n",
      "7091    48472.0     172    59.682511    88.458036    82.119184    77.656535   \n",
      "\n",
      "        PopSum2004  \n",
      "10838    29.924287  \n",
      "12308  1632.845113  \n",
      "5625    284.646671  \n",
      "13193    32.818144  \n",
      "11962    61.490647  \n",
      "15      225.434913  \n",
      "13407   238.521004  \n",
      "8843     11.783431  \n",
      "267      58.368198  \n",
      "7091     74.404603  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2005. Thu Dec  8 19:12:02 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2005. Thu Dec  8 19:14:35 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2005. Thu Dec  8 19:17:33 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2005. Thu Dec  8 19:17:36 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2005. Thu Dec  8 19:28:37 2022 \n",
      "\n",
      "                      X              Y          Z  \\\n",
      "93401533  -1.592432e+06  469487.973489  25.683674   \n",
      "67553599  -1.145406e+06  760238.902693   0.038353   \n",
      "93334708  -1.563839e+06  470242.925593   0.207742   \n",
      "78583886  -1.511369e+06  636143.650534   1.016023   \n",
      "61555094  -1.045375e+06  827712.747024   0.042844   \n",
      "104906595 -1.497969e+06  340108.056598   0.121901   \n",
      "72886530  -1.498818e+06  700220.210394   0.945168   \n",
      "81520433  -1.539963e+06  603114.495966   1.682845   \n",
      "92261896  -1.447198e+06  482322.159263   0.192739   \n",
      "97472834  -1.435214e+06  423719.002160   0.332976   \n",
      "\n",
      "                                  geometry  index_right   Sett_ID  ADM_ID  \n",
      "93401533   POINT (-1592432.386 469487.973)         3083   12521.0       9  \n",
      "67553599   POINT (-1145406.371 760238.903)        10950  122105.0     200  \n",
      "93334708   POINT (-1563838.575 470242.926)         2708    9169.0      25  \n",
      "78583886   POINT (-1511369.404 636143.651)         7161   49133.0       6  \n",
      "61555094   POINT (-1045375.217 827712.747)        11003  122238.0     202  \n",
      "104906595  POINT (-1497969.004 340108.057)          236     702.0      21  \n",
      "72886530   POINT (-1498818.325 700220.210)        10579   90490.0     151  \n",
      "81520433   POINT (-1539963.215 603114.496)         7293   49751.0     292  \n",
      "92261896   POINT (-1447198.475 482322.159)         3955   18986.0      34  \n",
      "97472834   POINT (-1435213.610 423719.002)         3652   15956.0      38  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Exported as table, year 2005. Thu Dec  8 19:29:35 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 19:29:36 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2005. Thu Dec  8 19:29:36 2022 \n",
      "\n",
      "        Sett_ID   PopSum2005\n",
      "3462    14656.0   143.404916\n",
      "8409    59697.0   112.879680\n",
      "10723   99142.0   699.170236\n",
      "8560    61323.0   191.171702\n",
      "3649    15941.0    38.951498\n",
      "10532   89285.0    74.171582\n",
      "10440   85486.0   877.788448\n",
      "945      2668.0    84.568843\n",
      "10850  111619.0  1434.658957\n",
      "5332    24656.0   132.939086\n",
      "\n",
      "Merged year 2005 onto latest year settlement feature layer. Thu Dec  8 19:29:36 2022 \n",
      "\n",
      "       Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "2546    7845.0     167  1638.420614  1687.255548  1583.490713  1716.008143   \n",
      "8006   56283.0     267   434.211739   450.195530   468.482166   450.711706   \n",
      "9053   67069.0     286    12.579460    12.374531    11.008186    11.596449   \n",
      "2066    6319.0      44   111.596905   137.708425   139.663374   149.141015   \n",
      "7193   49169.0     289   350.093398   354.232267   346.122959   362.441954   \n",
      "9942   73436.0     141    24.429173    26.778067    31.197187    26.857027   \n",
      "10606  91459.0     147  3542.121686  3768.736819  3632.856275  3704.272342   \n",
      "3939   18322.0      33    81.543613    70.888468    74.869624    72.462852   \n",
      "3257   13012.0     124   460.959742   499.084408   503.679121   475.214059   \n",
      "4727   22533.0     271   257.343832   259.292096   247.464640   290.580695   \n",
      "\n",
      "        PopSum2004   PopSum2005  \n",
      "2546   1627.848401  1587.517017  \n",
      "8006    439.106367   388.796394  \n",
      "9053     11.534672    11.932073  \n",
      "2066    141.176219   157.003773  \n",
      "7193    350.954751   370.542215  \n",
      "9942     25.062269    27.408928  \n",
      "10606  3985.824299  4063.146963  \n",
      "3939     69.379019    69.994925  \n",
      "3257    451.070485   469.183624  \n",
      "4727    269.236467   265.583129  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2006. Thu Dec  8 19:29:36 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2006. Thu Dec  8 19:32:10 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2006. Thu Dec  8 19:35:10 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2006. Thu Dec  8 19:35:13 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2006. Thu Dec  8 19:46:16 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "30179386  -1.211370e+06  1.180558e+06  0.657416   \n",
      "75729816  -1.618006e+06  6.682291e+05  0.704050   \n",
      "83752542  -1.530243e+06  5.780123e+05  0.103801   \n",
      "22124577  -1.159373e+06  1.271153e+06  1.373110   \n",
      "110864719 -1.449463e+06  2.731061e+05  0.035011   \n",
      "29793332  -1.217787e+06  1.184899e+06  1.615384   \n",
      "92154655  -1.273371e+06  4.835490e+05  0.249365   \n",
      "33125277  -1.150030e+06  1.147435e+06  6.672367   \n",
      "73330894  -1.532697e+06  6.952187e+05  1.819059   \n",
      "99881927  -1.352546e+06  3.966351e+05  0.170491   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "30179386   POINT (-1211370.311 1180558.487)        11775  161203.0     226  \n",
      "75729816    POINT (-1618006.388 668229.115)         6779   46432.0     182  \n",
      "83752542    POINT (-1530243.206 578012.338)        12882  196747.0     317  \n",
      "22124577   POINT (-1159372.985 1271152.739)        12045  175718.0     229  \n",
      "110864719   POINT (-1449463.331 273106.057)          752    2075.0     324  \n",
      "29793332   POINT (-1217787.404 1184899.461)        11742  156280.0     226  \n",
      "92154655    POINT (-1273370.753 483548.956)         6036   33991.0     309  \n",
      "33125277   POINT (-1150030.453 1147434.963)        11678  155469.0     224  \n",
      "73330894    POINT (-1532696.801 695218.653)         8520   60793.0     156  \n",
      "99881927    POINT (-1352546.355 396635.095)         2019    5872.0      79  \n",
      "\n",
      "Exported as table, year 2006. Thu Dec  8 19:47:13 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 19:47:14 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2006. Thu Dec  8 19:47:14 2022 \n",
      "\n",
      "        Sett_ID   PopSum2006\n",
      "231       673.0    53.804854\n",
      "4059    19234.0   256.835003\n",
      "2860     9793.0    22.037402\n",
      "5209    24451.0    16.055307\n",
      "1635     4331.0   340.704428\n",
      "7164    49140.0   446.311359\n",
      "8159    57221.0    77.015032\n",
      "1850     5152.0   153.181271\n",
      "11246  135914.0  2443.566029\n",
      "3641    15885.0    23.361638\n",
      "\n",
      "Merged year 2006 onto latest year settlement feature layer. Thu Dec  8 19:47:14 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "7831    54716.0       6   218.226013   214.842091   196.846244   229.529852   \n",
      "1329     3745.0      19   129.006523   111.228063    96.058880   103.951635   \n",
      "11176  126383.0     210   137.284993   122.034577   117.767867   125.972722   \n",
      "632      1753.0      81   105.278795   109.274431   110.679096   121.772163   \n",
      "2497     7789.0     319   662.533787   859.045587   745.520742   664.720361   \n",
      "2677     9132.0       5   433.760435   434.101370   423.017377   444.304867   \n",
      "12374  191373.0     264   210.129448   212.666644   227.067890   216.276243   \n",
      "11544  145982.0     216  1097.783414  1084.989480  1038.157777  1104.776277   \n",
      "9616    70440.0     139   427.366174   386.168587   368.284317   410.520909   \n",
      "7531    51718.0     282   207.346913   178.983430   127.215249   139.380379   \n",
      "\n",
      "        PopSum2004  PopSum2005   PopSum2006  \n",
      "7831    210.939903  234.223543   209.461753  \n",
      "1329    108.275464  107.719797   112.397360  \n",
      "11176   133.056629  133.122061   142.100453  \n",
      "632     132.443457  302.559040   299.478402  \n",
      "2497    790.397815  773.449870   766.001384  \n",
      "2677    492.336753  485.334904   413.516252  \n",
      "12374   214.965147  224.252439   209.646184  \n",
      "11544  1162.942990  993.742270  1058.309372  \n",
      "9616    399.452272  428.228049   445.968723  \n",
      "7531    141.639459  281.826186   245.717570  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2007. Thu Dec  8 19:47:14 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2007. Thu Dec  8 19:49:47 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2007. Thu Dec  8 19:52:45 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2007. Thu Dec  8 19:52:48 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2007. Thu Dec  8 20:03:50 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "90271444  -1.615647e+06  5.046876e+05  1.912503   \n",
      "95424388  -1.533546e+06  4.467450e+05  0.315344   \n",
      "88080788  -1.672740e+06  5.293179e+05  0.101555   \n",
      "108457704 -1.336032e+06  3.001900e+05  0.125089   \n",
      "96474832  -1.385481e+06  4.349489e+05  0.435590   \n",
      "100384632 -1.423795e+06  3.909730e+05  0.301960   \n",
      "1902711   -1.117473e+06  1.498582e+06  0.266914   \n",
      "84590923  -1.598095e+06  5.685754e+05  0.822904   \n",
      "100132614 -1.450973e+06  3.938040e+05  0.087011   \n",
      "88568611  -1.564688e+06  5.238445e+05  0.244185   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "90271444    POINT (-1615647.163 504687.615)         3163   12897.0     120  \n",
      "95424388    POINT (-1533546.122 446745.041)         2918   10102.0       5  \n",
      "88080788    POINT (-1672740.416 529317.928)        12808  196316.0     173  \n",
      "108457704   POINT (-1336031.777 300189.964)          984    2707.0      73  \n",
      "96474832    POINT (-1385481.140 434948.915)         5221   24517.0      92  \n",
      "100384632   POINT (-1423794.960 390972.955)         4781   22706.0      87  \n",
      "1902711    POINT (-1117473.143 1498582.061)        12248  190919.0     327  \n",
      "84590923    POINT (-1598094.527 568575.437)         7031   48398.0     170  \n",
      "100132614   POINT (-1450973.235 393804.025)          573    1590.0      18  \n",
      "88568611    POINT (-1564687.896 523844.525)        12865  196723.0     115  \n",
      "\n",
      "Exported as table, year 2007. Thu Dec  8 20:04:47 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 20:04:48 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2007. Thu Dec  8 20:04:48 2022 \n",
      "\n",
      "        Sett_ID    PopSum2007\n",
      "12717  196164.0    537.716808\n",
      "7155    49128.0   2463.463651\n",
      "557      1534.0     40.818533\n",
      "10499   89180.0   3788.681353\n",
      "2158     6411.0    108.596776\n",
      "2460     7749.0  12606.975629\n",
      "10849  111617.0    411.722914\n",
      "11682  155540.0   1267.462555\n",
      "5334    24658.0    383.802802\n",
      "1708     4536.0     25.362771\n",
      "\n",
      "Merged year 2007 onto latest year settlement feature layer. Thu Dec  8 20:04:48 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000  PopSum2001  PopSum2002  PopSum2003  \\\n",
      "12582  193415.0     257  110.157068  139.237069  137.011704  139.073578   \n",
      "4342    19688.0     277   79.282610   85.758879   79.004989   86.992072   \n",
      "449      1283.0      17  148.208117  188.141477  203.774831  178.270579   \n",
      "13038  197768.0     300  151.111221  153.034175  167.935796  160.197884   \n",
      "9443    68955.0     140  366.797387  326.256952  316.051104  361.663828   \n",
      "4955    23147.0      87  178.414261  206.878464  226.329221  184.907073   \n",
      "5333    24657.0     103  290.934419  308.809741  311.606383  343.515671   \n",
      "11760  156483.0     225  655.802637  597.022508  597.312556  662.810307   \n",
      "12860  196717.0       9  219.798468  243.409923  248.539453  214.188306   \n",
      "13021  197717.0     276  186.448851  160.593196  179.833548  151.745377   \n",
      "\n",
      "       PopSum2004  PopSum2005  PopSum2006  PopSum2007  \n",
      "12582  139.552757  144.561122  147.093642  162.509792  \n",
      "4342    84.463304   78.739741   78.519831   83.420517  \n",
      "449    187.318905  216.971172  205.273339  211.288445  \n",
      "13038  148.686090  153.010002  145.218046  161.198592  \n",
      "9443   377.503139  353.056393  354.619251  400.270646  \n",
      "4955   182.842515  176.887182  172.749462  190.387165  \n",
      "5333   323.226827  354.515276  341.467175  325.457061  \n",
      "11760  719.450419  653.703747  699.192963  730.642716  \n",
      "12860  217.282689  209.538665  228.094064  221.610717  \n",
      "13021  210.317674  178.188118  223.909330  201.276985  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2008. Thu Dec  8 20:04:48 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2008. Thu Dec  8 20:07:20 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2008. Thu Dec  8 20:10:16 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2008. Thu Dec  8 20:10:19 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2008. Thu Dec  8 20:21:19 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "45374853  -1.271200e+06  1.009656e+06  0.264824   \n",
      "58677190  -1.025652e+06  8.600813e+05  0.078509   \n",
      "83990429  -1.252893e+06  5.753700e+05  0.182034   \n",
      "84022137  -1.428042e+06  5.749925e+05  0.520043   \n",
      "91104082  -1.433609e+06  4.953451e+05  1.029662   \n",
      "60118805  -1.180134e+06  8.438498e+05  0.235516   \n",
      "79934793  -1.515522e+06  6.209502e+05  0.926240   \n",
      "113554466 -1.012818e+06  2.429080e+05  0.253989   \n",
      "73532901  -1.473905e+06  6.929538e+05  0.133780   \n",
      "111771079 -1.437007e+06  2.629142e+05  0.134126   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "45374853   POINT (-1271200.265 1009656.204)        11398  143011.0     213  \n",
      "58677190    POINT (-1025652.094 860081.318)        11365  137717.0     203  \n",
      "83990429    POINT (-1252892.677 575370.006)         5902   32264.0      96  \n",
      "84022137    POINT (-1428041.565 574992.530)         8737   63215.0     316  \n",
      "91104082    POINT (-1433609.337 495345.083)         4502   20517.0     273  \n",
      "60118805    POINT (-1180134.168 843849.848)        11098  126142.0     209  \n",
      "79934793    POINT (-1515521.640 620950.239)         7161   49133.0       6  \n",
      "113554466   POINT (-1012817.908 242907.973)         2213    6689.0      44  \n",
      "73532901    POINT (-1473904.906 692953.796)         9848   72195.0     143  \n",
      "111771079   POINT (-1437006.621 262914.204)          713    1965.0      83  \n",
      "\n",
      "Exported as table, year 2008. Thu Dec  8 20:22:16 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 20:22:17 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2008. Thu Dec  8 20:22:17 2022 \n",
      "\n",
      "        Sett_ID   PopSum2008\n",
      "8006    56284.0   487.518131\n",
      "11780  163125.0  5039.708622\n",
      "379      1117.0   458.729266\n",
      "12079  182917.0  7022.286601\n",
      "606      1724.0   147.124943\n",
      "12989  197690.0    37.014581\n",
      "2880     9928.0   294.297562\n",
      "6070    34249.0    80.277029\n",
      "5458    25295.0    73.333566\n",
      "11853  169357.0  1139.360170\n",
      "\n",
      "Merged year 2008 onto latest year settlement feature layer. Thu Dec  8 20:22:17 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "3456    14650.0      30   189.036177   225.576518   210.302050   226.322910   \n",
      "1746     4628.0      74    95.865345    92.102134    89.670584   104.411406   \n",
      "3627    15771.0      39    94.912579   103.434442   110.174753    94.508303   \n",
      "7634    52880.0     282   173.450781   169.618821   171.457338   177.272752   \n",
      "5301    24610.0     103   123.719360   101.005684   107.433962   103.443809   \n",
      "11528  145880.0     216  1303.668222  1426.558244  1473.585403  1556.964894   \n",
      "4995    23331.0      88    64.552251    58.481828    53.951597    62.148509   \n",
      "5232    24528.0      92    58.173874    45.527539    45.016485    49.916102   \n",
      "9731    71304.0     138   156.249479   163.998963   173.336542   167.363790   \n",
      "5737    28289.0     128   190.755890   166.507077   175.302714   188.425468   \n",
      "\n",
      "        PopSum2004   PopSum2005   PopSum2006   PopSum2007   PopSum2008  \n",
      "3456    227.745338   235.887668   240.184106   237.563849   241.435788  \n",
      "1746    107.290849   102.058762   110.436221   124.282462   113.755546  \n",
      "3627     93.088524    90.593721    86.782222    91.728361    91.595185  \n",
      "7634    170.737059   186.596880   168.954713   184.820572   171.455695  \n",
      "5301     99.293315    91.540604    97.311200    96.789749    98.384471  \n",
      "11528  1631.314616  1552.873109  1602.683017  1724.327656  1870.552921  \n",
      "4995     61.233695    58.348876    60.912141    62.570926    57.066147  \n",
      "5232     46.581198    46.495098    49.292815    49.372614    51.664031  \n",
      "9731    172.602563   169.363589   168.935565   189.675819   176.158521  \n",
      "5737    172.282684   189.351391   182.479871   208.403537   230.138722  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2009. Thu Dec  8 20:22:17 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2009. Thu Dec  8 20:24:50 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2009. Thu Dec  8 20:27:47 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2009. Thu Dec  8 20:27:50 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2009. Thu Dec  8 20:38:47 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "69001986  -1.452672e+06  7.439131e+05  0.664100   \n",
      "97253603  -1.535717e+06  4.261726e+05  0.060165   \n",
      "27628828  -1.182493e+06  1.209247e+06  2.718627   \n",
      "89501754  -1.400297e+06  5.133696e+05  0.261665   \n",
      "86572792  -1.447765e+06  5.463044e+05  0.220444   \n",
      "107986920 -1.419831e+06  3.054746e+05  0.075442   \n",
      "21546831  -1.043016e+06  1.277664e+06  2.027624   \n",
      "103471320 -1.537038e+06  3.562452e+05  0.133602   \n",
      "17451326  -1.108791e+06  1.323716e+06  0.286904   \n",
      "92050078  -1.639994e+06  4.846814e+05  2.477328   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "69001986    POINT (-1452671.878 743913.063)        10667   97761.0     191  \n",
      "97253603    POINT (-1535716.609 426172.596)         2650    9098.0      26  \n",
      "27628828   POINT (-1182493.393 1209246.667)        12006  174975.0     227  \n",
      "89501754    POINT (-1400297.075 513369.565)         4272   19499.0     271  \n",
      "86572792    POINT (-1447764.689 546304.350)         8869   65285.0     109  \n",
      "107986920   POINT (-1419831.461 305474.629)          826    2439.0       7  \n",
      "21546831   POINT (-1043015.992 1277664.201)        11899  169843.0     255  \n",
      "103471320   POINT (-1537037.775 356245.158)          209     595.0      15  \n",
      "17451326   POINT (-1108791.194 1323716.280)        12219  188260.0     260  \n",
      "92050078    POINT (-1639994.368 484681.385)         2509    7803.0     319  \n",
      "\n",
      "Exported as table, year 2009. Thu Dec  8 20:39:44 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 20:39:44 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2009. Thu Dec  8 20:39:45 2022 \n",
      "\n",
      "        Sett_ID  PopSum2009\n",
      "6023    33973.0  794.573789\n",
      "13338  198926.0   18.439348\n",
      "3262    13018.0  340.313028\n",
      "12890  196884.0    2.289523\n",
      "12191  188128.0  882.529441\n",
      "9251    68735.0  614.922364\n",
      "1315     3725.0  103.968886\n",
      "10582   90650.0  620.398589\n",
      "5995    33917.0  545.668844\n",
      "10227   79647.0  510.906051\n",
      "\n",
      "Merged year 2009 onto latest year settlement feature layer. Thu Dec  8 20:39:45 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "11039  123956.0     201    97.015623   188.155949   188.513145   214.884888   \n",
      "3941    18330.0      33   149.487988   153.751184   150.906239   159.714941   \n",
      "2494     7785.0     167  1467.203429  1378.450127  1578.894753  1632.066036   \n",
      "10236   79663.0     163  1091.653618  1038.146083   963.436209   969.474238   \n",
      "7526    51713.0     282  1051.088561  1049.235119  1038.918253  1062.967418   \n",
      "11480  145641.0     218   173.782202   176.291782   189.571630   169.147916   \n",
      "13536  199898.0     301   554.931159   710.282439   710.997553   676.255757   \n",
      "13089  197940.0     299    32.859293    37.499902    35.391698    35.167772   \n",
      "990      2714.0      74   119.262021   138.728975   127.640500   164.613425   \n",
      "756      2096.0      81    31.345629    30.679989    30.413797    36.598071   \n",
      "\n",
      "        PopSum2004   PopSum2005   PopSum2006   PopSum2007   PopSum2008  \\\n",
      "11039   206.024577   204.742058   187.739500   260.650540   265.068353   \n",
      "3941    164.235792   162.711838   167.542264   170.177824   165.772996   \n",
      "2494   1632.500790  1640.218229  1754.646950  1802.787210  1752.400805   \n",
      "10236   961.542167   903.032434   944.706681   963.517700   867.917208   \n",
      "7526   1071.527374   934.222700   982.271393  1110.023048  1008.393531   \n",
      "11480   185.152148   192.704887   187.925606   222.406226   221.189274   \n",
      "13536   657.229304   439.241458   444.544897   453.912674   436.576870   \n",
      "13089    34.619548    35.353346    34.388060    39.074747    39.762503   \n",
      "990     154.399787   174.854791   178.023077   172.629420   185.143209   \n",
      "756      34.808266    52.144284    49.264421    50.355051    48.406986   \n",
      "\n",
      "        PopSum2009  \n",
      "11039   199.031129  \n",
      "3941    170.195594  \n",
      "2494   1855.702976  \n",
      "10236   870.153240  \n",
      "7526    985.992599  \n",
      "11480   198.493577  \n",
      "13536   414.865161  \n",
      "13089    35.506179  \n",
      "990     185.621323  \n",
      "756      51.315938  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2010. Thu Dec  8 20:39:45 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2010. Thu Dec  8 20:42:17 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2010. Thu Dec  8 20:45:14 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2010. Thu Dec  8 20:45:17 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2010. Thu Dec  8 20:56:14 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "79464965  -1.509105e+06  6.262349e+05  0.783713   \n",
      "105375744 -1.568463e+06  3.348234e+05  2.674244   \n",
      "63207533  -1.100864e+06  8.091221e+05  0.056628   \n",
      "101073663 -1.332351e+06  3.832347e+05  0.187392   \n",
      "93821599  -1.543738e+06  4.647695e+05  0.202552   \n",
      "76479522  -1.343676e+06  6.598303e+05  0.057565   \n",
      "22619688  -1.155409e+06  1.265585e+06  1.005828   \n",
      "97985178  -1.388690e+06  4.179625e+05  1.194914   \n",
      "87460306  -1.630086e+06  5.363012e+05  1.292169   \n",
      "5377235   -1.056133e+06  1.459513e+06  1.910453   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "79464965    POINT (-1509104.547 626234.904)         7161   49133.0       6  \n",
      "105375744   POINT (-1568462.657 334823.392)           35      43.0      11  \n",
      "63207533    POINT (-1100864.197 809122.051)        11113  126196.0     209  \n",
      "101073663   POINT (-1332351.386 383234.696)         1651    4354.0      78  \n",
      "93821599    POINT (-1543737.975 464769.523)         2974   10524.0       5  \n",
      "76479522    POINT (-1343675.668 659830.273)         5806   28439.0     126  \n",
      "22619688   POINT (-1155409.486 1265584.968)        11991  174889.0     232  \n",
      "97985178    POINT (-1388689.687 417962.492)         5222   24518.0      92  \n",
      "87460306    POINT (-1630085.622 536301.235)         6659   45239.0       4  \n",
      "5377235    POINT (-1056133.285 1459513.289)        12441  191589.0     264  \n",
      "\n",
      "Exported as table, year 2010. Thu Dec  8 20:57:12 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 20:57:12 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2010. Thu Dec  8 20:57:12 2022 \n",
      "\n",
      "        Sett_ID   PopSum2010\n",
      "1777     4759.0    97.465270\n",
      "5258    24559.0  2496.498304\n",
      "9646    70811.0  1912.847060\n",
      "5116    23969.0   177.790262\n",
      "6700    45296.0  2873.105316\n",
      "12369  191374.0   199.491311\n",
      "3942    18336.0   214.979408\n",
      "3416    14520.0   161.255177\n",
      "9359    68845.0   784.358254\n",
      "10035   74871.0   122.287063\n",
      "\n",
      "Merged year 2010 onto latest year settlement feature layer. Thu Dec  8 20:57:12 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000   PopSum2001  PopSum2002  PopSum2003  \\\n",
      "801      2337.0     324   57.044514    49.154199   52.621205   61.423661   \n",
      "5757    28327.0     128  151.410981   223.608810  223.222233  227.997655   \n",
      "10953  122109.0     200  359.379468   616.581383  609.849808  577.516183   \n",
      "10194   79589.0     162  432.671515   488.981095  510.314148  543.967306   \n",
      "4763    22688.0      88   83.943540    85.619139   83.263554   89.439395   \n",
      "8629    63075.0     109  299.666929   284.795898  238.061510  233.656356   \n",
      "6839    46498.0     183   92.233885   129.737123  119.296835  134.400645   \n",
      "5066    23695.0      91   92.883694   102.957527   95.475299  102.925232   \n",
      "5185    24347.0      93   46.463071    38.287097   37.641091   39.293462   \n",
      "10227   79643.0     162  966.621806  1027.780300  919.287619  939.875214   \n",
      "\n",
      "       PopSum2004  PopSum2005  PopSum2006   PopSum2007   PopSum2008  \\\n",
      "801     63.596655   58.408251   61.740360    63.318634    62.731374   \n",
      "5757   228.325279  227.916100  233.190381   240.548875   256.864207   \n",
      "10953  635.280285  542.903061  991.709782  1125.494002  1026.022173   \n",
      "10194  520.781428  550.385270  524.540548   569.494650   609.364605   \n",
      "4763    86.397767   86.709592   86.556780    95.048314    87.273967   \n",
      "8629   238.960010  210.690145  227.134906   231.632515   238.319022   \n",
      "6839   139.908366  118.987531  134.555661   137.916728   137.358672   \n",
      "5066    97.950985  105.184828  106.485232   115.750455   108.732078   \n",
      "5185    39.523036   41.817575   43.973864    46.464230    42.979608   \n",
      "10227  905.697745  904.936357  911.469748   964.265830   883.316377   \n",
      "\n",
      "        PopSum2009  PopSum2010  \n",
      "801      63.976400   58.827278  \n",
      "5757    250.034829  269.484917  \n",
      "10953  1115.621349  924.178061  \n",
      "10194   524.995425  590.533305  \n",
      "4763     99.705669   86.664226  \n",
      "8629    250.298178  530.355260  \n",
      "6839    140.462878  118.696571  \n",
      "5066    128.841258  104.959719  \n",
      "5185     46.189227   40.574710  \n",
      "10227   859.132043  863.993449  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2011. Thu Dec  8 20:57:12 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2011. Thu Dec  8 20:59:44 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2011. Thu Dec  8 21:02:39 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2011. Thu Dec  8 21:02:42 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2011. Thu Dec  8 21:13:40 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "93880358  -1.541662e+06  4.641089e+05  0.167772   \n",
      "4948805   -1.102280e+06  1.464326e+06  0.327101   \n",
      "99487719  -1.336598e+06  4.010704e+05  0.150358   \n",
      "84406361  -1.594320e+06  5.706516e+05  1.358079   \n",
      "110302096 -1.489665e+06  2.794288e+05  0.165302   \n",
      "89910989  -1.581863e+06  5.087455e+05  0.088081   \n",
      "80765382  -1.526846e+06  6.116077e+05  1.530100   \n",
      "87882239  -1.405204e+06  5.315828e+05  0.132864   \n",
      "89385988  -1.239115e+06  5.146907e+05  0.106337   \n",
      "98983914  -1.369155e+06  4.067326e+05  0.239062   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "93880358    POINT (-1541661.857 464108.940)         2700    9159.0       5  \n",
      "4948805    POINT (-1102279.732 1464326.109)        12456  191637.0     328  \n",
      "99487719    POINT (-1336597.992 401070.439)         5978   33892.0      79  \n",
      "84406361    POINT (-1594319.766 570651.555)         7034   48402.0     170  \n",
      "110302096   POINT (-1489664.531 279428.781)          633    1755.0     324  \n",
      "89910989    POINT (-1581863.056 508745.483)         3328   13470.0     119  \n",
      "80765382    POINT (-1526845.922 611607.707)         7160   49132.0     288  \n",
      "87882239    POINT (-1405204.264 531582.784)         5749   28313.0     129  \n",
      "89385988    POINT (-1239114.801 514690.731)         6274   36448.0      54  \n",
      "98983914    POINT (-1369155.301 406732.580)         4842   22775.0      91  \n",
      "\n",
      "Exported as table, year 2011. Thu Dec  8 21:14:38 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 21:14:38 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2011. Thu Dec  8 21:14:38 2022 \n",
      "\n",
      "        Sett_ID   PopSum2011\n",
      "9862    72529.0   127.881060\n",
      "1320     3734.0    75.401005\n",
      "1433     4021.0   138.670610\n",
      "5690    27612.0   105.736756\n",
      "11849  169301.0   299.725436\n",
      "42         54.0  1914.020168\n",
      "3927    18224.0    26.687475\n",
      "8151    57163.0   230.551595\n",
      "2983    10670.0   423.670685\n",
      "7339    50131.0   460.129293\n",
      "\n",
      "Merged year 2011 onto latest year settlement feature layer. Thu Dec  8 21:14:38 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000  PopSum2001  PopSum2002  PopSum2003  \\\n",
      "2751     9251.0     107  296.053440  278.354873  255.585851  304.158499   \n",
      "1259     3554.0       7    6.239213    6.876177    6.812180    7.681709   \n",
      "11         13.0      14  152.408934  192.816098  174.943112  222.322595   \n",
      "9821    71920.0     141   14.429020   14.265001   15.160708   15.371216   \n",
      "195       550.0      15   33.671163   43.465708   37.433449   42.052884   \n",
      "1937     5433.0      78   75.714038   60.660384   53.745490   72.734011   \n",
      "12472  191684.0     328  277.022909  225.975588  201.323053  238.022390   \n",
      "5746    28303.0     126  116.566420  143.470720  139.508000  159.386096   \n",
      "1845     5143.0      73   63.493904   73.029892   80.396594   78.467315   \n",
      "84         98.0      18  109.211805  103.267000   93.344080  112.569339   \n",
      "\n",
      "       PopSum2004  PopSum2005  PopSum2006  PopSum2007  PopSum2008  PopSum2009  \\\n",
      "2751   320.350454  273.173093  305.706080  315.179249  269.037301  312.823253   \n",
      "1259     8.589714    7.961667    8.524483    8.177774    8.268440    8.693334   \n",
      "11     213.738612  216.342133  239.900783  254.671616  250.617138  294.934272   \n",
      "9821    15.299452   15.768582   16.731587   16.253731   16.000596   17.938907   \n",
      "195     42.145663   42.409641   45.674244   46.107092   44.761950   48.502442   \n",
      "1937    71.844132   80.476128   95.720887   80.113914   78.385736   89.569874   \n",
      "12472  240.503016  250.687419  261.564827  326.266287  411.725648  399.730931   \n",
      "5746   167.187890  159.001122  167.178486  176.861860  182.029875  211.273638   \n",
      "1845    77.875372   87.787924   85.343026   83.520512   96.148090   87.595499   \n",
      "84     105.550814  119.795581  117.119317  113.148859  119.232706  133.676650   \n",
      "\n",
      "       PopSum2010  PopSum2011  \n",
      "2751   241.566658  297.903690  \n",
      "1259     8.552094    9.774266  \n",
      "11     275.019179  330.697231  \n",
      "9821    17.790701   17.420860  \n",
      "195     46.536622   55.973356  \n",
      "1937    67.828616   97.524977  \n",
      "12472  400.603674  389.693532  \n",
      "5746   175.177388  190.199252  \n",
      "1845   103.023091  108.401623  \n",
      "84     113.567253  133.346011  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2012. Thu Dec  8 21:14:38 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2012. Thu Dec  8 21:17:11 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2012. Thu Dec  8 21:20:08 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2012. Thu Dec  8 21:20:11 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2012. Thu Dec  8 21:31:10 2022 \n",
      "\n",
      "                      X             Y          Z  \\\n",
      "99285913  -1.376422e+06  4.033353e+05   0.315939   \n",
      "101627426 -1.336409e+06  3.770063e+05   0.407626   \n",
      "100091665 -1.356038e+06  3.942759e+05   0.220534   \n",
      "77769897  -1.517220e+06  6.452974e+05  28.306953   \n",
      "95333319  -1.417283e+06  4.477831e+05   0.199549   \n",
      "76712824  -1.499007e+06  6.571879e+05   2.309147   \n",
      "95199380  -1.387369e+06  4.492930e+05   0.428813   \n",
      "19632736  -1.132383e+06  1.299180e+06   6.770020   \n",
      "42640666  -1.150502e+06  1.040421e+06   0.242711   \n",
      "60151165  -1.293754e+06  8.434724e+05   0.113996   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "99285913    POINT (-1376421.715 403335.295)         5066   23695.0      91  \n",
      "101627426   POINT (-1336409.254 377006.341)         1635    4331.0      78  \n",
      "100091665   POINT (-1356038.008 394275.870)         1667    4375.0      79  \n",
      "77769897    POINT (-1517220.283 645297.445)         7161   49133.0       6  \n",
      "95333319    POINT (-1417283.498 447783.100)        13462  199642.0     298  \n",
      "76712824    POINT (-1499007.063 657187.940)         9242   68724.0     144  \n",
      "95199380    POINT (-1387368.521 449293.005)         5492   25559.0     265  \n",
      "19632736   POINT (-1132383.447 1299180.336)        12155  187858.0     260  \n",
      "42640666   POINT (-1150502.298 1040420.502)        11346  136379.0     206  \n",
      "60151165    POINT (-1293754.460 843472.372)        10906  116778.0     197  \n",
      "\n",
      "Exported as table, year 2012. Thu Dec  8 21:32:07 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 21:32:08 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2012. Thu Dec  8 21:32:08 2022 \n",
      "\n",
      "        Sett_ID   PopSum2012\n",
      "4770    22695.0   257.568163\n",
      "8059    56379.0   306.642651\n",
      "4385    19854.0    79.607663\n",
      "9709    71214.0    24.911938\n",
      "3359    13854.0   700.799020\n",
      "11423  145476.0  6703.079596\n",
      "5968    33825.0   470.357044\n",
      "2613     8562.0   420.921893\n",
      "9940    73436.0    27.046615\n",
      "11671  155468.0  3383.376107\n",
      "\n",
      "Merged year 2012 onto latest year settlement feature layer. Thu Dec  8 21:32:08 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000  PopSum2001  PopSum2002  PopSum2003  \\\n",
      "5052    23603.0      90  378.286585  445.577992  442.726474  407.727671   \n",
      "8158    57190.0     267  106.937025  112.814985  116.674636  117.396636   \n",
      "11401  143025.0     213         NaN         NaN         NaN         NaN   \n",
      "11717  155930.0     226  892.063677  777.514828  751.090921  841.371144   \n",
      "5127    24055.0      90  149.320895  141.882714  125.877870  133.123574   \n",
      "5401    24919.0      92   97.160025  133.946147  132.195534  147.487246   \n",
      "10110   75035.0      70   87.210973  137.323622  142.272188  135.794103   \n",
      "1759     4672.0      74  210.560445  183.998809  175.846673  197.331469   \n",
      "8428    59849.0     156  117.716398  139.805215  154.572024  175.793742   \n",
      "9967    73616.0     318   68.819974   70.868021   75.210938   73.375330   \n",
      "\n",
      "       PopSum2004  PopSum2005  PopSum2006  PopSum2007  PopSum2008  PopSum2009  \\\n",
      "5052   403.694100  351.669413  357.677228  354.075776  341.348114  365.923188   \n",
      "8158   109.029963   97.307412   99.293119  112.917844  121.788648  126.694252   \n",
      "11401         NaN         NaN         NaN         NaN         NaN         NaN   \n",
      "11717  861.424590  774.994596  904.587338  876.909797  859.709999  907.555306   \n",
      "5127   144.212121  138.170352  137.255266  131.481351  125.054800  139.252714   \n",
      "5401   139.898589  142.550633  138.077595  139.581705  150.889063  165.203383   \n",
      "10110  133.405076  131.274273  139.081512  123.453027  126.547482  142.072313   \n",
      "1759   206.616709  178.928249  185.538726  214.725818  194.260550  191.976731   \n",
      "8428   168.082044  180.207551  176.972457  181.140092  191.274687  221.755242   \n",
      "9967    75.094979   82.597754   81.813601   89.301133   83.378639   79.427259   \n",
      "\n",
      "       PopSum2010  PopSum2011  PopSum2012  \n",
      "5052   340.531405  380.792390  381.218828  \n",
      "8158   120.612793  131.560945  156.544858  \n",
      "11401         NaN         NaN         NaN  \n",
      "11717  823.071066  928.158759  980.616461  \n",
      "5127   135.830890  146.699405  128.917958  \n",
      "5401   164.957772  153.667114  166.024393  \n",
      "10110  133.220391  151.574900  162.763324  \n",
      "1759   179.186703  189.258272  193.573046  \n",
      "8428   181.130687  196.441336  196.348972  \n",
      "9967    83.310681   84.473803   74.396958  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2013. Thu Dec  8 21:32:08 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2013. Thu Dec  8 21:34:41 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2013. Thu Dec  8 21:37:37 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2013. Thu Dec  8 21:37:41 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2013. Thu Dec  8 21:49:16 2022 \n",
      "\n",
      "                      X             Y         Z  \\\n",
      "94696141  -1.366513e+06  4.549551e+05  0.159125   \n",
      "75445707  -1.506179e+06  6.714377e+05  5.243060   \n",
      "29988712  -9.925286e+05  1.182729e+06  0.220837   \n",
      "83495742  -1.216844e+06  5.809378e+05  6.363528   \n",
      "21697862  -1.043677e+06  1.275966e+06  4.098082   \n",
      "97793485  -1.266010e+06  4.201330e+05  0.159137   \n",
      "110167545 -1.517503e+06  2.809387e+05  0.282448   \n",
      "105326887 -1.427947e+06  3.353896e+05  0.346631   \n",
      "87277644  -1.447010e+06  5.383774e+05  0.298969   \n",
      "83814545  -1.222034e+06  5.773518e+05  0.981611   \n",
      "\n",
      "                                   geometry  index_right   Sett_ID  ADM_ID  \n",
      "94696141    POINT (-1366512.969 454955.145)         5653   27271.0     265  \n",
      "75445707    POINT (-1506179.108 671437.661)         9308   68792.0     144  \n",
      "29988712    POINT (-992528.570 1182728.974)        11855  169301.0     249  \n",
      "83495742    POINT (-1216843.714 580937.778)        10040   74883.0      67  \n",
      "21697862   POINT (-1043676.575 1275965.559)        11894  169831.0     255  \n",
      "97793485    POINT (-1266009.970 420132.980)         6371   36905.0      52  \n",
      "110167545   POINT (-1517503.390 280938.685)          670    1827.0     324  \n",
      "105326887   POINT (-1427947.196 335389.606)         1125    3025.0      19  \n",
      "87277644    POINT (-1447009.737 538377.353)         8812   64671.0     109  \n",
      "83814545    POINT (-1222034.010 577351.755)        10052   74896.0      67  \n",
      "\n",
      "Exported as table, year 2013. Thu Dec  8 21:50:18 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 21:50:18 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2013. Thu Dec  8 21:50:19 2022 \n",
      "\n",
      "      Sett_ID  PopSum2013\n",
      "6183  36279.0   81.949788\n",
      "183     528.0  280.629035\n",
      "5366  24754.0   36.384007\n",
      "1863   5207.0  129.610446\n",
      "6752  46405.0  264.366768\n",
      "5750  28315.0   95.625825\n",
      "5573  26222.0  445.036697\n",
      "3393  14186.0  300.729002\n",
      "1600   4288.0  602.396309\n",
      "4482  20410.0   57.390465\n",
      "\n",
      "Merged year 2013 onto latest year settlement feature layer. Thu Dec  8 21:50:19 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "1358     3804.0      76   174.380839   142.065188   136.274394   151.910175   \n",
      "10979  122151.0     201  1271.327385  1783.713683  1790.403621  1963.314617   \n",
      "12880  196743.0     317   153.264933   156.109873   143.696400   134.230046   \n",
      "11548  145993.0       2   325.967242   336.565453   341.966157   371.832815   \n",
      "13006  197701.0      85  2870.175856  3250.398380  3262.539616  3116.142437   \n",
      "452      1295.0      18   151.304264   161.998576   155.556916   168.235343   \n",
      "1494     4153.0      74   161.683398   179.473634   186.589590   180.458288   \n",
      "9126    67781.0     286    31.779840    32.099396    31.397280    31.641524   \n",
      "5995    33916.0     106   376.680693   441.335242   454.391915   450.706676   \n",
      "12503  191758.0     263   406.940038   321.480385   297.073303   351.061889   \n",
      "\n",
      "        PopSum2004   PopSum2005   PopSum2006   PopSum2007   PopSum2008  \\\n",
      "1358    155.164191   148.279313   150.244109   142.517723   155.647412   \n",
      "10979  1979.051982  2004.177902  1769.109500  1973.945857  2095.391906   \n",
      "12880   129.562951   130.688623   135.426423   127.748833   122.728539   \n",
      "11548   381.694378   350.618888   370.800261   393.707095   398.567219   \n",
      "13006  3364.161808  3615.622292  3576.628538  4638.038647  4976.772415   \n",
      "452     174.768797   163.264637   174.423694   181.315742   181.937138   \n",
      "1494    190.955109   182.690623   194.295106   225.574221   214.425056   \n",
      "9126     32.590058    36.677952    37.068440    36.065565    35.132921   \n",
      "5995    457.113336   438.251577   468.548391   475.788041   471.697281   \n",
      "12503   346.657622   368.204492   389.122904   405.564358   474.401078   \n",
      "\n",
      "        PopSum2009   PopSum2010   PopSum2011   PopSum2012   PopSum2013  \n",
      "1358    319.545583   349.623164   340.955486   311.139181   291.536531  \n",
      "10979  2115.078104  2031.184192  1947.934117  1788.218603  1933.269173  \n",
      "12880   136.896791   118.876137   236.364904   236.359684   229.178422  \n",
      "11548   442.827701   420.545989   533.740464   558.674792   678.909665  \n",
      "13006  4693.758213  4746.057850  4910.788223  3286.037689  3599.943766  \n",
      "452     189.009785   190.147788   219.177975   229.980627   212.372659  \n",
      "1494    186.481384   199.507370   197.112046   209.993568   211.119239  \n",
      "9126     38.224339    36.064164    32.049618    38.593353    30.848280  \n",
      "5995    476.246287   440.528447   490.126197   447.315160   463.842855  \n",
      "12503   450.744802   640.059567   627.744920   755.838163   690.186912  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2014. Thu Dec  8 21:50:19 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2014. Thu Dec  8 21:53:13 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2014. Thu Dec  8 21:56:25 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2014. Thu Dec  8 21:56:29 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2014. Thu Dec  8 22:11:41 2022 \n",
      "\n",
      "                     X              Y         Z  \\\n",
      "96146494 -1.488249e+06  438629.306222  0.292688   \n",
      "75520895 -1.537415e+06  670588.340297  2.020557   \n",
      "72248139 -1.562517e+06  707392.255386  1.316793   \n",
      "74942134 -1.516843e+06  677099.802197  1.335178   \n",
      "94334222 -1.470885e+06  459013.013040  0.397369   \n",
      "87051286 -1.428230e+06  540925.316367  2.055139   \n",
      "92798794 -1.459089e+06  476282.542428  0.260160   \n",
      "90498276 -1.589696e+06  502139.652004  0.563185   \n",
      "47188904 -1.120682e+06  989272.497364  0.727111   \n",
      "89300432 -1.394446e+06  515634.420870  0.182080   \n",
      "\n",
      "                                 geometry  index_right   Sett_ID  ADM_ID  \n",
      "96146494  POINT (-1488248.996 438629.306)         3757   16859.0      31  \n",
      "75520895  POINT (-1537415.251 670588.340)         8196   57668.0     269  \n",
      "72248139  POINT (-1562517.409 707392.255)        10184   79579.0     160  \n",
      "74942134  POINT (-1516842.806 677099.802)         8242   58195.0     267  \n",
      "94334222  POINT (-1470885.097 459013.013)         3874   17809.0      35  \n",
      "87051286  POINT (-1428230.303 540925.316)         8815   64721.0     110  \n",
      "92798794  POINT (-1459088.970 476282.542)         3508   14714.0      33  \n",
      "90498276  POINT (-1589695.685 502139.652)         3192   12934.0     119  \n",
      "47188904  POINT (-1120681.690 989272.497)        11239  135904.0     205  \n",
      "89300432  POINT (-1394446.197 515634.421)         5819   28512.0     128  \n",
      "\n",
      "Exported as table, year 2014. Thu Dec  8 22:13:56 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:13:57 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2014. Thu Dec  8 22:13:57 2022 \n",
      "\n",
      "       Sett_ID   PopSum2014\n",
      "7757   54157.0   206.962325\n",
      "3201   12944.0  5050.878923\n",
      "10039  74884.0   761.339957\n",
      "2722    9185.0    98.336291\n",
      "4257   19475.0   201.515451\n",
      "5120   24025.0     8.671898\n",
      "5130   24063.0    37.685632\n",
      "2043    6144.0   118.043632\n",
      "8746   63230.0   207.436192\n",
      "4889   22843.0   162.755364\n",
      "\n",
      "Merged year 2014 onto latest year settlement feature layer. Thu Dec  8 22:13:57 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  PopSum2000  PopSum2001  PopSum2002  PopSum2003  \\\n",
      "13175  198210.0     299  114.508766  105.739483  125.308854   86.083421   \n",
      "8956    66361.0     316  185.810010  177.216214  151.252029  183.299096   \n",
      "3220    12966.0     123  484.227015  531.324139  569.026704  500.301034   \n",
      "5559    26131.0     265   20.455822   18.955035   18.163507   18.203476   \n",
      "12575  192851.0     327  206.867524  145.416377  130.551576  160.828339   \n",
      "10709   97882.0     188  771.718902  560.283095  548.670123  553.871237   \n",
      "12565  191877.0     303   48.821957   44.141165   45.054410   49.307825   \n",
      "1919     5329.0      80  124.971055  222.141713  224.384006  215.946216   \n",
      "527      1440.0      17   59.453866   70.677965   77.139511   70.193693   \n",
      "2330     7294.0      46   18.893784   22.208857   17.761134   24.412298   \n",
      "\n",
      "       PopSum2004  PopSum2005  PopSum2006  PopSum2007  PopSum2008  PopSum2009  \\\n",
      "13175   89.306840   90.559354   80.584317   79.537701   85.378911   74.646682   \n",
      "8956   181.297624  171.613470  176.156502  177.031757  180.857389  203.730858   \n",
      "3220   482.632137  455.929525  486.778313  480.378832  463.775679  476.056945   \n",
      "5559    17.683959   16.142488   16.166262   16.231676   16.356180   16.427739   \n",
      "12575  157.657843  168.667581  171.485861  247.515846  303.450761  299.321229   \n",
      "10709  950.606131  827.265845  783.567156  765.437910  867.918711  894.680011   \n",
      "12565   49.658673   51.116178   53.333082   54.544558   53.651992   56.489605   \n",
      "1919   235.387718  203.416617  205.542016  243.464319  227.013417  209.594950   \n",
      "527     70.589365   73.258138   78.358172   78.008567   81.418059   84.825048   \n",
      "2330    23.652870   59.866578   55.550143   59.960110   63.748074   59.632511   \n",
      "\n",
      "        PopSum2010  PopSum2011   PopSum2012  PopSum2013  PopSum2014  \n",
      "13175    96.905471   70.118970    56.350358   64.540661   51.146182  \n",
      "8956    144.491949  171.511644   227.019848  203.458357  195.443802  \n",
      "3220    447.900708  482.838947   485.027883  520.834309  447.230381  \n",
      "5559     15.270271   16.973781    25.482149   23.585577   21.464843  \n",
      "12575   282.209364  302.500785   372.234315  333.030720  372.160655  \n",
      "10709  1026.957928  837.065206  1029.548850  839.117368  956.581105  \n",
      "12565    52.276024   89.448804   109.846725  125.676638  109.381838  \n",
      "1919    196.382665  220.350096   231.438612  235.473736  226.371324  \n",
      "527      97.655148   92.827890    85.233524   82.856890   86.534835  \n",
      "2330     63.373521   71.965047    70.113943   70.030531   70.239971  \n",
      "\n",
      "\n",
      "\n",
      "Loading data for year 2015. Thu Dec  8 22:13:57 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2015. Thu Dec  8 22:19:41 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2015. Thu Dec  8 22:22:39 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2015. Thu Dec  8 22:22:43 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2015. Thu Dec  8 22:34:08 2022 \n",
      "\n",
      "                     X             Y         Z  \\\n",
      "90892599 -1.594792e+06  4.977043e+05  0.768722   \n",
      "82825676 -1.102280e+06  5.884873e+05  0.543422   \n",
      "87787058 -1.676987e+06  5.326208e+05  0.091943   \n",
      "75017915 -1.492118e+06  6.762505e+05  0.963208   \n",
      "37622354 -1.197121e+06  1.096853e+06  5.164711   \n",
      "97061803 -1.423134e+06  4.283431e+05  0.136059   \n",
      "8121070  -1.058209e+06  1.428655e+06  1.602315   \n",
      "77057203 -1.466167e+06  6.533188e+05  0.412987   \n",
      "8792167  -1.075479e+06  1.421105e+06  0.535933   \n",
      "98374999 -1.026785e+06  4.136215e+05  0.296255   \n",
      "\n",
      "                                  geometry  index_right   Sett_ID  ADM_ID  \n",
      "90892599   POINT (-1594791.611 497704.308)         3186   12924.0     118  \n",
      "82825676   POINT (-1102279.732 588487.299)        10046   74890.0      70  \n",
      "87787058   POINT (-1676987.022 532620.843)        12739  196183.0     178  \n",
      "75017915   POINT (-1492118.125 676250.481)         9327   68811.0     143  \n",
      "37622354  POINT (-1197120.590 1096853.172)        11572  146121.0       2  \n",
      "97061803   POINT (-1423134.376 428343.084)        13137  198077.0     299  \n",
      "8121070   POINT (-1058209.403 1428654.622)        12366  191355.0     264  \n",
      "77057203   POINT (-1466166.646 653318.811)         9597   69166.0     138  \n",
      "8792167   POINT (-1075478.932 1421105.101)        12344  191311.0     257  \n",
      "98374999   POINT (-1026784.522 413621.518)         2135    6388.0      46  \n",
      "\n",
      "Exported as table, year 2015. Thu Dec  8 22:35:09 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:35:09 2022 \n",
      "\n",
      "\n",
      "Values aggregated to settlement level, year 2015. Thu Dec  8 22:35:09 2022 \n",
      "\n",
      "        Sett_ID    PopSum2015\n",
      "2364     7367.0     22.606146\n",
      "2471     7760.0   4021.916022\n",
      "5272    24576.0     87.819747\n",
      "7641    52988.0     19.916737\n",
      "11434  145488.0  17381.217869\n",
      "2526     7824.0   1604.398652\n",
      "4370    19804.0     98.841195\n",
      "8681    63143.0    380.877364\n",
      "9023    66540.0    116.391441\n",
      "3632    15815.0     74.116524\n",
      "\n",
      "Merged year 2015 onto latest year settlement feature layer. Thu Dec  8 22:35:09 2022 \n",
      "\n",
      "       Sett_ID  ADM_ID   PopSum2000   PopSum2001   PopSum2002   PopSum2003  \\\n",
      "132      278.0      14    49.621753    65.199283    61.488151    60.066116   \n",
      "8497   60607.0     157   125.706544   121.876690   120.611414   134.536142   \n",
      "10251  79695.0     163  2133.220868  2205.549746  2076.609597  2019.079465   \n",
      "1555    4229.0      41   181.806864   219.648713   245.836608   237.880702   \n",
      "10563  89772.0     143   219.412091   163.731791   285.299599   241.589898   \n",
      "1423    3991.0      22   226.953288   243.985914   293.599190   251.236167   \n",
      "10157  76340.0      67   623.082971   635.647007  1170.816895  1260.848190   \n",
      "3042   11109.0      27    58.381570    54.040640    55.181075    57.249938   \n",
      "3974   19130.0     277   236.991882   261.177636   227.579403   216.013244   \n",
      "5215   24489.0      93    54.748021    43.495182    41.261075    44.259579   \n",
      "\n",
      "        PopSum2004   PopSum2005   PopSum2006   PopSum2007   PopSum2008  \\\n",
      "132      72.114316    73.787221    64.049655    90.495086    87.014424   \n",
      "8497    133.069541   138.938888   141.390505   133.960726   137.062423   \n",
      "10251  1976.672747  1651.079103  1706.086034  1747.817381  1608.547025   \n",
      "1555    226.880159   290.602502   252.563870   249.276486   286.660822   \n",
      "10563   263.086238   230.772312   270.460971   299.207631   272.216091   \n",
      "1423    247.007712   268.032797   266.567517   260.347954   294.885890   \n",
      "10157  1289.633215  1523.075567  1514.070253  1517.230314  1610.608137   \n",
      "3042     58.055145    57.976081    60.475747    63.470802    61.711863   \n",
      "3974    213.267550   184.998520   182.925941   181.019826   183.191848   \n",
      "5215     46.101400    44.717109    46.724073    47.142810    46.661161   \n",
      "\n",
      "        PopSum2009   PopSum2010   PopSum2011   PopSum2012   PopSum2013  \\\n",
      "132      80.677128    78.373379    96.141188   126.450226   359.083944   \n",
      "8497    175.286164   143.066982   147.703019   145.445510   147.431952   \n",
      "10251  1650.511265  1511.281363  1641.840466  1686.871946  1788.797913   \n",
      "1555    287.408492   339.825910   326.093398   329.776084   315.451420   \n",
      "10563   232.990350   247.323486   229.282778   289.159375   266.790705   \n",
      "1423    271.130442   288.676188   296.294532   268.687122   262.289447   \n",
      "10157  1387.708066  1236.761613  1046.703183  1118.189386   944.917016   \n",
      "3042     58.938045    59.970395    57.009495    49.537229    52.314052   \n",
      "3974    191.240964   167.153978   181.047760   193.962634   166.301902   \n",
      "5215     50.764717    44.671234    48.448653    58.150347    55.624833   \n",
      "\n",
      "        PopSum2014   PopSum2015  \n",
      "132     414.944765   455.244565  \n",
      "8497    169.696325   173.232370  \n",
      "10251  1612.158031  1576.520862  \n",
      "1555    316.004421   378.566325  \n",
      "10563   526.932289   557.240821  \n",
      "1423    264.886865   258.982924  \n",
      "10157  1048.501611  1203.059786  \n",
      "3042     70.365729    65.754804  \n",
      "3974    143.856348   166.539598  \n",
      "5215     73.246728    70.776691  \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Finished. All years masked and assigned their nearest settlement. Thu Dec  8 22:35:10 2022\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Thu Dec  8 22:35:10 2022 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for YearFile in AnnualizedMaskedFiles:\n",
    "    \n",
    "### STEP 1: TIF TO XYZ ###\n",
    "    InputRasterName = os.path.join(ProjectFolder, \"Population\", YearFile)\n",
    "    Year = str(re.sub(r'[^0-9]', '', YearFile))\n",
    "    print('Loading data for year %s. %s \\n' % (Year, time.ctime()))\n",
    "    InputRasterObject = gdal.Open(InputRasterName)\n",
    "    XYZOutputPath = r'Population/{}'.format(\n",
    "        YearFile.replace('.tif', '.xyz')) # New file path will be the same as original, but .tif is replaced with .xyz\n",
    "    \n",
    "    # Create an .xyz version of the .tif\n",
    "    XYZ = gdal.Translate(XYZOutputPath, # Specify a destination path\n",
    "                         InputRasterObject, # Input is the masked .tif file\n",
    "                         format='XYZ', \n",
    "                         creationOptions=[\"ADD_HEADER_LINE=YES\"])\n",
    "    print('Finished gdal.Translate() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "\n",
    "#     # Remove the temporary masked tif file.\n",
    "#     try:  \n",
    "#         os.remove(InputRasterName)\n",
    "#     except OSError:\n",
    "#         pass\n",
    "#     print('Removed (or skipped if error) intermediate tif file. %s \\n' % time.ctime())\n",
    "    \n",
    "    InputRasterObject = None\n",
    "    XYZ = None # Reload XYZ as a point geodataframe\n",
    "\n",
    "    \n",
    "### STEP 2: GENERATE GEODATAFRAME WITH SETT_ID FIELD ###\n",
    "    InputXYZName = ''.join(['Masked_', Year, '.xyz'])\n",
    "    InputXYZ = pd.read_table(os.path.join(ProjectFolder, 'Population', InputXYZName), delim_whitespace=True)\n",
    "    InputXYZ = InputXYZ.loc[InputXYZ['Z'] != NoDataVal] # Subset to only the features that have a raster value.\n",
    "    print('Loaded XYZ file as a pandas dataframe, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    ValObject = gpd.GeoDataFrame(InputXYZ,\n",
    "                                 geometry = gpd.points_from_xy(InputXYZ['X'], InputXYZ['Y']),\n",
    "                                 crs = 'ESRI:102022')\n",
    "    print('Created geodataframe from non-NoData points, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    del InputXYZ\n",
    "    \n",
    "    # Sjoin_nearest: No need to group by ADM this time. \n",
    "    ValObject_withID = gpd.sjoin_nearest(ValObject, \n",
    "                                    Settlements, \n",
    "                                    how='left') # No need for max_distance parameter this time. We've already narrowed down to nearby raster cells.\n",
    "    \n",
    "    print('\\nJoined settlement ID onto vectorized raster cells for year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValObject_withID.sample(10))\n",
    "    del ValObject\n",
    "    \n",
    "    # We no longer need the spatial information of the raster values because we have their unique settlement ID.\n",
    "    ValObject_withID = pd.DataFrame(ValObject_withID).drop(columns='geometry')\n",
    "    \n",
    "    ValObject_withID.to_csv(''.join([r'Population/', 'Masked_', Year, '.csv']))\n",
    "    print('\\nExported as table, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    \n",
    "    # Remove the temporary xyz file.\n",
    "    try:  \n",
    "        os.remove(os.path.join(ProjectFolder, 'Population', InputXYZName))\n",
    "    except OSError:\n",
    "        pass\n",
    "    print('Removed (or skipped if error) intermediate xyz file. %s \\n' % time.ctime())\n",
    "\n",
    "    \n",
    "\n",
    "### STEP 3: AGGREGATE BY SETTLEMENT AND MERGE ONTO SUMMARIES TABLE ###\n",
    "    VariableName = ''.join(['PopSum', Year])\n",
    "    \n",
    "    ValAggregated = ValObject_withID.groupby('Sett_ID', \n",
    "                                      as_index=False)['Z'].sum().rename(columns={\"Z\": VariableName})\n",
    "    print('\\nValues aggregated to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValAggregated.sample(10))\n",
    "    \n",
    "    AllSummaries = AllSummaries.merge(ValAggregated, how='left', on='Sett_ID')\n",
    "    print('\\nMerged year %s onto latest year settlement feature layer. %s \\n' % (Year, time.ctime()))\n",
    "    print(AllSummaries.sample(10))\n",
    "    \n",
    "    del ValObject_withID, ValAggregated\n",
    "    print('\\n\\n')\n",
    "    \n",
    "\n",
    "print('\\n\\nFinished. All years masked and assigned their nearest settlement. %s' % time.ctime())\n",
    "\n",
    "AllSummaries.to_csv(os.path.join(ResultsFolder, 'Pop%sto%s.csv' % (2000, 2015)))\n",
    "print('Saved to file. %s \\n' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "7fa7e159",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>PopSum2000</th>\n",
       "      <th>PopSum2001</th>\n",
       "      <th>PopSum2002</th>\n",
       "      <th>PopSum2003</th>\n",
       "      <th>PopSum2004</th>\n",
       "      <th>PopSum2005</th>\n",
       "      <th>PopSum2006</th>\n",
       "      <th>PopSum2007</th>\n",
       "      <th>PopSum2008</th>\n",
       "      <th>PopSum2009</th>\n",
       "      <th>PopSum2010</th>\n",
       "      <th>PopSum2011</th>\n",
       "      <th>PopSum2012</th>\n",
       "      <th>PopSum2013</th>\n",
       "      <th>PopSum2014</th>\n",
       "      <th>PopSum2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3964</th>\n",
       "      <td>19120.0</td>\n",
       "      <td>12</td>\n",
       "      <td>1.371739e+06</td>\n",
       "      <td>1.433621e+06</td>\n",
       "      <td>1.532100e+06</td>\n",
       "      <td>1.611711e+06</td>\n",
       "      <td>1.677785e+06</td>\n",
       "      <td>1.768640e+06</td>\n",
       "      <td>1.869954e+06</td>\n",
       "      <td>1.947824e+06</td>\n",
       "      <td>2.055855e+06</td>\n",
       "      <td>2.155441e+06</td>\n",
       "      <td>2.260273e+06</td>\n",
       "      <td>2.370697e+06</td>\n",
       "      <td>2.559540e+06</td>\n",
       "      <td>2.697901e+06</td>\n",
       "      <td>2.856110e+06</td>\n",
       "      <td>2.999944e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3083</th>\n",
       "      <td>12521.0</td>\n",
       "      <td>9</td>\n",
       "      <td>7.554348e+05</td>\n",
       "      <td>7.229169e+05</td>\n",
       "      <td>8.067938e+05</td>\n",
       "      <td>8.695919e+05</td>\n",
       "      <td>8.789665e+05</td>\n",
       "      <td>9.419342e+05</td>\n",
       "      <td>1.003072e+06</td>\n",
       "      <td>9.917892e+05</td>\n",
       "      <td>1.040045e+06</td>\n",
       "      <td>1.135986e+06</td>\n",
       "      <td>1.178489e+06</td>\n",
       "      <td>1.235285e+06</td>\n",
       "      <td>1.295360e+06</td>\n",
       "      <td>1.351160e+06</td>\n",
       "      <td>1.472478e+06</td>\n",
       "      <td>1.600271e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13493</th>\n",
       "      <td>199779.0</td>\n",
       "      <td>12</td>\n",
       "      <td>4.813840e+05</td>\n",
       "      <td>4.957936e+05</td>\n",
       "      <td>5.327978e+05</td>\n",
       "      <td>5.885219e+05</td>\n",
       "      <td>5.925901e+05</td>\n",
       "      <td>6.358964e+05</td>\n",
       "      <td>6.710046e+05</td>\n",
       "      <td>7.193855e+05</td>\n",
       "      <td>7.252042e+05</td>\n",
       "      <td>7.640128e+05</td>\n",
       "      <td>8.210750e+05</td>\n",
       "      <td>8.430945e+05</td>\n",
       "      <td>9.956889e+05</td>\n",
       "      <td>1.077169e+06</td>\n",
       "      <td>1.116253e+06</td>\n",
       "      <td>1.144993e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13494</th>\n",
       "      <td>199780.0</td>\n",
       "      <td>12</td>\n",
       "      <td>4.813840e+05</td>\n",
       "      <td>4.957936e+05</td>\n",
       "      <td>5.327978e+05</td>\n",
       "      <td>5.885219e+05</td>\n",
       "      <td>5.925901e+05</td>\n",
       "      <td>6.358964e+05</td>\n",
       "      <td>6.710046e+05</td>\n",
       "      <td>7.193855e+05</td>\n",
       "      <td>7.252042e+05</td>\n",
       "      <td>7.640128e+05</td>\n",
       "      <td>8.210750e+05</td>\n",
       "      <td>8.430945e+05</td>\n",
       "      <td>9.956889e+05</td>\n",
       "      <td>1.077169e+06</td>\n",
       "      <td>1.116253e+06</td>\n",
       "      <td>1.144993e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13495</th>\n",
       "      <td>199782.0</td>\n",
       "      <td>12</td>\n",
       "      <td>4.813840e+05</td>\n",
       "      <td>4.957936e+05</td>\n",
       "      <td>5.327978e+05</td>\n",
       "      <td>5.885219e+05</td>\n",
       "      <td>5.925901e+05</td>\n",
       "      <td>6.358964e+05</td>\n",
       "      <td>6.710046e+05</td>\n",
       "      <td>7.193855e+05</td>\n",
       "      <td>7.252042e+05</td>\n",
       "      <td>7.640128e+05</td>\n",
       "      <td>8.210750e+05</td>\n",
       "      <td>8.430945e+05</td>\n",
       "      <td>9.956889e+05</td>\n",
       "      <td>1.077169e+06</td>\n",
       "      <td>1.116253e+06</td>\n",
       "      <td>1.144993e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13496</th>\n",
       "      <td>199787.0</td>\n",
       "      <td>12</td>\n",
       "      <td>4.813840e+05</td>\n",
       "      <td>4.957936e+05</td>\n",
       "      <td>5.327978e+05</td>\n",
       "      <td>5.885219e+05</td>\n",
       "      <td>5.925901e+05</td>\n",
       "      <td>6.358964e+05</td>\n",
       "      <td>6.710046e+05</td>\n",
       "      <td>7.193855e+05</td>\n",
       "      <td>7.252042e+05</td>\n",
       "      <td>7.640128e+05</td>\n",
       "      <td>8.210750e+05</td>\n",
       "      <td>8.430945e+05</td>\n",
       "      <td>9.956889e+05</td>\n",
       "      <td>1.077169e+06</td>\n",
       "      <td>1.116253e+06</td>\n",
       "      <td>1.144993e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3135</th>\n",
       "      <td>12674.0</td>\n",
       "      <td>9</td>\n",
       "      <td>3.434185e+05</td>\n",
       "      <td>3.265064e+05</td>\n",
       "      <td>3.668424e+05</td>\n",
       "      <td>3.962720e+05</td>\n",
       "      <td>3.954759e+05</td>\n",
       "      <td>4.298850e+05</td>\n",
       "      <td>4.585172e+05</td>\n",
       "      <td>4.555575e+05</td>\n",
       "      <td>4.595139e+05</td>\n",
       "      <td>5.213460e+05</td>\n",
       "      <td>5.465248e+05</td>\n",
       "      <td>5.724127e+05</td>\n",
       "      <td>6.236011e+05</td>\n",
       "      <td>6.653021e+05</td>\n",
       "      <td>7.385664e+05</td>\n",
       "      <td>7.991168e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3105</th>\n",
       "      <td>12544.0</td>\n",
       "      <td>9</td>\n",
       "      <td>3.430208e+05</td>\n",
       "      <td>3.262101e+05</td>\n",
       "      <td>3.664446e+05</td>\n",
       "      <td>3.958616e+05</td>\n",
       "      <td>3.950851e+05</td>\n",
       "      <td>4.295021e+05</td>\n",
       "      <td>4.580760e+05</td>\n",
       "      <td>4.551785e+05</td>\n",
       "      <td>4.589896e+05</td>\n",
       "      <td>5.208793e+05</td>\n",
       "      <td>5.459382e+05</td>\n",
       "      <td>5.719066e+05</td>\n",
       "      <td>6.229827e+05</td>\n",
       "      <td>6.646893e+05</td>\n",
       "      <td>7.379481e+05</td>\n",
       "      <td>7.982845e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3137</th>\n",
       "      <td>12684.0</td>\n",
       "      <td>9</td>\n",
       "      <td>3.428078e+05</td>\n",
       "      <td>3.259993e+05</td>\n",
       "      <td>3.662116e+05</td>\n",
       "      <td>3.956150e+05</td>\n",
       "      <td>3.948304e+05</td>\n",
       "      <td>4.292218e+05</td>\n",
       "      <td>4.577869e+05</td>\n",
       "      <td>4.548766e+05</td>\n",
       "      <td>4.587047e+05</td>\n",
       "      <td>5.205358e+05</td>\n",
       "      <td>5.455942e+05</td>\n",
       "      <td>5.715448e+05</td>\n",
       "      <td>6.225841e+05</td>\n",
       "      <td>6.642601e+05</td>\n",
       "      <td>7.374647e+05</td>\n",
       "      <td>7.977807e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3133</th>\n",
       "      <td>12670.0</td>\n",
       "      <td>9</td>\n",
       "      <td>3.427151e+05</td>\n",
       "      <td>3.258997e+05</td>\n",
       "      <td>3.661126e+05</td>\n",
       "      <td>3.955014e+05</td>\n",
       "      <td>3.947171e+05</td>\n",
       "      <td>4.290967e+05</td>\n",
       "      <td>4.576536e+05</td>\n",
       "      <td>4.547388e+05</td>\n",
       "      <td>4.585783e+05</td>\n",
       "      <td>5.203804e+05</td>\n",
       "      <td>5.454478e+05</td>\n",
       "      <td>5.713789e+05</td>\n",
       "      <td>6.224143e+05</td>\n",
       "      <td>6.640756e+05</td>\n",
       "      <td>7.372510e+05</td>\n",
       "      <td>7.975707e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3149</th>\n",
       "      <td>12883.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2.144976e+05</td>\n",
       "      <td>2.007684e+05</td>\n",
       "      <td>2.239637e+05</td>\n",
       "      <td>2.427078e+05</td>\n",
       "      <td>2.406728e+05</td>\n",
       "      <td>2.483941e+05</td>\n",
       "      <td>2.732043e+05</td>\n",
       "      <td>2.695429e+05</td>\n",
       "      <td>2.904986e+05</td>\n",
       "      <td>2.988567e+05</td>\n",
       "      <td>3.151554e+05</td>\n",
       "      <td>3.023021e+05</td>\n",
       "      <td>3.495991e+05</td>\n",
       "      <td>3.373360e+05</td>\n",
       "      <td>3.728275e+05</td>\n",
       "      <td>4.030610e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7161</th>\n",
       "      <td>49133.0</td>\n",
       "      <td>6</td>\n",
       "      <td>2.803588e+05</td>\n",
       "      <td>2.841899e+05</td>\n",
       "      <td>2.868460e+05</td>\n",
       "      <td>2.810916e+05</td>\n",
       "      <td>2.891247e+05</td>\n",
       "      <td>2.877936e+05</td>\n",
       "      <td>3.001541e+05</td>\n",
       "      <td>2.880852e+05</td>\n",
       "      <td>2.894043e+05</td>\n",
       "      <td>2.983901e+05</td>\n",
       "      <td>2.949910e+05</td>\n",
       "      <td>2.970187e+05</td>\n",
       "      <td>2.925320e+05</td>\n",
       "      <td>2.845031e+05</td>\n",
       "      <td>2.970372e+05</td>\n",
       "      <td>2.969485e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7970</th>\n",
       "      <td>56244.0</td>\n",
       "      <td>136</td>\n",
       "      <td>1.778574e+05</td>\n",
       "      <td>1.878714e+05</td>\n",
       "      <td>1.923771e+05</td>\n",
       "      <td>1.858594e+05</td>\n",
       "      <td>1.919140e+05</td>\n",
       "      <td>1.981118e+05</td>\n",
       "      <td>2.033877e+05</td>\n",
       "      <td>1.912813e+05</td>\n",
       "      <td>1.887865e+05</td>\n",
       "      <td>1.993785e+05</td>\n",
       "      <td>1.993248e+05</td>\n",
       "      <td>1.945615e+05</td>\n",
       "      <td>1.978140e+05</td>\n",
       "      <td>1.882577e+05</td>\n",
       "      <td>1.972394e+05</td>\n",
       "      <td>1.984549e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8284</th>\n",
       "      <td>59177.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.288019e+05</td>\n",
       "      <td>1.332201e+05</td>\n",
       "      <td>1.403131e+05</td>\n",
       "      <td>1.371364e+05</td>\n",
       "      <td>1.512828e+05</td>\n",
       "      <td>1.425807e+05</td>\n",
       "      <td>1.565300e+05</td>\n",
       "      <td>1.401330e+05</td>\n",
       "      <td>1.556225e+05</td>\n",
       "      <td>1.677686e+05</td>\n",
       "      <td>1.910720e+05</td>\n",
       "      <td>1.854920e+05</td>\n",
       "      <td>1.896172e+05</td>\n",
       "      <td>1.853751e+05</td>\n",
       "      <td>2.081000e+05</td>\n",
       "      <td>2.093947e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11448</th>\n",
       "      <td>145497.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5.892181e+04</td>\n",
       "      <td>5.888261e+04</td>\n",
       "      <td>7.395893e+04</td>\n",
       "      <td>7.290782e+04</td>\n",
       "      <td>8.052636e+04</td>\n",
       "      <td>8.037211e+04</td>\n",
       "      <td>8.494039e+04</td>\n",
       "      <td>7.597019e+04</td>\n",
       "      <td>8.599988e+04</td>\n",
       "      <td>9.409743e+04</td>\n",
       "      <td>9.148051e+04</td>\n",
       "      <td>9.600152e+04</td>\n",
       "      <td>9.236770e+04</td>\n",
       "      <td>8.683710e+04</td>\n",
       "      <td>9.772199e+04</td>\n",
       "      <td>1.019649e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2461</th>\n",
       "      <td>7750.0</td>\n",
       "      <td>319</td>\n",
       "      <td>3.913348e+04</td>\n",
       "      <td>3.978288e+04</td>\n",
       "      <td>4.790403e+04</td>\n",
       "      <td>4.723983e+04</td>\n",
       "      <td>5.038749e+04</td>\n",
       "      <td>5.365393e+04</td>\n",
       "      <td>5.559046e+04</td>\n",
       "      <td>5.121100e+04</td>\n",
       "      <td>5.617969e+04</td>\n",
       "      <td>5.606005e+04</td>\n",
       "      <td>6.128149e+04</td>\n",
       "      <td>5.917228e+04</td>\n",
       "      <td>5.527594e+04</td>\n",
       "      <td>6.161304e+04</td>\n",
       "      <td>6.957245e+04</td>\n",
       "      <td>7.191043e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7853</th>\n",
       "      <td>54759.0</td>\n",
       "      <td>6</td>\n",
       "      <td>5.060669e+04</td>\n",
       "      <td>5.198594e+04</td>\n",
       "      <td>5.667637e+04</td>\n",
       "      <td>5.262453e+04</td>\n",
       "      <td>5.767159e+04</td>\n",
       "      <td>5.535435e+04</td>\n",
       "      <td>5.877044e+04</td>\n",
       "      <td>5.460515e+04</td>\n",
       "      <td>5.442380e+04</td>\n",
       "      <td>5.894248e+04</td>\n",
       "      <td>5.749671e+04</td>\n",
       "      <td>6.117021e+04</td>\n",
       "      <td>5.555759e+04</td>\n",
       "      <td>5.396133e+04</td>\n",
       "      <td>5.891667e+04</td>\n",
       "      <td>5.727985e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>7747.0</td>\n",
       "      <td>10</td>\n",
       "      <td>4.417151e+04</td>\n",
       "      <td>4.083041e+04</td>\n",
       "      <td>4.609906e+04</td>\n",
       "      <td>4.731392e+04</td>\n",
       "      <td>4.729234e+04</td>\n",
       "      <td>4.877721e+04</td>\n",
       "      <td>5.148892e+04</td>\n",
       "      <td>5.074520e+04</td>\n",
       "      <td>5.121763e+04</td>\n",
       "      <td>5.456370e+04</td>\n",
       "      <td>5.198147e+04</td>\n",
       "      <td>5.679496e+04</td>\n",
       "      <td>5.597647e+04</td>\n",
       "      <td>5.683178e+04</td>\n",
       "      <td>6.047453e+04</td>\n",
       "      <td>6.434348e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3150</th>\n",
       "      <td>12884.0</td>\n",
       "      <td>118</td>\n",
       "      <td>3.597064e+04</td>\n",
       "      <td>3.566363e+04</td>\n",
       "      <td>3.746600e+04</td>\n",
       "      <td>3.603278e+04</td>\n",
       "      <td>3.957360e+04</td>\n",
       "      <td>3.781210e+04</td>\n",
       "      <td>4.128707e+04</td>\n",
       "      <td>4.140770e+04</td>\n",
       "      <td>4.225558e+04</td>\n",
       "      <td>4.566015e+04</td>\n",
       "      <td>5.083184e+04</td>\n",
       "      <td>4.744620e+04</td>\n",
       "      <td>4.627393e+04</td>\n",
       "      <td>4.162727e+04</td>\n",
       "      <td>4.425315e+04</td>\n",
       "      <td>4.603448e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12089</th>\n",
       "      <td>182922.0</td>\n",
       "      <td>241</td>\n",
       "      <td>3.240442e+04</td>\n",
       "      <td>3.514619e+04</td>\n",
       "      <td>4.279843e+04</td>\n",
       "      <td>3.976002e+04</td>\n",
       "      <td>4.424597e+04</td>\n",
       "      <td>3.950003e+04</td>\n",
       "      <td>4.430519e+04</td>\n",
       "      <td>4.432669e+04</td>\n",
       "      <td>3.934169e+04</td>\n",
       "      <td>4.334620e+04</td>\n",
       "      <td>4.872681e+04</td>\n",
       "      <td>4.454238e+04</td>\n",
       "      <td>3.799858e+04</td>\n",
       "      <td>3.690470e+04</td>\n",
       "      <td>4.262694e+04</td>\n",
       "      <td>4.479763e+04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Sett_ID  ADM_ID    PopSum2000    PopSum2001    PopSum2002  \\\n",
       "3964    19120.0      12  1.371739e+06  1.433621e+06  1.532100e+06   \n",
       "3083    12521.0       9  7.554348e+05  7.229169e+05  8.067938e+05   \n",
       "13493  199779.0      12  4.813840e+05  4.957936e+05  5.327978e+05   \n",
       "13494  199780.0      12  4.813840e+05  4.957936e+05  5.327978e+05   \n",
       "13495  199782.0      12  4.813840e+05  4.957936e+05  5.327978e+05   \n",
       "13496  199787.0      12  4.813840e+05  4.957936e+05  5.327978e+05   \n",
       "3135    12674.0       9  3.434185e+05  3.265064e+05  3.668424e+05   \n",
       "3105    12544.0       9  3.430208e+05  3.262101e+05  3.664446e+05   \n",
       "3137    12684.0       9  3.428078e+05  3.259993e+05  3.662116e+05   \n",
       "3133    12670.0       9  3.427151e+05  3.258997e+05  3.661126e+05   \n",
       "3149    12883.0       9  2.144976e+05  2.007684e+05  2.239637e+05   \n",
       "7161    49133.0       6  2.803588e+05  2.841899e+05  2.868460e+05   \n",
       "7970    56244.0     136  1.778574e+05  1.878714e+05  1.923771e+05   \n",
       "8284    59177.0       1  1.288019e+05  1.332201e+05  1.403131e+05   \n",
       "11448  145497.0       2  5.892181e+04  5.888261e+04  7.395893e+04   \n",
       "2461     7750.0     319  3.913348e+04  3.978288e+04  4.790403e+04   \n",
       "7853    54759.0       6  5.060669e+04  5.198594e+04  5.667637e+04   \n",
       "2458     7747.0      10  4.417151e+04  4.083041e+04  4.609906e+04   \n",
       "3150    12884.0     118  3.597064e+04  3.566363e+04  3.746600e+04   \n",
       "12089  182922.0     241  3.240442e+04  3.514619e+04  4.279843e+04   \n",
       "\n",
       "         PopSum2003    PopSum2004    PopSum2005    PopSum2006    PopSum2007  \\\n",
       "3964   1.611711e+06  1.677785e+06  1.768640e+06  1.869954e+06  1.947824e+06   \n",
       "3083   8.695919e+05  8.789665e+05  9.419342e+05  1.003072e+06  9.917892e+05   \n",
       "13493  5.885219e+05  5.925901e+05  6.358964e+05  6.710046e+05  7.193855e+05   \n",
       "13494  5.885219e+05  5.925901e+05  6.358964e+05  6.710046e+05  7.193855e+05   \n",
       "13495  5.885219e+05  5.925901e+05  6.358964e+05  6.710046e+05  7.193855e+05   \n",
       "13496  5.885219e+05  5.925901e+05  6.358964e+05  6.710046e+05  7.193855e+05   \n",
       "3135   3.962720e+05  3.954759e+05  4.298850e+05  4.585172e+05  4.555575e+05   \n",
       "3105   3.958616e+05  3.950851e+05  4.295021e+05  4.580760e+05  4.551785e+05   \n",
       "3137   3.956150e+05  3.948304e+05  4.292218e+05  4.577869e+05  4.548766e+05   \n",
       "3133   3.955014e+05  3.947171e+05  4.290967e+05  4.576536e+05  4.547388e+05   \n",
       "3149   2.427078e+05  2.406728e+05  2.483941e+05  2.732043e+05  2.695429e+05   \n",
       "7161   2.810916e+05  2.891247e+05  2.877936e+05  3.001541e+05  2.880852e+05   \n",
       "7970   1.858594e+05  1.919140e+05  1.981118e+05  2.033877e+05  1.912813e+05   \n",
       "8284   1.371364e+05  1.512828e+05  1.425807e+05  1.565300e+05  1.401330e+05   \n",
       "11448  7.290782e+04  8.052636e+04  8.037211e+04  8.494039e+04  7.597019e+04   \n",
       "2461   4.723983e+04  5.038749e+04  5.365393e+04  5.559046e+04  5.121100e+04   \n",
       "7853   5.262453e+04  5.767159e+04  5.535435e+04  5.877044e+04  5.460515e+04   \n",
       "2458   4.731392e+04  4.729234e+04  4.877721e+04  5.148892e+04  5.074520e+04   \n",
       "3150   3.603278e+04  3.957360e+04  3.781210e+04  4.128707e+04  4.140770e+04   \n",
       "12089  3.976002e+04  4.424597e+04  3.950003e+04  4.430519e+04  4.432669e+04   \n",
       "\n",
       "         PopSum2008    PopSum2009    PopSum2010    PopSum2011    PopSum2012  \\\n",
       "3964   2.055855e+06  2.155441e+06  2.260273e+06  2.370697e+06  2.559540e+06   \n",
       "3083   1.040045e+06  1.135986e+06  1.178489e+06  1.235285e+06  1.295360e+06   \n",
       "13493  7.252042e+05  7.640128e+05  8.210750e+05  8.430945e+05  9.956889e+05   \n",
       "13494  7.252042e+05  7.640128e+05  8.210750e+05  8.430945e+05  9.956889e+05   \n",
       "13495  7.252042e+05  7.640128e+05  8.210750e+05  8.430945e+05  9.956889e+05   \n",
       "13496  7.252042e+05  7.640128e+05  8.210750e+05  8.430945e+05  9.956889e+05   \n",
       "3135   4.595139e+05  5.213460e+05  5.465248e+05  5.724127e+05  6.236011e+05   \n",
       "3105   4.589896e+05  5.208793e+05  5.459382e+05  5.719066e+05  6.229827e+05   \n",
       "3137   4.587047e+05  5.205358e+05  5.455942e+05  5.715448e+05  6.225841e+05   \n",
       "3133   4.585783e+05  5.203804e+05  5.454478e+05  5.713789e+05  6.224143e+05   \n",
       "3149   2.904986e+05  2.988567e+05  3.151554e+05  3.023021e+05  3.495991e+05   \n",
       "7161   2.894043e+05  2.983901e+05  2.949910e+05  2.970187e+05  2.925320e+05   \n",
       "7970   1.887865e+05  1.993785e+05  1.993248e+05  1.945615e+05  1.978140e+05   \n",
       "8284   1.556225e+05  1.677686e+05  1.910720e+05  1.854920e+05  1.896172e+05   \n",
       "11448  8.599988e+04  9.409743e+04  9.148051e+04  9.600152e+04  9.236770e+04   \n",
       "2461   5.617969e+04  5.606005e+04  6.128149e+04  5.917228e+04  5.527594e+04   \n",
       "7853   5.442380e+04  5.894248e+04  5.749671e+04  6.117021e+04  5.555759e+04   \n",
       "2458   5.121763e+04  5.456370e+04  5.198147e+04  5.679496e+04  5.597647e+04   \n",
       "3150   4.225558e+04  4.566015e+04  5.083184e+04  4.744620e+04  4.627393e+04   \n",
       "12089  3.934169e+04  4.334620e+04  4.872681e+04  4.454238e+04  3.799858e+04   \n",
       "\n",
       "         PopSum2013    PopSum2014    PopSum2015  \n",
       "3964   2.697901e+06  2.856110e+06  2.999944e+06  \n",
       "3083   1.351160e+06  1.472478e+06  1.600271e+06  \n",
       "13493  1.077169e+06  1.116253e+06  1.144993e+06  \n",
       "13494  1.077169e+06  1.116253e+06  1.144993e+06  \n",
       "13495  1.077169e+06  1.116253e+06  1.144993e+06  \n",
       "13496  1.077169e+06  1.116253e+06  1.144993e+06  \n",
       "3135   6.653021e+05  7.385664e+05  7.991168e+05  \n",
       "3105   6.646893e+05  7.379481e+05  7.982845e+05  \n",
       "3137   6.642601e+05  7.374647e+05  7.977807e+05  \n",
       "3133   6.640756e+05  7.372510e+05  7.975707e+05  \n",
       "3149   3.373360e+05  3.728275e+05  4.030610e+05  \n",
       "7161   2.845031e+05  2.970372e+05  2.969485e+05  \n",
       "7970   1.882577e+05  1.972394e+05  1.984549e+05  \n",
       "8284   1.853751e+05  2.081000e+05  2.093947e+05  \n",
       "11448  8.683710e+04  9.772199e+04  1.019649e+05  \n",
       "2461   6.161304e+04  6.957245e+04  7.191043e+04  \n",
       "7853   5.396133e+04  5.891667e+04  5.727985e+04  \n",
       "2458   5.683178e+04  6.047453e+04  6.434348e+04  \n",
       "3150   4.162727e+04  4.425315e+04  4.603448e+04  \n",
       "12089  3.690470e+04  4.262694e+04  4.479763e+04  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "AllSummaries.sort_values('PopSum2010', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47a37640",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e49e4cf",
   "metadata": {},
   "source": [
    "## 9. PREPARE YEARLY DATASETS: NIGHTTIME LIGHTS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4667dd97",
   "metadata": {},
   "source": [
    "### 9.1 Reproject and reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "a333acd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Harmonized_DN_NTL_1999_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2000_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2001_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2002_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2003_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2004_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2005_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2006_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2007_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2008_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2009_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2010_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2011_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2012_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2013_calDMSP.tif',\n",
       " 'Harmonized_DN_NTL_2014_simVIIRS.tif',\n",
       " 'Harmonized_DN_NTL_2015_simVIIRS.tif']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ProjCRS = gdal.WarpOptions(dstSRS='ESRI:102022')\n",
    "AnnualizedSourceFiles = [i for i in os.listdir('NTL/') if i.endswith('.tif')]\n",
    "\n",
    "with fiona.open(r'Results/Catchment.gpkg', mode=\"r\", layer=\"Buff250m_2015\") as shapefile:\n",
    "    MaskGeom = [feature[\"geometry\"] for feature in shapefile] # Identify the bounding areas of the mask.\n",
    "# Mask_out = './LatestYearBuffer.tif'\n",
    "AnnualizedSourceFiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "803f26b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "ValStart = 1999\n",
    "ValEnd = 2015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "cb2c70a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Warp() for year 1999. Thu Dec  8 22:36:22 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 1999. Thu Dec  8 22:36:25 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:36:25 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:36:25 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2000. Thu Dec  8 22:37:35 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2000. Thu Dec  8 22:37:38 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:37:38 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:37:38 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2001. Thu Dec  8 22:38:47 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2001. Thu Dec  8 22:38:50 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:38:50 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:38:50 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2002. Thu Dec  8 22:39:58 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2002. Thu Dec  8 22:40:01 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:40:01 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:40:01 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2003. Thu Dec  8 22:41:09 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2003. Thu Dec  8 22:41:12 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:41:12 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:41:12 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2004. Thu Dec  8 22:42:22 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2004. Thu Dec  8 22:42:25 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:42:25 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:42:25 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2005. Thu Dec  8 22:43:36 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2005. Thu Dec  8 22:43:39 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:43:39 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:43:39 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2006. Thu Dec  8 22:44:49 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2006. Thu Dec  8 22:44:51 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:44:51 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:44:51 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2007. Thu Dec  8 22:46:02 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2007. Thu Dec  8 22:46:05 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:46:05 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:46:05 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2008. Thu Dec  8 22:47:14 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2008. Thu Dec  8 22:47:16 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:47:16 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:47:16 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2009. Thu Dec  8 22:48:25 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2009. Thu Dec  8 22:48:27 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:48:27 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:48:27 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2010. Thu Dec  8 22:49:36 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2010. Thu Dec  8 22:49:38 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:49:38 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:49:38 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2011. Thu Dec  8 22:50:46 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2011. Thu Dec  8 22:50:49 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:50:49 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:50:49 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2012. Thu Dec  8 22:51:58 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2012. Thu Dec  8 22:52:01 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:52:01 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:52:01 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2013. Thu Dec  8 22:53:12 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2013. Thu Dec  8 22:53:14 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:53:14 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:53:14 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2014. Thu Dec  8 22:54:26 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2014. Thu Dec  8 22:54:29 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:54:29 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:54:29 2022 \n",
      "\n",
      "Finished gdal.Warp() for year 2015. Thu Dec  8 22:55:39 2022 \n",
      "\n",
      "Finished rasterio.mask.mask() for year 2015. Thu Dec  8 22:55:41 2022 \n",
      "\n",
      "Written to file. Thu Dec  8 22:55:41 2022 \n",
      "\n",
      "Removed intermediate file. Thu Dec  8 22:55:41 2022 \n",
      "\n",
      "\n",
      " \n",
      " Finished all years in list. Thu Dec  8 22:55:41 2022\n"
     ]
    }
   ],
   "source": [
    "# This codeblock changes each annual population raster's projection (gdal.Warp()), \n",
    "# then masks it to within a specified distance of the settlements (rasterio.mask.mask()).\n",
    "\n",
    "for YearFile in AnnualizedSourceFiles:\n",
    "    InputRasterName = os.path.join(ProjectFolder, \"NTL\", YearFile)\n",
    "    Year = str(re.sub(r'[^0-9]', '', YearFile))\n",
    "    InputRasterObject = gdal.Open(InputRasterName)\n",
    "    TempOutputName = \"Temp_\" + Year + \"_albers.tif\"\n",
    "    TempOutputPath = os.path.join(ProjectFolder, \"NTL\", TempOutputName)\n",
    "    if exists(TempOutputPath):\n",
    "        pass\n",
    "    else:\n",
    "        # Reproject to same CRS as settlements.\n",
    "        Warp = gdal.Warp(TempOutputPath, # Where to store the warped raster\n",
    "                     InputRasterObject, # Which raster to warp\n",
    "                     format='GTiff', \n",
    "                     options=ProjCRS) # Reproject to Africa Albers Equal Area Conic\n",
    "        print('Finished gdal.Warp() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "        \n",
    "        Warp = None # Close the files\n",
    "        InputRasterObject = None\n",
    "\n",
    "        # Reclassify as nodata if outside settlement buffer zones.\n",
    "        with rasterio.open(TempOutputPath) as InputRasterObject:\n",
    "            MaskedOutputRaster, OutTransform = rasterio.mask.mask(\n",
    "                InputRasterObject, MaskGeom, crop=True) # Anything outside the mask is reclassed to the raster's NoData value.\n",
    "            OutMetaData = InputRasterObject.meta.copy()\n",
    "        print('Finished rasterio.mask.mask() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "        OutMetaData.update({\"driver\": \"GTiff\",\n",
    "                         \"height\": MaskedOutputRaster.shape[1],\n",
    "                         \"width\": MaskedOutputRaster.shape[2],\n",
    "                         \"transform\": OutTransform})\n",
    "        FinalOutputPath = os.path.join(ProjectFolder, \"NTL\", ''.join(['Masked_', Year, '.tif']))\n",
    "        with rasterio.open(FinalOutputPath, \"w\", **OutMetaData) as dest:\n",
    "            dest.write(MaskedOutputRaster)\n",
    "        print('Written to file. %s \\n' % time.ctime())\n",
    "    InputRasterObject = None\n",
    "    \n",
    "    try:  # Finally, remove the intermediate file from disk\n",
    "        os.remove(TempOutputPath)\n",
    "    except OSError:\n",
    "        pass\n",
    "    print('Removed intermediate file. %s \\n' % time.ctime())\n",
    "\n",
    "print('\\n \\n Finished all years in list. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "8762b626",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Harmonized_DN_NTL_1999_calDMSP.tif', 'Harmonized_DN_NTL_2000_calDMSP.tif', 'Harmonized_DN_NTL_2001_calDMSP.tif', 'Harmonized_DN_NTL_2002_calDMSP.tif', 'Harmonized_DN_NTL_2003_calDMSP.tif', 'Harmonized_DN_NTL_2004_calDMSP.tif', 'Harmonized_DN_NTL_2005_calDMSP.tif', 'Harmonized_DN_NTL_2006_calDMSP.tif', 'Harmonized_DN_NTL_2007_calDMSP.tif', 'Harmonized_DN_NTL_2008_calDMSP.tif', 'Harmonized_DN_NTL_2009_calDMSP.tif', 'Harmonized_DN_NTL_2010_calDMSP.tif', 'Harmonized_DN_NTL_2011_calDMSP.tif', 'Harmonized_DN_NTL_2012_calDMSP.tif', 'Harmonized_DN_NTL_2013_calDMSP.tif', 'Harmonized_DN_NTL_2014_simVIIRS.tif', 'Harmonized_DN_NTL_2015_simVIIRS.tif', 'Masked_1999.tif', 'Masked_2000.tif', 'Masked_2001.tif', 'Masked_2002.tif', 'Masked_2003.tif', 'Masked_2004.tif', 'Masked_2005.tif', 'Masked_2006.tif', 'Masked_2007.tif', 'Masked_2008.tif', 'Masked_2009.tif', 'Masked_2010.tif', 'Masked_2011.tif', 'Masked_2012.tif', 'Masked_2013.tif', 'Masked_2014.tif', 'Masked_2015.tif']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('NTL/'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "85ea1535",
   "metadata": {},
   "outputs": [],
   "source": [
    "AnnualizedSourceFiles = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c52640a6",
   "metadata": {},
   "source": [
    "### 9.2 Raster values summarized by settlement.\n",
    "1. Convert each annualized raster to .xyz, \n",
    "2. then bring them to vector space and assign their Sett_ID,\n",
    "3. and finally, aggregate the value as appropriate to the settlement level and save table to file.\n",
    "\n",
    "XYZ is similar to .csv. Raster cell centers are stored as x and y, and their value is stored as z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "3e920861",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Masked_1999.tif',\n",
       " 'Masked_2000.tif',\n",
       " 'Masked_2001.tif',\n",
       " 'Masked_2002.tif',\n",
       " 'Masked_2003.tif',\n",
       " 'Masked_2004.tif',\n",
       " 'Masked_2005.tif',\n",
       " 'Masked_2006.tif',\n",
       " 'Masked_2007.tif',\n",
       " 'Masked_2008.tif',\n",
       " 'Masked_2009.tif',\n",
       " 'Masked_2010.tif',\n",
       " 'Masked_2011.tif',\n",
       " 'Masked_2012.tif',\n",
       " 'Masked_2013.tif',\n",
       " 'Masked_2014.tif',\n",
       " 'Masked_2015.tif']"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NoDataVal = 0\n",
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS')\n",
    "AllSummaries = pd.DataFrame(Settlements).drop(columns='geometry')\n",
    "\n",
    "AnnualizedMaskedFiles = [i for i in os.listdir('NTL/') if i.startswith('Masked') and i.endswith('.tif')]\n",
    "AnnualizedMaskedFiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "16fceb02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data for year 1999. Thu Dec  8 22:55:43 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 1999. Thu Dec  8 22:55:44 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 1999. Thu Dec  8 22:55:46 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 1999. Thu Dec  8 22:55:46 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 1999. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "1090570 -1.399430e+06  4.586734e+05  61   POINT (-1399430.443 458673.358)   \n",
      "896720  -1.517622e+06  6.469044e+05  17   POINT (-1517622.054 646904.443)   \n",
      "1087870 -1.396804e+06  4.612998e+05  17   POINT (-1396803.962 461299.838)   \n",
      "1093269 -1.402932e+06  4.560469e+05  61   POINT (-1402932.416 456046.877)   \n",
      "1076843 -1.585035e+06  4.718058e+05  49   POINT (-1585035.047 471805.759)   \n",
      "1074136 -1.588537e+06  4.744322e+05  61   POINT (-1588537.021 474432.239)   \n",
      "1074139 -1.585911e+06  4.744322e+05  59   POINT (-1585910.541 474432.239)   \n",
      "1070537 -1.584160e+06  4.779342e+05  48   POINT (-1584159.554 477934.213)   \n",
      "1091469 -1.401181e+06  4.577979e+05  61   POINT (-1401181.429 457797.864)   \n",
      "464626  -1.179682e+06  1.067141e+06  11  POINT (-1179681.594 1067141.284)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1090570        13496  199787.0      12  \n",
      "896720          7853   54759.0       6  \n",
      "1087870         3964   19120.0      12  \n",
      "1093269         3964   19120.0      12  \n",
      "1076843         3105   12544.0       9  \n",
      "1074136         3133   12670.0       9  \n",
      "1074139         3137   12684.0       9  \n",
      "1070537         3137   12684.0       9  \n",
      "1091469        13493  199779.0      12  \n",
      "464626         11439  145488.0     216  \n",
      "\n",
      "Exported as table, year 1999. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 1999. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "     Sett_ID  NTLct1999\n",
      "2     2572.0         11\n",
      "70  155471.0         14\n",
      "17    7768.0          3\n",
      "11    5430.0          3\n",
      "31   12546.0          1\n",
      "61  126287.0          1\n",
      "9     4298.0          3\n",
      "20    7892.0          1\n",
      "7     4152.0          1\n",
      "30   12544.0        125\n",
      "\n",
      "Values summed to settlement level, year 1999. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "     Sett_ID  NTLsum1999\n",
      "20    7892.0          13\n",
      "78  199793.0          22\n",
      "51   25096.0          17\n",
      "70  155471.0         177\n",
      "48   25047.0          15\n",
      "16    7765.0         148\n",
      "26   10181.0          40\n",
      "3     3115.0         106\n",
      "49   25054.0          15\n",
      "14    6337.0          11\n",
      "\n",
      "Values averaged to settlement level, year 1999. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "     Sett_ID  NTLavg1999\n",
      "19    7891.0   30.000000\n",
      "29   12521.0   38.000000\n",
      "26   10181.0   20.000000\n",
      "28   10205.0   11.000000\n",
      "78  199793.0   22.000000\n",
      "59  126049.0   17.928571\n",
      "72  190914.0   17.846154\n",
      "52   38188.0   11.666667\n",
      "43   19120.0   35.130872\n",
      "51   25096.0   17.000000\n",
      "\n",
      "Merged year 1999 onto latest year settlement feature layer. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999\n",
      "5611    26664.0      86        NaN         NaN         NaN\n",
      "10565   89798.0     280        NaN         NaN         NaN\n",
      "7368    50532.0     306        NaN         NaN         NaN\n",
      "9189    68669.0     138        NaN         NaN         NaN\n",
      "7866    54823.0       6        NaN         NaN         NaN\n",
      "11628  146438.0     223        NaN         NaN         NaN\n",
      "11640  146541.0     220        NaN         NaN         NaN\n",
      "6248    36392.0      59        NaN         NaN         NaN\n",
      "3785    17056.0      38        NaN         NaN         NaN\n",
      "4025    19191.0     275        NaN         NaN         NaN\n",
      "Loading data for year 2000. Thu Dec  8 22:55:47 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2000. Thu Dec  8 22:55:48 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2000. Thu Dec  8 22:55:50 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2000. Thu Dec  8 22:55:50 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2000. Thu Dec  8 22:55:50 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "841731  -1.542136e+06  7.003095e+05  13   POINT (-1542135.870 700309.541)   \n",
      "1075945 -1.582409e+06  4.726813e+05  29   POINT (-1582408.567 472681.252)   \n",
      "1091461 -1.408185e+06  4.577979e+05  16   POINT (-1408185.377 457797.864)   \n",
      "1096875 -1.401181e+06  4.525449e+05  59   POINT (-1401181.429 452544.904)   \n",
      "1090573 -1.396804e+06  4.586734e+05  44   POINT (-1396803.962 458673.358)   \n",
      "361944  -1.151666e+06  1.166948e+06  11  POINT (-1151665.805 1166947.533)   \n",
      "1089669 -1.399430e+06  4.595489e+05  60   POINT (-1399430.443 459548.851)   \n",
      "1090574 -1.395928e+06  4.586734e+05  31   POINT (-1395928.469 458673.358)   \n",
      "1082460 -1.400306e+06  4.665528e+05  11   POINT (-1400305.936 466552.798)   \n",
      "1104084 -1.400306e+06  4.455410e+05  12   POINT (-1400305.936 445540.956)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "841731          8284   59177.0       1  \n",
      "1075945         3083   12521.0       9  \n",
      "1091461         3964   19120.0      12  \n",
      "1096875        13493  199779.0      12  \n",
      "1090573         3964   19120.0      12  \n",
      "361944         11680  155471.0     225  \n",
      "1089669        13493  199779.0      12  \n",
      "1090574        13496  199787.0      12  \n",
      "1082460         3964   19120.0      12  \n",
      "1104084         3964   19120.0      12  \n",
      "\n",
      "Exported as table, year 2000. Thu Dec  8 22:55:50 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2000. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "     Sett_ID  NTLct2000\n",
      "64  146160.0          1\n",
      "37   12763.0          1\n",
      "18    8269.0          3\n",
      "38   12883.0         66\n",
      "33   12670.0        126\n",
      "26   10205.0          1\n",
      "12    7803.0          3\n",
      "7     5435.0          2\n",
      "34   12674.0        126\n",
      "19    8282.0          1\n",
      "\n",
      "Values summed to settlement level, year 2000. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2000\n",
      "60  133551.0          16\n",
      "9     7750.0          33\n",
      "49   25096.0          15\n",
      "4     4298.0          12\n",
      "41   19120.0       10540\n",
      "68  155469.0          11\n",
      "32   12662.0          48\n",
      "74  199780.0        7090\n",
      "22    9135.0          35\n",
      "19    8282.0          11\n",
      "\n",
      "Values averaged to settlement level, year 2000. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "    Sett_ID  NTLavg2000\n",
      "56  74882.0   13.842105\n",
      "26  10205.0   12.000000\n",
      "7    5435.0   12.000000\n",
      "0     464.0   11.000000\n",
      "32  12662.0   48.000000\n",
      "52  49133.0   17.106383\n",
      "13   7887.0   24.000000\n",
      "12   7803.0   11.000000\n",
      "18   8269.0   11.000000\n",
      "14   7891.0   24.000000\n",
      "\n",
      "Merged year 2000 onto latest year settlement feature layer. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "8671    63129.0     110        NaN         NaN         NaN        NaN   \n",
      "780      2195.0      81        NaN         NaN         NaN        NaN   \n",
      "1544     4216.0      41        NaN         NaN         NaN        NaN   \n",
      "6786    46440.0     181        NaN         NaN         NaN        NaN   \n",
      "6376    37309.0     308        NaN         NaN         NaN        NaN   \n",
      "10521   89207.0     151        NaN         NaN         NaN        NaN   \n",
      "9135    67925.0     133        NaN         NaN         NaN        NaN   \n",
      "11129  126298.0      13        NaN         NaN         NaN        NaN   \n",
      "12945  197632.0     299        NaN         NaN         NaN        NaN   \n",
      "3483    14683.0      35        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  \n",
      "8671          NaN         NaN  \n",
      "780           NaN         NaN  \n",
      "1544          NaN         NaN  \n",
      "6786          NaN         NaN  \n",
      "6376          NaN         NaN  \n",
      "10521         NaN         NaN  \n",
      "9135          NaN         NaN  \n",
      "11129         NaN         NaN  \n",
      "12945         NaN         NaN  \n",
      "3483          NaN         NaN  \n",
      "Loading data for year 2001. Thu Dec  8 22:55:51 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2001. Thu Dec  8 22:55:52 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1074139 -1.585911e+06  474432.239176  59  POINT (-1585910.541 474432.239)   \n",
      "1068734 -1.585035e+06  479685.199688  44  POINT (-1585035.047 479685.200)   \n",
      "1085164 -1.399430e+06  463926.318151  20  POINT (-1399430.443 463926.318)   \n",
      "1067837 -1.581533e+06  480560.693107  16  POINT (-1581533.074 480560.693)   \n",
      "1073237 -1.586786e+06  475307.732595  61  POINT (-1586786.034 475307.733)   \n",
      "1103184 -1.399430e+06  446416.449777  17  POINT (-1399430.443 446416.450)   \n",
      "1134716 -1.402057e+06  415774.180122  12  POINT (-1402056.923 415774.180)   \n",
      "1091472 -1.398555e+06  457797.864220  61  POINT (-1398554.949 457797.864)   \n",
      "1099578 -1.401181e+06  449918.423452  53  POINT (-1401181.429 449918.423)   \n",
      "1095076 -1.398555e+06  454295.890545  60  POINT (-1398554.949 454295.891)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1074139         3105   12544.0       9  \n",
      "1068734         3135   12674.0       9  \n",
      "1085164        13493  199779.0      12  \n",
      "1067837         3083   12521.0       9  \n",
      "1073237         3133   12670.0       9  \n",
      "1103184         3964   19120.0      12  \n",
      "1134716         4752   22677.0      90  \n",
      "1091472        13496  199787.0      12  \n",
      "1099578         3964   19120.0      12  \n",
      "1095076        13495  199782.0      12  \n",
      "\n",
      "Exported as table, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "     Sett_ID  NTLct2001\n",
      "48   24544.0          1\n",
      "45   22677.0         12\n",
      "94  199793.0          1\n",
      "80  151678.0          1\n",
      "49   24546.0          3\n",
      "43   12898.0          1\n",
      "88  190914.0         10\n",
      "54   25093.0          1\n",
      "92  199782.0        146\n",
      "65  122070.0          4\n",
      "\n",
      "Values summed to settlement level, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2001\n",
      "91  199780.0        7282\n",
      "59   54759.0         506\n",
      "61   59177.0         520\n",
      "31   10205.0          11\n",
      "72  136568.0          11\n",
      "88  190914.0         206\n",
      "37   12662.0          48\n",
      "66  126049.0         973\n",
      "56   38188.0          11\n",
      "90  199779.0        7282\n",
      "\n",
      "Values averaged to settlement level, year 2001. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2001\n",
      "86  174895.0   11.000000\n",
      "3     3147.0   11.818182\n",
      "24    8282.0   11.000000\n",
      "40   12684.0   41.826772\n",
      "13    7747.0   15.708333\n",
      "48   24544.0   11.000000\n",
      "30   10186.0   18.000000\n",
      "44   19120.0   36.072607\n",
      "58   49133.0   16.795918\n",
      "45   22677.0   14.083333\n",
      "\n",
      "Merged year 2001 onto latest year settlement feature layer. Thu Dec  8 22:55:54 2022 \n",
      "\n",
      "       Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "4397   19911.0     276        NaN         NaN         NaN        NaN   \n",
      "1164    3236.0      76        NaN         NaN         NaN        NaN   \n",
      "1299    3692.0       7        NaN         NaN         NaN        NaN   \n",
      "3099   12537.0       9        NaN         NaN         NaN        NaN   \n",
      "2971   10458.0       5        NaN         NaN         NaN        NaN   \n",
      "4747   22659.0     276        NaN         NaN         NaN        NaN   \n",
      "242      754.0      16        NaN         NaN         NaN        NaN   \n",
      "10449  85507.0     187        NaN         NaN         NaN        NaN   \n",
      "3820   17385.0      35        NaN         NaN         NaN        NaN   \n",
      "7577   51785.0     281        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  NTLavg2001  \n",
      "4397          NaN         NaN        NaN         NaN         NaN  \n",
      "1164          NaN         NaN        NaN         NaN         NaN  \n",
      "1299          NaN         NaN        NaN         NaN         NaN  \n",
      "3099          NaN         NaN        NaN         NaN         NaN  \n",
      "2971          NaN         NaN        NaN         NaN         NaN  \n",
      "4747          NaN         NaN        NaN         NaN         NaN  \n",
      "242           NaN         NaN        NaN         NaN         NaN  \n",
      "10449         NaN         NaN        NaN         NaN         NaN  \n",
      "3820          NaN         NaN        NaN         NaN         NaN  \n",
      "7577          NaN         NaN        NaN         NaN         NaN  \n",
      "Loading data for year 2002. Thu Dec  8 22:55:54 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2002. Thu Dec  8 22:55:56 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2002. Thu Dec  8 22:55:57 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2002. Thu Dec  8 22:55:57 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2002. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "1070538 -1.583284e+06  4.779342e+05  41   POINT (-1583284.060 477934.213)   \n",
      "1096875 -1.401181e+06  4.525449e+05  62   POINT (-1401181.429 452544.904)   \n",
      "1097777 -1.400306e+06  4.516694e+05  60   POINT (-1400305.936 451669.410)   \n",
      "1094165 -1.407310e+06  4.551714e+05  29   POINT (-1407309.883 455171.384)   \n",
      "1067835 -1.583284e+06  4.805607e+05  24   POINT (-1583284.060 480560.693)   \n",
      "101680  -1.042229e+06  1.419965e+06  12  POINT (-1042229.128 1419965.131)   \n",
      "1071443 -1.579782e+06  4.770587e+05  22   POINT (-1579782.087 477058.719)   \n",
      "898526  -1.514120e+06  6.451535e+05  12   POINT (-1514120.080 645153.456)   \n",
      "1073236 -1.587662e+06  4.753077e+05  62   POINT (-1587661.528 475307.733)   \n",
      "361044  -1.150790e+06  1.167823e+06  12  POINT (-1150790.312 1167823.027)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1070538         3083   12521.0       9  \n",
      "1096875        13496  199787.0      12  \n",
      "1097777        13496  199787.0      12  \n",
      "1094165        13494  199780.0      12  \n",
      "1067835         3135   12674.0       9  \n",
      "101680         12243  190914.0     262  \n",
      "1071443         3133   12670.0       9  \n",
      "898526          7161   49133.0       6  \n",
      "1073236         3083   12521.0       9  \n",
      "361044         11680  155471.0     225  \n",
      "\n",
      "Exported as table, year 2002. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2002. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "     Sett_ID  NTLct2002\n",
      "87  145497.0         69\n",
      "74  111590.0         10\n",
      "15    6301.0          1\n",
      "12    4303.0          4\n",
      "19    7768.0          1\n",
      "51   24544.0          1\n",
      "72   76272.0          1\n",
      "90  151387.0          1\n",
      "92  151749.0          2\n",
      "22    7891.0          2\n",
      "\n",
      "Values summed to settlement level, year 2002. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2002\n",
      "84  142842.0          22\n",
      "38   12650.0          22\n",
      "64   59177.0         613\n",
      "88  146152.0          22\n",
      "66   65886.0          33\n",
      "42   12684.0        5693\n",
      "32   10186.0          38\n",
      "67   68648.0          88\n",
      "69   74883.0          79\n",
      "59   38188.0          23\n",
      "\n",
      "Values averaged to settlement level, year 2002. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2002\n",
      "74   111590.0   16.600000\n",
      "53    24548.0   13.000000\n",
      "44    12884.0   11.000000\n",
      "3       464.0   13.666667\n",
      "102  199782.0   51.856164\n",
      "51    24544.0   11.000000\n",
      "7      4086.0   11.000000\n",
      "30    10118.0   26.000000\n",
      "38    12650.0   22.000000\n",
      "80   133551.0   17.000000\n",
      "\n",
      "Merged year 2002 onto latest year settlement feature layer. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "309       942.0      15        NaN         NaN         NaN        NaN   \n",
      "3988    19144.0     276        NaN         NaN         NaN        NaN   \n",
      "3534    14865.0      30        NaN         NaN         NaN        NaN   \n",
      "10384   80859.0     163        NaN         NaN         NaN        NaN   \n",
      "5016    23423.0      88        NaN         NaN         NaN        NaN   \n",
      "13584  201258.0     301        NaN         NaN         NaN        NaN   \n",
      "6083    34374.0      79        NaN         NaN         NaN        NaN   \n",
      "346      1033.0      15        NaN         NaN         NaN        NaN   \n",
      "6592    41715.0      71        NaN         NaN         NaN        NaN   \n",
      "11869  169611.0     253        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  NTLavg2001  NTLct2002  \\\n",
      "309           NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "3988          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "3534          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "10384         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "5016          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "13584         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6083          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "346           NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6592          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "11869         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2002  NTLavg2002  \n",
      "309           NaN         NaN  \n",
      "3988          NaN         NaN  \n",
      "3534          NaN         NaN  \n",
      "10384         NaN         NaN  \n",
      "5016          NaN         NaN  \n",
      "13584         NaN         NaN  \n",
      "6083          NaN         NaN  \n",
      "346           NaN         NaN  \n",
      "6592          NaN         NaN  \n",
      "11869         NaN         NaN  \n",
      "Loading data for year 2003. Thu Dec  8 22:55:58 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2003. Thu Dec  8 22:55:59 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2003. Thu Dec  8 22:56:01 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2003. Thu Dec  8 22:56:01 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2003. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "1076835 -1.592039e+06  4.718058e+05  53   POINT (-1592038.995 471805.759)   \n",
      "101680  -1.042229e+06  1.419965e+06  11  POINT (-1042229.128 1419965.131)   \n",
      "1075939 -1.587662e+06  4.726813e+05  60   POINT (-1587661.528 472681.252)   \n",
      "1072339 -1.584160e+06  4.761832e+05  55   POINT (-1584159.554 476183.226)   \n",
      "1209457 -1.438828e+06  3.431082e+05  13   POINT (-1438827.646 343108.226)   \n",
      "1186165 -1.321512e+06  3.658711e+05  12   POINT (-1321511.528 365871.055)   \n",
      "1078645 -1.585035e+06  4.700548e+05  36   POINT (-1585035.047 470054.772)   \n",
      "1075036 -1.589413e+06  4.735567e+05  62   POINT (-1589412.514 473556.746)   \n",
      "1080452 -1.580658e+06  4.683038e+05  11   POINT (-1580657.580 468303.785)   \n",
      "1092365 -1.405559e+06  4.569224e+05  39   POINT (-1405558.896 456922.371)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1076835         3083   12521.0       9  \n",
      "101680         12243  190914.0     262  \n",
      "1075939         3105   12544.0       9  \n",
      "1072339         3135   12674.0       9  \n",
      "1209457         1136    3115.0       7  \n",
      "1186165         1938    5435.0      78  \n",
      "1078645         3133   12670.0       9  \n",
      "1075036         3135   12674.0       9  \n",
      "1080452         3133   12670.0       9  \n",
      "1092365         3964   19120.0      12  \n",
      "\n",
      "Exported as table, year 2003. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2003. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "     Sett_ID  NTLct2003\n",
      "65   74961.0          2\n",
      "24    9087.0         10\n",
      "4     3147.0         10\n",
      "36   12650.0          1\n",
      "2     2572.0         12\n",
      "71  126287.0          2\n",
      "11    5430.0          2\n",
      "63   74883.0          5\n",
      "8     4158.0          1\n",
      "94  199804.0          1\n",
      "\n",
      "Values summed to settlement level, year 2003. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2003\n",
      "22    8258.0          23\n",
      "8     4158.0          11\n",
      "54   25093.0          13\n",
      "46   19120.0       10911\n",
      "50   24548.0          13\n",
      "78  146152.0          24\n",
      "92  199787.0        7282\n",
      "14    7750.0          34\n",
      "57   45189.0          33\n",
      "17    7803.0          23\n",
      "\n",
      "Values averaged to settlement level, year 2003. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2003\n",
      "31   10205.0   11.000000\n",
      "23    8269.0   11.500000\n",
      "1     1707.0   11.000000\n",
      "79  146160.0   18.000000\n",
      "52   25047.0   15.000000\n",
      "55   25096.0   16.000000\n",
      "14    7750.0   11.333333\n",
      "86  182922.0   11.666667\n",
      "67   76287.0   11.666667\n",
      "25    9088.0   15.875000\n",
      "\n",
      "Merged year 2003 onto latest year settlement feature layer. Thu Dec  8 22:56:02 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "10298   79812.0     161        NaN         NaN         NaN        NaN   \n",
      "11266  136019.0     205        NaN         NaN         NaN        NaN   \n",
      "12530  191819.0     327        NaN         NaN         NaN        NaN   \n",
      "7049    48421.0     170        NaN         NaN         NaN        NaN   \n",
      "6840    46499.0     183        NaN         NaN         NaN        NaN   \n",
      "7023    48389.0     170        NaN         NaN         NaN        NaN   \n",
      "10708   97880.0     189        NaN         NaN         NaN        NaN   \n",
      "11186  126395.0     210        NaN         NaN         NaN        NaN   \n",
      "3726    16490.0      38        NaN         NaN         NaN        NaN   \n",
      "214       610.0      11        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  NTLavg2001  NTLct2002  \\\n",
      "10298         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "11266         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "12530         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "7049          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6840          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "7023          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "10708         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "11186         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "3726          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "214           NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2002  NTLavg2002  NTLct2003  NTLsum2003  NTLavg2003  \n",
      "10298         NaN         NaN        NaN         NaN         NaN  \n",
      "11266         NaN         NaN        NaN         NaN         NaN  \n",
      "12530         NaN         NaN        NaN         NaN         NaN  \n",
      "7049          NaN         NaN        NaN         NaN         NaN  \n",
      "6840          NaN         NaN        NaN         NaN         NaN  \n",
      "7023          NaN         NaN        NaN         NaN         NaN  \n",
      "10708         NaN         NaN        NaN         NaN         NaN  \n",
      "11186         NaN         NaN        NaN         NaN         NaN  \n",
      "3726          NaN         NaN        NaN         NaN         NaN  \n",
      "214           NaN         NaN        NaN         NaN         NaN  \n",
      "Loading data for year 2004. Thu Dec  8 22:56:02 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2004. Thu Dec  8 22:56:03 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1072334 -1.588537e+06  476183.226013  58  POINT (-1588537.021 476183.226)   \n",
      "1075945 -1.582409e+06  472681.252338  47  POINT (-1582408.567 472681.252)   \n",
      "1071440 -1.582409e+06  477058.719432  45  POINT (-1582408.567 477058.719)   \n",
      "900326  -1.515871e+06  643402.468988  26  POINT (-1515871.067 643402.469)   \n",
      "1105724 -1.542136e+06  443789.969521  11  POINT (-1542135.870 443789.970)   \n",
      "1094179 -1.395053e+06  455171.383964  41  POINT (-1395052.975 455171.384)   \n",
      "1098675 -1.402932e+06  450793.916870  54  POINT (-1402932.416 450793.917)   \n",
      "1096871 -1.404683e+06  452544.903708  49  POINT (-1404683.403 452544.904)   \n",
      "1099581 -1.398555e+06  449918.423452  51  POINT (-1398554.949 449918.423)   \n",
      "1208557 -1.437952e+06  343983.719787  10  POINT (-1437952.153 343983.720)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1072334         3135   12674.0       9  \n",
      "1075945         3135   12674.0       9  \n",
      "1071440         3135   12674.0       9  \n",
      "900326          7161   49133.0       6  \n",
      "1105724         2641    9087.0       5  \n",
      "1094179        13493  199779.0      12  \n",
      "1098675        13495  199782.0      12  \n",
      "1096871        13493  199779.0      12  \n",
      "1099581        13493  199779.0      12  \n",
      "1208557          864    2572.0       7  \n",
      "\n",
      "Exported as table, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "      Sett_ID  NTLct2004\n",
      "27     7768.0          3\n",
      "69    38188.0          4\n",
      "25     7750.0         22\n",
      "71    38255.0          1\n",
      "99   146160.0          1\n",
      "43    10186.0          2\n",
      "70    38193.0          1\n",
      "80    74883.0          7\n",
      "108  174895.0          4\n",
      "103  155469.0          3\n",
      "\n",
      "Values summed to settlement level, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2004\n",
      "121  199809.0          19\n",
      "73    49133.0         956\n",
      "42    10181.0          39\n",
      "37     9087.0         308\n",
      "6       458.0          10\n",
      "48    12547.0          11\n",
      "36     8282.0          10\n",
      "70    38193.0          10\n",
      "30     7891.0          68\n",
      "112  190914.0         314\n",
      "\n",
      "Values averaged to settlement level, year 2004. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2004\n",
      "69    38188.0   10.750000\n",
      "77    65886.0   10.200000\n",
      "53    12674.0   46.852713\n",
      "105  163120.0   10.000000\n",
      "101  151678.0   21.000000\n",
      "84    76287.0   10.333333\n",
      "60    22677.0   12.666667\n",
      "49    12591.0   13.000000\n",
      "78    68648.0   10.181818\n",
      "25     7750.0   10.363636\n",
      "\n",
      "Merged year 2004 onto latest year settlement feature layer. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "1647     4349.0      80        NaN         NaN         NaN        NaN   \n",
      "4186    19393.0     278        NaN         NaN         NaN        NaN   \n",
      "6359    36718.0      49        NaN         NaN         NaN        NaN   \n",
      "8801    64552.0     109        NaN         NaN         NaN        NaN   \n",
      "8047    56327.0     267        NaN         NaN         NaN        NaN   \n",
      "12704  196081.0     133        NaN         NaN         NaN        NaN   \n",
      "5271    24575.0     265        NaN         NaN         NaN        NaN   \n",
      "6998    48347.0     172        NaN         NaN         NaN        NaN   \n",
      "12522  191803.0     327        NaN         NaN         NaN        NaN   \n",
      "485      1359.0      18        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  NTLavg2001  NTLct2002  \\\n",
      "1647          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "4186          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6359          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "8801          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "8047          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "12704         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "5271          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6998          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "12522         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "485           NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2002  NTLavg2002  NTLct2003  NTLsum2003  NTLavg2003  NTLct2004  \\\n",
      "1647          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "4186          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6359          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "8801          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "8047          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "12704         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "5271          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "6998          NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "12522         NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "485           NaN         NaN        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2004  NTLavg2004  \n",
      "1647          NaN         NaN  \n",
      "4186          NaN         NaN  \n",
      "6359          NaN         NaN  \n",
      "8801          NaN         NaN  \n",
      "8047          NaN         NaN  \n",
      "12704         NaN         NaN  \n",
      "5271          NaN         NaN  \n",
      "6998          NaN         NaN  \n",
      "12522         NaN         NaN  \n",
      "485           NaN         NaN  \n",
      "Loading data for year 2005. Thu Dec  8 22:56:05 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2005. Thu Dec  8 22:56:07 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1104819 -1.545638e+06  444665.462939  14  POINT (-1545637.843 444665.463)   \n",
      "1103178 -1.404683e+06  446416.449777  11  POINT (-1404683.403 446416.450)   \n",
      "1091475 -1.395928e+06  457797.864220  46  POINT (-1395928.469 457797.864)   \n",
      "1074136 -1.588537e+06  474432.239176  59  POINT (-1588537.021 474432.239)   \n",
      "1060564 -1.638440e+06  487564.640457  10  POINT (-1638440.146 487564.640)   \n",
      "1095976 -1.399430e+06  453420.397127  58  POINT (-1399430.443 453420.397)   \n",
      "1067817 -1.599043e+06  480560.693107  10  POINT (-1599042.942 480560.693)   \n",
      "1069627 -1.592039e+06  478809.706269  47  POINT (-1592038.995 478809.706)   \n",
      "1002024 -1.616553e+06  544471.712673  11  POINT (-1616552.810 544471.713)   \n",
      "1096876 -1.400306e+06  452544.903708  58  POINT (-1400305.936 452544.904)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1104819         2641    9087.0       5  \n",
      "1103178         3964   19120.0      12  \n",
      "1091475        13493  199779.0      12  \n",
      "1074136         3133   12670.0       9  \n",
      "1060564         2605    8282.0     319  \n",
      "1095976        13495  199782.0      12  \n",
      "1067817         3149   12883.0       9  \n",
      "1069627         3149   12883.0       9  \n",
      "1002024         6611   45189.0       4  \n",
      "1096876        13493  199779.0      12  \n",
      "\n",
      "Exported as table, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "     Sett_ID  NTLct2005\n",
      "12    5435.0          4\n",
      "11    5430.0          3\n",
      "63   74882.0         24\n",
      "64   74960.0          1\n",
      "18    7887.0          2\n",
      "66   76340.0          1\n",
      "4      473.0          1\n",
      "76  135966.0          2\n",
      "31   10205.0          1\n",
      "23    8282.0          1\n",
      "\n",
      "Values summed to settlement level, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2005\n",
      "81  146152.0          20\n",
      "77  136085.0          10\n",
      "54   25096.0          14\n",
      "75  133862.0          10\n",
      "2      464.0         133\n",
      "32   12521.0        7846\n",
      "12    5435.0          49\n",
      "47   22793.0          10\n",
      "45   19120.0       11902\n",
      "76  135966.0          61\n",
      "\n",
      "Values averaged to settlement level, year 2005. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2005\n",
      "42   12763.0   14.000000\n",
      "58   54759.0   17.676471\n",
      "11    5430.0   11.666667\n",
      "95  199793.0   21.000000\n",
      "20    7894.0   20.000000\n",
      "90  199749.0   37.000000\n",
      "47   22793.0   10.000000\n",
      "48   24546.0   10.000000\n",
      "68  122070.0   10.000000\n",
      "63   74882.0   13.583333\n",
      "\n",
      "Merged year 2005 onto latest year settlement feature layer. Thu Dec  8 22:56:09 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "11252  135917.0     206        NaN         NaN         NaN        NaN   \n",
      "12337  191299.0     262        NaN         NaN         NaN        NaN   \n",
      "2147     6400.0      47        NaN         NaN         NaN        NaN   \n",
      "5502    25614.0      92        NaN         NaN         NaN        NaN   \n",
      "2740     9205.0      28        NaN         NaN         NaN        NaN   \n",
      "9382    68873.0     138        NaN         NaN         NaN        NaN   \n",
      "9453    68968.0     140        NaN         NaN         NaN        NaN   \n",
      "32         40.0      14        NaN         NaN         NaN        NaN   \n",
      "11954  172307.0     322        NaN         NaN         NaN        NaN   \n",
      "2819     9503.0      23        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2002  \\\n",
      "11252         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12337         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2147          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5502          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2740          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9382          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9453          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "32            NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11954         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2819          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2003  NTLsum2003  NTLavg2003  NTLct2004  NTLsum2004  NTLavg2004  \\\n",
      "11252        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12337        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2147         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5502         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2740         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9382         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9453         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "32           NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11954        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2819         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2005  NTLsum2005  NTLavg2005  \n",
      "11252        NaN         NaN         NaN  \n",
      "12337        NaN         NaN         NaN  \n",
      "2147         NaN         NaN         NaN  \n",
      "5502         NaN         NaN         NaN  \n",
      "2740         NaN         NaN         NaN  \n",
      "9382         NaN         NaN         NaN  \n",
      "9453         NaN         NaN         NaN  \n",
      "32           NaN         NaN         NaN  \n",
      "11954        NaN         NaN         NaN  \n",
      "2819         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 23 columns]\n",
      "Loading data for year 2006. Thu Dec  8 22:56:09 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2006. Thu Dec  8 22:56:11 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1076839 -1.588537e+06  471805.758920  54  POINT (-1588537.021 471805.759)   \n",
      "1069635 -1.585035e+06  478809.706269  52  POINT (-1585035.047 478809.706)   \n",
      "1075946 -1.581533e+06  472681.252338  32  POINT (-1581533.074 472681.252)   \n",
      "1071439 -1.583284e+06  477058.719432  51  POINT (-1583284.060 477058.719)   \n",
      "1185265 -1.320636e+06  366746.548674  11  POINT (-1320636.035 366746.549)   \n",
      "1008835 -1.175304e+06  538343.258742  11  POINT (-1175304.127 538343.259)   \n",
      "1095065 -1.408185e+06  454295.890545  15  POINT (-1408185.377 454295.891)   \n",
      "963742  -1.212950e+06  582117.929678  16  POINT (-1212950.344 582117.930)   \n",
      "1007029 -1.178806e+06  540094.245580  10  POINT (-1178806.101 540094.246)   \n",
      "674551  -1.186686e+06  863151.317086  21  POINT (-1186685.542 863151.317)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1076839         3135   12674.0       9  \n",
      "1069635         3105   12544.0       9  \n",
      "1075946         3105   12544.0       9  \n",
      "1071439         3137   12684.0       9  \n",
      "1185265         1613    4303.0      78  \n",
      "1008835        10039   74882.0       8  \n",
      "1095065         3964   19120.0      12  \n",
      "963742         10087   74960.0      67  \n",
      "1007029        10039   74882.0       8  \n",
      "674551         11072  126049.0      13  \n",
      "\n",
      "Exported as table, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "     Sett_ID  NTLct2006\n",
      "11    5435.0          4\n",
      "8     4298.0          3\n",
      "67  126049.0         52\n",
      "28   10118.0          1\n",
      "69  126287.0          1\n",
      "30   10186.0          2\n",
      "49   24546.0          2\n",
      "90  199782.0        146\n",
      "61   68648.0          8\n",
      "43   12883.0         61\n",
      "\n",
      "Values summed to settlement level, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2006\n",
      "37   12662.0          46\n",
      "35   12547.0          14\n",
      "71  135904.0          20\n",
      "36   12650.0          14\n",
      "29   10181.0          37\n",
      "19    7892.0          11\n",
      "78  146160.0          15\n",
      "18    7891.0          38\n",
      "42   12763.0          13\n",
      "85  190914.0         254\n",
      "\n",
      "Values averaged to settlement level, year 2006. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2006\n",
      "81  151749.0   21.500000\n",
      "28   10118.0   17.000000\n",
      "76  145497.0   22.073529\n",
      "77  146152.0   20.000000\n",
      "49   24546.0   10.000000\n",
      "67  126049.0   15.942308\n",
      "51   25047.0   10.000000\n",
      "75  145488.0   10.000000\n",
      "20    7894.0   22.000000\n",
      "9     4303.0   11.333333\n",
      "\n",
      "Merged year 2006 onto latest year settlement feature layer. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "12420  191518.0     264        NaN         NaN         NaN        NaN   \n",
      "12816  196467.0     174        NaN         NaN         NaN        NaN   \n",
      "7976    56250.0     283        NaN         NaN         NaN        NaN   \n",
      "4306    19546.0     276        NaN         NaN         NaN        NaN   \n",
      "11215  133586.0      13        NaN         NaN         NaN        NaN   \n",
      "1557     4232.0      41        NaN         NaN         NaN        NaN   \n",
      "8471    60430.0     157        NaN         NaN         NaN        NaN   \n",
      "3304    13100.0     118        NaN         NaN         NaN        NaN   \n",
      "12067  175925.0     227        NaN         NaN         NaN        NaN   \n",
      "1000     2725.0       7        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2003  \\\n",
      "12420         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12816         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "7976          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4306          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11215         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "1557          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "8471          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3304          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12067         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "1000          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2004  NTLsum2004  NTLavg2004  NTLct2005  NTLsum2005  NTLavg2005  \\\n",
      "12420        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12816        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "7976         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4306         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11215        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "1557         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "8471         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3304         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12067        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "1000         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2006  NTLsum2006  NTLavg2006  \n",
      "12420        NaN         NaN         NaN  \n",
      "12816        NaN         NaN         NaN  \n",
      "7976         NaN         NaN         NaN  \n",
      "4306         NaN         NaN         NaN  \n",
      "11215        NaN         NaN         NaN  \n",
      "1557         NaN         NaN         NaN  \n",
      "8471         NaN         NaN         NaN  \n",
      "3304         NaN         NaN         NaN  \n",
      "12067        NaN         NaN         NaN  \n",
      "1000         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 26 columns]\n",
      "Loading data for year 2007. Thu Dec  8 22:56:13 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2007. Thu Dec  8 22:56:15 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2007. Thu Dec  8 22:56:16 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2007. Thu Dec  8 22:56:16 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2007. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1099583 -1.396804e+06  449918.423452  33  POINT (-1396803.962 449918.423)   \n",
      "1101382 -1.399430e+06  448167.436614  44  POINT (-1399430.443 448167.437)   \n",
      "1098673 -1.404683e+06  450793.916870  45  POINT (-1404683.403 450793.917)   \n",
      "1078645 -1.585035e+06  470054.772082  43  POINT (-1585035.047 470054.772)   \n",
      "1097779 -1.398555e+06  451669.410289  53  POINT (-1398554.949 451669.410)   \n",
      "1100481 -1.399430e+06  449042.930033  47  POINT (-1399430.443 449042.930)   \n",
      "1075044 -1.582409e+06  473556.745757  49  POINT (-1582408.567 473556.746)   \n",
      "1076839 -1.588537e+06  471805.758920  54  POINT (-1588537.021 471805.759)   \n",
      "1095068 -1.405559e+06  454295.890545  43  POINT (-1405558.896 454295.891)   \n",
      "1073234 -1.589413e+06  475307.732595  60  POINT (-1589412.514 475307.733)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1099583        13495  199782.0      12  \n",
      "1101382        13495  199782.0      12  \n",
      "1098673        13493  199779.0      12  \n",
      "1078645         3083   12521.0       9  \n",
      "1097779        13493  199779.0      12  \n",
      "1100481        13496  199787.0      12  \n",
      "1075044         3135   12674.0       9  \n",
      "1076839         3133   12670.0       9  \n",
      "1095068         3964   19120.0      12  \n",
      "1073234         3137   12684.0       9  \n",
      "\n",
      "Exported as table, year 2007. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2007. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "     Sett_ID  NTLct2007\n",
      "84  199779.0        146\n",
      "88  199793.0          1\n",
      "4     2572.0          6\n",
      "32   12544.0        129\n",
      "43   12893.0         11\n",
      "33   12546.0          1\n",
      "7     4298.0          3\n",
      "10    5435.0          4\n",
      "70  135966.0          2\n",
      "73  145497.0         73\n",
      "\n",
      "Values summed to settlement level, year 2007. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "     Sett_ID  NTLsum2007\n",
      "28   10181.0          41\n",
      "10    5435.0          49\n",
      "24    9088.0         102\n",
      "86  199782.0        7444\n",
      "21    8269.0          67\n",
      "5     3115.0          51\n",
      "60   74882.0         355\n",
      "32   12544.0        6090\n",
      "67  126282.0          11\n",
      "16    7887.0          63\n",
      "\n",
      "Values averaged to settlement level, year 2007. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2007\n",
      "16    7887.0   31.500000\n",
      "21    8269.0   11.166667\n",
      "17    7891.0   31.500000\n",
      "66  126049.0   17.222222\n",
      "79  155469.0   10.000000\n",
      "90  199809.0   23.000000\n",
      "63   76340.0   14.000000\n",
      "27   10118.0   21.000000\n",
      "48   25047.0   11.000000\n",
      "80  155471.0   11.230769\n",
      "\n",
      "Merged year 2007 onto latest year settlement feature layer. Thu Dec  8 22:56:17 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "1207     3382.0      22        NaN         NaN         NaN        NaN   \n",
      "9509    69040.0     141        NaN         NaN         NaN        NaN   \n",
      "13187  198267.0     298        NaN         NaN         NaN        NaN   \n",
      "5511    25668.0     265        NaN         NaN         NaN        NaN   \n",
      "3111    12553.0       9        NaN         NaN         NaN        NaN   \n",
      "4124    19319.0     276        NaN         NaN         NaN        NaN   \n",
      "5363    24746.0      92        NaN         NaN         NaN        NaN   \n",
      "7977    56251.0     270        NaN         NaN         NaN        NaN   \n",
      "8210    57791.0     267        NaN         NaN         NaN        NaN   \n",
      "5877    32217.0      99        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2004  \\\n",
      "1207          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9509          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "13187         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5511          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3111          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4124          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5363          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "7977          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "8210          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5877          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2005  NTLsum2005  NTLavg2005  NTLct2006  NTLsum2006  NTLavg2006  \\\n",
      "1207         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9509         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "13187        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5511         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3111         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4124         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5363         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "7977         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "8210         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5877         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2007  NTLsum2007  NTLavg2007  \n",
      "1207         NaN         NaN         NaN  \n",
      "9509         NaN         NaN         NaN  \n",
      "13187        NaN         NaN         NaN  \n",
      "5511         NaN         NaN         NaN  \n",
      "3111         NaN         NaN         NaN  \n",
      "4124         NaN         NaN         NaN  \n",
      "5363         NaN         NaN         NaN  \n",
      "7977         NaN         NaN         NaN  \n",
      "8210         NaN         NaN         NaN  \n",
      "5877         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 29 columns]\n",
      "Loading data for year 2008. Thu Dec  8 22:56:17 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2008. Thu Dec  8 22:56:18 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2008. Thu Dec  8 22:56:20 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2008. Thu Dec  8 22:56:20 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2008. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1091474 -1.396804e+06  457797.864220  55  POINT (-1396803.962 457797.864)   \n",
      "1075943 -1.584160e+06  472681.252338  56  POINT (-1584159.554 472681.252)   \n",
      "1081347 -1.585911e+06  467428.291826  13  POINT (-1585910.541 467428.292)   \n",
      "1103179 -1.403808e+06  446416.449777  22  POINT (-1403807.910 446416.450)   \n",
      "1094165 -1.407310e+06  455171.383964  24  POINT (-1407309.883 455171.384)   \n",
      "1081351 -1.582409e+06  467428.291826  16  POINT (-1582408.567 467428.292)   \n",
      "1075949 -1.578907e+06  472681.252338  16  POINT (-1578906.593 472681.252)   \n",
      "1077745 -1.584160e+06  470930.265501  51  POINT (-1584159.554 470930.266)   \n",
      "1091470 -1.400306e+06  457797.864220  60  POINT (-1400305.936 457797.864)   \n",
      "1097771 -1.405559e+06  451669.410289  46  POINT (-1405558.896 451669.410)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1091474         3964   19120.0      12  \n",
      "1075943         3137   12684.0       9  \n",
      "1081347         3083   12521.0       9  \n",
      "1103179         3964   19120.0      12  \n",
      "1094165        13493  199779.0      12  \n",
      "1081351         3133   12670.0       9  \n",
      "1075949         3083   12521.0       9  \n",
      "1077745         3133   12670.0       9  \n",
      "1091470        13493  199779.0      12  \n",
      "1097771         3964   19120.0      12  \n",
      "\n",
      "Exported as table, year 2008. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2008. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "     Sett_ID  NTLct2008\n",
      "14    4298.0          3\n",
      "94  199779.0        146\n",
      "21    7768.0          3\n",
      "97  199787.0        146\n",
      "13    3147.0          5\n",
      "37   10205.0          1\n",
      "96  199782.0        146\n",
      "99  199804.0          1\n",
      "10     473.0          1\n",
      "7      458.0          1\n",
      "\n",
      "Values summed to settlement level, year 2008. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2008\n",
      "32     9135.0          55\n",
      "23     7887.0          57\n",
      "100  199809.0          22\n",
      "85   146160.0          14\n",
      "80   126287.0          22\n",
      "97   199787.0        7735\n",
      "87   151678.0          25\n",
      "35    10181.0          55\n",
      "67    63000.0          30\n",
      "62    38188.0          32\n",
      "\n",
      "Values averaged to settlement level, year 2008. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "     Sett_ID  NTLavg2008\n",
      "32    9135.0   27.500000\n",
      "34   10118.0   27.000000\n",
      "80  126287.0   11.000000\n",
      "37   10205.0   11.000000\n",
      "17    5435.0   10.500000\n",
      "62   38188.0   10.666667\n",
      "11    2572.0   10.000000\n",
      "4      451.0   10.000000\n",
      "66   59177.0   12.137931\n",
      "56   24548.0   13.000000\n",
      "\n",
      "Merged year 2008 onto latest year settlement feature layer. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "1018     2743.0      19        NaN         NaN         NaN        NaN   \n",
      "7478    51657.0     136        NaN         NaN         NaN        NaN   \n",
      "2726     9189.0      27        NaN         NaN         NaN        NaN   \n",
      "6067    34205.0      79        NaN         NaN         NaN        NaN   \n",
      "3843    17595.0      35        NaN         NaN         NaN        NaN   \n",
      "827      2446.0      81        NaN         NaN         NaN        NaN   \n",
      "12738  196182.0     173        NaN         NaN         NaN        NaN   \n",
      "379      1117.0      15        NaN         NaN         NaN        NaN   \n",
      "11555  146039.0       2        NaN         NaN         NaN        NaN   \n",
      "695      1908.0      84        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2005  \\\n",
      "1018          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "7478          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2726          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6067          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3843          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "827           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12738         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "379           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11555         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "695           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2006  NTLsum2006  NTLavg2006  NTLct2007  NTLsum2007  NTLavg2007  \\\n",
      "1018         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "7478         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2726         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6067         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3843         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "827          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12738        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "379          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11555        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "695          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2008  NTLsum2008  NTLavg2008  \n",
      "1018         NaN         NaN         NaN  \n",
      "7478         NaN         NaN         NaN  \n",
      "2726         NaN         NaN         NaN  \n",
      "6067         NaN         NaN         NaN  \n",
      "3843         NaN         NaN         NaN  \n",
      "827          NaN         NaN         NaN  \n",
      "12738        NaN         NaN         NaN  \n",
      "379          NaN         NaN         NaN  \n",
      "11555        NaN         NaN         NaN  \n",
      "695          NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 32 columns]\n",
      "Loading data for year 2009. Thu Dec  8 22:56:21 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2009. Thu Dec  8 22:56:22 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2009. Thu Dec  8 22:56:24 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2009. Thu Dec  8 22:56:24 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2009. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "1098677 -1.401168e+06  4.506659e+05  58   POINT (-1401167.537 450665.898)   \n",
      "436661  -1.209434e+06  1.094154e+06  35  POINT (-1209434.423 1094153.746)   \n",
      "1091468 -1.402043e+06  4.576698e+05  57   POINT (-1402043.030 457669.847)   \n",
      "433960  -1.207683e+06  1.096780e+06  31  POINT (-1207683.436 1096780.227)   \n",
      "987819  -1.431810e+06  5.583516e+05  10   POINT (-1431809.815 558351.619)   \n",
      "1072333 -1.589399e+06  4.760552e+05  58   POINT (-1589398.676 476055.214)   \n",
      "1098680 -1.398541e+06  4.506659e+05  54   POINT (-1398541.056 450665.898)   \n",
      "1069642 -1.578893e+06  4.786817e+05  29   POINT (-1578892.752 478681.695)   \n",
      "1075945 -1.582395e+06  4.725532e+05  52   POINT (-1582394.727 472553.239)   \n",
      "1067834 -1.584146e+06  4.804327e+05  50   POINT (-1584145.714 480432.682)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1098677        13495  199782.0      12  \n",
      "436661         11448  145497.0       2  \n",
      "1091468        13493  199779.0      12  \n",
      "433960         11448  145497.0       2  \n",
      "987819          8916   65886.0     111  \n",
      "1072333         3105   12544.0       9  \n",
      "1098680        13493  199779.0      12  \n",
      "1069642         3105   12544.0       9  \n",
      "1075945         3135   12674.0       9  \n",
      "1067834         3105   12544.0       9  \n",
      "\n",
      "Exported as table, year 2009. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2009. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "      Sett_ID  NTLct2009\n",
      "46    12893.0         10\n",
      "70    54759.0         34\n",
      "118  199793.0          1\n",
      "85   126282.0          1\n",
      "108  193510.0          1\n",
      "107  190914.0         25\n",
      "104  169041.0          2\n",
      "28     9088.0         12\n",
      "60    25096.0          1\n",
      "91   145488.0          6\n",
      "\n",
      "Values summed to settlement level, year 2009. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2009\n",
      "57    25001.0          10\n",
      "2       352.0          16\n",
      "24     8258.0         123\n",
      "92   145497.0        1587\n",
      "6      1707.0          41\n",
      "40    12662.0          47\n",
      "118  199793.0          25\n",
      "102  155469.0          41\n",
      "60    25096.0          17\n",
      "80    76272.0          10\n",
      "\n",
      "Values averaged to settlement level, year 2009. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2009\n",
      "60    25096.0   17.000000\n",
      "40    12662.0   47.000000\n",
      "94   146156.0   10.000000\n",
      "8      3115.0   10.166667\n",
      "114  199780.0   52.441379\n",
      "118  199793.0   25.000000\n",
      "61    25114.0   16.000000\n",
      "85   126282.0   11.000000\n",
      "22     7891.0   27.000000\n",
      "19     7803.0   13.666667\n",
      "\n",
      "Merged year 2009 onto latest year settlement feature layer. Thu Dec  8 22:56:25 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "12097  183006.0     240        NaN         NaN         NaN        NaN   \n",
      "12341  191306.0     262        NaN         NaN         NaN        NaN   \n",
      "3613    15590.0      39        NaN         NaN         NaN        NaN   \n",
      "10442   85484.0     186        NaN         NaN         NaN        NaN   \n",
      "6174    36268.0      43        NaN         NaN         NaN        NaN   \n",
      "6957    46876.0     183        NaN         NaN         NaN        NaN   \n",
      "11945  170089.0     247        NaN         NaN         NaN        NaN   \n",
      "9611    70212.0     139        NaN         NaN         NaN        NaN   \n",
      "3015    10947.0      27        NaN         NaN         NaN        NaN   \n",
      "2283     7122.0      46        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2006  \\\n",
      "12097         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12341         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3613          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "10442         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6174          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6957          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11945         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9611          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3015          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2283          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2007  NTLsum2007  NTLavg2007  NTLct2008  NTLsum2008  NTLavg2008  \\\n",
      "12097        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12341        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3613         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "10442        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6174         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6957         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11945        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9611         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3015         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2283         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2009  NTLsum2009  NTLavg2009  \n",
      "12097        NaN         NaN         NaN  \n",
      "12341        NaN         NaN         NaN  \n",
      "3613         NaN         NaN         NaN  \n",
      "10442        NaN         NaN         NaN  \n",
      "6174         NaN         NaN         NaN  \n",
      "6957         NaN         NaN         NaN  \n",
      "11945        NaN         NaN         NaN  \n",
      "9611         NaN         NaN         NaN  \n",
      "3015         NaN         NaN         NaN  \n",
      "2283         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 35 columns]\n",
      "Loading data for year 2010. Thu Dec  8 22:56:25 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2010. Thu Dec  8 22:56:27 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2010. Thu Dec  8 22:56:28 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2010. Thu Dec  8 22:56:28 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2010. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1077746 -1.583270e+06  470802.252092  54  POINT (-1583270.220 470802.252)   \n",
      "1095080 -1.395039e+06  454167.872351  42  POINT (-1395039.081 454167.872)   \n",
      "1095968 -1.406420e+06  453292.378681  44  POINT (-1406420.499 453292.379)   \n",
      "1071437 -1.585021e+06  476930.707786  57  POINT (-1585021.208 476930.708)   \n",
      "1211261 -1.437063e+06  341229.188850  14  POINT (-1437062.777 341229.189)   \n",
      "670044  -1.188423e+06  867400.884853  12  POINT (-1188422.575 867400.885)   \n",
      "1078647 -1.583270e+06  469926.758421  50  POINT (-1583270.220 469926.758)   \n",
      "1105887 -1.399417e+06  443661.948304  35  POINT (-1399416.549 443661.948)   \n",
      "903930  -1.515857e+06  639772.530509  12  POINT (-1515857.208 639772.531)   \n",
      "1078655 -1.576266e+06  469926.758421  13  POINT (-1576266.271 469926.758)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1077746         3105   12544.0       9  \n",
      "1095080         3964   19120.0      12  \n",
      "1095968         3964   19120.0      12  \n",
      "1071437         3137   12684.0       9  \n",
      "1211261          864    2572.0       7  \n",
      "670044         11072  126049.0      13  \n",
      "1078647         3133   12670.0       9  \n",
      "1105887         3964   19120.0      12  \n",
      "903930          7161   49133.0       6  \n",
      "1078655         3083   12521.0       9  \n",
      "\n",
      "Exported as table, year 2010. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2010. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "      Sett_ID  NTLct2010\n",
      "32    10012.0          1\n",
      "31     9135.0          3\n",
      "19     7750.0         27\n",
      "3       451.0          1\n",
      "48    12883.0         64\n",
      "106  190912.0          2\n",
      "23     7887.0          1\n",
      "117  199787.0        145\n",
      "46    12684.0        131\n",
      "109  193578.0          1\n",
      "\n",
      "Values summed to settlement level, year 2010. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2010\n",
      "104  174895.0          20\n",
      "7      2572.0         236\n",
      "42    12657.0          13\n",
      "91   135966.0          53\n",
      "71    45189.0         271\n",
      "72    49133.0         915\n",
      "107  190914.0         571\n",
      "74    59177.0        1210\n",
      "49    12893.0          71\n",
      "82    74961.0          17\n",
      "\n",
      "Values averaged to settlement level, year 2010. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2010\n",
      "60    24542.0   11.000000\n",
      "2       352.0   10.000000\n",
      "17     5435.0   11.750000\n",
      "18     7747.0   13.181818\n",
      "113  199749.0   55.000000\n",
      "34    10181.0   20.000000\n",
      "26     8258.0   10.555556\n",
      "106  190912.0   10.000000\n",
      "82    74961.0   17.000000\n",
      "14     4298.0   11.750000\n",
      "\n",
      "Merged year 2010 onto latest year settlement feature layer. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "5052    23603.0      90        NaN         NaN         NaN        NaN   \n",
      "6024    33973.0      94        NaN         NaN         NaN        NaN   \n",
      "10956  122114.0     200        NaN         NaN         NaN        NaN   \n",
      "10851  111607.0     198        NaN         NaN         NaN        NaN   \n",
      "11776  161707.0     226        NaN         NaN         NaN        NaN   \n",
      "2724     9187.0      27        NaN         NaN         NaN        NaN   \n",
      "7880    54911.0       6        NaN         NaN         NaN        NaN   \n",
      "703      1936.0      84        NaN         NaN         NaN        NaN   \n",
      "2983    10670.0      25        NaN         NaN         NaN        NaN   \n",
      "3743    16626.0      38        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2007  \\\n",
      "5052          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6024          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "10956         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "10851         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11776         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2724          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "7880          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "703           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2983          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3743          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2008  NTLsum2008  NTLavg2008  NTLct2009  NTLsum2009  NTLavg2009  \\\n",
      "5052         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6024         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "10956        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "10851        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11776        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2724         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "7880         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "703          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2983         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3743         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2010  NTLsum2010  NTLavg2010  \n",
      "5052         NaN         NaN         NaN  \n",
      "6024         NaN         NaN         NaN  \n",
      "10956        NaN         NaN         NaN  \n",
      "10851        NaN         NaN         NaN  \n",
      "11776        NaN         NaN         NaN  \n",
      "2724         NaN         NaN         NaN  \n",
      "7880         NaN         NaN         NaN  \n",
      "703          NaN         NaN         NaN  \n",
      "2983         NaN         NaN         NaN  \n",
      "3743         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 38 columns]\n",
      "Loading data for year 2011. Thu Dec  8 22:56:29 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2011. Thu Dec  8 22:56:31 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1087866 -1.400292e+06  461171.821716  53  POINT (-1400292.043 461171.822)   \n",
      "1076844 -1.584146e+06  471677.745762  61  POINT (-1584145.714 471677.746)   \n",
      "1094174 -1.399417e+06  455043.366022  63  POINT (-1399416.549 455043.366)   \n",
      "1095974 -1.401168e+06  453292.378681  63  POINT (-1401167.537 453292.379)   \n",
      "1074133 -1.591150e+06  474304.226774  58  POINT (-1591149.663 474304.227)   \n",
      "1091473 -1.397666e+06  457669.847033  62  POINT (-1397665.562 457669.847)   \n",
      "1096881 -1.395915e+06  452416.885010  56  POINT (-1395914.575 452416.885)   \n",
      "1094167 -1.405545e+06  455043.366022  46  POINT (-1405545.005 455043.366)   \n",
      "1093267 -1.404670e+06  455918.859692  50  POINT (-1404669.511 455918.860)   \n",
      "1081885 -1.114881e+06  467300.277409  10  POINT (-1114881.106 467300.277)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1087866        13496  199787.0      12  \n",
      "1076844         3135   12674.0       9  \n",
      "1094174         3964   19120.0      12  \n",
      "1095974        13495  199782.0      12  \n",
      "1074133         3083   12521.0       9  \n",
      "1091473        13496  199787.0      12  \n",
      "1096881        13493  199779.0      12  \n",
      "1094167        13495  199782.0      12  \n",
      "1093267        13495  199782.0      12  \n",
      "1081885         6482   38255.0      63  \n",
      "\n",
      "Exported as table, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "      Sett_ID  NTLct2011\n",
      "19     7750.0         23\n",
      "78    74883.0          1\n",
      "86   135966.0          2\n",
      "112  199804.0          1\n",
      "70    49133.0         51\n",
      "18     7747.0         34\n",
      "66    25567.0          1\n",
      "35    12521.0        237\n",
      "85   133551.0          1\n",
      "49    19118.0          2\n",
      "\n",
      "Values summed to settlement level, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2011\n",
      "108  199782.0        8106\n",
      "93   151678.0          25\n",
      "83   126049.0        1028\n",
      "48    12893.0          76\n",
      "72    59177.0         947\n",
      "13     4298.0          34\n",
      "25     8258.0          42\n",
      "59    24602.0          12\n",
      "61    25054.0          18\n",
      "45    12684.0        7358\n",
      "\n",
      "Values averaged to settlement level, year 2011. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2011\n",
      "20     7765.0    19.20000\n",
      "93   151678.0    25.00000\n",
      "86   135966.0    19.00000\n",
      "40    12650.0    29.00000\n",
      "89   146152.0    20.00000\n",
      "35    12521.0    46.43038\n",
      "68    38255.0    10.00000\n",
      "110  199791.0    21.00000\n",
      "97   155469.0    10.00000\n",
      "104  193510.0    14.00000\n",
      "\n",
      "Merged year 2011 onto latest year settlement feature layer. Thu Dec  8 22:56:33 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "4724    22518.0     271        NaN         NaN         NaN        NaN   \n",
      "13598  201476.0     297        NaN         NaN         NaN        NaN   \n",
      "10363   80289.0     162        NaN         NaN         NaN        NaN   \n",
      "4114    19307.0     272        NaN         NaN         NaN        NaN   \n",
      "6356    36709.0      49        NaN         NaN         NaN        NaN   \n",
      "9964    73578.0     318        NaN         NaN         NaN        NaN   \n",
      "82         96.0      17        NaN         NaN         NaN        NaN   \n",
      "670      1827.0     324        NaN         NaN         NaN        NaN   \n",
      "7413    51471.0     288        NaN         NaN         NaN        NaN   \n",
      "6126    35288.0     106        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2008  \\\n",
      "4724          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "13598         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "10363         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4114          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6356          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9964          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "82            NaN         NaN        NaN         NaN  ...         NaN   \n",
      "670           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "7413          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6126          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2009  NTLsum2009  NTLavg2009  NTLct2010  NTLsum2010  NTLavg2010  \\\n",
      "4724         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "13598        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "10363        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4114         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6356         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9964         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "82           NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "670          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "7413         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6126         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2011  NTLsum2011  NTLavg2011  \n",
      "4724         NaN         NaN         NaN  \n",
      "13598        NaN         NaN         NaN  \n",
      "10363        NaN         NaN         NaN  \n",
      "4114         NaN         NaN         NaN  \n",
      "6356         NaN         NaN         NaN  \n",
      "9964         NaN         NaN         NaN  \n",
      "82           NaN         NaN         NaN  \n",
      "670          NaN         NaN         NaN  \n",
      "7413         NaN         NaN         NaN  \n",
      "6126         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 41 columns]\n",
      "Loading data for year 2012. Thu Dec  8 22:56:33 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2012. Thu Dec  8 22:56:35 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "                    X             Y   Z                          geometry  \\\n",
      "1076837 -1.590288e+06  4.718058e+05  56   POINT (-1590288.008 471805.759)   \n",
      "1077668 -1.651573e+06  4.709303e+05  11   POINT (-1651572.547 470930.266)   \n",
      "1137421 -1.400306e+06  4.131477e+05  11   POINT (-1400305.936 413147.700)   \n",
      "1082253 -1.581533e+06  4.665528e+05  20   POINT (-1581533.074 466552.798)   \n",
      "433959  -1.208573e+06  1.096908e+06  58  POINT (-1208572.877 1096908.060)   \n",
      "1002026 -1.614802e+06  5.444717e+05  11   POINT (-1614801.824 544471.713)   \n",
      "1092374 -1.397679e+06  4.569224e+05  63   POINT (-1397679.456 456922.371)   \n",
      "898525  -1.514996e+06  6.451535e+05  24   POINT (-1514995.574 645153.456)   \n",
      "1069642 -1.578907e+06  4.788097e+05  26   POINT (-1578906.593 478809.706)   \n",
      "1105880 -1.405559e+06  4.437900e+05  10   POINT (-1405558.896 443789.970)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1076837         3105   12544.0       9  \n",
      "1077668         2564    7892.0      10  \n",
      "1137421         4752   22677.0      90  \n",
      "1082253         3083   12521.0       9  \n",
      "433959         11448  145497.0       2  \n",
      "1002026         6611   45189.0       4  \n",
      "1092374         3964   19120.0      12  \n",
      "898525          7161   49133.0       6  \n",
      "1069642         3083   12521.0       9  \n",
      "1105880        13488  199738.0      12  \n",
      "\n",
      "Exported as table, year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "      Sett_ID  NTLct2012\n",
      "61    12893.0         10\n",
      "66    19314.0          1\n",
      "68    19322.0          1\n",
      "30     8119.0          1\n",
      "85    25054.0          1\n",
      "16     4303.0          1\n",
      "138  199793.0          1\n",
      "35     9087.0         15\n",
      "42    10181.0          2\n",
      "34     8282.0          1\n",
      "\n",
      "Values summed to settlement level, year 2012. Thu Dec  8 22:56:37 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2012\n",
      "77    22796.0          10\n",
      "97    68648.0          73\n",
      "22     7765.0         117\n",
      "95    63000.0          84\n",
      "7      2572.0         498\n",
      "92    56244.0          20\n",
      "34     8282.0          11\n",
      "83    25037.0          11\n",
      "50    12591.0          12\n",
      "130  197714.0          17\n",
      "\n",
      "Values averaged to settlement level, year 2012. Thu Dec  8 22:56:38 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2012\n",
      "126  193578.0   13.000000\n",
      "139  199804.0   33.000000\n",
      "29     7894.0   26.000000\n",
      "37     9135.0   27.000000\n",
      "1       453.0   11.000000\n",
      "77    22796.0   10.000000\n",
      "48    12546.0   26.000000\n",
      "60    12884.0   10.571429\n",
      "108  133551.0   21.000000\n",
      "70    20935.0   15.000000\n",
      "\n",
      "Merged year 2012 onto latest year settlement feature layer. Thu Dec  8 22:56:38 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "9584    69136.0     142        NaN         NaN         NaN        NaN   \n",
      "4932    23005.0      88        NaN         NaN         NaN        NaN   \n",
      "6169    36258.0     106        NaN         NaN         NaN        NaN   \n",
      "4702    22289.0     272        NaN         NaN         NaN        NaN   \n",
      "5983    33899.0      79        NaN         NaN         NaN        NaN   \n",
      "12232  190146.0     261        NaN         NaN         NaN        NaN   \n",
      "11833  169033.0     250        NaN         NaN         NaN        NaN   \n",
      "4581    21065.0     276        NaN         NaN         NaN        NaN   \n",
      "11856  169328.0     248        NaN         NaN         NaN        NaN   \n",
      "2424     7578.0      47        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2009  \\\n",
      "9584          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4932          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6169          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4702          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5983          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12232         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11833         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4581          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11856         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2424          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2010  NTLsum2010  NTLavg2010  NTLct2011  NTLsum2011  NTLavg2011  \\\n",
      "9584         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4932         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6169         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4702         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5983         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12232        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11833        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4581         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11856        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2424         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2012  NTLsum2012  NTLavg2012  \n",
      "9584         NaN         NaN         NaN  \n",
      "4932         NaN         NaN         NaN  \n",
      "6169         NaN         NaN         NaN  \n",
      "4702         NaN         NaN         NaN  \n",
      "5983         NaN         NaN         NaN  \n",
      "12232        NaN         NaN         NaN  \n",
      "11833        NaN         NaN         NaN  \n",
      "4581         NaN         NaN         NaN  \n",
      "11856        NaN         NaN         NaN  \n",
      "2424         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 44 columns]\n",
      "Loading data for year 2013. Thu Dec  8 22:56:38 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2013. Thu Dec  8 22:56:39 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1084261 -1.401181e+06  464801.811570  30  POINT (-1401181.429 464801.812)   \n",
      "1094164 -1.408185e+06  455171.383964  29  POINT (-1408185.377 455171.384)   \n",
      "1101382 -1.399430e+06  448167.436614  57  POINT (-1399430.443 448167.437)   \n",
      "1075944 -1.583284e+06  472681.252338  59  POINT (-1583284.060 472681.252)   \n",
      "1073236 -1.587662e+06  475307.732595  62  POINT (-1587661.528 475307.733)   \n",
      "903030  -1.514996e+06  640775.988732  10  POINT (-1514995.574 640775.989)   \n",
      "1086968 -1.397679e+06  462175.331314  38  POINT (-1397679.456 462175.331)   \n",
      "1094181 -1.393302e+06  455171.383964  29  POINT (-1393301.989 455171.384)   \n",
      "1066034 -1.582409e+06  482311.679944  26  POINT (-1582408.567 482311.680)   \n",
      "1094176 -1.397679e+06  455171.383964  62  POINT (-1397679.456 455171.384)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1084261         3964   19120.0      12  \n",
      "1094164         3964   19120.0      12  \n",
      "1101382        13496  199787.0      12  \n",
      "1075944         3135   12674.0       9  \n",
      "1073236         3105   12544.0       9  \n",
      "903030          7161   49133.0       6  \n",
      "1086968         3964   19120.0      12  \n",
      "1094181         3964   19120.0      12  \n",
      "1066034         3107   12547.0       9  \n",
      "1094176        13496  199787.0      12  \n",
      "\n",
      "Exported as table, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "      Sett_ID  NTLct2013\n",
      "29     7747.0         36\n",
      "93    59256.0          1\n",
      "114  151387.0          1\n",
      "137  199782.0        146\n",
      "21     4086.0          6\n",
      "63    12684.0        129\n",
      "8       463.0          1\n",
      "5       451.0          1\n",
      "22     4141.0          4\n",
      "70    13113.0          1\n",
      "\n",
      "Values summed to settlement level, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2013\n",
      "7       458.0          10\n",
      "60    12662.0          56\n",
      "16     2563.0          13\n",
      "3        57.0          10\n",
      "27     5430.0          51\n",
      "105  126049.0        1215\n",
      "36     7887.0          49\n",
      "94    63000.0          93\n",
      "40     8258.0         195\n",
      "63    12684.0        7110\n",
      "\n",
      "Values averaged to settlement level, year 2013. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2013\n",
      "120  174895.0   10.000000\n",
      "15     1707.0   16.750000\n",
      "42     8269.0   15.000000\n",
      "62    12674.0   55.116279\n",
      "136  199780.0   58.582192\n",
      "0         4.0   14.000000\n",
      "96    68648.0   10.250000\n",
      "87    25093.0   21.000000\n",
      "37     7891.0   24.500000\n",
      "91    54759.0   19.030303\n",
      "\n",
      "Merged year 2013 onto latest year settlement feature layer. Thu Dec  8 22:56:41 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "11536  145933.0     216        NaN         NaN         NaN        NaN   \n",
      "2487     7778.0      10        NaN         NaN         NaN        NaN   \n",
      "9789    71687.0     144        NaN         NaN         NaN        NaN   \n",
      "11169  126368.0      13        NaN         NaN         NaN        NaN   \n",
      "5379    24814.0      92        NaN         NaN         NaN        NaN   \n",
      "8617    63054.0     110        NaN         NaN         NaN        NaN   \n",
      "12707  196124.0     135        NaN         NaN         NaN        NaN   \n",
      "5639    27199.0     105        NaN         NaN         NaN        NaN   \n",
      "13180  198250.0     299        NaN         NaN         NaN        NaN   \n",
      "6224    36355.0     308        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2010  \\\n",
      "11536         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "2487          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "9789          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11169         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5379          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "8617          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "12707         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5639          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "13180         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6224          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2011  NTLsum2011  NTLavg2011  NTLct2012  NTLsum2012  NTLavg2012  \\\n",
      "11536        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "2487         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "9789         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11169        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5379         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "8617         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "12707        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5639         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "13180        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6224         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2013  NTLsum2013  NTLavg2013  \n",
      "11536        NaN         NaN         NaN  \n",
      "2487         NaN         NaN         NaN  \n",
      "9789         NaN         NaN         NaN  \n",
      "11169        NaN         NaN         NaN  \n",
      "5379         NaN         NaN         NaN  \n",
      "8617         NaN         NaN         NaN  \n",
      "12707        NaN         NaN         NaN  \n",
      "5639         NaN         NaN         NaN  \n",
      "13180        NaN         NaN         NaN  \n",
      "6224         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 47 columns]\n",
      "Loading data for year 2014. Thu Dec  8 22:56:41 2022 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2014. Thu Dec  8 22:56:43 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2014. Thu Dec  8 22:56:45 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2014. Thu Dec  8 22:56:45 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2014. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1092370 -1.401181e+06  456922.370801  61  POINT (-1401181.429 456922.371)   \n",
      "1097773 -1.403808e+06  451669.410289  57  POINT (-1403807.910 451669.410)   \n",
      "1076767 -1.651573e+06  471805.758920  10  POINT (-1651572.547 471805.759)   \n",
      "1074144 -1.581533e+06  474432.239176  49  POINT (-1581533.074 474432.239)   \n",
      "1071440 -1.582409e+06  477058.719432  52  POINT (-1582408.567 477058.719)   \n",
      "1097780 -1.397679e+06  451669.410289  58  POINT (-1397679.456 451669.410)   \n",
      "1098681 -1.397679e+06  450793.916870  56  POINT (-1397679.456 450793.917)   \n",
      "1098676 -1.402057e+06  450793.916870  58  POINT (-1402056.923 450793.917)   \n",
      "1074141 -1.584160e+06  474432.239176  56  POINT (-1584159.554 474432.239)   \n",
      "1185264 -1.321512e+06  366746.548674  11  POINT (-1321511.528 366746.549)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1092370        13495  199782.0      12  \n",
      "1097773        13494  199780.0      12  \n",
      "1076767         2478    7768.0      10  \n",
      "1074144         3135   12674.0       9  \n",
      "1071440         3135   12674.0       9  \n",
      "1097780         3964   19120.0      12  \n",
      "1098681        13493  199779.0      12  \n",
      "1098676         3964   19120.0      12  \n",
      "1074141         3105   12544.0       9  \n",
      "1185264         1613    4303.0      78  \n",
      "\n",
      "Exported as table, year 2014. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2014. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "     Sett_ID  NTLct2014\n",
      "83  126282.0          1\n",
      "2     2563.0          1\n",
      "58   24546.0          4\n",
      "41   12670.0        129\n",
      "48   12893.0         18\n",
      "1     1707.0          4\n",
      "11    7750.0         41\n",
      "33   12534.0          1\n",
      "78   74960.0          1\n",
      "62   24569.0          1\n",
      "\n",
      "Values summed to settlement level, year 2014. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2014\n",
      "91   146160.0          14\n",
      "24     9087.0         230\n",
      "11     7750.0         540\n",
      "97   191289.0          10\n",
      "46    12883.0        2234\n",
      "102  197714.0          19\n",
      "14     7803.0         225\n",
      "109  199779.0        8216\n",
      "100  197701.0          14\n",
      "98   193523.0          10\n",
      "\n",
      "Values averaged to settlement level, year 2014. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2014\n",
      "23     8282.0   15.000000\n",
      "51    19120.0   40.015086\n",
      "35    12546.0   18.000000\n",
      "73    49133.0   13.090909\n",
      "101  197705.0   14.000000\n",
      "9      5435.0   10.750000\n",
      "33    12534.0   11.000000\n",
      "39    12657.0   17.000000\n",
      "100  197701.0   14.000000\n",
      "31    12521.0   43.945378\n",
      "\n",
      "Merged year 2014 onto latest year settlement feature layer. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "11033  123732.0     209        NaN         NaN         NaN        NaN   \n",
      "442      1270.0      17        NaN         NaN         NaN        NaN   \n",
      "656      1794.0     324        NaN         NaN         NaN        NaN   \n",
      "1866     5214.0      42        NaN         NaN         NaN        NaN   \n",
      "6198    36303.0      52        NaN         NaN         NaN        NaN   \n",
      "13145  198117.0     299        NaN         NaN         NaN        NaN   \n",
      "3772    16938.0      37        NaN         NaN         NaN        NaN   \n",
      "13308  198804.0      85        NaN         NaN         NaN        NaN   \n",
      "6474    38241.0      64        NaN         NaN         NaN        NaN   \n",
      "10827  108585.0     195        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2011  \\\n",
      "11033         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "442           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "656           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "1866          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6198          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "13145         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "3772          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "13308         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6474          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "10827         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2012  NTLsum2012  NTLavg2012  NTLct2013  NTLsum2013  NTLavg2013  \\\n",
      "11033        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "442          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "656          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "1866         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6198         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "13145        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "3772         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "13308        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6474         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "10827        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2014  NTLsum2014  NTLavg2014  \n",
      "11033        NaN         NaN         NaN  \n",
      "442          NaN         NaN         NaN  \n",
      "656          NaN         NaN         NaN  \n",
      "1866         NaN         NaN         NaN  \n",
      "6198         NaN         NaN         NaN  \n",
      "13145        NaN         NaN         NaN  \n",
      "3772         NaN         NaN         NaN  \n",
      "13308        NaN         NaN         NaN  \n",
      "6474         NaN         NaN         NaN  \n",
      "10827        NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 50 columns]\n",
      "Loading data for year 2015. Thu Dec  8 22:56:46 2022 \n",
      "\n",
      "Finished gdal.Translate() for year 2015. Thu Dec  8 22:56:47 2022 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe, year 2015. Thu Dec  8 22:56:49 2022 \n",
      "\n",
      "Created geodataframe from non-NoData points, year 2015. Thu Dec  8 22:56:49 2022 \n",
      "\n",
      "\n",
      "Joined settlement ID onto vectorized raster cells for year 2015. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "                    X              Y   Z                         geometry  \\\n",
      "1065121 -1.592914e+06  483187.173363  13  POINT (-1592914.488 483187.173)   \n",
      "1188595 -1.560521e+06  363244.574999  10  POINT (-1560521.232 363244.575)   \n",
      "1089667 -1.401181e+06  459548.851058  59  POINT (-1401181.429 459548.851)   \n",
      "1080451 -1.581533e+06  468303.785245  37  POINT (-1581533.074 468303.785)   \n",
      "1097774 -1.402932e+06  451669.410289  60  POINT (-1402932.416 451669.410)   \n",
      "1091474 -1.396804e+06  457797.864220  58  POINT (-1396803.962 457797.864)   \n",
      "1062366 -1.638440e+06  485813.653619  12  POINT (-1638440.146 485813.654)   \n",
      "1067832 -1.585911e+06  480560.693107  43  POINT (-1585910.541 480560.693)   \n",
      "1206610 -1.564899e+06  345734.706625  10  POINT (-1564898.699 345734.707)   \n",
      "1086064 -1.400306e+06  463050.824733  51  POINT (-1400305.936 463050.825)   \n",
      "\n",
      "         index_right   Sett_ID  ADM_ID  \n",
      "1065121         3149   12883.0       9  \n",
      "1188595          260     802.0      11  \n",
      "1089667        13494  199780.0      12  \n",
      "1080451         3135   12674.0       9  \n",
      "1097774        13494  199780.0      12  \n",
      "1091474        13493  199779.0      12  \n",
      "1062366         2461    7750.0     319  \n",
      "1067832         3083   12521.0       9  \n",
      "1206610          167     464.0      11  \n",
      "1086064        13496  199787.0      12  \n",
      "\n",
      "Exported as table, year 2015. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "\n",
      "Cells per settlement counted, year 2015. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "      Sett_ID  NTLct2015\n",
      "38    10079.0          1\n",
      "138  197715.0          1\n",
      "152  199780.0        146\n",
      "54    12674.0        129\n",
      "80    21650.0          7\n",
      "22     7750.0         42\n",
      "7      2572.0         24\n",
      "36     9088.0          6\n",
      "142  199333.0          1\n",
      "134  197701.0          1\n",
      "\n",
      "Values summed to settlement level, year 2015. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "      Sett_ID  NTLsum2015\n",
      "66    12905.0          26\n",
      "89    24557.0          13\n",
      "97    25047.0          17\n",
      "15     4298.0          46\n",
      "92    24589.0          68\n",
      "46    12544.0        6621\n",
      "79    21036.0          11\n",
      "117  126287.0          10\n",
      "55    12684.0        6621\n",
      "128  182922.0          48\n",
      "\n",
      "Values averaged to settlement level, year 2015. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "      Sett_ID  NTLavg2015\n",
      "126  151749.0   25.000000\n",
      "92    24589.0   22.666667\n",
      "110   74883.0   10.500000\n",
      "3       464.0   10.285714\n",
      "141  199330.0   10.000000\n",
      "28     7891.0   28.500000\n",
      "44    12534.0   13.000000\n",
      "34     8282.0   13.000000\n",
      "93    24592.0   11.000000\n",
      "14     4158.0   10.250000\n",
      "\n",
      "Merged year 2015 onto latest year settlement feature layer. Thu Dec  8 22:56:50 2022 \n",
      "\n",
      "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
      "13090  197944.0     299        NaN         NaN         NaN        NaN   \n",
      "5966    33701.0      97        NaN         NaN         NaN        NaN   \n",
      "282       874.0      15        NaN         NaN         NaN        NaN   \n",
      "426      1236.0      18        NaN         NaN         NaN        NaN   \n",
      "11572  146121.0       2        NaN         NaN         NaN        NaN   \n",
      "8568    61443.0     158        NaN         NaN         NaN        NaN   \n",
      "921      2642.0       7        NaN         NaN         NaN        NaN   \n",
      "7002    48363.0     172        NaN         NaN         NaN        NaN   \n",
      "6803    46457.0     181        NaN         NaN         NaN        NaN   \n",
      "4520    20599.0     272        NaN         NaN         NaN        NaN   \n",
      "\n",
      "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2012  \\\n",
      "13090         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "5966          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "282           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "426           NaN         NaN        NaN         NaN  ...         NaN   \n",
      "11572         NaN         NaN        NaN         NaN  ...         NaN   \n",
      "8568          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "921           NaN         NaN        NaN         NaN  ...        10.0   \n",
      "7002          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "6803          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "4520          NaN         NaN        NaN         NaN  ...         NaN   \n",
      "\n",
      "       NTLct2013  NTLsum2013  NTLavg2013  NTLct2014  NTLsum2014  NTLavg2014  \\\n",
      "13090        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "5966         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "282          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "426          NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "11572        NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "8568         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "921          1.0        10.0        10.0        NaN         NaN         NaN   \n",
      "7002         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "6803         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "4520         NaN         NaN         NaN        NaN         NaN         NaN   \n",
      "\n",
      "       NTLct2015  NTLsum2015  NTLavg2015  \n",
      "13090        NaN         NaN         NaN  \n",
      "5966         NaN         NaN         NaN  \n",
      "282          NaN         NaN         NaN  \n",
      "426          NaN         NaN         NaN  \n",
      "11572        NaN         NaN         NaN  \n",
      "8568         NaN         NaN         NaN  \n",
      "921          1.0        10.0        10.0  \n",
      "7002         NaN         NaN         NaN  \n",
      "6803         NaN         NaN         NaN  \n",
      "4520         NaN         NaN         NaN  \n",
      "\n",
      "[10 rows x 53 columns]\n",
      "\n",
      "\n",
      "Finished. All years masked and assigned their nearest settlement. Thu Dec  8 22:56:50 2022\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Thu Dec  8 22:56:50 2022 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for YearFile in AnnualizedMaskedFiles:\n",
    "    \n",
    "### STEP 1: TIF TO XYZ ###\n",
    "    InputRasterName = os.path.join(ProjectFolder, \"NTL\", YearFile)\n",
    "    Year = str(re.sub(r'[^0-9]', '', YearFile))\n",
    "    print('Loading data for year %s. %s \\n' % (Year, time.ctime()))\n",
    "    InputRasterObject = gdal.Open(InputRasterName)\n",
    "    XYZOutputPath = r'NTL/{}'.format(\n",
    "        YearFile.replace('.tif', '.xyz')) # New file path will be the same as original, but .tif is replaced with .xyz\n",
    "    \n",
    "    # Create an .xyz version of the .tif\n",
    "    XYZ = gdal.Translate(XYZOutputPath, # Specify a destination path\n",
    "                         InputRasterObject, # Input is the masked .tif file\n",
    "                         format='XYZ', \n",
    "                         creationOptions=[\"ADD_HEADER_LINE=YES\"])\n",
    "    print('Finished gdal.Translate() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "\n",
    "#     # Remove the temporary masked tif file.\n",
    "#     try:  \n",
    "#         os.remove(InputRasterName)\n",
    "#     except OSError:\n",
    "#         pass\n",
    "#     print('Removed (or skipped if error) intermediate tif file. %s \\n' % time.ctime())\n",
    "    \n",
    "    InputRasterObject = None\n",
    "    XYZ = None # Reload XYZ as a point geodataframe\n",
    "\n",
    "    \n",
    "### STEP 2: GENERATE GEODATAFRAME WITH SETT_ID FIELD ###\n",
    "    InputXYZName = ''.join(['Masked_', Year, '.xyz'])\n",
    "    InputXYZ = pd.read_table(os.path.join(ProjectFolder, 'NTL', InputXYZName), delim_whitespace=True)\n",
    "    InputXYZ = InputXYZ.loc[InputXYZ['Z'] >= 10] # Subset to only the cells that contained values of 10 or higher.\n",
    "    print('Loaded XYZ file as a pandas dataframe, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    ValObject = gpd.GeoDataFrame(InputXYZ,\n",
    "                                 geometry = gpd.points_from_xy(InputXYZ['X'], InputXYZ['Y']),\n",
    "                                 crs = 'ESRI:102022')\n",
    "    print('Created geodataframe from non-NoData points, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    del InputXYZ\n",
    "    \n",
    "    # Sjoin_nearest: No need to group by ADM this time. \n",
    "    ValObject_withID = gpd.sjoin_nearest(ValObject, \n",
    "                                    Settlements, \n",
    "                                    how='left') # No need for max_distance parameter this time. We've already narrowed down to nearby raster cells.\n",
    "    \n",
    "    print('\\nJoined settlement ID onto vectorized raster cells for year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValObject_withID.sample(10))\n",
    "    del ValObject\n",
    "    \n",
    "    # We no longer need the spatial information of the raster values because we have their unique settlement ID.\n",
    "    ValObject_withID = pd.DataFrame(ValObject_withID).drop(columns='geometry')\n",
    "    \n",
    "    ValObject_withID.to_csv(''.join([r'NTL/', 'Masked_', Year, '.csv']))\n",
    "    print('\\nExported as table, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    \n",
    "    # Remove the temporary xyz file.\n",
    "    try:  \n",
    "        os.remove(os.path.join(ProjectFolder, 'NTL', InputXYZName))\n",
    "    except OSError:\n",
    "        pass\n",
    "    print('Removed (or skipped if error) intermediate xyz file. %s \\n' % time.ctime())\n",
    "\n",
    "    \n",
    "\n",
    "### STEP 3: AGGREGATE BY SETTLEMENT AND MERGE ONTO SUMMARIES TABLE ###\n",
    "    \n",
    "    # Cell count\n",
    "    VariableName = ''.join(['NTLct', Year])\n",
    "    ValAggregated = ValObject_withID[\n",
    "        ValObject_withID['Z'].notna()].groupby(\n",
    "        'Sett_ID', as_index=False)['Z'].count().rename(columns={\"Z\": VariableName})\n",
    "    print('\\nCells per settlement counted, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValAggregated.sample(10))\n",
    "    AllSummaries = AllSummaries.merge(ValAggregated, how='left', on='Sett_ID')\n",
    "    \n",
    "    # Sum\n",
    "    VariableName = ''.join(['NTLsum', Year])\n",
    "    ValAggregated = ValObject_withID[\n",
    "        ValObject_withID['Z'].notna()].groupby(\n",
    "        'Sett_ID', as_index=False)['Z'].sum().rename(columns={\"Z\": VariableName})\n",
    "    print('\\nValues summed to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValAggregated.sample(10))\n",
    "    AllSummaries = AllSummaries.merge(ValAggregated, how='left', on='Sett_ID')\n",
    "    \n",
    "    # Average\n",
    "    VariableName = ''.join(['NTLavg', Year])\n",
    "    ValAggregated = ValObject_withID[\n",
    "        ValObject_withID['Z'].notna()].groupby(\n",
    "        'Sett_ID', as_index=False)['Z'].mean().rename(columns={\"Z\": VariableName})\n",
    "    print('\\nValues averaged to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "    print(ValAggregated.sample(10))\n",
    "    AllSummaries = AllSummaries.merge(ValAggregated, how='left', on='Sett_ID')\n",
    "    print('\\nMerged year %s onto latest year settlement feature layer. %s \\n' % (Year, time.ctime()))\n",
    "    \n",
    "    \n",
    "    print(AllSummaries.sample(10))\n",
    "    del ValObject_withID, ValAggregated\n",
    "    \n",
    "    \n",
    "\n",
    "print('\\n\\nFinished. All years masked and assigned their nearest settlement. %s' % time.ctime())\n",
    "\n",
    "AllSummaries.to_csv(os.path.join(ResultsFolder, 'NTL%sto%s.csv' % (ValStart, ValEnd)))\n",
    "print('Saved to file. %s \\n' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "19996d8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>NTLct1999</th>\n",
       "      <th>NTLsum1999</th>\n",
       "      <th>NTLavg1999</th>\n",
       "      <th>NTLct2000</th>\n",
       "      <th>NTLsum2000</th>\n",
       "      <th>NTLavg2000</th>\n",
       "      <th>NTLct2001</th>\n",
       "      <th>NTLsum2001</th>\n",
       "      <th>...</th>\n",
       "      <th>NTLavg2012</th>\n",
       "      <th>NTLct2013</th>\n",
       "      <th>NTLsum2013</th>\n",
       "      <th>NTLavg2013</th>\n",
       "      <th>NTLct2014</th>\n",
       "      <th>NTLsum2014</th>\n",
       "      <th>NTLavg2014</th>\n",
       "      <th>NTLct2015</th>\n",
       "      <th>NTLsum2015</th>\n",
       "      <th>NTLavg2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3964</th>\n",
       "      <td>19120.0</td>\n",
       "      <td>12</td>\n",
       "      <td>298.0</td>\n",
       "      <td>10469.0</td>\n",
       "      <td>35.130872</td>\n",
       "      <td>291.0</td>\n",
       "      <td>10540.0</td>\n",
       "      <td>36.219931</td>\n",
       "      <td>303.0</td>\n",
       "      <td>10930.0</td>\n",
       "      <td>...</td>\n",
       "      <td>39.063205</td>\n",
       "      <td>444.0</td>\n",
       "      <td>17058.0</td>\n",
       "      <td>38.418919</td>\n",
       "      <td>464.0</td>\n",
       "      <td>18567.0</td>\n",
       "      <td>40.015086</td>\n",
       "      <td>476.0</td>\n",
       "      <td>20262.0</td>\n",
       "      <td>42.567227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3083</th>\n",
       "      <td>12521.0</td>\n",
       "      <td>9</td>\n",
       "      <td>185.0</td>\n",
       "      <td>7030.0</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>185.0</td>\n",
       "      <td>6926.0</td>\n",
       "      <td>37.437838</td>\n",
       "      <td>185.0</td>\n",
       "      <td>6955.0</td>\n",
       "      <td>...</td>\n",
       "      <td>44.029536</td>\n",
       "      <td>237.0</td>\n",
       "      <td>10674.0</td>\n",
       "      <td>45.037975</td>\n",
       "      <td>238.0</td>\n",
       "      <td>10459.0</td>\n",
       "      <td>43.945378</td>\n",
       "      <td>238.0</td>\n",
       "      <td>10476.0</td>\n",
       "      <td>44.016807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13493</th>\n",
       "      <td>199779.0</td>\n",
       "      <td>12</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7079.0</td>\n",
       "      <td>48.820690</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7090.0</td>\n",
       "      <td>48.896552</td>\n",
       "      <td>146.0</td>\n",
       "      <td>7282.0</td>\n",
       "      <td>...</td>\n",
       "      <td>59.493151</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8553.0</td>\n",
       "      <td>58.582192</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8216.0</td>\n",
       "      <td>56.273973</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8436.0</td>\n",
       "      <td>57.780822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13494</th>\n",
       "      <td>199780.0</td>\n",
       "      <td>12</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7079.0</td>\n",
       "      <td>48.820690</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7090.0</td>\n",
       "      <td>48.896552</td>\n",
       "      <td>146.0</td>\n",
       "      <td>7282.0</td>\n",
       "      <td>...</td>\n",
       "      <td>59.493151</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8553.0</td>\n",
       "      <td>58.582192</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8216.0</td>\n",
       "      <td>56.273973</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8436.0</td>\n",
       "      <td>57.780822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13496</th>\n",
       "      <td>199787.0</td>\n",
       "      <td>12</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7079.0</td>\n",
       "      <td>48.820690</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7090.0</td>\n",
       "      <td>48.896552</td>\n",
       "      <td>146.0</td>\n",
       "      <td>7282.0</td>\n",
       "      <td>...</td>\n",
       "      <td>59.493151</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8553.0</td>\n",
       "      <td>58.582192</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8216.0</td>\n",
       "      <td>56.273973</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8436.0</td>\n",
       "      <td>57.780822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13495</th>\n",
       "      <td>199782.0</td>\n",
       "      <td>12</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7079.0</td>\n",
       "      <td>48.820690</td>\n",
       "      <td>145.0</td>\n",
       "      <td>7090.0</td>\n",
       "      <td>48.896552</td>\n",
       "      <td>146.0</td>\n",
       "      <td>7282.0</td>\n",
       "      <td>...</td>\n",
       "      <td>59.493151</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8553.0</td>\n",
       "      <td>58.582192</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8216.0</td>\n",
       "      <td>56.273973</td>\n",
       "      <td>146.0</td>\n",
       "      <td>8436.0</td>\n",
       "      <td>57.780822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3137</th>\n",
       "      <td>12684.0</td>\n",
       "      <td>9</td>\n",
       "      <td>125.0</td>\n",
       "      <td>5382.0</td>\n",
       "      <td>43.056000</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5283.0</td>\n",
       "      <td>41.928571</td>\n",
       "      <td>127.0</td>\n",
       "      <td>5312.0</td>\n",
       "      <td>...</td>\n",
       "      <td>54.782946</td>\n",
       "      <td>129.0</td>\n",
       "      <td>7110.0</td>\n",
       "      <td>55.116279</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6669.0</td>\n",
       "      <td>51.697674</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6621.0</td>\n",
       "      <td>51.325581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3105</th>\n",
       "      <td>12544.0</td>\n",
       "      <td>9</td>\n",
       "      <td>125.0</td>\n",
       "      <td>5382.0</td>\n",
       "      <td>43.056000</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5283.0</td>\n",
       "      <td>41.928571</td>\n",
       "      <td>127.0</td>\n",
       "      <td>5312.0</td>\n",
       "      <td>...</td>\n",
       "      <td>54.782946</td>\n",
       "      <td>129.0</td>\n",
       "      <td>7110.0</td>\n",
       "      <td>55.116279</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6669.0</td>\n",
       "      <td>51.697674</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6621.0</td>\n",
       "      <td>51.325581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3135</th>\n",
       "      <td>12674.0</td>\n",
       "      <td>9</td>\n",
       "      <td>125.0</td>\n",
       "      <td>5382.0</td>\n",
       "      <td>43.056000</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5283.0</td>\n",
       "      <td>41.928571</td>\n",
       "      <td>127.0</td>\n",
       "      <td>5312.0</td>\n",
       "      <td>...</td>\n",
       "      <td>54.782946</td>\n",
       "      <td>129.0</td>\n",
       "      <td>7110.0</td>\n",
       "      <td>55.116279</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6669.0</td>\n",
       "      <td>51.697674</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6621.0</td>\n",
       "      <td>51.325581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3133</th>\n",
       "      <td>12670.0</td>\n",
       "      <td>9</td>\n",
       "      <td>125.0</td>\n",
       "      <td>5382.0</td>\n",
       "      <td>43.056000</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5283.0</td>\n",
       "      <td>41.928571</td>\n",
       "      <td>127.0</td>\n",
       "      <td>5312.0</td>\n",
       "      <td>...</td>\n",
       "      <td>54.782946</td>\n",
       "      <td>129.0</td>\n",
       "      <td>7110.0</td>\n",
       "      <td>55.116279</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6669.0</td>\n",
       "      <td>51.697674</td>\n",
       "      <td>129.0</td>\n",
       "      <td>6621.0</td>\n",
       "      <td>51.325581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3149</th>\n",
       "      <td>12883.0</td>\n",
       "      <td>9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>30.769231</td>\n",
       "      <td>66.0</td>\n",
       "      <td>2064.0</td>\n",
       "      <td>31.272727</td>\n",
       "      <td>66.0</td>\n",
       "      <td>2061.0</td>\n",
       "      <td>...</td>\n",
       "      <td>39.393939</td>\n",
       "      <td>66.0</td>\n",
       "      <td>2506.0</td>\n",
       "      <td>37.969697</td>\n",
       "      <td>66.0</td>\n",
       "      <td>2234.0</td>\n",
       "      <td>33.848485</td>\n",
       "      <td>66.0</td>\n",
       "      <td>2172.0</td>\n",
       "      <td>32.909091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11448</th>\n",
       "      <td>145497.0</td>\n",
       "      <td>2</td>\n",
       "      <td>65.0</td>\n",
       "      <td>1520.0</td>\n",
       "      <td>23.384615</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1618.0</td>\n",
       "      <td>24.149254</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1835.0</td>\n",
       "      <td>...</td>\n",
       "      <td>31.093333</td>\n",
       "      <td>76.0</td>\n",
       "      <td>2315.0</td>\n",
       "      <td>30.460526</td>\n",
       "      <td>74.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>27.324324</td>\n",
       "      <td>75.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>26.760000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8284</th>\n",
       "      <td>59177.0</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>502.0</td>\n",
       "      <td>14.342857</td>\n",
       "      <td>36.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>13.777778</td>\n",
       "      <td>36.0</td>\n",
       "      <td>520.0</td>\n",
       "      <td>...</td>\n",
       "      <td>17.825397</td>\n",
       "      <td>62.0</td>\n",
       "      <td>1050.0</td>\n",
       "      <td>16.935484</td>\n",
       "      <td>52.0</td>\n",
       "      <td>775.0</td>\n",
       "      <td>14.903846</td>\n",
       "      <td>54.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>14.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11072</th>\n",
       "      <td>126049.0</td>\n",
       "      <td>13</td>\n",
       "      <td>56.0</td>\n",
       "      <td>1004.0</td>\n",
       "      <td>17.928571</td>\n",
       "      <td>48.0</td>\n",
       "      <td>872.0</td>\n",
       "      <td>18.166667</td>\n",
       "      <td>54.0</td>\n",
       "      <td>973.0</td>\n",
       "      <td>...</td>\n",
       "      <td>19.724138</td>\n",
       "      <td>60.0</td>\n",
       "      <td>1215.0</td>\n",
       "      <td>20.250000</td>\n",
       "      <td>52.0</td>\n",
       "      <td>914.0</td>\n",
       "      <td>17.576923</td>\n",
       "      <td>56.0</td>\n",
       "      <td>1085.0</td>\n",
       "      <td>19.375000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7161</th>\n",
       "      <td>49133.0</td>\n",
       "      <td>6</td>\n",
       "      <td>50.0</td>\n",
       "      <td>830.0</td>\n",
       "      <td>16.600000</td>\n",
       "      <td>47.0</td>\n",
       "      <td>804.0</td>\n",
       "      <td>17.106383</td>\n",
       "      <td>49.0</td>\n",
       "      <td>823.0</td>\n",
       "      <td>...</td>\n",
       "      <td>17.301587</td>\n",
       "      <td>63.0</td>\n",
       "      <td>1087.0</td>\n",
       "      <td>17.253968</td>\n",
       "      <td>44.0</td>\n",
       "      <td>576.0</td>\n",
       "      <td>13.090909</td>\n",
       "      <td>52.0</td>\n",
       "      <td>689.0</td>\n",
       "      <td>13.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7853</th>\n",
       "      <td>54759.0</td>\n",
       "      <td>6</td>\n",
       "      <td>29.0</td>\n",
       "      <td>517.0</td>\n",
       "      <td>17.827586</td>\n",
       "      <td>27.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>18.370370</td>\n",
       "      <td>28.0</td>\n",
       "      <td>506.0</td>\n",
       "      <td>...</td>\n",
       "      <td>18.696970</td>\n",
       "      <td>33.0</td>\n",
       "      <td>628.0</td>\n",
       "      <td>19.030303</td>\n",
       "      <td>28.0</td>\n",
       "      <td>382.0</td>\n",
       "      <td>13.642857</td>\n",
       "      <td>33.0</td>\n",
       "      <td>451.0</td>\n",
       "      <td>13.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12243</th>\n",
       "      <td>190914.0</td>\n",
       "      <td>262</td>\n",
       "      <td>13.0</td>\n",
       "      <td>232.0</td>\n",
       "      <td>17.846154</td>\n",
       "      <td>14.0</td>\n",
       "      <td>265.0</td>\n",
       "      <td>18.928571</td>\n",
       "      <td>10.0</td>\n",
       "      <td>206.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.750000</td>\n",
       "      <td>26.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>23.038462</td>\n",
       "      <td>24.0</td>\n",
       "      <td>703.0</td>\n",
       "      <td>29.291667</td>\n",
       "      <td>23.0</td>\n",
       "      <td>678.0</td>\n",
       "      <td>29.478261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10039</th>\n",
       "      <td>74882.0</td>\n",
       "      <td>8</td>\n",
       "      <td>15.0</td>\n",
       "      <td>218.0</td>\n",
       "      <td>14.533333</td>\n",
       "      <td>19.0</td>\n",
       "      <td>263.0</td>\n",
       "      <td>13.842105</td>\n",
       "      <td>22.0</td>\n",
       "      <td>308.0</td>\n",
       "      <td>...</td>\n",
       "      <td>19.161290</td>\n",
       "      <td>31.0</td>\n",
       "      <td>541.0</td>\n",
       "      <td>17.451613</td>\n",
       "      <td>34.0</td>\n",
       "      <td>489.0</td>\n",
       "      <td>14.382353</td>\n",
       "      <td>35.0</td>\n",
       "      <td>506.0</td>\n",
       "      <td>14.457143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>7747.0</td>\n",
       "      <td>10</td>\n",
       "      <td>22.0</td>\n",
       "      <td>363.0</td>\n",
       "      <td>16.500000</td>\n",
       "      <td>22.0</td>\n",
       "      <td>315.0</td>\n",
       "      <td>14.318182</td>\n",
       "      <td>24.0</td>\n",
       "      <td>377.0</td>\n",
       "      <td>...</td>\n",
       "      <td>14.937500</td>\n",
       "      <td>36.0</td>\n",
       "      <td>612.0</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>38.0</td>\n",
       "      <td>589.0</td>\n",
       "      <td>15.500000</td>\n",
       "      <td>38.0</td>\n",
       "      <td>603.0</td>\n",
       "      <td>15.868421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2641</th>\n",
       "      <td>9087.0</td>\n",
       "      <td>5</td>\n",
       "      <td>11.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>14.818182</td>\n",
       "      <td>9.0</td>\n",
       "      <td>139.0</td>\n",
       "      <td>15.444444</td>\n",
       "      <td>10.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>...</td>\n",
       "      <td>17.066667</td>\n",
       "      <td>16.0</td>\n",
       "      <td>270.0</td>\n",
       "      <td>16.875000</td>\n",
       "      <td>17.0</td>\n",
       "      <td>230.0</td>\n",
       "      <td>13.529412</td>\n",
       "      <td>17.0</td>\n",
       "      <td>245.0</td>\n",
       "      <td>14.411765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20 rows × 53 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Sett_ID  ADM_ID  NTLct1999  NTLsum1999  NTLavg1999  NTLct2000  \\\n",
       "3964    19120.0      12      298.0     10469.0   35.130872      291.0   \n",
       "3083    12521.0       9      185.0      7030.0   38.000000      185.0   \n",
       "13493  199779.0      12      145.0      7079.0   48.820690      145.0   \n",
       "13494  199780.0      12      145.0      7079.0   48.820690      145.0   \n",
       "13496  199787.0      12      145.0      7079.0   48.820690      145.0   \n",
       "13495  199782.0      12      145.0      7079.0   48.820690      145.0   \n",
       "3137    12684.0       9      125.0      5382.0   43.056000      126.0   \n",
       "3105    12544.0       9      125.0      5382.0   43.056000      126.0   \n",
       "3135    12674.0       9      125.0      5382.0   43.056000      126.0   \n",
       "3133    12670.0       9      125.0      5382.0   43.056000      126.0   \n",
       "3149    12883.0       9       65.0      2000.0   30.769231       66.0   \n",
       "11448  145497.0       2       65.0      1520.0   23.384615       67.0   \n",
       "8284    59177.0       1       35.0       502.0   14.342857       36.0   \n",
       "11072  126049.0      13       56.0      1004.0   17.928571       48.0   \n",
       "7161    49133.0       6       50.0       830.0   16.600000       47.0   \n",
       "7853    54759.0       6       29.0       517.0   17.827586       27.0   \n",
       "12243  190914.0     262       13.0       232.0   17.846154       14.0   \n",
       "10039   74882.0       8       15.0       218.0   14.533333       19.0   \n",
       "2458     7747.0      10       22.0       363.0   16.500000       22.0   \n",
       "2641     9087.0       5       11.0       163.0   14.818182        9.0   \n",
       "\n",
       "       NTLsum2000  NTLavg2000  NTLct2001  NTLsum2001  ...  NTLavg2012  \\\n",
       "3964      10540.0   36.219931      303.0     10930.0  ...   39.063205   \n",
       "3083       6926.0   37.437838      185.0      6955.0  ...   44.029536   \n",
       "13493      7090.0   48.896552      146.0      7282.0  ...   59.493151   \n",
       "13494      7090.0   48.896552      146.0      7282.0  ...   59.493151   \n",
       "13496      7090.0   48.896552      146.0      7282.0  ...   59.493151   \n",
       "13495      7090.0   48.896552      146.0      7282.0  ...   59.493151   \n",
       "3137       5283.0   41.928571      127.0      5312.0  ...   54.782946   \n",
       "3105       5283.0   41.928571      127.0      5312.0  ...   54.782946   \n",
       "3135       5283.0   41.928571      127.0      5312.0  ...   54.782946   \n",
       "3133       5283.0   41.928571      127.0      5312.0  ...   54.782946   \n",
       "3149       2064.0   31.272727       66.0      2061.0  ...   39.393939   \n",
       "11448      1618.0   24.149254       70.0      1835.0  ...   31.093333   \n",
       "8284        496.0   13.777778       36.0       520.0  ...   17.825397   \n",
       "11072       872.0   18.166667       54.0       973.0  ...   19.724138   \n",
       "7161        804.0   17.106383       49.0       823.0  ...   17.301587   \n",
       "7853        496.0   18.370370       28.0       506.0  ...   18.696970   \n",
       "12243       265.0   18.928571       10.0       206.0  ...   23.750000   \n",
       "10039       263.0   13.842105       22.0       308.0  ...   19.161290   \n",
       "2458        315.0   14.318182       24.0       377.0  ...   14.937500   \n",
       "2641        139.0   15.444444       10.0       155.0  ...   17.066667   \n",
       "\n",
       "       NTLct2013  NTLsum2013  NTLavg2013  NTLct2014  NTLsum2014  NTLavg2014  \\\n",
       "3964       444.0     17058.0   38.418919      464.0     18567.0   40.015086   \n",
       "3083       237.0     10674.0   45.037975      238.0     10459.0   43.945378   \n",
       "13493      146.0      8553.0   58.582192      146.0      8216.0   56.273973   \n",
       "13494      146.0      8553.0   58.582192      146.0      8216.0   56.273973   \n",
       "13496      146.0      8553.0   58.582192      146.0      8216.0   56.273973   \n",
       "13495      146.0      8553.0   58.582192      146.0      8216.0   56.273973   \n",
       "3137       129.0      7110.0   55.116279      129.0      6669.0   51.697674   \n",
       "3105       129.0      7110.0   55.116279      129.0      6669.0   51.697674   \n",
       "3135       129.0      7110.0   55.116279      129.0      6669.0   51.697674   \n",
       "3133       129.0      7110.0   55.116279      129.0      6669.0   51.697674   \n",
       "3149        66.0      2506.0   37.969697       66.0      2234.0   33.848485   \n",
       "11448       76.0      2315.0   30.460526       74.0      2022.0   27.324324   \n",
       "8284        62.0      1050.0   16.935484       52.0       775.0   14.903846   \n",
       "11072       60.0      1215.0   20.250000       52.0       914.0   17.576923   \n",
       "7161        63.0      1087.0   17.253968       44.0       576.0   13.090909   \n",
       "7853        33.0       628.0   19.030303       28.0       382.0   13.642857   \n",
       "12243       26.0       599.0   23.038462       24.0       703.0   29.291667   \n",
       "10039       31.0       541.0   17.451613       34.0       489.0   14.382353   \n",
       "2458        36.0       612.0   17.000000       38.0       589.0   15.500000   \n",
       "2641        16.0       270.0   16.875000       17.0       230.0   13.529412   \n",
       "\n",
       "       NTLct2015  NTLsum2015  NTLavg2015  \n",
       "3964       476.0     20262.0   42.567227  \n",
       "3083       238.0     10476.0   44.016807  \n",
       "13493      146.0      8436.0   57.780822  \n",
       "13494      146.0      8436.0   57.780822  \n",
       "13496      146.0      8436.0   57.780822  \n",
       "13495      146.0      8436.0   57.780822  \n",
       "3137       129.0      6621.0   51.325581  \n",
       "3105       129.0      6621.0   51.325581  \n",
       "3135       129.0      6621.0   51.325581  \n",
       "3133       129.0      6621.0   51.325581  \n",
       "3149        66.0      2172.0   32.909091  \n",
       "11448       75.0      2007.0   26.760000  \n",
       "8284        54.0       768.0   14.222222  \n",
       "11072       56.0      1085.0   19.375000  \n",
       "7161        52.0       689.0   13.250000  \n",
       "7853        33.0       451.0   13.666667  \n",
       "12243       23.0       678.0   29.478261  \n",
       "10039       35.0       506.0   14.457143  \n",
       "2458        38.0       603.0   15.868421  \n",
       "2641        17.0       245.0   14.411765  \n",
       "\n",
       "[20 rows x 53 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "AllSummaries.sort_values('NTLsum2010', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1416ef51",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3343240",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30df1869",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}