Settlement_Panel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0f2016b3",
   "metadata": {},
   "source": [
    "# Spatiotemporal Trends in Urban and Rural Settlements\n",
    "## Overview\n",
    "This repository investigates year-by-year change in cities and settlements in Central and Western Africa (CWA). The goal is to capture activity for every settlement locality in a country to produce indicators that are high frequency, spatially granular, and timely. The Jupyter Notebook is the primary script used to construct each country's dataset. It tracks population, built-area, and economic and climate indicators across a 16-year timeframe from 2000 to 2015. \n",
    "\n",
    "The repository is split into three sections: methodology and notebooks, source data, and outputs. Outputs are organized by country and include growth tables (\"urban panel datasets\"), charts, and country briefs.\n",
    "\n",
    "\n",
    "## Datasets\n",
    "Datasets used to create each African country's urban panel data are as follows:\n",
    "1. Most up-to-date administrative boundaries.\n",
    "2. City names: **UCDB, Africapolis, and GeoNames.**\n",
    "3. Settlement types: **GRID3 settlement extents.** Captured between 2009-2019.\n",
    "4. Built-up area, yearly: **World Settlement Footprint Evolution.** Resolution: 30m.\n",
    "5. Population, yearly: **WorldPop.** UN-adjusted, unconstrained. Resolution: 100m.\n",
    "6. Nighttime lights, yearly: **Harmonization of DMSP and VIIRS.** Resolution: 1km and 500m.\n",
    "7. Flood extents, by return period: **FATHOM.** Resolution: 90m.\n",
    "\n",
    "\n",
    "## Accessing Data\n",
    "Source data are available to the public by providers listed in the previous section, with the exception of flood data. Please note that the source data files in this repository have been fit for purpose and may not cover your area of interest. Some sources are also not global; GRID3 settlement extents are only available for sub-Saharan Africa, and Africapolis names for Africa.\n",
    "\n",
    "Results from the analysis are currently available for Cameroon and are under development for Central African Republic and four Sahel countries: Burkina Faso, Chad, Mali, and Niger. Results are available in the outputs folder by country. Please contact the CWA Geospatial team to inquire about new locations.\n",
    "<br>\n",
    "> **Walker Kosmidou-Bradley**, wkosmidoubradley@worldbank.org\n",
    "<br>\n",
    "> **Grace Doherty**, gdoherty2@worldbank.org\n",
    "\n",
    "## License\n",
    "Materials under this repository are open-source under an MIT license. The community is invited to test, adapt, and re-purpose materials as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85fd14a1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9276b447",
   "metadata": {},
   "source": [
    "## 1. PREPARE WORKSPACE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0190f8",
   "metadata": {},
   "source": [
    "### 1.1 Off-script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9093fd6",
   "metadata": {},
   "source": [
    "##### Off-script: Create folders in your working directory. (The folder where you are storing this script).\n",
    "> *ADM\n",
    "<br>Buildup\n",
    "<br>PlaceName\n",
    "<br>Population\n",
    "<br>Settlement\n",
    "<br>NTL*\n",
    "\n",
    "##### Before starting: Download datasets (as shapefile, GeoJSON, or tif where possible) and place or extract into corresponding folder. You can download the cleaned files from our [GitHub Repository](https://github.com/worldbank/Urban_Spatio_Temporal_Trends) or access original sources here:\n",
    "- ADM: *Varies by source.*\n",
    "- Buildup: https://download.geoservice.dlr.de/WSF_EVO/files/\n",
    "    - If more than one tif, name the tifs as follows: WSFE1.tif, WSFE2.tif... etc.\n",
    "- PlaceName: \n",
    "    - GeoNames: (file: cities500.zip) https://download.geonames.org/export/dump/\n",
    "    - Africapolis: https://africapolis.org/en/data\n",
    "    - Urban Centres Database: https://ghsl.jrc.ec.europa.eu/ghs_stat_ucdb2015mt_r2019a.php\n",
    "- Population: https://hub.worldpop.org/geodata/listing?id=69\n",
    "- Settlement: https://data.grid3.org/datasets/GRID3::grid3-cameroon-settlement-extents-version-01-01-/explore\n",
    "- Nighttime Lights: https://eogdata.mines.edu/products/dmsp/#v4 and https://eogdata.mines.edu/products/vnl/#annual_v2\n",
    "\n",
    "##### Other off-script:\n",
    "- Convert GeoNames from .txt file to shape (delimiter = tab, header rows = 0) and rename fields.\n",
    "- If necessary, mosaic WSFE rasters that cover the area of interest to create a single file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2a2548",
   "metadata": {},
   "source": [
    "### 1.2 Load all packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "8c42795f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Built-in:\n",
    "# dir(), print(), range(), format(), int(), len(), list(), max(), min(), zip(), sorted(), sum(), open(), del, = None, try except, with as, for in, if elif else\n",
    "# Also: list.append(), list.insert(), list.remove(), count(), startswith(), endswith(), contains(), replace()\n",
    "\n",
    "import os, sys, glob, re, time, subprocess, string # os.getcwd(), os.path.join(), os.listdir(), os.remove(), time.ctime(), glob.glob(), string.zfill(), string.join()\n",
    "from os.path import exists # exists()\n",
    "from functools import reduce # reduce()\n",
    "\n",
    "import geopandas as gpd # read_file(), GeoDataFrame(), sjoin_nearest(), to_crs(), to_file(), .crs, buffer(), dissolve()\n",
    "import pandas as pd # .dtypes, Series(), concat(), DataFrame(), read_table(), merge(), to_csv(), .loc[], head(), sample(), astype(), unique(), rename(), between(), drop(), fillna(), idxmax(), isna(), isin(), apply(), info(), sort_values(), notna(), groupby(), value_counts(), duplicated(), drop_duplicates()\n",
    "from shapely.geometry import Point, LineString, Polygon, shape, MultiPoint\n",
    "from shapely.ops import cascaded_union\n",
    "from shapely.validation import make_valid  # in apply(make_valid)\n",
    "import shapely.wkt\n",
    "\n",
    "import numpy as np # median(), mean(), tolist(), .inf\n",
    "import fiona, rioxarray # fiona.open()\n",
    "import rasterio # open(), write_band(), .name, .count, .width, .height. nodatavals, .meta, update(), copy(), write()\n",
    "from rasterio.plot import show\n",
    "from rasterio import features # features.rasterize()\n",
    "from rasterio.features import shapes\n",
    "from rasterio import mask # rasterio.mask.mask()\n",
    "from rasterio.enums import Resampling # rasterio.enums.Resampling()\n",
    "from rasterstats import zonal_stats # zonal_stats()\n",
    "from osgeo import gdal, osr, ogr, gdal_array, gdalconst # Open(), SpatialReference, WarpOptions(), Warp(), GetDataTypeName(), GetRasterBand(), GetNoDataValue(), Translate(), GetProjection(), GetAttrValue()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be5f9d7",
   "metadata": {},
   "source": [
    "### 1.3 Set up workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "a20dfb25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q:\\GIS\\povertyequity\\urban_growth\\Syria\n",
      "Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\n",
      "Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\n",
      "Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Results\n",
      "Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\n"
     ]
    }
   ],
   "source": [
    "Workspace = os.getcwd()\n",
    "Source = os.path.join(Workspace, 'Source')\n",
    "Intermediate = os.path.join(Workspace, 'Intermediate')\n",
    "Results = os.path.join(Workspace, 'Results')\n",
    "NTL = os.path.join(os.path.dirname(Workspace), 'NighttimeLights_VIIRS_DMSP', 'Temp')\n",
    "\n",
    "print('\\n'.join([Workspace, Source, Intermediate, Results, NTL]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "267db21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ListFromRange(r1, r2):\n",
    "    return [item for item in range(r1, r2+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff97aa01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eligible date range: 1900 to 2030\n",
      "[1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030]\n"
     ]
    }
   ],
   "source": [
    "# These are beyond the maximum expected timeframe. \n",
    "# For instance, some sources we use don't go past even 2015, and Earth Observation wouldn't have started until later than 1900.\n",
    "\n",
    "# Why include this? To ensure that numerically assigned, non-year values (like NoData or a QC marker) are excluded.\n",
    "# The next block of code will re-write these objects with the accurate timeframe.\n",
    "Year_end = 2030\n",
    "Year_start = 1900\n",
    "\n",
    "print(\"Eligible date range:\", Year_start, 'to', Year_end)\n",
    "\n",
    "EligibleYears = ListFromRange(Year_start, Year_end)\n",
    "print(EligibleYears)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bc3ea3",
   "metadata": {},
   "source": [
    "### 1.4 User-defined functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "317d1355",
   "metadata": {},
   "outputs": [],
   "source": [
    "# From Stack Exchange @RutgerH\n",
    "# https://gis.stackexchange.com/questions/163685/reclassify-a-raster-value-to-9999-and-set-it-to-the-nodata-value-using-python-a\n",
    "def readRaster(filename):\n",
    "    filehandle = gdal.Open(filename)\n",
    "    band1 = filehandle.GetRasterBand(1)\n",
    "    geotransform = filehandle.GetGeoTransform()\n",
    "    geoproj = filehandle.GetProjection()\n",
    "    Z = band1.ReadAsArray()\n",
    "    xsize = filehandle.RasterXSize\n",
    "    ysize = filehandle.RasterYSize\n",
    "    return xsize,ysize,geotransform,geoproj,Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "48d2e64f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default arguments can be changed here, or can be specified below when running the functions.\n",
    "def writeRaster(filename,geotransform,geoprojection,data, NoDataVal=0, dst_datatype=gdal.GDT_UInt32):\n",
    "    (x,y) = data.shape\n",
    "    Dformat = \"GTiff\"\n",
    "    driver = gdal.GetDriverByName(Dformat)\n",
    "    # you can change the dataformat but be sure to be able to store negative values including -9999\n",
    "    dst_ds = driver.Create(filename,y,x,1,dst_datatype)\n",
    "    dst_ds.GetRasterBand(1).WriteArray(data)\n",
    "    dst_ds.SetGeoTransform(geotransform)\n",
    "    dst_ds.SetProjection(geoprojection)\n",
    "    dst_ds.GetRasterBand(1).SetNoDataValue(NoDataVal)\n",
    "    return 1\n",
    "    dst_ds = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ff2bd0dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Based on Stack Exchange @Kurt Schwehr:\n",
    "# https://stackoverflow.com/questions/10454316/how-to-project-and-resample-a-grid-to-match-another-grid-with-gdal-python\n",
    "def resampleRaster(InRaster_Path, MatchRaster_Path, OutFile_Path, \n",
    "                   DataType = gdalconst.GDT_UInt32, \n",
    "                   ResampType = gdal.GRA_Bilinear, NoDataVal = 0):\n",
    "    print('Loading for %s. %s' % (InRaster_Path, time.ctime()))\n",
    "    \n",
    "    RasterObject = gdal.Open(InRaster_Path)\n",
    "    In_proj = RasterObject.GetProjection()\n",
    "    [Match_x, Match_y, Match_geo, Match_proj, Match_Z] = readRaster(MatchRaster_Path)\n",
    "    print('---Specs to match to: \\n', \n",
    "      Match_proj, '\\n', Match_geo, '\\n', Match_x, '\\n', Match_y, '\\n')\n",
    "        \n",
    "    OutFile = gdal.GetDriverByName('GTiff').Create(OutFile_Path, Match_x, Match_y, 1, DataType)\n",
    "    OutFile.SetGeoTransform(Match_geo)\n",
    "    OutFile.SetProjection(Match_proj)\n",
    "    print('---Created raster file for upsampled version. %s' % time.ctime())\n",
    "    \n",
    "    gdal.ReprojectImage(RasterObject, OutFile, In_proj, Match_proj, ResampType)\n",
    "    print('---Resampled values onto an empty raster matching the dimensions of the buildup layer. %s \\n\\n' % time.ctime())\n",
    "    \n",
    "    OutFile.GetRasterBand(1).SetNoDataValue(NoDataVal)\n",
    "    \n",
    "    RasterObject = Outfile = None\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "65c5efde",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calcShell(A, OutFile, Calculation, OutType = '',\n",
    "              B=None, C=None, D=None, E=None, F=None, G=None):\n",
    "    \"\"\"Raster math using gdal_calc.py.\n",
    "\n",
    "    The OSgeo package for Python API does not make raster calculations\n",
    "    easy outside of the shell. This function plugs up to 6 raster files\n",
    "    into a string which subprocess.call() then commits to the terminal.\n",
    "\n",
    "        A : str\n",
    "            File path to the first raster for the calculation.\n",
    "        B : str\n",
    "            File path to the second raster for the calculation.\n",
    "        NoDataVal : numeric\n",
    "            Optional value to assign as No Data in the output raster.\n",
    "        OutFile : str\n",
    "            File path where to store the raster generated from the calculation.\n",
    "        Calculation : str\n",
    "            Algebra that uses A and B to create a new raster. Use double quotes.\n",
    "    \"\"\"\n",
    "    print('Running for %s. %s' % (A, time.ctime()))\n",
    "    cmd = 'gdal_calc.py -A ' + A\n",
    "    if B is not None:\n",
    "        cmd = cmd + ' -B ' + B \n",
    "    if C is not None:\n",
    "        cmd = cmd + ' -C ' + C \n",
    "    if D is not None:\n",
    "        cmd = cmd + ' -D ' + D\n",
    "    if E is not None:\n",
    "        cmd = cmd + ' -E ' + E\n",
    "    if F is not None:\n",
    "        cmd = cmd + ' -F ' + F\n",
    "    if G is not None:\n",
    "        cmd = cmd + ' -G ' + G\n",
    "        \n",
    "    cmd = cmd + OutType + ' --outfile=' + OutFile + ' --overwrite --calc=' + Calculation\n",
    "    subprocess.call(cmd, shell=True)\n",
    "    cmd = A = B = C = D = E = F = G = None\n",
    "    print('Ran in shell. See OutFile folder to inspect results. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ac123cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mosaicShell(A, B, OutFile, Band = 1, OutType = '',\n",
    "                  C=None, D=None, E=None, F=None, G=None):\n",
    "    print('Running for %s. %s' % (A, time.ctime()))\n",
    "    \n",
    "    StringFiles = ' '.join([A,B])\n",
    "    \n",
    "    for RasterName in [C,D,E,F,G]:\n",
    "        if RasterName is not None:\n",
    "            StringFiles = ' '.join([StringFiles, RasterName])\n",
    "        else:\n",
    "            pass\n",
    "        \n",
    "    cmd = 'gdal_merge.py -o ' + OutFile + OutType + ' -of gtiff ' + StringFiles\n",
    "    \n",
    "    subprocess.call(cmd, shell=True)\n",
    "    print('Ran in shell. See OutFile folder to inspect results. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c7ae0d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def RasterToShapefile(InRasterPath, OutFilePath = 'RastToShp.shp', Band=1, \n",
    "                      OutName='RastToShp', VariableName='value', Driver = 'ESRI Shapefile'):\n",
    "    \"\"\"Raster tiff to vector polygon shapefile.\n",
    "    Can also be used for other file types like geopackage, but note that this code\n",
    "    currently does not account for writing into an existing file. It will write over\n",
    "    the file if specified as the file path.\n",
    "    \n",
    "    \"\"\"\n",
    "    Raster = gdal.Open(InRasterPath)\n",
    "    RasterBand = Raster.GetRasterBand(Band)\n",
    "    \n",
    "    OutDriver = ogr.GetDriverByName(Driver)\n",
    "    InProj = Raster.GetProjectionRef()\n",
    "    SpatRef = osr.SpatialReference()\n",
    "    SpatRef.ImportFromWkt(InProj)\n",
    "    print(InProj, '\\n\\n', SpatRef)\n",
    "    \n",
    "    if exists(OutFilePath):\n",
    "        OutFile = ogr.Open(OutFilePath)\n",
    "    else:\n",
    "        OutFile = OutDriver.CreateDataSource(OutFilePath)\n",
    "    OutLayer = OutFile.CreateLayer(OutName, srs = SpatRef, geom_type = ogr.wkbPolygon)\n",
    "    OutField = ogr.FieldDefn(VariableName, ogr.OFTInteger)\n",
    "    OutLayer.CreateField(OutField)\n",
    "    OutField = OutLayer.GetLayerDefn().GetFieldIndex(VariableName)\n",
    "    print('\\n', OutFile, '\\n', OutLayer, '\\n', OutField)\n",
    "    \n",
    "    print('Vectorizing. Input: %s. %s' % (InRasterPath, time.ctime()))\n",
    "    gdal.Polygonize(RasterBand, None, OutLayer, 0, [], callback=None)\n",
    "    print('Completed polygons. Stored as: %s. %s' % (OutFilePath, time.ctime()))\n",
    "\n",
    "    del Raster, RasterBand, OutFile, OutLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d23cfdbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rioStats(InRasterPath, Band = 1):\n",
    "    out = rasterio.open(InRasterPath)\n",
    "    stats = []\n",
    "    band = out.read(Band)\n",
    "    stats.append({\n",
    "        'raster': out.name,\n",
    "        'bands': out.count,\n",
    "        'data type': out.dtypes,\n",
    "        'no data value': out.nodatavals,\n",
    "        'width': out.width,\n",
    "        'height': out.height,\n",
    "        'min': band.min(),\n",
    "        'mean': band.mean(),\n",
    "        'median': np.median(band),\n",
    "        'max': band.max()})\n",
    "    print(\"\\n\", stats)\n",
    "    \n",
    "    out = band = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "95b9a04e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ShapeToRaster(Shapefile, ValueVar, MetaRasterPath, OutFilePath = 'ShpToRast.tif', Band=1, NewDType=None):\n",
    "    \"\"\"\n",
    "    Polygon spatial object to raster tiff.\n",
    "    \"\"\"\n",
    "    # Copy and update the metadata from another raster for the output\n",
    "    MetaRaster = rasterio.open(MetaRasterPath)\n",
    "    meta = MetaRaster.meta.copy()\n",
    "    meta.update(compress='lzw')\n",
    "    if NewDType is not None:\n",
    "        meta.update(dtype=NewDType)\n",
    "    MetaRaster.meta\n",
    "\n",
    "    print(\"Rasterizing dataset. %s\" % time.ctime())\n",
    "    with rasterio.open(OutFilePath, 'w+', **meta) as out:\n",
    "        out_arr = out.read(Band)\n",
    "\n",
    "        # this is where we create a generator of geom, value pairs to use in rasterizing\n",
    "        shapes = ((geom,value) for geom, value in zip(Shapefile.geometry, Shapefile[ValueVar]))\n",
    "\n",
    "        burned = features.rasterize(shapes=shapes, fill=0, out=out_arr, transform=out.transform)\n",
    "        out.write_band(1, burned)\n",
    "    out = burned = shapes = None\n",
    "    \n",
    "    print(\"Finished rasterizing. Checking contents. %s\" % time.ctime())\n",
    "    rioStats(OutFilePath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0b9a81ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def BatchZonal(RasterFileList, Zones, KeepFields = None, \n",
    "               RasterDirectory = os.getcwd(),\n",
    "               OutPath = 'BatchZonal.csv',\n",
    "               Statistics=['count', 'sum', 'mean'],\n",
    "               NoDataVal = None,\n",
    "               Prefix = '',\n",
    "               Suffix = '', \n",
    "               DropStatName = False):\n",
    "    '''\n",
    "    Choose a single file for the raster source, or multiple. If multiple, then each stat field\n",
    "    will receive a suffix with the raster's position in the RasterFileList.\n",
    "    \n",
    "        RasterFileList: list\n",
    "                        List of file paths, e.g. tifs. Loaded with rasterio.\n",
    "        Zones:          geodataframe object\n",
    "        KeepFields:     list\n",
    "                        List of field names (string format).\n",
    "        OutPath:        string\n",
    "                        File path for the results.\n",
    "        Statistics:     list\n",
    "                        Full Statistics options: \n",
    "                        ['max', 'min', 'mean', 'count', 'sum', 'median', 'std', 'majority', 'minority', \n",
    "                        'unique', 'range']\n",
    "                        There's also a percentile option in zonal_stats(). I believe you \n",
    "                        write it after an underscore in string format, e.g.: 'percentile_25'.\n",
    "        NoDataVal:      numeric\n",
    "        Prefix:         string\n",
    "        Suffix:         string\n",
    "        DropStatName:   boolean\n",
    "    \n",
    "    '''\n",
    "    if KeepFields is None:\n",
    "        AllSummaries = pd.DataFrame(Zones)[[]]\n",
    "    else:\n",
    "        AllSummaries = pd.DataFrame(Zones)[KeepFields]\n",
    "    print('Dataframe to merge with: \\n', AllSummaries)\n",
    "    \n",
    "    \n",
    "    for idx, RasterFile in enumerate(RasterFileList):\n",
    "        print('Loading with rasterio. %s\\nRaster: %s' % (time.ctime(), RasterFile))\n",
    "        \n",
    "        InRasterPath = os.path.join(RasterDirectory, RasterFile)\n",
    "        \n",
    "        Year = re.search('\\d{4}', os.path.basename(RasterFile))\n",
    "        if Year is None:\n",
    "            Year = '_' + str(idx)\n",
    "        else:\n",
    "            Year = Year.group(0)\n",
    "\n",
    "        with rasterio.open(InRasterPath) as src:\n",
    "            transform = src.meta['transform']\n",
    "            array = src.read(1)\n",
    "\n",
    "        print(transform)\n",
    "        print(array)\n",
    "        \n",
    "        print('Zonal statistics. %s' % time.ctime())\n",
    "        zStats = zonal_stats(Zones, array, affine=transform, stats = Statistics, nodata=NoDataVal)\n",
    "        #print(zStats)\n",
    "\n",
    "        ValByZone = pd.DataFrame(Zones)[[]].join(pd.DataFrame(zStats))\n",
    "        \n",
    "        if DropStatName is False:\n",
    "            for stat in Statistics:\n",
    "                StatField = ''.join([Prefix, stat, Year, Suffix])\n",
    "                ValByZone = ValByZone.rename(columns={stat:StatField})\n",
    "            print('%s stat output field for %s: %s' % (stat, RasterFile, StatField))\n",
    "        else:\n",
    "            for stat in Statistics:\n",
    "                StatField = ''.join([Prefix, Year, Suffix])\n",
    "                ValByZone = ValByZone.rename(columns={stat:StatField})\n",
    "            print('%s stat output field for %s: %s' % (stat, RasterFile, StatField))\n",
    "\n",
    "        AllSummaries = AllSummaries.join(ValByZone)\n",
    "        print(AllSummaries.sample(5))\n",
    "        \n",
    "    print('Final dataframe: \\n', AllSummaries.sample(5))\n",
    "\n",
    "    AllSummaries.to_csv(OutPath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7ccda52e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def MaskByZone(MaskPath, SourceFolder, DestFolder, SourceList = None,\n",
    "               MaskLayerName = None, dstSRS = 'ESRI:102022'):\n",
    "    \"\"\"\n",
    "    Reduces the size of a raster's valid data cells to vector areas of interest.\n",
    "    This is useful if the raster data needs to be vectorized later to save space.\n",
    "    \n",
    "    The script prepares the vector zones as a list of geometries in the desired\n",
    "    spatial reference system, then warps each raster in the specified source\n",
    "    folder to the same SRS. Masking in rasterio then reclassifies any raster cells\n",
    "    falling outside of a mask polygon as NoData.\n",
    "    \"\"\"\n",
    "    \n",
    "    ProjSRS = osr.SpatialReference()\n",
    "    ProjSRS.SetFromUserInput(dstSRS)\n",
    "    ProjWarp = gdal.WarpOptions(dstSRS = dstSRS)\n",
    "    \n",
    "    if SourceList is not None:\n",
    "        SourceFiles = SourceList\n",
    "    else:\n",
    "        SourceFiles = []\n",
    "        SourceFiles = SourceFiles + [i for i in os.listdir(''.join([SourceFolder, r'/'])) if i.endswith('tif')]\n",
    "        print(SourceFiles)\n",
    "\n",
    "    \n",
    "    ### 1. ASSIGN SPATIAL REFERENCE SYSTEM OF VECTOR MASK AND LOAD GEOMETRIES\n",
    "    Vector = gpd.read_file(filename=MaskPath, layer=MaskLayerName)\n",
    "    if Vector.crs != dstSRS:\n",
    "        if MaskLayerName == None:\n",
    "            MaskPath = MaskPath + '_temp'\n",
    "        else:\n",
    "            MaskLayerName = MaskLayerName + '_temp'\n",
    "        Vector.to_crs(dstSRS).to_file(filename=MaskPath, layer=MaskLayerName)\n",
    "    Vector = None # We're reloading the geometries with fiona\n",
    "    \n",
    "    with fiona.open(MaskPath, mode=\"r\", layer=MaskLayerName) as Vector:\n",
    "        MaskGeom = [feature[\"geometry\"] for feature in Vector] # Identify the bounding areas of the mask.\n",
    "    \n",
    "    \n",
    "    ### 2. PREPARE DESTINATION FILES\n",
    "    for FileName in SourceFiles:\n",
    "    \n",
    "        InputRasterPath = os.path.join(Workspace, SourceFolder, FileName)\n",
    "        \n",
    "        Sensor = re.search('[A-Z]+_', FileName)\n",
    "        if Sensor is None:\n",
    "            Sensor = ''\n",
    "        else:\n",
    "            Sensor = Sensor.group(0)\n",
    "\n",
    "        Year = re.search('\\d{4}', FileName)\n",
    "        if Year is None:\n",
    "            Year = ''\n",
    "        else:\n",
    "            Year = Year.group(0)\n",
    "\n",
    "        if FileName.endswith('avg.tif') == True:\n",
    "            IndicType = '_avg'\n",
    "        elif FileName.endswith('cfc.tif') == True:\n",
    "            IndicType = '_cfc'\n",
    "        else:\n",
    "            IndicType = ''\n",
    "\n",
    "        TempOutputName = 'Temp_' + Sensor + Year + IndicType + '.tif'\n",
    "        TempOutputPath = os.path.join(Workspace, DestFolder, TempOutputName)\n",
    "        FinalOutputName = 'Msk_' + Sensor + Year + IndicType + '.tif'\n",
    "        FinalOutputPath = os.path.join(Workspace, DestFolder, FinalOutputName)\n",
    "\n",
    "    ### 3. ASSIGN SPATIAL REFERENCE SYSTEM OF RASTER(S)\n",
    "        InputRasterObject = gdal.Open(InputRasterPath)\n",
    "        SourceSRS = osr.SpatialReference(wkt=InputRasterObject.GetProjection())\n",
    "        print('Source projection: ', SourceSRS.GetAttrValue('projcs'))\n",
    "        print('Destination projection: ', ProjSRS.GetAttrValue('projcs'))\n",
    "\n",
    "        if SourceSRS.GetAttrValue('projcs') != ProjSRS.GetAttrValue('projcs'):\n",
    "            Warp = gdal.Warp(TempOutputPath, # Where to store the warped raster\n",
    "                         InputRasterObject, # Which raster to warp\n",
    "                         format='GTiff', \n",
    "                         options=ProjWarp) # Reproject to Africa Albers Equal Area Conic\n",
    "            print('Finished gdal.Warp() for %s. %s \\n' % (FileName, time.ctime()))\n",
    "\n",
    "            Warp = None # Close the files\n",
    "        else:\n",
    "            pass\n",
    "        InputRasterObject = None\n",
    "        \n",
    "    ### 4. RECLASSIFY AS NODATA IF OUTSIDE OF SETTLEMENT BUFFER ZONE.\n",
    "        if exists(TempOutputPath):\n",
    "            NewInputPath = TempOutputPath \n",
    "            print(\"We warped the data, so we'll use that file for next step.\")\n",
    "        else:\n",
    "            NewInputPath = InputRasterPath \n",
    "            print(\"We skipped the warp, so we continue to use the source file.\")\n",
    "\n",
    "        with rasterio.open(NewInputPath) as InputRasterObject:\n",
    "            MaskedOutputRaster, OutTransform = rasterio.mask.mask(\n",
    "                InputRasterObject, MaskGeom, crop=True) # Anything outside the mask is reclassed to the raster's NoData value.\n",
    "            OutMetaData = InputRasterObject.meta.copy()\n",
    "        print('Finished rasterio.mask.mask() for %s. %s \\n' % (FileName, time.ctime()))\n",
    "\n",
    "        OutMetaData.update({\"driver\": \"GTiff\",\n",
    "                         \"height\": MaskedOutputRaster.shape[1],\n",
    "                         \"width\": MaskedOutputRaster.shape[2],\n",
    "                         \"transform\": OutTransform})\n",
    "\n",
    "        with rasterio.open(FinalOutputPath, \"w\", **OutMetaData) as dest:\n",
    "            dest.write(MaskedOutputRaster)\n",
    "        print('Written to file. %s \\n' % time.ctime())\n",
    "        InputRasterObject = None\n",
    "\n",
    "        if exists(TempOutputPath):\n",
    "            try:  # Finally, remove the intermediate file from disk\n",
    "                os.remove(TempOutputPath)\n",
    "            except OSError:\n",
    "                pass\n",
    "            print('Removed intermediate file. %s \\n' % time.ctime())\n",
    "        else:\n",
    "            pass\n",
    "\n",
    "\n",
    "    print('\\n \\n Finished all years in list. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "55a3e947",
   "metadata": {},
   "outputs": [],
   "source": [
    "def BatchZonalStats(FolderName, Zones, \n",
    "                    CRS = 'ESRI:102022', \n",
    "                    JoinField = 'Sett_ID',\n",
    "                    StatsWanted = ['count', 'sum', 'mean', 'max', 'min'],\n",
    "                    SeriesStart = 1999, SeriesEnd = 2021, \n",
    "                    AnnualizedFiles = None, VarPrefix = None):\n",
    "    \"\"\"\n",
    "    Normally, we would use numpy to generate a point gdf from the raster's matrix. \n",
    "    However, I was running into a lot of memory errors with that method.\n",
    "    This method uses some extra steps: tif to xyz to df to gdf. But it saves to file\n",
    "    and deletes intermediate files along the way, circumventing memory issues.\n",
    "    \n",
    "    Run MaskByZone() prior to reduce the raster to only your area(s) of interest.\n",
    "    \n",
    "    \"\"\"\n",
    "    if AnnualizedFiles is None:\n",
    "        AnnualizedFiles = [i for i in os.listdir(FolderName) if i.endswith('.tif')]\n",
    "    print(AnnualizedFiles)\n",
    "    AllSummaries = pd.DataFrame(Zones).drop(columns='geometry')[[JoinField]]\n",
    "    print(AllSummaries)\n",
    "    \n",
    "    if VarPrefix is None:\n",
    "        VarPrefix = FolderName[:3].upper()\n",
    "    \n",
    "    for FileName in AnnualizedFiles:\n",
    "    ### STEP 1: TIF TO XYZ ###\n",
    "        print('Loading data for %s. %s \\n' % (FileName, time.ctime()))\n",
    "        \n",
    "        Sensor = re.search('[A-Z]+_', FileName)\n",
    "        if Sensor is None:\n",
    "            Sensor = ''\n",
    "        else:\n",
    "            Sensor = Sensor.group(0)\n",
    "            \n",
    "        Year = re.search('\\d{4}', FileName)\n",
    "        if Year is None:\n",
    "            Year = ''\n",
    "        else:\n",
    "            Year = Year.group(0)\n",
    "        \n",
    "        InputRasterPath = os.path.join(Workspace, FolderName, FileName)\n",
    "        InputRasterObject = gdal.Open(InputRasterPath)\n",
    "        XYZOutputPath = FolderName + r'/{}'.format(\n",
    "            FileName.replace('.tif', '.xyz')) # New file path will be the same as original, but .tif is replaced with .xyz\n",
    "\n",
    "        # Create an .xyz version of the .tif\n",
    "        if exists(XYZOutputPath):\n",
    "            print(\"Already created xyz file.\")\n",
    "        else:\n",
    "            print(\"Creating XYZ (gdal.Translate()).\")\n",
    "            XYZ = gdal.Translate(XYZOutputPath, # Specify a destination path\n",
    "                                 InputRasterObject, # Input is the masked .tif file\n",
    "                                 format='XYZ', \n",
    "                                 creationOptions=[\"ADD_HEADER_LINE=YES\"])\n",
    "            print('Finished gdal.Translate() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "            XYZ = None # Reload XYZ as a point geodataframe\n",
    "\n",
    "        InputRasterObject = None\n",
    "\n",
    "\n",
    "    ### STEP 2: GENERATE GEODATAFRAME WITH JOIN FIELD ###\n",
    "        InputXYZ = pd.read_table(XYZOutputPath, delim_whitespace=True)\n",
    "        InputXYZ = InputXYZ.loc[InputXYZ['Z'] > 0] # Subset to only the features that have a value.\n",
    "        \n",
    "        if re.search('WSFE', FileName) is not None: # Scale back up to years if working with flood/building data.\n",
    "            InputXYZ['Z'] = InputXYZ['Z'] + 1900\n",
    "            \n",
    "        print('Loaded XYZ file as a pandas dataframe. %s \\n' % time.ctime())\n",
    "        ValObject = gpd.GeoDataFrame(InputXYZ,\n",
    "                                     geometry = gpd.points_from_xy(InputXYZ['X'], InputXYZ['Y']),\n",
    "                                     crs = CRS)\n",
    "        print('Created geodataframe from non-NoData points. %s \\n' % time.ctime())\n",
    "        del InputXYZ\n",
    "\n",
    "        # Sjoin_nearest: No need to group by ADM this time. \n",
    "        ValObject_withID = pd.DataFrame(gpd.sjoin_nearest(ValObject, \n",
    "                                        Zones, \n",
    "                                        how='left')).drop(columns='geometry')[['Z', JoinField]] # No need for max_distance parameter this time. We've already narrowed down to nearby raster cells.\n",
    "\n",
    "        print('\\nJoined zone ID onto vectorized raster cells. %s \\n' % time.ctime())\n",
    "        print(ValObject_withID.sample(10))\n",
    "        del ValObject\n",
    "\n",
    "        ValObject_withID.to_csv(''.join([FolderName, r'/', FileName.replace('.tif', '.csv')]))\n",
    "        print('\\nExported as table. %s \\n' % time.ctime())\n",
    "\n",
    "#         # Remove the temporary xyz file.\n",
    "#         try:  \n",
    "#             os.remove(os.path.join(XYZOutputPath))\n",
    "#         except OSError:\n",
    "#             pass\n",
    "#         print('Removed (or skipped if error) intermediate xyz file. %s \\n' % time.ctime())\n",
    "\n",
    "\n",
    "    ### STEP 3: AGGREGATE BY SETTLEMENT AND MERGE ONTO SUMMARIES TABLE ###\n",
    "        GroupedVals = ValObject_withID[ValObject_withID['Z'].notna()].groupby(JoinField, as_index=False)\n",
    "        \n",
    "        # Run this block if the variable is about cloud-free coverage.\n",
    "        if re.search('cfc', FileName) is not None:\n",
    "            VariableName = ''.join([VarPrefix, 'cfc_', Sensor, Year])\n",
    "            AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            print('\\nCount of cloud-free observations averaged to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "            # Save in-progress results\n",
    "            AllSummaries.to_csv(os.path.join(Results, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "            print(AllSummaries.sample(10))\n",
    "        \n",
    "        # Run this block if we're working with the flooded buildings data.\n",
    "        elif re.search('WSFE', FileName) is not None:\n",
    "            for BuiltYear in EligibleYears:\n",
    "                Grouped_Subset = GroupedVals[GroupedVals['Z'].between(\n",
    "                    1985, BuiltYear, inclusive=True)] # Inclusive parameter means we include the years 1985 and the named year.\n",
    "                if 'count' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'ct', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'sum' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'sum', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.sum().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'mean' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'avg', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'max' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'max', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.max().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'min' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'min', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.min().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "\n",
    "                # Save in-progress results\n",
    "                AllSummaries.to_csv(os.path.join(Results, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "                print(AllSummaries.sample(10))\n",
    "        \n",
    "        # Anything else takes the standard aggregation method.\n",
    "        else:\n",
    "            if 'count' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'ct', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'sum' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'sum', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.sum().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'mean' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'avg', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'max' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'max', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.max().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'min' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'min', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.min().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "            # Save in-progress results\n",
    "            AllSummaries.to_csv(os.path.join(Results, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "            print(AllSummaries.sample(10))\n",
    "\n",
    "    \n",
    "    print('\\n\\nFinished. All years masked and assigned their nearest settlement. %s' % time.ctime())\n",
    "    print(AllSummaries.sample(10))\n",
    "    AllSummaries.to_csv(os.path.join(Results, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "    print('Saved to file. %s \\n' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5537c1ce",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c08550",
   "metadata": {},
   "source": [
    "## 2. PREPARE BUILDUP, SETTLEMENT, AND ADMIN DATASETS\n",
    "Projection for all datasets: Africa Albers Equal Area Conic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e591ea5f",
   "metadata": {},
   "source": [
    "### 2.1 Prepare GRID3 and Admin area files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc10ab7",
   "metadata": {},
   "source": [
    "#### Admin areas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "47a759f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pull the first file ([0]) which ends in '.shp' from the specified folder, drop all variables, and reproject.\n",
    "ADM_vec = gpd.read_file(glob.glob('Source/ADM/*.shp')[0])[['geometry']].to_crs('ESRI:102022')\n",
    "\n",
    "# Create a fresh unique ID\n",
    "ADM_vec['ADM_ID'] = range(0, len(ADM_vec))\n",
    "ADM_vec['ADM_ID'] = ADM_vec['ADM_ID'] + 1  # Plus one allows us to assign 0 as No Data when rasterizing, to match WSFE.\n",
    "# We have to add 1 if we want our rasterized version's NoData value to be 0. Otherwise the first feature won't be valid.\n",
    "ADM_vec.to_file(driver='GPKG', filename=os.path.join(Source, 'ADM', 'ADM_withID.gpkg'), layer='ADM')\n",
    "\n",
    "# Now reload.\n",
    "ADM_vec = gpd.read_file(os.path.join(Source, 'ADM', 'ADM_withID.gpkg'), layer='ADM')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "732244bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 272 entries, 0 to 271\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   ADM_ID    272 non-null    int64   \n",
      " 1   geometry  272 non-null    geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 4.4 KB\n",
      "None \n",
      "\n",
      "      ADM_ID                                           geometry\n",
      "50       51  POLYGON ((1184519.255 3881539.664, 1184522.540...\n",
      "174     175  POLYGON ((1189990.139 3892919.557, 1190113.966...\n",
      "95       96  POLYGON ((1190359.520 3725033.179, 1190481.824...\n",
      "66       67  POLYGON ((1340425.094 3983571.296, 1339706.926...\n",
      "150     151  POLYGON ((1226443.094 3854947.081, 1226283.464... PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " Number of digits:  3\n"
     ]
    }
   ],
   "source": [
    "# We need to know how many digits need to be allocated to each dataset in the \"join\" serial.\n",
    "len_ADM = len(str(ADM_vec['ADM_ID'].max()))\n",
    "\n",
    "print(ADM_vec.info(), \"\\n\\n\", \n",
    "      ADM_vec.sample(5),\n",
    "      ADM_vec.crs, \"\\n\\n\", \n",
    "      'Number of digits: ', len_ADM) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae24f3f9",
   "metadata": {},
   "source": [
    "#### Settlements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f096ac4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For GRID3, we will still retain the 'type' field.\n",
    "GRID3_vec = gpd.read_file(glob.glob('Source/Settlement/*.shp')[0])[['class_v1','geometry']].to_crs(\"ESRI:102022\")\n",
    "\n",
    "GRID3_vec['G3_ID'] = range(0,len(GRID3_vec))\n",
    "GRID3_vec['G3_ID'] = GRID3_vec['G3_ID'] +1 # Plus one allows us to assign 0 as No Data when rasterizing, to match WSFE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e7946e25",
   "metadata": {},
   "outputs": [],
   "source": [
    "GRID3_vec.to_file(driver='GPKG', filename=os.path.join(Source, 'Settlement', 'Settlement_withID.gpkg'), layer='GRID3')\n",
    "GRID3_vec = gpd.read_file(os.path.join(Source, 'Settlement', 'Settlement_withID.gpkg'), layer='GRID3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d44280f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 157080 entries, 0 to 157079\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   class_v1  157080 non-null  object  \n",
      " 1   G3_ID     157080 non-null  int64   \n",
      " 2   geometry  157080 non-null  geometry\n",
      "dtypes: geometry(1), int64(1), object(1)\n",
      "memory usage: 3.6+ MB\n",
      "None \n",
      "\n",
      "        class_v1   G3_ID                                           geometry\n",
      "2385        SSA    2386  POLYGON ((1157156.777 3652077.065, 1157244.456...\n",
      "146381   Hamlet  146382  POLYGON ((1653157.319 3977659.526, 1653420.703...\n",
      "96226    Hamlet   96227  POLYGON ((1275960.840 3974436.382, 1276048.634...\n",
      "111234   Hamlet  111235  POLYGON ((1347453.247 3979275.227, 1347541.043...\n",
      "22091    Hamlet   22092  POLYGON ((1156671.016 3923360.306, 1156758.791... PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " Number of digits:  6\n"
     ]
    }
   ],
   "source": [
    "len_G3 =  len(str(GRID3_vec['G3_ID'].max()))\n",
    "\n",
    "print(GRID3_vec.info(), \"\\n\\n\",\n",
    "      GRID3_vec.sample(5),\n",
    "      GRID3_vec.crs, \"\\n\\n\", \n",
    "      'Number of digits: ', len_G3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5816d0c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "GRID3_vec = GRID3_vec.rename(columns={'class_v1':'type'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca195d70",
   "metadata": {},
   "source": [
    "### 2.2 Reproject WSFE to project CRS, and ensure file contains only valid date range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "47d96d71",
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in os.listdir(os.path.join(Source, 'Buildup')):\n",
    "    if exists(filename.endswith('.tif')):\n",
    "        SourceWSFE = glob.glob(os.path.join(Source, 'Buildup', '*.tif'))[0]\n",
    "    else:\n",
    "        # If WSFE hasn't been mosaicked yet:\n",
    "        A=os.path.join(Source, 'Buildup', 'Premosaic', 'WSFE1.tif')\n",
    "        B=os.path.join(Source, 'Buildup', 'Premosaic', 'WSFE2.tif')\n",
    "        C=os.path.join(Source, 'Buildup', 'Premosaic', 'WSFE3.tif')\n",
    "        D=os.path.join(Source, 'Buildup', 'Premosaic', 'WSFE4.tif')\n",
    "\n",
    "        SourceWSFE = os.path.join(Intermediate, 'WSFE.tif')\n",
    "        \n",
    "        mosaicShell(A=A, B=B, C=C, D=D, OutFile=SourceWSFE, OutType = ' -ot UInt32 ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "472ca668",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Buildup\\\\GAIA_WSFevolution_1985-2022.tif'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SourceWSFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76bc7548",
   "metadata": {},
   "outputs": [],
   "source": [
    "InPath = SourceWSFE\n",
    "OutWGS = os.path.join(Intermediate, 'WSFE_reclass.tif')\n",
    "\n",
    "# Together, x and y define the data's \"shape\".\n",
    "# geotransform contains the parameters detailing how the raster should be stretched and aligned.\n",
    "# geoproj is the map projection\n",
    "# Z are the values in the raster band.\n",
    "[xsize,ysize,geotransform,geoproj,Z] = readRaster(InPath)\n",
    "Z[Z<Year_start] = 0\n",
    "Z[Z>Year_end] = 0\n",
    "\n",
    "writeRaster(OutWGS,geotransform,geoproj,Z, NoDataVal=0, dst_datatype=gdal.GDT_UInt32) # Unsigned 32 so that we can concatenate.\n",
    "print('Wrote the reclassed raster to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "a9a61cf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source date range:  1985 - 2021\n"
     ]
    }
   ],
   "source": [
    "# In case the raster doesn't go up to our preferred end year, let's make sure we've recorded the accurate one.\n",
    "WSFE_start = int(Z[Z>0].min())\n",
    "WSFE_end = int(Z[Z>0].max())\n",
    "print('Source date range: ', WSFE_start, '-', WSFE_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "0d006840",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021] \n",
      "\n",
      " [2021, 2020, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999, 1998, 1997, 1996, 1995, 1994, 1993, 1992, 1991, 1990, 1989, 1988, 1987, 1986, 1985]\n"
     ]
    }
   ],
   "source": [
    "WSFE_Years = ListFromRange(WSFE_start, WSFE_end)\n",
    "Reversed_WSFE_Years = []\n",
    "for i in WSFE_Years:\n",
    "    Reversed_WSFE_Years.insert(0,i)\n",
    "print(WSFE_Years, '\\n\\n', Reversed_WSFE_Years)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "71ebd2eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote the reclassed and reprojected raster to file. Wed Sep 13 11:25:25 2023\n"
     ]
    }
   ],
   "source": [
    "OutEqArea = os.path.join(Intermediate, 'WSFE_equalarea.tif')\n",
    "\n",
    "# Whenever we want to work in a projected CRS, we'll use Africa Albers Equal Area Conic.\n",
    "ProjEqArea = gdal.WarpOptions(dstSRS='ESRI:102022')\n",
    "Warp = gdal.Warp(OutEqArea, # Where to store the warped raster\n",
    "                 OutWGS, # Which raster to warp\n",
    "                 format='GTiff', \n",
    "                 options=ProjEqArea)\n",
    "print('Wrote the reclassed and reprojected raster to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "695d3cd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/WSFE_reclass.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 31912, 'height': 21152, 'min': 0, 'mean': 43.49248497291767, 'median': 0.0, 'max': 2021}]\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/WSFE_equalarea.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 32854, 'height': 20036, 'min': 0, 'mean': 42.82576195744719, 'median': 0.0, 'max': 2021}]\n"
     ]
    }
   ],
   "source": [
    "rioStats(OutWGS)\n",
    "rioStats(OutEqArea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "de7dbcaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "InPath = Warp = OutWGS = OutEqArea = SourceWSFE = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b44be4f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440aa2a2",
   "metadata": {},
   "source": [
    "## 3. WSFE AND ADM; GRID3 AND ADM\n",
    "RASTERIZE: Bring ADM and GRID3 into raster space.\n",
    "\n",
    "RASTER MATH: \"Join\" ADM ID onto GRID3 and onto WSFE by creating unique concatenation string.\n",
    "\n",
    "VECTORIZE: Bring joined data into vector space.\n",
    "\n",
    "VECTOR MATH: Split unique ID from raster math step into separate columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9663bd1",
   "metadata": {},
   "source": [
    "### 3.1 Rasterize admin areas and GRID3 using WSFE specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1bbb96a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "157080 1 272 1\n"
     ]
    }
   ],
   "source": [
    "G3_max = GRID3_vec.G3_ID.max()\n",
    "G3_min = GRID3_vec.G3_ID.min()\n",
    "ADM_max = ADM_vec.ADM_ID.max()\n",
    "ADM_min = ADM_vec.ADM_ID.min()\n",
    "print(G3_max, G3_min, ADM_max, ADM_min) # Min should not be zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ea871242",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy and update the metadata from WSFE for the output\n",
    "WSFE = os.path.join(Intermediate, 'WSFE_equalarea.tif')\n",
    "ADM_out = os.path.join(Intermediate, 'ADM_rasterized.tif')\n",
    "GRID3_out = os.path.join(Intermediate, 'GRID3_rasterized.tif')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "af504c4b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rasterizing dataset. Wed Sep 13 11:25:35 2023\n",
      "Finished rasterizing. Checking contents. Wed Sep 13 11:25:43 2023\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/ADM_rasterized.tif', 'bands': 1, 'data type': ('uint16',), 'no data value': (0.0,), 'width': 32854, 'height': 20036, 'min': 0, 'mean': 58.275021362898215, 'median': 0.0, 'max': 272}]\n",
      "Rasterizing dataset. Wed Sep 13 11:25:46 2023\n",
      "Finished rasterizing. Checking contents. Wed Sep 13 11:26:16 2023\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/GRID3_rasterized.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 32854, 'height': 20036, 'min': 0, 'mean': 1511.3145033375913, 'median': 0.0, 'max': 157080}]\n"
     ]
    }
   ],
   "source": [
    "# ADM raster can be unsigned int 16. Consider 32 if there are thousands of admin areas.\n",
    "ShapeToRaster(Shapefile=ADM_vec, ValueVar=\"ADM_ID\", MetaRasterPath=WSFE, OutFilePath=ADM_out, NewDType = 'uint16')\n",
    "\n",
    "# To be safe, we'll do unsigned 32 for settlements.\n",
    "ShapeToRaster(GRID3_vec, \"G3_ID\", WSFE, GRID3_out, NewDType='uint32')\n",
    "\n",
    "# Check printed stats here to make sure the NoData and min values are 0 \n",
    "# and the max is the number of vector features in the input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406ceaa3",
   "metadata": {},
   "source": [
    "### 3.2 Raster math to \"join\" admin to GRID3 and to WSFE.\n",
    "Processing is more rapid when \"joining,\" i.e. creating serial codes out of two datasets, in raster rather than vector space.\n",
    "Here, we are concatenating the ID fields of the two datasets to create a serial number that we can then split in vector space later to create two ID fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f28140ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# In paths\n",
    "InG3 = os.path.join(Intermediate, 'GRID3_rasterized.tif')\n",
    "InWSFE = os.path.join(Intermediate, 'WSFE_equalarea.tif')\n",
    "InADM = os.path.join(Intermediate, 'ADM_rasterized.tif')\n",
    "\n",
    "# Out paths\n",
    "G3_ADM = os.path.join(Intermediate, 'GRID3_ADM.tif')\n",
    "WSFE_ADM = os.path.join(Intermediate, 'WSFE_ADM.tif')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2d495a53",
   "metadata": {},
   "outputs": [],
   "source": [
    "G3_rio = rasterio.open(InG3).read(1)\n",
    "WSFE_rio = rasterio.open(InWSFE).read(1)\n",
    "ADM_rio = rasterio.open(InADM).read(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "fb845aff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "G3 range accurate?  True\n",
      "WSFE range accurate?  True\n",
      "ADM range accurate?  True\n"
     ]
    }
   ],
   "source": [
    "print(\"G3 range accurate? \", G3_rio.max()==G3_max)\n",
    "print(\"WSFE range accurate? \", WSFE_rio.max()==WSFE_end)\n",
    "print(\"ADM range accurate? \", ADM_rio.max()==ADM_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c4b79129",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recalculate number of digits for each dataset if starting from Section 3\n",
    "len_G3 = len(str(G3_rio.max()))\n",
    "len_WSFE = len(str(WSFE_rio.max()))\n",
    "len_ADM = len(str(ADM_rio.max()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8166a334",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "DIGITS \n",
      "GRID3:  6 \n",
      "WSFE:  4 \n",
      "ADM:  3\n"
     ]
    }
   ],
   "source": [
    "G3_rio = WSFE_rio = ADM_rio = None\n",
    "print('\\nDIGITS', '\\nGRID3: ', len_G3, '\\nWSFE: ', len_WSFE, '\\nADM: ', len_ADM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3e831cd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\GRID3_rasterized.tif. Wed Sep 13 11:27:39 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Wed Sep 13 11:28:42 2023\n",
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\WSFE_equalarea.tif. Wed Sep 13 11:28:42 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Wed Sep 13 11:29:44 2023\n"
     ]
    }
   ],
   "source": [
    "# Calculations\n",
    "# The number of digits in the largest ADM index value (len_ADM) is \n",
    "# the number of zeroes we tack onto the first variable in the serial.\n",
    "\n",
    "Calc = \"(A*\" + str(10**len_ADM) + \")+B\" \n",
    "\n",
    "calcShell(A=InG3, B=InADM, OutFile=G3_ADM, Calculation=Calc)\n",
    "calcShell(A=InWSFE, B=InADM, OutFile=WSFE_ADM, Calculation=Calc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e46a2406",
   "metadata": {},
   "source": [
    "*Adding together the values to create join IDs. This is in effect a concatenation of their ID strings, by way of summation. The number of zeros in the calc multiplication corresponds with number of digits of the maximum value in the \"B\" dataset. (e.g. Chad ADM codes go up 4 digits, so it's calc=(A*10000)+B).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b8df5020",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/GRID3_ADM.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (4294967293.0,), 'width': 32854, 'height': 20036, 'min': 1063, 'mean': 4208011649.944821, 'median': 4294967293.0, 'max': 4294967293}]\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/Syria/Intermediate/WSFE_ADM.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (4294967293.0,), 'width': 32854, 'height': 20036, 'min': 1985001, 'mean': 4271188891.221792, 'median': 4294967293.0, 'max': 4294967293}]\n"
     ]
    }
   ],
   "source": [
    "rioStats(G3_ADM)\n",
    "rioStats(WSFE_ADM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e6188bce",
   "metadata": {},
   "outputs": [],
   "source": [
    "G3_ADM = WSFE_ADM = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ea3639",
   "metadata": {},
   "source": [
    "### 3.3 Vectorize serialized layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "1d90d3df",
   "metadata": {},
   "outputs": [],
   "source": [
    "G3_in = os.path.join(Intermediate, 'GRID3_ADM.tif')\n",
    "G3_out = os.path.join(Intermediate, 'GRID3_ADM.shp')\n",
    "WSFE_in = os.path.join(Intermediate, 'WSFE_ADM.tif')\n",
    "WSFE_out = os.path.join(Intermediate, 'WSFE_ADM.shp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a9a76c62",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]] \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",\n",
      "    GEOGCS[\"WGS 84\",\n",
      "        DATUM[\"WGS_1984\",\n",
      "            SPHEROID[\"WGS 84\",6378137,298.257223563,\n",
      "                AUTHORITY[\"EPSG\",\"7030\"]],\n",
      "            AUTHORITY[\"EPSG\",\"6326\"]],\n",
      "        PRIMEM[\"Greenwich\",0],\n",
      "        UNIT[\"degree\",0.0174532925199433,\n",
      "            AUTHORITY[\"EPSG\",\"9122\"]],\n",
      "        AUTHORITY[\"EPSG\",\"4326\"]],\n",
      "    PROJECTION[\"Albers_Conic_Equal_Area\"],\n",
      "    PARAMETER[\"latitude_of_center\",0],\n",
      "    PARAMETER[\"longitude_of_center\",25],\n",
      "    PARAMETER[\"standard_parallel_1\",20],\n",
      "    PARAMETER[\"standard_parallel_2\",-23],\n",
      "    PARAMETER[\"false_easting\",0],\n",
      "    PARAMETER[\"false_northing\",0],\n",
      "    UNIT[\"metre\",1,\n",
      "        AUTHORITY[\"EPSG\",\"9001\"]],\n",
      "    AXIS[\"Easting\",EAST],\n",
      "    AXIS[\"Northing\",NORTH]]\n",
      "\n",
      " <osgeo.ogr.DataSource; proxy of <Swig Object of type 'OGRDataSourceShadow *' at 0x0000022F0049C2D0> > \n",
      " <osgeo.ogr.Layer; proxy of <Swig Object of type 'OGRLayerShadow *' at 0x0000022F0049F750> > \n",
      " 0\n",
      "Vectorizing. Input: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\GRID3_ADM.tif. Wed Sep 13 11:30:22 2023\n",
      "Completed polygons. Stored as: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\GRID3_ADM.shp. Wed Sep 13 11:39:48 2023\n"
     ]
    }
   ],
   "source": [
    "RasterToShapefile(G3_in, G3_out, OutName='GRID3_ADM', VariableName='gridcode', Driver = 'ESRI Shapefile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "85dfc892",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]] \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",\n",
      "    GEOGCS[\"WGS 84\",\n",
      "        DATUM[\"WGS_1984\",\n",
      "            SPHEROID[\"WGS 84\",6378137,298.257223563,\n",
      "                AUTHORITY[\"EPSG\",\"7030\"]],\n",
      "            AUTHORITY[\"EPSG\",\"6326\"]],\n",
      "        PRIMEM[\"Greenwich\",0],\n",
      "        UNIT[\"degree\",0.0174532925199433,\n",
      "            AUTHORITY[\"EPSG\",\"9122\"]],\n",
      "        AUTHORITY[\"EPSG\",\"4326\"]],\n",
      "    PROJECTION[\"Albers_Conic_Equal_Area\"],\n",
      "    PARAMETER[\"latitude_of_center\",0],\n",
      "    PARAMETER[\"longitude_of_center\",25],\n",
      "    PARAMETER[\"standard_parallel_1\",20],\n",
      "    PARAMETER[\"standard_parallel_2\",-23],\n",
      "    PARAMETER[\"false_easting\",0],\n",
      "    PARAMETER[\"false_northing\",0],\n",
      "    UNIT[\"metre\",1,\n",
      "        AUTHORITY[\"EPSG\",\"9001\"]],\n",
      "    AXIS[\"Easting\",EAST],\n",
      "    AXIS[\"Northing\",NORTH]]\n",
      "\n",
      " <osgeo.ogr.DataSource; proxy of <Swig Object of type 'OGRDataSourceShadow *' at 0x0000022F0049CF60> > \n",
      " <osgeo.ogr.Layer; proxy of <Swig Object of type 'OGRLayerShadow *' at 0x0000022F0049F750> > \n",
      " 0\n",
      "Vectorizing. Input: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\WSFE_ADM.tif. Wed Sep 13 11:39:48 2023\n",
      "Completed polygons. Stored as: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Intermediate\\WSFE_ADM.shp. Wed Sep 13 11:44:41 2023\n"
     ]
    }
   ],
   "source": [
    "RasterToShapefile(WSFE_in, WSFE_out, OutName='WSFE_ADM', VariableName='gridcode', Driver = 'ESRI Shapefile')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03937b47",
   "metadata": {},
   "source": [
    "### 3.4 Vector math to split raster strings into admin area, GRID3, and WSFE year assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bb900b13",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 197953 entries, 0 to 197952\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   gridcode  197953 non-null  int64   \n",
      " 1   geometry  197953 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 3.0 MB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 1375670 entries, 0 to 1375669\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count    Dtype   \n",
      "---  ------    --------------    -----   \n",
      " 0   gridcode  1375670 non-null  int64   \n",
      " 1   geometry  1375670 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 21.0 MB\n",
      "None \n",
      "\n",
      "          gridcode                                           geometry\n",
      "82893   110514166  POLYGON ((1399028.929 3952698.588, 1399111.913...\n",
      "46466   100721076  POLYGON ((1313362.258 3991617.866, 1313528.225...\n",
      "81225    77211030  POLYGON ((1213892.659 3954551.887, 1213975.642...\n",
      "111583   67734246  POLYGON ((1234638.543 3909436.504, 1234804.510...\n",
      "50587   112310211  POLYGON ((1334218.787 3986057.969, 1334329.432...\n",
      "116649   17983060  POLYGON ((1168141.069 3898095.421, 1168307.036...\n",
      "89180    66307225  POLYGON ((1245426.403 3945755.632, 1245537.047...\n",
      "71039   145975229  POLYGON ((1622503.593 3964426.928, 1622697.222...\n",
      "128777   86662190  POLYGON ((1305451.161 3874694.063, 1305561.806...\n",
      "3146    152264200  POLYGON ((1718432.562 4060577.185, 1718515.545... \n",
      "\n",
      " ESRI:102022 \n",
      "\n",
      " None \n",
      "\n",
      "          gridcode                                           geometry\n",
      "156384    2013114  POLYGON ((1715030.237 4009846.583, 1715057.898...\n",
      "832332    2021044  POLYGON ((1170796.542 3751684.801, 1170824.203...\n",
      "587853    2013036  POLYGON ((1270100.174 3873338.665, 1270127.835...\n",
      "474065    2021050  POLYGON ((1155223.298 3897182.602, 1155278.621...\n",
      "23749     1987014  POLYGON ((1808746.311 4066164.743, 1808773.972...\n",
      "1091289   1990045  POLYGON ((1189689.127 3722280.967, 1189716.788...\n",
      "207666    2010110  POLYGON ((1293390.887 3990732.709, 1293418.548...\n",
      "548523    2012099  POLYGON ((1245177.452 3883241.367, 1245205.113...\n",
      "594210    2021107  POLYGON ((1226782.768 3871430.044, 1226810.429...\n",
      "1359430   2010063  POLYGON ((1181114.162 3637361.148, 1181169.484... \n",
      "\n",
      " ESRI:102022 \n",
      "\n",
      " 2147483647 2147483647\n"
     ]
    }
   ],
   "source": [
    "# Load newly created vectorized datasets.\n",
    "GRID3_ADM = gpd.read_file(G3_out).to_crs(\"ESRI:102022\")\n",
    "WSFE_ADM = gpd.read_file(WSFE_out).to_crs(\"ESRI:102022\")\n",
    "print(GRID3_ADM.info(), \"\\n\\n\", GRID3_ADM.sample(10), \"\\n\\n\", GRID3_ADM.crs, \"\\n\\n\", \n",
    "      WSFE_ADM.info(), \"\\n\\n\", WSFE_ADM.sample(10), \"\\n\\n\", WSFE_ADM.crs, \"\\n\\n\", \n",
    "      GRID3_ADM['gridcode'].max(), WSFE_ADM['gridcode'].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "8f41b97c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          gridcode                                           geometry  \\\n",
      "51885     76723062  POLYGON ((1233781.046 3984619.588, 1233864.030...   \n",
      "70140    139073152  POLYGON ((1596612.730 3965229.102, 1596806.358...   \n",
      "181541    36864155  POLYGON ((1232646.938 3686708.691, 1232729.922...   \n",
      "164896  2147483647  POLYGON ((1163217.379 3746069.581, 1163383.346...   \n",
      "92625    109807012  POLYGON ((1361216.098 3940997.909, 1361299.081...   \n",
      "149351    50530016  POLYGON ((1223657.055 3837351.472, 1223933.667...   \n",
      "164443    38225070  POLYGON ((1255661.039 3747812.236, 1255771.684...   \n",
      "17368    135560243  POLYGON ((1509839.612 4030315.856, 1510005.579...   \n",
      "25205    115715177  POLYGON ((1377204.259 4022128.147, 1377287.242...   \n",
      "65909    139683152  POLYGON ((1564940.680 3968825.055, 1565051.325...   \n",
      "\n",
      "        gridstring  Sett_ID  ADM_ID  \n",
      "51885    076723062    76723      62  \n",
      "70140    139073152   139073     152  \n",
      "181541   036864155    36864     155  \n",
      "164896  2147483647  2147483     647  \n",
      "92625    109807012   109807      12  \n",
      "149351   050530016    50530      16  \n",
      "164443   038225070    38225      70  \n",
      "17368    135560243   135560     243  \n",
      "25205    115715177   115715     177  \n",
      "65909    139683152   139683     152            gridcode                                           geometry  \\\n",
      "783941    2005111  POLYGON ((1240198.440 3811654.236, 1240226.101...   \n",
      "1142849   2021197  POLYGON ((1144518.422 3715531.640, 1144573.744...   \n",
      "930456    2010073  POLYGON ((1206783.736 3734894.465, 1206811.397...   \n",
      "875101    1986073  POLYGON ((1208249.778 3741892.743, 1208277.439...   \n",
      "1133276   1986045  POLYGON ((1196770.389 3717716.873, 1196798.050...   \n",
      "223795    2005129  POLYGON ((1279145.380 3988824.087, 1279173.041...   \n",
      "981393    1991061  POLYGON ((1187310.266 3731187.867, 1187337.927...   \n",
      "696645    2010151  POLYGON ((1222605.930 3847254.174, 1222744.236...   \n",
      "474831    2017050  POLYGON ((1155250.960 3897044.296, 1155278.621...   \n",
      "1257943   2018230  POLYGON ((1224874.147 3663445.640, 1224901.808...   \n",
      "\n",
      "        gridstring  year  ADM_ID  \n",
      "783941     2005111  2005     111  \n",
      "1142849    2021197  2021     197  \n",
      "930456     2010073  2010      73  \n",
      "875101     1986073  1986      73  \n",
      "1133276    1986045  1986      45  \n",
      "223795     2005129  2005     129  \n",
      "981393     1991061  1991      61  \n",
      "696645     2010151  2010     151  \n",
      "474831     2017050  2017      50  \n",
      "1257943    2018230  2018     230  \n"
     ]
    }
   ],
   "source": [
    "# Split serial back into separate dataset fields.\n",
    "# For example, Burkina: WSFE and ADM: 4+3=7 digits. GRID3 and ADM: 6+3=9 digits.\n",
    "G3_Fill = len_G3 + len_ADM\n",
    "WSFE_Fill = len_WSFE + len_ADM\n",
    "\n",
    "GRID3_ADM['gridstring'] = GRID3_ADM['gridcode'].astype(str).str.zfill(G3_Fill)\n",
    "WSFE_ADM['gridstring'] = WSFE_ADM['gridcode'].astype(str).str.zfill(WSFE_Fill)\n",
    "\n",
    "GRID3_ADM['Sett_ID'] = GRID3_ADM['gridstring'].str[:-len_ADM].astype(int) # Remove the last 3 digits to get the GRID3 portion.\n",
    "GRID3_ADM['ADM_ID'] = GRID3_ADM['gridstring'].str[-len_ADM:].astype(int) # Keep only the last 3 digits to get the ADM portion.\n",
    "WSFE_ADM['year'] = WSFE_ADM['gridstring'].str[:-len_ADM].astype(int)\n",
    "WSFE_ADM['ADM_ID'] = WSFE_ADM['gridstring'].str[-len_ADM:].astype(int)\n",
    "\n",
    "print(GRID3_ADM.sample(10), WSFE_ADM.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "3127bbf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed Sep 13 11:46:11 2023\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 158685 entries, 0 to 158684\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count   Dtype   \n",
      "---  ------      --------------   -----   \n",
      " 0   Sett_ID     158685 non-null  int64   \n",
      " 1   ADM_ID      158685 non-null  int64   \n",
      " 2   geometry    158685 non-null  geometry\n",
      " 3   gridcode    158685 non-null  int64   \n",
      " 4   gridstring  158685 non-null  object  \n",
      "dtypes: geometry(1), int64(3), object(1)\n",
      "memory usage: 6.1+ MB\n",
      "None    Sett_ID  ADM_ID                                           geometry  \\\n",
      "0        1      63  POLYGON ((1185042.049 3636946.231, 1185069.710...   \n",
      "1        1     180  POLYGON ((1185069.710 3636946.231, 1185235.677...   \n",
      "2        2      40  MULTIPOLYGON (((1143660.926 3656253.733, 11436...   \n",
      "3        3      63  MULTIPOLYGON (((1167117.605 3647263.850, 11671...   \n",
      "4        4      63  POLYGON ((1171598.716 3648314.975, 1171792.344...   \n",
      "\n",
      "   gridcode gridstring  \n",
      "0      1063  000001063  \n",
      "1      1180  000001180  \n",
      "2      2040  000002040  \n",
      "3      3063  000003063  \n",
      "4      4063  000004063   \n",
      "\n",
      " Wed Sep 13 11:48:01 2023\n"
     ]
    }
   ],
   "source": [
    "# Dissolve any features that have the same G3 and ADM values so that we have a single unique feature per settlement.\n",
    "# Note: we do NOT want to dissolve the WSFE features. Distinct features for noncontiguous builtup areas of the same year is necessary to separate them in the Near tool step.\n",
    "\n",
    "print(time.ctime())\n",
    "GRID3_ADM = GRID3_ADM.dissolve(by=['Sett_ID', 'ADM_ID'], as_index=False)\n",
    "print(GRID3_ADM.info(), GRID3_ADM.head(), \"\\n\\n\", time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "f172cebe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: WSFE (1375670, 5) and GRID3 (158685, 5)\n",
      "\n",
      "After: WSFE (1375670, 5) and GRID3 (158685, 5)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Remove features where year, settlement, or admin area = 0.\n",
    "# This was supposed to be resolved earlier with the gdal_calc NoDataValue parameter. Being thorough.\n",
    "\n",
    "print(\"Before: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))\n",
    "WSFE_ADM = WSFE_ADM.loc[(WSFE_ADM[\"year\"] != 0) & (WSFE_ADM[\"ADM_ID\"] != 0)] # Since we change the datatype to integer, no need to include all digits. Otherwise, it would need to be: != '0000'\n",
    "GRID3_ADM = GRID3_ADM.loc[(GRID3_ADM[\"Sett_ID\"] != 0) & (GRID3_ADM[\"ADM_ID\"] != 0)]\n",
    "print(\"After: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9c6f651d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The Bounded_ID is our new unique settlement identifier for subsequent matching steps.\n",
    "GRID3_ADM['Bounded_ID'] = GRID3_ADM.index\n",
    "WSFE_ADM['WSFE_ID'] = WSFE_ADM.index\n",
    "GRID3_ADM = GRID3_ADM[['Sett_ID', 'Bounded_ID', 'ADM_ID', 'geometry']]\n",
    "WSFE_ADM = WSFE_ADM[['WSFE_ID', 'year', 'ADM_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "35ec5fdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "158685 158685 1375670 5725\n"
     ]
    }
   ],
   "source": [
    "# Validation: \n",
    "# The first two printed numbers should be the same. There shouldn't be any GRID3 rows with matching Sett_ID and ADM_IDs.\n",
    "# The latter two numbers should be different, and the first should be larger. We never dissolved WSFE by any column.\n",
    "\n",
    "print(len(GRID3_ADM[['Sett_ID', 'ADM_ID']]),\n",
    "      len(GRID3_ADM[['Sett_ID', 'ADM_ID']].drop_duplicates()),\n",
    "      len(WSFE_ADM[['year', 'ADM_ID']]),\n",
    "      len(WSFE_ADM[['year', 'ADM_ID']].drop_duplicates()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "82419540",
   "metadata": {},
   "outputs": [],
   "source": [
    "GRID3_ADM.to_file(\n",
    "    driver='GPKG', filename=os.path.join(Intermediate,'GRID3_ADM.gpkg'), layer='GRID3_ADM_cleaned')\n",
    "WSFE_ADM.to_file(\n",
    "    driver='GPKG', filename=os.path.join(Intermediate,' WSFE_ADM.gpkg'), layer='WSFE_ADM_cleaned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abd7f5e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "013f0a8e",
   "metadata": {},
   "source": [
    "## 4. UNIQUE SETTLEMENTS FROM WSFE AND GRID3: TWO VERSIONS\n",
    "\n",
    "Note that there are 2 versions here, so that we can create a fragmentation index:\n",
    "1. **Boundless, aka boundary-agnostic settlements**: Unique settlements are linked to GRID3 settlement IDs. Administrative areas do not influence the extents of the settlement.\n",
    "2. **Bounded, aka politically-defined settlements**: Settlements in the Boundless dataset which spread across more than one administrative area are split into separate settlements in the Bounded dataset. The largest polygon after the split is considered the \"principal\" settlement, and polygons in other admin areas are considered \"fragments.\" By dividing the fragment area(s) of the Bounded settlement by the area of the Boundless settlement, we can acquire a fragmentation index for each locality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32df6624",
   "metadata": {},
   "source": [
    "### 4.1 BOUNDED SETTLEMENTS: Near Join by ADM group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "8d294c3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of admin areas with GRID3 features: 273\n",
      "Number of admin areas with WSFE features: 272\n",
      "Number of admin areas where one dataset is observed but the other is not: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of admin areas with GRID3 features: %s\" % len(GRID3_ADM['ADM_ID'].unique().tolist()))\n",
    "print(\"Number of admin areas with WSFE features: %s\" % len(WSFE_ADM['ADM_ID'].unique().tolist()))\n",
    "print(\"Number of admin areas where one dataset is observed but the other is not: %s\" % (\n",
    "    len(GRID3_ADM['ADM_ID'].unique().tolist()) - len(WSFE_ADM['ADM_ID'].unique().tolist())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "74e1c2f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 2,\n",
       " 3,\n",
       " 4,\n",
       " 5,\n",
       " 6,\n",
       " 7,\n",
       " 8,\n",
       " 9,\n",
       " 10,\n",
       " 11,\n",
       " 12,\n",
       " 13,\n",
       " 14,\n",
       " 15,\n",
       " 16,\n",
       " 17,\n",
       " 18,\n",
       " 19,\n",
       " 20,\n",
       " 21,\n",
       " 22,\n",
       " 23,\n",
       " 24,\n",
       " 25,\n",
       " 26,\n",
       " 27,\n",
       " 28,\n",
       " 29,\n",
       " 30,\n",
       " 31,\n",
       " 32,\n",
       " 33,\n",
       " 34,\n",
       " 35,\n",
       " 36,\n",
       " 37,\n",
       " 38,\n",
       " 39,\n",
       " 40,\n",
       " 41,\n",
       " 42,\n",
       " 43,\n",
       " 44,\n",
       " 45,\n",
       " 46,\n",
       " 47,\n",
       " 48,\n",
       " 49,\n",
       " 50,\n",
       " 51,\n",
       " 52,\n",
       " 53,\n",
       " 54,\n",
       " 55,\n",
       " 56,\n",
       " 57,\n",
       " 58,\n",
       " 59,\n",
       " 60,\n",
       " 61,\n",
       " 62,\n",
       " 63,\n",
       " 64,\n",
       " 65,\n",
       " 66,\n",
       " 67,\n",
       " 68,\n",
       " 69,\n",
       " 70,\n",
       " 71,\n",
       " 72,\n",
       " 73,\n",
       " 74,\n",
       " 75,\n",
       " 76,\n",
       " 77,\n",
       " 78,\n",
       " 79,\n",
       " 80,\n",
       " 81,\n",
       " 82,\n",
       " 83,\n",
       " 84,\n",
       " 85,\n",
       " 86,\n",
       " 87,\n",
       " 88,\n",
       " 89,\n",
       " 90,\n",
       " 91,\n",
       " 92,\n",
       " 93,\n",
       " 94,\n",
       " 95,\n",
       " 96,\n",
       " 97,\n",
       " 98,\n",
       " 99,\n",
       " 100,\n",
       " 101,\n",
       " 102,\n",
       " 103,\n",
       " 104,\n",
       " 105,\n",
       " 106,\n",
       " 107,\n",
       " 108,\n",
       " 109,\n",
       " 110,\n",
       " 111,\n",
       " 112,\n",
       " 113,\n",
       " 114,\n",
       " 115,\n",
       " 116,\n",
       " 117,\n",
       " 118,\n",
       " 119,\n",
       " 120,\n",
       " 121,\n",
       " 122,\n",
       " 123,\n",
       " 124,\n",
       " 125,\n",
       " 126,\n",
       " 127,\n",
       " 128,\n",
       " 129,\n",
       " 130,\n",
       " 131,\n",
       " 132,\n",
       " 133,\n",
       " 134,\n",
       " 135,\n",
       " 136,\n",
       " 137,\n",
       " 138,\n",
       " 139,\n",
       " 140,\n",
       " 141,\n",
       " 142,\n",
       " 143,\n",
       " 144,\n",
       " 145,\n",
       " 146,\n",
       " 147,\n",
       " 148,\n",
       " 149,\n",
       " 150,\n",
       " 151,\n",
       " 152,\n",
       " 153,\n",
       " 154,\n",
       " 155,\n",
       " 156,\n",
       " 157,\n",
       " 158,\n",
       " 159,\n",
       " 160,\n",
       " 161,\n",
       " 162,\n",
       " 163,\n",
       " 164,\n",
       " 165,\n",
       " 166,\n",
       " 167,\n",
       " 168,\n",
       " 169,\n",
       " 170,\n",
       " 171,\n",
       " 172,\n",
       " 173,\n",
       " 174,\n",
       " 175,\n",
       " 176,\n",
       " 177,\n",
       " 178,\n",
       " 179,\n",
       " 180,\n",
       " 181,\n",
       " 182,\n",
       " 183,\n",
       " 184,\n",
       " 185,\n",
       " 186,\n",
       " 187,\n",
       " 188,\n",
       " 189,\n",
       " 190,\n",
       " 191,\n",
       " 192,\n",
       " 193,\n",
       " 194,\n",
       " 195,\n",
       " 196,\n",
       " 197,\n",
       " 198,\n",
       " 199,\n",
       " 200,\n",
       " 201,\n",
       " 202,\n",
       " 203,\n",
       " 204,\n",
       " 205,\n",
       " 206,\n",
       " 207,\n",
       " 208,\n",
       " 209,\n",
       " 210,\n",
       " 211,\n",
       " 212,\n",
       " 213,\n",
       " 214,\n",
       " 215,\n",
       " 216,\n",
       " 217,\n",
       " 218,\n",
       " 219,\n",
       " 220,\n",
       " 221,\n",
       " 222,\n",
       " 223,\n",
       " 224,\n",
       " 225,\n",
       " 226,\n",
       " 227,\n",
       " 228,\n",
       " 229,\n",
       " 230,\n",
       " 231,\n",
       " 232,\n",
       " 233,\n",
       " 234,\n",
       " 235,\n",
       " 236,\n",
       " 237,\n",
       " 238,\n",
       " 239,\n",
       " 240,\n",
       " 241,\n",
       " 242,\n",
       " 243,\n",
       " 244,\n",
       " 245,\n",
       " 246,\n",
       " 247,\n",
       " 248,\n",
       " 249,\n",
       " 250,\n",
       " 251,\n",
       " 252,\n",
       " 253,\n",
       " 254,\n",
       " 255,\n",
       " 256,\n",
       " 257,\n",
       " 258,\n",
       " 259,\n",
       " 260,\n",
       " 261,\n",
       " 262,\n",
       " 263,\n",
       " 264,\n",
       " 265,\n",
       " 266,\n",
       " 267,\n",
       " 268,\n",
       " 269,\n",
       " 270,\n",
       " 271,\n",
       " 272,\n",
       " 647]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ADM_IDs = sorted(GRID3_ADM['ADM_ID'].unique().tolist())\n",
    "ADM_IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "72693d07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're creating this field to help in removing duplicates from the sjoin_nearest, next section.\n",
    "GRID3_ADM['G3_Area'] = GRID3_ADM['geometry'].area / 10**6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f3bafaff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 0 entries\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   Sett_ID     0 non-null      int64   \n",
      " 1   Bounded_ID  0 non-null      int64   \n",
      " 2   ADM_ID      0 non-null      int64   \n",
      " 3   geometry    0 non-null      geometry\n",
      " 4   G3_Area     0 non-null      float64 \n",
      " 5   year        0 non-null      int32   \n",
      "dtypes: float64(1), geometry(1), int32(1), int64(3)\n",
      "memory usage: 0.0 bytes\n"
     ]
    }
   ],
   "source": [
    "# Create empty geodataframe to append onto using the dataframe whose geometry we want to retain.\n",
    "Bounded = GRID3_ADM[0:0]\n",
    "Bounded[\"year\"] = pd.Series(dtype='int')\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "1b40bc28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 1. Wed Sep 13 11:52:44 2023 \n",
      "\n",
      "Completed near join in admin area 2. Wed Sep 13 11:52:44 2023 \n",
      "\n",
      "Completed near join in admin area 3. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 4. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 5. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 6. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 7. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 8. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 9. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 10. Wed Sep 13 11:52:45 2023 \n",
      "\n",
      "Completed near join in admin area 11. Wed Sep 13 11:52:46 2023 \n",
      "\n",
      "Completed near join in admin area 12. Wed Sep 13 11:52:46 2023 \n",
      "\n",
      "Completed near join in admin area 13. Wed Sep 13 11:52:46 2023 \n",
      "\n",
      "Completed near join in admin area 14. Wed Sep 13 11:52:47 2023 \n",
      "\n",
      "Completed near join in admin area 15. Wed Sep 13 11:52:47 2023 \n",
      "\n",
      "Completed near join in admin area 16. Wed Sep 13 11:52:48 2023 \n",
      "\n",
      "Completed near join in admin area 17. Wed Sep 13 11:52:48 2023 \n",
      "\n",
      "Completed near join in admin area 18. Wed Sep 13 11:52:48 2023 \n",
      "\n",
      "Completed near join in admin area 19. Wed Sep 13 11:52:48 2023 \n",
      "\n",
      "Completed near join in admin area 20. Wed Sep 13 11:52:49 2023 \n",
      "\n",
      "Completed near join in admin area 21. Wed Sep 13 11:52:49 2023 \n",
      "\n",
      "Completed near join in admin area 22. Wed Sep 13 11:52:49 2023 \n",
      "\n",
      "Completed near join in admin area 23. Wed Sep 13 11:52:50 2023 \n",
      "\n",
      "Completed near join in admin area 24. Wed Sep 13 11:52:50 2023 \n",
      "\n",
      "Completed near join in admin area 25. Wed Sep 13 11:52:50 2023 \n",
      "\n",
      "Completed near join in admin area 26. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 27. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 28. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 29. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 30. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 31. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 32. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 33. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 34. Wed Sep 13 11:52:51 2023 \n",
      "\n",
      "Completed near join in admin area 35. Wed Sep 13 11:52:52 2023 \n",
      "\n",
      "Completed near join in admin area 36. Wed Sep 13 11:52:52 2023 \n",
      "\n",
      "Completed near join in admin area 37. Wed Sep 13 11:52:52 2023 \n",
      "\n",
      "Completed near join in admin area 38. Wed Sep 13 11:52:53 2023 \n",
      "\n",
      "Completed near join in admin area 39. Wed Sep 13 11:52:54 2023 \n",
      "\n",
      "Completed near join in admin area 40. Wed Sep 13 11:52:54 2023 \n",
      "\n",
      "Completed near join in admin area 41. Wed Sep 13 11:52:54 2023 \n",
      "\n",
      "Completed near join in admin area 42. Wed Sep 13 11:52:55 2023 \n",
      "\n",
      "Completed near join in admin area 43. Wed Sep 13 11:52:56 2023 \n",
      "\n",
      "Completed near join in admin area 44. Wed Sep 13 11:52:56 2023 \n",
      "\n",
      "Completed near join in admin area 45. Wed Sep 13 11:52:57 2023 \n",
      "\n",
      "Completed near join in admin area 46. Wed Sep 13 11:52:57 2023 \n",
      "\n",
      "Completed near join in admin area 47. Wed Sep 13 11:52:57 2023 \n",
      "\n",
      "Completed near join in admin area 48. Wed Sep 13 11:52:57 2023 \n",
      "\n",
      "Completed near join in admin area 49. Wed Sep 13 11:52:57 2023 \n",
      "\n",
      "Completed near join in admin area 50. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 51. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 52. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 53. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 54. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 55. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 56. Wed Sep 13 11:52:58 2023 \n",
      "\n",
      "Completed near join in admin area 57. Wed Sep 13 11:52:59 2023 \n",
      "\n",
      "Completed near join in admin area 58. Wed Sep 13 11:52:59 2023 \n",
      "\n",
      "Completed near join in admin area 59. Wed Sep 13 11:52:59 2023 \n",
      "\n",
      "Completed near join in admin area 60. Wed Sep 13 11:52:59 2023 \n",
      "\n",
      "Completed near join in admin area 61. Wed Sep 13 11:53:02 2023 \n",
      "\n",
      "Completed near join in admin area 62. Wed Sep 13 11:53:03 2023 \n",
      "\n",
      "Completed near join in admin area 63. Wed Sep 13 11:53:04 2023 \n",
      "\n",
      "Completed near join in admin area 64. Wed Sep 13 11:53:04 2023 \n",
      "\n",
      "Completed near join in admin area 65. Wed Sep 13 11:53:04 2023 \n",
      "\n",
      "Completed near join in admin area 66. Wed Sep 13 11:53:04 2023 \n",
      "\n",
      "Completed near join in admin area 67. Wed Sep 13 11:53:04 2023 \n",
      "\n",
      "Completed near join in admin area 68. Wed Sep 13 11:53:05 2023 \n",
      "\n",
      "Completed near join in admin area 69. Wed Sep 13 11:53:05 2023 \n",
      "\n",
      "Completed near join in admin area 70. Wed Sep 13 11:53:05 2023 \n",
      "\n",
      "Completed near join in admin area 71. Wed Sep 13 11:53:06 2023 \n",
      "\n",
      "Completed near join in admin area 72. Wed Sep 13 11:53:06 2023 \n",
      "\n",
      "Completed near join in admin area 73. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 74. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 75. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 76. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 77. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 78. Wed Sep 13 11:53:07 2023 \n",
      "\n",
      "Completed near join in admin area 79. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 80. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 81. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 82. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 83. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 84. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 85. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 86. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 87. Wed Sep 13 11:53:08 2023 \n",
      "\n",
      "Completed near join in admin area 88. Wed Sep 13 11:53:09 2023 \n",
      "\n",
      "Completed near join in admin area 89. Wed Sep 13 11:53:09 2023 \n",
      "\n",
      "Completed near join in admin area 90. Wed Sep 13 11:53:09 2023 \n",
      "\n",
      "Completed near join in admin area 91. Wed Sep 13 11:53:09 2023 \n",
      "\n",
      "Completed near join in admin area 92. Wed Sep 13 11:53:09 2023 \n",
      "\n",
      "Completed near join in admin area 93. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 94. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 95. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 96. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 97. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 98. Wed Sep 13 11:53:10 2023 \n",
      "\n",
      "Completed near join in admin area 99. Wed Sep 13 11:53:13 2023 \n",
      "\n",
      "Completed near join in admin area 100. Wed Sep 13 11:53:13 2023 \n",
      "\n",
      "Completed near join in admin area 101. Wed Sep 13 11:53:13 2023 \n",
      "\n",
      "Completed near join in admin area 102. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 103. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 104. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 105. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 106. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 107. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 108. Wed Sep 13 11:53:14 2023 \n",
      "\n",
      "Completed near join in admin area 109. Wed Sep 13 11:53:15 2023 \n",
      "\n",
      "Completed near join in admin area 110. Wed Sep 13 11:53:15 2023 \n",
      "\n",
      "Completed near join in admin area 111. Wed Sep 13 11:53:15 2023 \n",
      "\n",
      "Completed near join in admin area 112. Wed Sep 13 11:53:16 2023 \n",
      "\n",
      "Completed near join in admin area 113. Wed Sep 13 11:53:16 2023 \n",
      "\n",
      "Completed near join in admin area 114. Wed Sep 13 11:53:16 2023 \n",
      "\n",
      "Completed near join in admin area 115. Wed Sep 13 11:53:19 2023 \n",
      "\n",
      "Completed near join in admin area 116. Wed Sep 13 11:53:19 2023 \n",
      "\n",
      "Completed near join in admin area 117. Wed Sep 13 11:53:20 2023 \n",
      "\n",
      "Completed near join in admin area 118. Wed Sep 13 11:53:20 2023 \n",
      "\n",
      "Completed near join in admin area 119. Wed Sep 13 11:53:20 2023 \n",
      "\n",
      "Completed near join in admin area 120. Wed Sep 13 11:53:20 2023 \n",
      "\n",
      "Completed near join in admin area 121. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 122. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 123. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 124. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 125. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 126. Wed Sep 13 11:53:21 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 127. Wed Sep 13 11:53:21 2023 \n",
      "\n",
      "Completed near join in admin area 128. Wed Sep 13 11:53:22 2023 \n",
      "\n",
      "Completed near join in admin area 129. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 130. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 131. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 132. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 133. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 134. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 135. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 136. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 137. Wed Sep 13 11:53:26 2023 \n",
      "\n",
      "Completed near join in admin area 138. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 139. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 140. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 141. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 142. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 143. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 144. Wed Sep 13 11:53:27 2023 \n",
      "\n",
      "Completed near join in admin area 145. Wed Sep 13 11:53:28 2023 \n",
      "\n",
      "Completed near join in admin area 146. Wed Sep 13 11:53:28 2023 \n",
      "\n",
      "Completed near join in admin area 147. Wed Sep 13 11:53:28 2023 \n",
      "\n",
      "Completed near join in admin area 148. Wed Sep 13 11:53:28 2023 \n",
      "\n",
      "Completed near join in admin area 149. Wed Sep 13 11:53:28 2023 \n",
      "\n",
      "Completed near join in admin area 150. Wed Sep 13 11:53:29 2023 \n",
      "\n",
      "Completed near join in admin area 151. Wed Sep 13 11:53:29 2023 \n",
      "\n",
      "Completed near join in admin area 152. Wed Sep 13 11:53:29 2023 \n",
      "\n",
      "Completed near join in admin area 153. Wed Sep 13 11:53:31 2023 \n",
      "\n",
      "Completed near join in admin area 154. Wed Sep 13 11:53:31 2023 \n",
      "\n",
      "Completed near join in admin area 155. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 156. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 157. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 158. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 159. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 160. Wed Sep 13 11:53:32 2023 \n",
      "\n",
      "Completed near join in admin area 161. Wed Sep 13 11:53:33 2023 \n",
      "\n",
      "Completed near join in admin area 162. Wed Sep 13 11:53:33 2023 \n",
      "\n",
      "Completed near join in admin area 163. Wed Sep 13 11:53:33 2023 \n",
      "\n",
      "Completed near join in admin area 164. Wed Sep 13 11:53:33 2023 \n",
      "\n",
      "Completed near join in admin area 165. Wed Sep 13 11:53:34 2023 \n",
      "\n",
      "Completed near join in admin area 166. Wed Sep 13 11:53:34 2023 \n",
      "\n",
      "Completed near join in admin area 167. Wed Sep 13 11:53:34 2023 \n",
      "\n",
      "Completed near join in admin area 168. Wed Sep 13 11:53:34 2023 \n",
      "\n",
      "Completed near join in admin area 169. Wed Sep 13 11:53:35 2023 \n",
      "\n",
      "Completed near join in admin area 170. Wed Sep 13 11:53:35 2023 \n",
      "\n",
      "Completed near join in admin area 171. Wed Sep 13 11:53:35 2023 \n",
      "\n",
      "Completed near join in admin area 172. Wed Sep 13 11:53:35 2023 \n",
      "\n",
      "Completed near join in admin area 173. Wed Sep 13 11:53:35 2023 \n",
      "\n",
      "Completed near join in admin area 174. Wed Sep 13 11:53:36 2023 \n",
      "\n",
      "Completed near join in admin area 175. Wed Sep 13 11:53:36 2023 \n",
      "\n",
      "Completed near join in admin area 176. Wed Sep 13 11:53:36 2023 \n",
      "\n",
      "Completed near join in admin area 177. Wed Sep 13 11:53:37 2023 \n",
      "\n",
      "Completed near join in admin area 178. Wed Sep 13 11:53:37 2023 \n",
      "\n",
      "Completed near join in admin area 179. Wed Sep 13 11:53:37 2023 \n",
      "\n",
      "Completed near join in admin area 180. Wed Sep 13 11:53:37 2023 \n",
      "\n",
      "Completed near join in admin area 181. Wed Sep 13 11:53:37 2023 \n",
      "\n",
      "Completed near join in admin area 182. Wed Sep 13 11:53:38 2023 \n",
      "\n",
      "Completed near join in admin area 183. Wed Sep 13 11:53:38 2023 \n",
      "\n",
      "Completed near join in admin area 184. Wed Sep 13 11:53:38 2023 \n",
      "\n",
      "Completed near join in admin area 185. Wed Sep 13 11:53:38 2023 \n",
      "\n",
      "Completed near join in admin area 186. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 187. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 188. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 189. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 190. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 191. Wed Sep 13 11:53:39 2023 \n",
      "\n",
      "Completed near join in admin area 192. Wed Sep 13 11:53:40 2023 \n",
      "\n",
      "Completed near join in admin area 193. Wed Sep 13 11:53:40 2023 \n",
      "\n",
      "Completed near join in admin area 194. Wed Sep 13 11:53:40 2023 \n",
      "\n",
      "Completed near join in admin area 195. Wed Sep 13 11:53:40 2023 \n",
      "\n",
      "Completed near join in admin area 196. Wed Sep 13 11:53:40 2023 \n",
      "\n",
      "Completed near join in admin area 197. Wed Sep 13 11:53:43 2023 \n",
      "\n",
      "Completed near join in admin area 198. Wed Sep 13 11:53:43 2023 \n",
      "\n",
      "Completed near join in admin area 199. Wed Sep 13 11:53:43 2023 \n",
      "\n",
      "Completed near join in admin area 200. Wed Sep 13 11:53:44 2023 \n",
      "\n",
      "Completed near join in admin area 201. Wed Sep 13 11:53:45 2023 \n",
      "\n",
      "Completed near join in admin area 202. Wed Sep 13 11:53:45 2023 \n",
      "\n",
      "Completed near join in admin area 203. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 204. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 205. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 206. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 207. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 208. Wed Sep 13 11:53:46 2023 \n",
      "\n",
      "Completed near join in admin area 209. Wed Sep 13 11:53:47 2023 \n",
      "\n",
      "Completed near join in admin area 210. Wed Sep 13 11:53:47 2023 \n",
      "\n",
      "Completed near join in admin area 211. Wed Sep 13 11:53:47 2023 \n",
      "\n",
      "Completed near join in admin area 212. Wed Sep 13 11:53:47 2023 \n",
      "\n",
      "Completed near join in admin area 213. Wed Sep 13 11:53:47 2023 \n",
      "\n",
      "Completed near join in admin area 214. Wed Sep 13 11:53:48 2023 \n",
      "\n",
      "Completed near join in admin area 215. Wed Sep 13 11:53:48 2023 \n",
      "\n",
      "Completed near join in admin area 216. Wed Sep 13 11:53:48 2023 \n",
      "\n",
      "Completed near join in admin area 217. Wed Sep 13 11:53:48 2023 \n",
      "\n",
      "Completed near join in admin area 218. Wed Sep 13 11:53:48 2023 \n",
      "\n",
      "Completed near join in admin area 219. Wed Sep 13 11:53:49 2023 \n",
      "\n",
      "Completed near join in admin area 220. Wed Sep 13 11:53:49 2023 \n",
      "\n",
      "Completed near join in admin area 221. Wed Sep 13 11:53:49 2023 \n",
      "\n",
      "Completed near join in admin area 222. Wed Sep 13 11:53:49 2023 \n",
      "\n",
      "Completed near join in admin area 223. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 224. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 225. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 226. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 227. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 228. Wed Sep 13 11:53:50 2023 \n",
      "\n",
      "Completed near join in admin area 229. Wed Sep 13 11:53:51 2023 \n",
      "\n",
      "Completed near join in admin area 230. Wed Sep 13 11:53:51 2023 \n",
      "\n",
      "Completed near join in admin area 231. Wed Sep 13 11:53:51 2023 \n",
      "\n",
      "Completed near join in admin area 232. Wed Sep 13 11:53:51 2023 \n",
      "\n",
      "Completed near join in admin area 233. Wed Sep 13 11:53:51 2023 \n",
      "\n",
      "Completed near join in admin area 234. Wed Sep 13 11:53:52 2023 \n",
      "\n",
      "Completed near join in admin area 235. Wed Sep 13 11:53:52 2023 \n",
      "\n",
      "Completed near join in admin area 236. Wed Sep 13 11:53:52 2023 \n",
      "\n",
      "Completed near join in admin area 237. Wed Sep 13 11:53:52 2023 \n",
      "\n",
      "Completed near join in admin area 238. Wed Sep 13 11:53:52 2023 \n",
      "\n",
      "Completed near join in admin area 239. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 240. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 241. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 242. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 243. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 244. Wed Sep 13 11:53:53 2023 \n",
      "\n",
      "Completed near join in admin area 245. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 246. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 247. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 248. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 249. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 250. Wed Sep 13 11:53:54 2023 \n",
      "\n",
      "Completed near join in admin area 251. Wed Sep 13 11:53:55 2023 \n",
      "\n",
      "Completed near join in admin area 252. Wed Sep 13 11:53:55 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 253. Wed Sep 13 11:53:55 2023 \n",
      "\n",
      "Completed near join in admin area 254. Wed Sep 13 11:53:55 2023 \n",
      "\n",
      "Completed near join in admin area 255. Wed Sep 13 11:53:55 2023 \n",
      "\n",
      "Completed near join in admin area 256. Wed Sep 13 11:53:55 2023 \n",
      "\n",
      "Completed near join in admin area 257. Wed Sep 13 11:53:56 2023 \n",
      "\n",
      "Completed near join in admin area 258. Wed Sep 13 11:53:56 2023 \n",
      "\n",
      "Completed near join in admin area 259. Wed Sep 13 11:53:56 2023 \n",
      "\n",
      "Completed near join in admin area 260. Wed Sep 13 11:53:57 2023 \n",
      "\n",
      "Completed near join in admin area 261. Wed Sep 13 11:53:57 2023 \n",
      "\n",
      "Completed near join in admin area 262. Wed Sep 13 11:53:57 2023 \n",
      "\n",
      "Completed near join in admin area 263. Wed Sep 13 11:53:57 2023 \n",
      "\n",
      "Completed near join in admin area 264. Wed Sep 13 11:53:57 2023 \n",
      "\n",
      "Completed near join in admin area 265. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 266. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 267. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 268. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 269. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 270. Wed Sep 13 11:53:58 2023 \n",
      "\n",
      "Completed near join in admin area 271. Wed Sep 13 11:53:59 2023 \n",
      "\n",
      "Completed near join in admin area 272. Wed Sep 13 11:53:59 2023 \n",
      "\n",
      "Completed near join in admin area 647. Wed Sep 13 12:14:02 2023 \n",
      "\n",
      "Completed near join for all ADMs. Wed Sep 13 12:14:02 2023 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for ID in ADM_IDs:\n",
    "    WSFE_shard = WSFE_ADM.loc[WSFE_ADM['ADM_ID'] == ID]\n",
    "    GRID3_shard = GRID3_ADM.loc[GRID3_ADM['ADM_ID'] == ID]\n",
    "    WSFE_GRID3_shard = gpd.sjoin_nearest(WSFE_shard, \n",
    "                                         GRID3_shard, \n",
    "                                         how='inner',\n",
    "                                         max_distance=500)\n",
    "    Bounded = pd.concat([Bounded, WSFE_GRID3_shard])\n",
    "    print('Completed near join in admin area %s. %s \\n' % (ID, time.ctime()))\n",
    "print('Completed near join for all ADMs. %s \\n' % time.ctime())\n",
    "\n",
    "del WSFE_shard, GRID3_shard, WSFE_GRID3_shard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "631c4985",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>Bounded_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>geometry</th>\n",
       "      <th>G3_Area</th>\n",
       "      <th>year</th>\n",
       "      <th>WSFE_ID</th>\n",
       "      <th>ADM_ID_left</th>\n",
       "      <th>index_right</th>\n",
       "      <th>ADM_ID_right</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>960567</th>\n",
       "      <td>10708</td>\n",
       "      <td>10886</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1163715.280 3732570.926, 1163742.941...</td>\n",
       "      <td>29.493118</td>\n",
       "      <td>1990</td>\n",
       "      <td>960567.0</td>\n",
       "      <td>197.0</td>\n",
       "      <td>10886.0</td>\n",
       "      <td>197.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1274853</th>\n",
       "      <td>31915</td>\n",
       "      <td>32458</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1208166.795 3658826.223, 1208194.456...</td>\n",
       "      <td>3.485982</td>\n",
       "      <td>2021</td>\n",
       "      <td>1274853.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>32458.0</td>\n",
       "      <td>176.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726468</th>\n",
       "      <td>50649</td>\n",
       "      <td>51510</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1225786.966 3842800.724, 1225814.627...</td>\n",
       "      <td>93.299741</td>\n",
       "      <td>2021</td>\n",
       "      <td>726468.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>51510.0</td>\n",
       "      <td>115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>600096</th>\n",
       "      <td>21451</td>\n",
       "      <td>21921</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1154421.124 3868497.959, 1154448.785...</td>\n",
       "      <td>0.853897</td>\n",
       "      <td>2020</td>\n",
       "      <td>600096.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>21921.0</td>\n",
       "      <td>260.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1030913</th>\n",
       "      <td>10549</td>\n",
       "      <td>10718</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1181529.079 3727730.220, 1181556.741...</td>\n",
       "      <td>0.009182</td>\n",
       "      <td>1990</td>\n",
       "      <td>1030913.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>10718.0</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1127932</th>\n",
       "      <td>42501</td>\n",
       "      <td>43291</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1208747.679 3718823.320, 1208775.341...</td>\n",
       "      <td>0.013773</td>\n",
       "      <td>1999</td>\n",
       "      <td>1127932.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>43291.0</td>\n",
       "      <td>93.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147717</th>\n",
       "      <td>147384</td>\n",
       "      <td>149113</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1657660.952 4015351.158, 1657688.613...</td>\n",
       "      <td>0.769732</td>\n",
       "      <td>2010</td>\n",
       "      <td>147717.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>149113.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388644</th>\n",
       "      <td>137972</td>\n",
       "      <td>139735</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1561400.049 3930237.710, 1561455.372...</td>\n",
       "      <td>3.586980</td>\n",
       "      <td>2010</td>\n",
       "      <td>388644.0</td>\n",
       "      <td>248.0</td>\n",
       "      <td>139735.0</td>\n",
       "      <td>248.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>681933</th>\n",
       "      <td>52279</td>\n",
       "      <td>53143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1224846.486 3848830.861, 1224901.808...</td>\n",
       "      <td>0.016068</td>\n",
       "      <td>2014</td>\n",
       "      <td>681933.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>53143.0</td>\n",
       "      <td>151.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167853</th>\n",
       "      <td>103121</td>\n",
       "      <td>104770</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1317594.418 4004563.298, 1317622.080...</td>\n",
       "      <td>21.638182</td>\n",
       "      <td>2010</td>\n",
       "      <td>167853.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>104770.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1133196</th>\n",
       "      <td>38534</td>\n",
       "      <td>39210</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1197268.290 3717744.534, 1197295.951...</td>\n",
       "      <td>48.538238</td>\n",
       "      <td>1995</td>\n",
       "      <td>1133196.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>39210.0</td>\n",
       "      <td>45.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>816222</th>\n",
       "      <td>44106</td>\n",
       "      <td>44921</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1215552.329 3762140.726, 1215579.991...</td>\n",
       "      <td>1.075788</td>\n",
       "      <td>2012</td>\n",
       "      <td>816222.0</td>\n",
       "      <td>159.0</td>\n",
       "      <td>44921.0</td>\n",
       "      <td>159.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251111</th>\n",
       "      <td>96393</td>\n",
       "      <td>97919</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1274664.269 3985504.746, 1274691.930...</td>\n",
       "      <td>137.108644</td>\n",
       "      <td>1985</td>\n",
       "      <td>251111.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>97919.0</td>\n",
       "      <td>129.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1364480</th>\n",
       "      <td>32090</td>\n",
       "      <td>32636</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1203602.700 3634760.997, 1203630.361...</td>\n",
       "      <td>4.207509</td>\n",
       "      <td>2012</td>\n",
       "      <td>1364480.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>32636.0</td>\n",
       "      <td>58.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138415</th>\n",
       "      <td>114572</td>\n",
       "      <td>116256</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1359916.022 4017840.664, 1359971.344...</td>\n",
       "      <td>39.303754</td>\n",
       "      <td>2021</td>\n",
       "      <td>138415.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>116256.0</td>\n",
       "      <td>177.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340902</th>\n",
       "      <td>124243</td>\n",
       "      <td>125912</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1464613.584 3961218.231, 1464641.245...</td>\n",
       "      <td>2.865452</td>\n",
       "      <td>2021</td>\n",
       "      <td>340902.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>125912.0</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960340</th>\n",
       "      <td>10708</td>\n",
       "      <td>10883</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1182026.981 3732598.587, 1182109.964...</td>\n",
       "      <td>10.996604</td>\n",
       "      <td>2012</td>\n",
       "      <td>960340.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>10883.0</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>587585</th>\n",
       "      <td>17098</td>\n",
       "      <td>17347</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1156772.324 3873393.988, 1156799.986...</td>\n",
       "      <td>2.579290</td>\n",
       "      <td>2021</td>\n",
       "      <td>587585.0</td>\n",
       "      <td>241.0</td>\n",
       "      <td>17347.0</td>\n",
       "      <td>241.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1246627</th>\n",
       "      <td>8958</td>\n",
       "      <td>9090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1164296.165 3666709.659, 1164379.149...</td>\n",
       "      <td>15.174273</td>\n",
       "      <td>2021</td>\n",
       "      <td>1246627.0</td>\n",
       "      <td>188.0</td>\n",
       "      <td>9090.0</td>\n",
       "      <td>188.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1213098</th>\n",
       "      <td>7239</td>\n",
       "      <td>7332</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((1170409.286 3682946.771, 1170436.947...</td>\n",
       "      <td>10.027170</td>\n",
       "      <td>2021</td>\n",
       "      <td>1213098.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>7332.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Sett_ID  Bounded_ID  ADM_ID  \\\n",
       "960567     10708       10886     NaN   \n",
       "1274853    31915       32458     NaN   \n",
       "726468     50649       51510     NaN   \n",
       "600096     21451       21921     NaN   \n",
       "1030913    10549       10718     NaN   \n",
       "1127932    42501       43291     NaN   \n",
       "147717    147384      149113     NaN   \n",
       "388644    137972      139735     NaN   \n",
       "681933     52279       53143     NaN   \n",
       "167853    103121      104770     NaN   \n",
       "1133196    38534       39210     NaN   \n",
       "816222     44106       44921     NaN   \n",
       "251111     96393       97919     NaN   \n",
       "1364480    32090       32636     NaN   \n",
       "138415    114572      116256     NaN   \n",
       "340902    124243      125912     NaN   \n",
       "960340     10708       10883     NaN   \n",
       "587585     17098       17347     NaN   \n",
       "1246627     8958        9090     NaN   \n",
       "1213098     7239        7332     NaN   \n",
       "\n",
       "                                                  geometry     G3_Area  year  \\\n",
       "960567   POLYGON ((1163715.280 3732570.926, 1163742.941...   29.493118  1990   \n",
       "1274853  POLYGON ((1208166.795 3658826.223, 1208194.456...    3.485982  2021   \n",
       "726468   POLYGON ((1225786.966 3842800.724, 1225814.627...   93.299741  2021   \n",
       "600096   POLYGON ((1154421.124 3868497.959, 1154448.785...    0.853897  2020   \n",
       "1030913  POLYGON ((1181529.079 3727730.220, 1181556.741...    0.009182  1990   \n",
       "1127932  POLYGON ((1208747.679 3718823.320, 1208775.341...    0.013773  1999   \n",
       "147717   POLYGON ((1657660.952 4015351.158, 1657688.613...    0.769732  2010   \n",
       "388644   POLYGON ((1561400.049 3930237.710, 1561455.372...    3.586980  2010   \n",
       "681933   POLYGON ((1224846.486 3848830.861, 1224901.808...    0.016068  2014   \n",
       "167853   POLYGON ((1317594.418 4004563.298, 1317622.080...   21.638182  2010   \n",
       "1133196  POLYGON ((1197268.290 3717744.534, 1197295.951...   48.538238  1995   \n",
       "816222   POLYGON ((1215552.329 3762140.726, 1215579.991...    1.075788  2012   \n",
       "251111   POLYGON ((1274664.269 3985504.746, 1274691.930...  137.108644  1985   \n",
       "1364480  POLYGON ((1203602.700 3634760.997, 1203630.361...    4.207509  2012   \n",
       "138415   POLYGON ((1359916.022 4017840.664, 1359971.344...   39.303754  2021   \n",
       "340902   POLYGON ((1464613.584 3961218.231, 1464641.245...    2.865452  2021   \n",
       "960340   POLYGON ((1182026.981 3732598.587, 1182109.964...   10.996604  2012   \n",
       "587585   POLYGON ((1156772.324 3873393.988, 1156799.986...    2.579290  2021   \n",
       "1246627  POLYGON ((1164296.165 3666709.659, 1164379.149...   15.174273  2021   \n",
       "1213098  POLYGON ((1170409.286 3682946.771, 1170436.947...   10.027170  2021   \n",
       "\n",
       "           WSFE_ID  ADM_ID_left  index_right  ADM_ID_right  \n",
       "960567    960567.0        197.0      10886.0         197.0  \n",
       "1274853  1274853.0        176.0      32458.0         176.0  \n",
       "726468    726468.0        115.0      51510.0         115.0  \n",
       "600096    600096.0        260.0      21921.0         260.0  \n",
       "1030913  1030913.0         61.0      10718.0          61.0  \n",
       "1127932  1127932.0         93.0      43291.0          93.0  \n",
       "147717    147717.0         11.0     149113.0          11.0  \n",
       "388644    388644.0        248.0     139735.0         248.0  \n",
       "681933    681933.0        151.0      53143.0         151.0  \n",
       "167853    167853.0         18.0     104770.0          18.0  \n",
       "1133196  1133196.0         45.0      39210.0          45.0  \n",
       "816222    816222.0        159.0      44921.0         159.0  \n",
       "251111    251111.0        129.0      97919.0         129.0  \n",
       "1364480  1364480.0         58.0      32636.0          58.0  \n",
       "138415    138415.0        177.0     116256.0         177.0  \n",
       "340902    340902.0         26.0     125912.0          26.0  \n",
       "960340    960340.0         61.0      10883.0          61.0  \n",
       "587585    587585.0        241.0      17347.0         241.0  \n",
       "1246627  1246627.0        188.0       9090.0         188.0  \n",
       "1213098  1213098.0         37.0       7332.0          37.0  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Bounded.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "c5b56278",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 1353979 entries, 124741 to 1375669\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count    Dtype   \n",
      "---  ------        --------------    -----   \n",
      " 0   Sett_ID       1353979 non-null  int64   \n",
      " 1   Bounded_ID    1353979 non-null  int64   \n",
      " 2   ADM_ID        0 non-null        float64 \n",
      " 3   geometry      1353979 non-null  geometry\n",
      " 4   G3_Area       1353979 non-null  float64 \n",
      " 5   year          1353979 non-null  int32   \n",
      " 6   WSFE_ID       1353979 non-null  float64 \n",
      " 7   ADM_ID_left   1353979 non-null  float64 \n",
      " 8   index_right   1353979 non-null  float64 \n",
      " 9   ADM_ID_right  1353979 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 108.5 MB\n"
     ]
    }
   ],
   "source": [
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "fdc68243",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 1353979 entries, 124741 to 1375669\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count    Dtype   \n",
      "---  ------        --------------    -----   \n",
      " 0   Sett_ID       1353979 non-null  int64   \n",
      " 1   Bounded_ID    1353979 non-null  int64   \n",
      " 2   ADM_ID        0 non-null        float64 \n",
      " 3   geometry      1353979 non-null  geometry\n",
      " 4   G3_Area       1353979 non-null  float64 \n",
      " 5   year          1353979 non-null  int32   \n",
      " 6   WSFE_ID       1353979 non-null  float64 \n",
      " 7   ADM_ID_left   1353979 non-null  float64 \n",
      " 8   index_right   1353979 non-null  float64 \n",
      " 9   ADM_ID_right  1353979 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 108.5 MB\n"
     ]
    }
   ],
   "source": [
    "# Remove WSFE features that did not match any GRID3 settlements.\n",
    "Bounded = Bounded.loc[~Bounded['Sett_ID'].isna()]\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "c5736e5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "del GRID3_ADM, ADM_IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "059bd29d",
   "metadata": {},
   "source": [
    "### 4.2 Remove duplicates: where buildup polygons intersected with more than one GRID3 settlement extent.\n",
    "This happens when the first dataset (WSFE) intersects (distance = 0) with more than one feature of the second dataset (GRID3). More common for large cities. For example, Yaoundé, CMN has a large contiguous 1985 WSFE polygon which overlaps several small GRID3 features that are not Yaoundé."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "49786b23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 8498\n"
     ]
    }
   ],
   "source": [
    "# The first number should always be zero. \n",
    "# The second tells us whether/how many WSFE polygons were duplicated by the Near join.\n",
    "\n",
    "print(len(WSFE_ADM[WSFE_ADM.duplicated('WSFE_ID')]), len(Bounded[Bounded.duplicated('WSFE_ID')]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "0b92f155",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 1345481 entries, 1375669 to 579652\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count    Dtype   \n",
      "---  ------        --------------    -----   \n",
      " 0   Sett_ID       1345481 non-null  int64   \n",
      " 1   Bounded_ID    1345481 non-null  int64   \n",
      " 2   ADM_ID        0 non-null        float64 \n",
      " 3   geometry      1345481 non-null  geometry\n",
      " 4   G3_Area       1345481 non-null  float64 \n",
      " 5   year          1345481 non-null  int32   \n",
      " 6   WSFE_ID       1345481 non-null  float64 \n",
      " 7   ADM_ID_left   1345481 non-null  float64 \n",
      " 8   index_right   1345481 non-null  float64 \n",
      " 9   ADM_ID_right  1345481 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 107.8 MB\n"
     ]
    }
   ],
   "source": [
    "# If there are duplicate WSFE_IDs, then we need to choose between them.\n",
    "# We'll pick the one that joined with the largest GRID3 polygon.\n",
    "# To do that, we can just sort the dataframe by GRID3 areas, then drop_duplicates. \n",
    "# It will retain the first row of each WSFE_ID group.\n",
    "Bounded = Bounded.sort_values('G3_Area', ascending=False).drop_duplicates(['WSFE_ID'])\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "cd6141b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(len(Bounded[Bounded.duplicated('WSFE_ID')]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "7d22854d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 97655 entries, 0 to 97654\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count  Dtype   \n",
      "---  ------        --------------  -----   \n",
      " 0   year          97655 non-null  int64   \n",
      " 1   Bounded_ID    97655 non-null  int64   \n",
      " 2   geometry      97655 non-null  geometry\n",
      " 3   Sett_ID       97655 non-null  int64   \n",
      " 4   ADM_ID        0 non-null      float64 \n",
      " 5   G3_Area       97655 non-null  float64 \n",
      " 6   WSFE_ID       97655 non-null  float64 \n",
      " 7   ADM_ID_left   97655 non-null  float64 \n",
      " 8   index_right   97655 non-null  float64 \n",
      " 9   ADM_ID_right  97655 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int64(3)\n",
      "memory usage: 7.5 MB\n",
      "None        year  Bounded_ID                                           geometry  \\\n",
      "73876  2018      139736  MULTIPOLYGON (((1582533.190 3916573.088, 15825...   \n",
      "11642  1997       43147  POLYGON ((1214030.965 3718629.692, 1214058.626...   \n",
      "85183  2020       78755  POLYGON ((1215552.329 3988243.203, 1215579.991...   \n",
      "83123  2020       31003  MULTIPOLYGON (((1191736.054 3638605.901, 11917...   \n",
      "3941   1988       10884  MULTIPOLYGON (((1172594.519 3734147.613, 11725...   \n",
      "47251  2014        6970  MULTIPOLYGON (((1145514.225 3699875.413, 11455...   \n",
      "84529  2020       59180  POLYGON ((1228968.001 3888635.297, 1228995.662...   \n",
      "49333  2014       21908  POLYGON ((1168279.375 3898206.065, 1168307.036...   \n",
      "89443  2021       16555  MULTIPOLYGON (((1178790.623 3846534.983, 11787...   \n",
      "39402  2013        6903  MULTIPOLYGON (((1150908.154 3700788.231, 11509...   \n",
      "\n",
      "       Sett_ID  ADM_ID   G3_Area    WSFE_ID  ADM_ID_left  index_right  \\\n",
      "73876   137973     NaN  3.513527   420426.0        248.0     139736.0   \n",
      "11642    42359     NaN  0.006886  1129042.0        105.0      43147.0   \n",
      "85183    77327     NaN  0.079575   229681.0        109.0      78755.0   \n",
      "83123    30477     NaN  0.250201  1357493.0        132.0      31003.0   \n",
      "3941     10708     NaN  4.815031   938498.0         71.0      10884.0   \n",
      "47251     6884     NaN  0.020659  1184829.0        145.0       6970.0   \n",
      "84529    58144     NaN  0.006886   514365.0         99.0      59180.0   \n",
      "49333    21438     NaN  0.013773   469519.0         60.0      21908.0   \n",
      "89443    16331     NaN  0.022954   700749.0        257.0      16555.0   \n",
      "39402     6819     NaN  0.013773  1182720.0        145.0       6903.0   \n",
      "\n",
      "       ADM_ID_right  \n",
      "73876         248.0  \n",
      "11642         105.0  \n",
      "85183         109.0  \n",
      "83123         132.0  \n",
      "3941           71.0  \n",
      "47251         145.0  \n",
      "84529          99.0  \n",
      "49333          60.0  \n",
      "89443         257.0  \n",
      "39402         145.0  \n"
     ]
    }
   ],
   "source": [
    "# Now we can dissolve with the WSFE years, now that we can group them by their administratively split ID.\n",
    "Bounded = Bounded.dissolve(by=['year', 'Bounded_ID'], as_index=False)\n",
    "print(Bounded.info(), Bounded.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "8277aa99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       ADM_ID  year  Bounded_ID  Sett_ID  \\\n",
      "68637      18  2017      108810   107158   \n",
      "37793     193  2012      154246   152602   \n",
      "4910      151  1989       47593    46745   \n",
      "46280      40  2014        2043     2034   \n",
      "67645     175  2017       58533    57518   \n",
      "17424      70  2005       40448    39728   \n",
      "33805     270  2012       45622    44792   \n",
      "30835      13  2012        9580     9437   \n",
      "29216      99  2011       59170    58135   \n",
      "51072     239  2014       39993    39284   \n",
      "\n",
      "                                                geometry  \n",
      "68637  MULTIPOLYGON (((1330429.205 4007661.350, 13304...  \n",
      "37793  POLYGON ((1739593.364 4063785.882, 1739648.686...  \n",
      "4910   MULTIPOLYGON (((1222854.881 3845926.437, 12228...  \n",
      "46280  POLYGON ((1150548.559 3657664.454, 1150576.220...  \n",
      "67645  MULTIPOLYGON (((1194170.238 3877819.776, 11941...  \n",
      "17424  POLYGON ((1231236.218 3735143.416, 1231263.879...  \n",
      "33805  MULTIPOLYGON (((1222412.302 3770632.708, 12224...  \n",
      "30835  POLYGON ((1143080.041 3675893.170, 1143107.702...  \n",
      "29216  MULTIPOLYGON (((1236353.536 3884956.361, 12364...  \n",
      "51072  POLYGON ((1200366.342 3748061.186, 1200394.003...  \n"
     ]
    }
   ],
   "source": [
    "# Clean up and save to file.\n",
    "Bounded = Bounded[['ADM_ID_left', 'year', 'Bounded_ID', 'Sett_ID', 'geometry']].rename(columns={\"ADM_ID_left\": \"ADM_ID\"})\n",
    "Bounded = Bounded.astype({\"ADM_ID\":'int', \"Bounded_ID\":'int', \"Sett_ID\":'int', \"year\":'int'})\n",
    "print(Bounded.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "e45ca6e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "Bounded.to_file(\n",
    "    driver='GPKG', filename=os.path.join(Intermediate,'NonCumulativeSettlements.gpkg'), layer='Settlements_Bounded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "75e43acf",
   "metadata": {},
   "outputs": [],
   "source": [
    "del WSFE_ADM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea6eadab",
   "metadata": {},
   "source": [
    "### 4.3 BOUNDLESS SETTLEMENTS: Dissolve features that were split by an ADM boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "f2638d9c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 93908 entries, 0 to 93907\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   year        93908 non-null  int64   \n",
      " 1   Sett_ID     93908 non-null  int64   \n",
      " 2   geometry    93908 non-null  geometry\n",
      " 3   ADM_ID      93908 non-null  int32   \n",
      " 4   Bounded_ID  93908 non-null  int32   \n",
      "dtypes: geometry(1), int32(2), int64(2)\n",
      "memory usage: 2.9 MB\n",
      "None        year  Sett_ID                                           geometry  \\\n",
      "48085  2014    31001  MULTIPOLYGON (((1232840.566 3655894.138, 12328...   \n",
      "24313  2010    59900  MULTIPOLYGON (((1258012.239 3898814.611, 12580...   \n",
      "6940   1990   100777  POLYGON ((1308327.924 3997316.069, 1308383.246...   \n",
      "72051  2019      363  MULTIPOLYGON (((1146205.754 3658217.677, 11461...   \n",
      "21250  2010    26282  POLYGON ((1150437.914 3934276.243, 1150465.576...   \n",
      "1730   1985   135003  MULTIPOLYGON (((1487074.462 3962048.066, 14870...   \n",
      "3539   1987    84886  POLYGON ((1268495.826 3835553.495, 1268523.487...   \n",
      "66407  2017   143215  MULTIPOLYGON (((1627759.217 3871015.126, 16277...   \n",
      "72177  2019     2019  MULTIPOLYGON (((1172262.585 3647540.462, 11722...   \n",
      "86784  2021    24340  MULTIPOLYGON (((1168749.615 3920500.975, 11687...   \n",
      "\n",
      "       ADM_ID  Bounded_ID  \n",
      "48085     171       31533  \n",
      "24313      36       61001  \n",
      "6940       18      102403  \n",
      "72051      40         369  \n",
      "21250      47       26815  \n",
      "1730       26      136753  \n",
      "3539       88       86242  \n",
      "66407      20      144907  \n",
      "72177      63        2027  \n",
      "86784      15       24858  \n"
     ]
    }
   ],
   "source": [
    "# Fragments of any bounded settlement will be combined into a single \"boundless\" settlement in this version.\n",
    "# It is based on their \"Sett_ID\", which is a direct loan from the GRID3 settlement features.\n",
    "Boundless = Bounded.dissolve(by=['year', 'Sett_ID'], as_index=False)\n",
    "print(Boundless.info(), Boundless.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "b76e2a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up and save to file.\n",
    "Boundless.to_file(driver='GPKG', \n",
    "                  filename=os.path.join(Intermediate,'NonCumulativeSettlements.gpkg'), layer='Settlements_Boundless')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbba12eb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c027eba",
   "metadata": {},
   "source": [
    "## 5. CUMULATIVE ANNUALIZED SETTLEMENT EXTENTS\n",
    "DISSOLVE BY YEAR SETS: Create separate feature layers of each cumulative year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d70322",
   "metadata": {},
   "source": [
    "### 5.1 Define study years for each for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "cda3995e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Boundless = gpd.read_file(os.path.join(Intermediate,'NonCumulativeSettlements.gpkg'), layer='Settlements_Boundless')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "60c3aff1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " [2020, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999, 1998, 1997, 1996, 1995, 1994, 1993, 1992, 1991, 1990, 1989, 1988, 1987, 1986, 1985]\n"
     ]
    }
   ],
   "source": [
    "Reversed_WSFE_Years.remove(WSFE_end) # We'll call the last year in the study, then use this list for subsequent sets.\n",
    "print(Reversed_WSFE_Years)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15de5331",
   "metadata": {},
   "source": [
    "### 5.2 Starting with main Boundless dataset, create a cumulative area feature layer for each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "b3c1be53",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1985. Wed Sep 13 15:32:59 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:32:59 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:33:06 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1986. Wed Sep 13 15:33:09 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:33:09 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:33:18 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1987. Wed Sep 13 15:33:21 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:33:21 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:33:33 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1988. Wed Sep 13 15:33:38 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:33:38 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:33:51 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1989. Wed Sep 13 15:33:55 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:33:55 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:34:09 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1990. Wed Sep 13 15:34:13 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:34:13 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:34:31 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1991. Wed Sep 13 15:34:36 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:34:36 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:34:54 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1992. Wed Sep 13 15:34:59 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:34:59 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:35:18 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1993. Wed Sep 13 15:35:22 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:35:22 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:35:41 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1994. Wed Sep 13 15:35:45 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:35:45 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:36:05 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1995. Wed Sep 13 15:36:09 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:36:09 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:36:30 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1996. Wed Sep 13 15:36:35 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:36:35 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:36:58 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1997. Wed Sep 13 15:37:04 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:37:04 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:37:26 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1998. Wed Sep 13 15:37:31 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:37:31 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:37:55 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1999. Wed Sep 13 15:37:59 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:37:59 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:38:23 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Wed Sep 13 15:38:27 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:38:27 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:38:51 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Wed Sep 13 15:38:56 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:38:56 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:39:20 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Wed Sep 13 15:39:24 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:39:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:39:48 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Wed Sep 13 15:39:53 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:39:53 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:40:19 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Wed Sep 13 15:40:23 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:40:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:40:49 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Wed Sep 13 15:40:54 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:40:54 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:41:21 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Wed Sep 13 15:41:25 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:41:25 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:41:52 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Wed Sep 13 15:41:56 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:41:57 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:42:24 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Wed Sep 13 15:42:28 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:42:28 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:42:57 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Wed Sep 13 15:43:02 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:43:02 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:43:28 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Wed Sep 13 15:43:32 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:43:32 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:44:16 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Wed Sep 13 15:44:35 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:44:35 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:46:02 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Wed Sep 13 15:46:24 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:46:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:47:40 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Wed Sep 13 15:47:51 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:47:51 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:48:48 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Wed Sep 13 15:48:59 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:48:59 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:50:13 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Wed Sep 13 15:50:26 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:50:26 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:51:44 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2016. Wed Sep 13 15:51:57 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:51:57 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:53:13 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2017. Wed Sep 13 15:53:27 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:53:27 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:54:52 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2018. Wed Sep 13 15:55:05 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:55:05 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:56:29 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2019. Wed Sep 13 15:56:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:56:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:58:11 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2020. Wed Sep 13 15:58:25 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 15:58:25 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 15:59:56 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2021. Wed Sep 13 16:00:10 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Wed Sep 13 16:00:10 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\1314417394.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 16:01:47 2023\n",
      "\n",
      "Done with all years in set. Wed Sep 13 16:02:00 2023\n"
     ]
    }
   ],
   "source": [
    "# For each year in the growth stats study, we are taking features from all years prior to and including that year, \n",
    "# dissolving those features, and exporting as its own file.\n",
    "\n",
    "for item in WSFE_Years:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    YearSet = Boundless[Boundless['year'].between(\n",
    "        WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the start year and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    YearDissolve = YearSet.dissolve(by='Sett_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\"}, # Though ADM_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    YearName = ''.join(['Cu', str(item), '_Boundless'])\n",
    "    YearDissolve.to_file(driver='GPKG', filename=os.path.join(Results,'CumulativeSettlements.gpkg'), layer=YearName)\n",
    "    del YearSet, YearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "307c4746",
   "metadata": {},
   "source": [
    "##### Join area information from each cumulative layer onto the latest year dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "8acdcfc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The latest year in the study contains all settlements. Merge all other years' areas onto this dataset.\n",
    "SettAreas = gpd.read_file(os.path.join(Results,'CumulativeSettlements.gpkg'), layer=\n",
    "                          ''.join(['Cu', str(WSFE_end), '_Boundless'])) \n",
    "SettAreas['AREA'+str(WSFE_end)] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry') # We have settlement IDs, so no need to join spatially!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "b33dc9c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2020. Wed Sep 13 16:02:03 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:06 2023\n",
      "\n",
      "Merging variables from 2020 onto our latest year (2021) via table join. Wed Sep 13 16:02:06 2023\n",
      "\n",
      "Loading cumulative layer for year 2019. Wed Sep 13 16:02:06 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:08 2023\n",
      "\n",
      "Merging variables from 2019 onto our latest year (2021) via table join. Wed Sep 13 16:02:08 2023\n",
      "\n",
      "Loading cumulative layer for year 2018. Wed Sep 13 16:02:08 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:11 2023\n",
      "\n",
      "Merging variables from 2018 onto our latest year (2021) via table join. Wed Sep 13 16:02:11 2023\n",
      "\n",
      "Loading cumulative layer for year 2017. Wed Sep 13 16:02:11 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:13 2023\n",
      "\n",
      "Merging variables from 2017 onto our latest year (2021) via table join. Wed Sep 13 16:02:14 2023\n",
      "\n",
      "Loading cumulative layer for year 2016. Wed Sep 13 16:02:14 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:16 2023\n",
      "\n",
      "Merging variables from 2016 onto our latest year (2021) via table join. Wed Sep 13 16:02:16 2023\n",
      "\n",
      "Loading cumulative layer for year 2015. Wed Sep 13 16:02:16 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:19 2023\n",
      "\n",
      "Merging variables from 2015 onto our latest year (2021) via table join. Wed Sep 13 16:02:19 2023\n",
      "\n",
      "Loading cumulative layer for year 2014. Wed Sep 13 16:02:19 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:21 2023\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2021) via table join. Wed Sep 13 16:02:22 2023\n",
      "\n",
      "Loading cumulative layer for year 2013. Wed Sep 13 16:02:22 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:24 2023\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2021) via table join. Wed Sep 13 16:02:24 2023\n",
      "\n",
      "Loading cumulative layer for year 2012. Wed Sep 13 16:02:24 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:26 2023\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2021) via table join. Wed Sep 13 16:02:26 2023\n",
      "\n",
      "Loading cumulative layer for year 2011. Wed Sep 13 16:02:27 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:29 2023\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2021) via table join. Wed Sep 13 16:02:29 2023\n",
      "\n",
      "Loading cumulative layer for year 2010. Wed Sep 13 16:02:29 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:31 2023\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2021) via table join. Wed Sep 13 16:02:31 2023\n",
      "\n",
      "Loading cumulative layer for year 2009. Wed Sep 13 16:02:31 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:32 2023\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2021) via table join. Wed Sep 13 16:02:32 2023\n",
      "\n",
      "Loading cumulative layer for year 2008. Wed Sep 13 16:02:32 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:33 2023\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2021) via table join. Wed Sep 13 16:02:33 2023\n",
      "\n",
      "Loading cumulative layer for year 2007. Wed Sep 13 16:02:33 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:34 2023\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2021) via table join. Wed Sep 13 16:02:34 2023\n",
      "\n",
      "Loading cumulative layer for year 2006. Wed Sep 13 16:02:34 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:35 2023\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2021) via table join. Wed Sep 13 16:02:35 2023\n",
      "\n",
      "Loading cumulative layer for year 2005. Wed Sep 13 16:02:35 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:36 2023\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2021) via table join. Wed Sep 13 16:02:36 2023\n",
      "\n",
      "Loading cumulative layer for year 2004. Wed Sep 13 16:02:36 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:37 2023\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2021) via table join. Wed Sep 13 16:02:37 2023\n",
      "\n",
      "Loading cumulative layer for year 2003. Wed Sep 13 16:02:37 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:38 2023\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2021) via table join. Wed Sep 13 16:02:38 2023\n",
      "\n",
      "Loading cumulative layer for year 2002. Wed Sep 13 16:02:38 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:39 2023\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2021) via table join. Wed Sep 13 16:02:39 2023\n",
      "\n",
      "Loading cumulative layer for year 2001. Wed Sep 13 16:02:39 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:40 2023\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2021) via table join. Wed Sep 13 16:02:40 2023\n",
      "\n",
      "Loading cumulative layer for year 2000. Wed Sep 13 16:02:40 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:41 2023\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2021) via table join. Wed Sep 13 16:02:41 2023\n",
      "\n",
      "Loading cumulative layer for year 1999. Wed Sep 13 16:02:41 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:42 2023\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2021) via table join. Wed Sep 13 16:02:42 2023\n",
      "\n",
      "Loading cumulative layer for year 1998. Wed Sep 13 16:02:42 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:43 2023\n",
      "\n",
      "Merging variables from 1998 onto our latest year (2021) via table join. Wed Sep 13 16:02:43 2023\n",
      "\n",
      "Loading cumulative layer for year 1997. Wed Sep 13 16:02:43 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:44 2023\n",
      "\n",
      "Merging variables from 1997 onto our latest year (2021) via table join. Wed Sep 13 16:02:44 2023\n",
      "\n",
      "Loading cumulative layer for year 1996. Wed Sep 13 16:02:44 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:45 2023\n",
      "\n",
      "Merging variables from 1996 onto our latest year (2021) via table join. Wed Sep 13 16:02:45 2023\n",
      "\n",
      "Loading cumulative layer for year 1995. Wed Sep 13 16:02:45 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:46 2023\n",
      "\n",
      "Merging variables from 1995 onto our latest year (2021) via table join. Wed Sep 13 16:02:46 2023\n",
      "\n",
      "Loading cumulative layer for year 1994. Wed Sep 13 16:02:46 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:47 2023\n",
      "\n",
      "Merging variables from 1994 onto our latest year (2021) via table join. Wed Sep 13 16:02:47 2023\n",
      "\n",
      "Loading cumulative layer for year 1993. Wed Sep 13 16:02:47 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:49 2023\n",
      "\n",
      "Merging variables from 1993 onto our latest year (2021) via table join. Wed Sep 13 16:02:49 2023\n",
      "\n",
      "Loading cumulative layer for year 1992. Wed Sep 13 16:02:49 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:50 2023\n",
      "\n",
      "Merging variables from 1992 onto our latest year (2021) via table join. Wed Sep 13 16:02:50 2023\n",
      "\n",
      "Loading cumulative layer for year 1991. Wed Sep 13 16:02:50 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:51 2023\n",
      "\n",
      "Merging variables from 1991 onto our latest year (2021) via table join. Wed Sep 13 16:02:51 2023\n",
      "\n",
      "Loading cumulative layer for year 1990. Wed Sep 13 16:02:51 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:53 2023\n",
      "\n",
      "Merging variables from 1990 onto our latest year (2021) via table join. Wed Sep 13 16:02:53 2023\n",
      "\n",
      "Loading cumulative layer for year 1989. Wed Sep 13 16:02:53 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:54 2023\n",
      "\n",
      "Merging variables from 1989 onto our latest year (2021) via table join. Wed Sep 13 16:02:54 2023\n",
      "\n",
      "Loading cumulative layer for year 1988. Wed Sep 13 16:02:54 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:55 2023\n",
      "\n",
      "Merging variables from 1988 onto our latest year (2021) via table join. Wed Sep 13 16:02:55 2023\n",
      "\n",
      "Loading cumulative layer for year 1987. Wed Sep 13 16:02:55 2023\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:56 2023\n",
      "\n",
      "Merging variables from 1987 onto our latest year (2021) via table join. Wed Sep 13 16:02:56 2023\n",
      "\n",
      "Loading cumulative layer for year 1986. Wed Sep 13 16:02:56 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:57 2023\n",
      "\n",
      "Merging variables from 1986 onto our latest year (2021) via table join. Wed Sep 13 16:02:57 2023\n",
      "\n",
      "Loading cumulative layer for year 1985. Wed Sep 13 16:02:57 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 16:02:58 2023\n",
      "\n",
      "Merging variables from 1985 onto our latest year (2021) via table join. Wed Sep 13 16:02:58 2023\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Wed Sep 13 16:02:58 2023\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 40 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   18552 non-null  int64  \n",
      " 1   year      18552 non-null  int64  \n",
      " 2   ADM_ID    18552 non-null  int64  \n",
      " 3   AREA2021  18552 non-null  float64\n",
      " 4   AREA2020  17807 non-null  float64\n",
      " 5   AREA2019  17460 non-null  float64\n",
      " 6   AREA2018  17058 non-null  float64\n",
      " 7   AREA2017  16845 non-null  float64\n",
      " 8   AREA2016  16629 non-null  float64\n",
      " 9   AREA2015  16479 non-null  float64\n",
      " 10  AREA2014  15938 non-null  float64\n",
      " 11  AREA2013  14137 non-null  float64\n",
      " 12  AREA2012  13533 non-null  float64\n",
      " 13  AREA2011  11442 non-null  float64\n",
      " 14  AREA2010  11415 non-null  float64\n",
      " 15  AREA2009  4755 non-null   float64\n",
      " 16  AREA2008  4744 non-null   float64\n",
      " 17  AREA2007  4739 non-null   float64\n",
      " 18  AREA2006  4730 non-null   float64\n",
      " 19  AREA2005  4723 non-null   float64\n",
      " 20  AREA2004  4715 non-null   float64\n",
      " 21  AREA2003  4699 non-null   float64\n",
      " 22  AREA2002  4689 non-null   float64\n",
      " 23  AREA2001  4680 non-null   float64\n",
      " 24  AREA2000  4670 non-null   float64\n",
      " 25  AREA1999  4075 non-null   float64\n",
      " 26  AREA1998  4058 non-null   float64\n",
      " 27  AREA1997  4031 non-null   float64\n",
      " 28  AREA1996  4017 non-null   float64\n",
      " 29  AREA1995  3949 non-null   float64\n",
      " 30  AREA1994  3907 non-null   float64\n",
      " 31  AREA1993  3885 non-null   float64\n",
      " 32  AREA1992  3864 non-null   float64\n",
      " 33  AREA1991  3853 non-null   float64\n",
      " 34  AREA1990  3835 non-null   float64\n",
      " 35  AREA1989  2507 non-null   float64\n",
      " 36  AREA1988  2484 non-null   float64\n",
      " 37  AREA1987  2441 non-null   float64\n",
      " 38  AREA1986  2372 non-null   float64\n",
      " 39  AREA1985  2139 non-null   float64\n",
      "dtypes: float64(37), int64(3)\n",
      "memory usage: 5.8 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "for item in Reversed_WSFE_Years:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(os.path.join(Results,'CumulativeSettlements.gpkg'), \n",
    "                              layer=''.join(['Cu', str(item), '_Boundless']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['AREA', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Sett_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, WSFE_end, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Sett_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(Results, 'Area.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "1d0879f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a1cafcd",
   "metadata": {},
   "source": [
    "### 5.3 Repeat for Bounded dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "ad1dcf6f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1985. Wed Sep 13 13:44:20 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:44:20 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:44:25 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1986. Wed Sep 13 13:44:27 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:44:27 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:44:35 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1987. Wed Sep 13 13:44:37 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:44:37 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:44:47 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1988. Wed Sep 13 13:44:50 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:44:50 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:45:02 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1989. Wed Sep 13 13:45:05 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:45:05 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:45:17 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1990. Wed Sep 13 13:45:20 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:45:20 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:45:37 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1991. Wed Sep 13 13:45:40 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:45:40 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:45:57 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1992. Wed Sep 13 13:46:01 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:46:01 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:46:18 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1993. Wed Sep 13 13:46:23 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:46:23 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:46:39 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1994. Wed Sep 13 13:46:43 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:46:43 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:46:59 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1995. Wed Sep 13 13:47:03 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:47:03 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:47:22 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1996. Wed Sep 13 13:47:26 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:47:26 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:47:45 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1997. Wed Sep 13 13:47:49 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:47:49 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:48:08 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1998. Wed Sep 13 13:48:12 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:48:12 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:48:31 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 1999. Wed Sep 13 13:48:35 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:48:35 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:48:54 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Wed Sep 13 13:48:58 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:48:58 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:49:20 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Wed Sep 13 13:49:24 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:49:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:49:49 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Wed Sep 13 13:49:54 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:49:54 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:50:17 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Wed Sep 13 13:50:21 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:50:21 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:50:45 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Wed Sep 13 13:50:50 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:50:50 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:51:15 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Wed Sep 13 13:51:20 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:51:20 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:51:43 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Wed Sep 13 13:51:49 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:51:49 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:52:14 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Wed Sep 13 13:52:20 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:52:20 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:52:45 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Wed Sep 13 13:52:49 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:52:49 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:53:14 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Wed Sep 13 13:53:19 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:53:19 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:53:45 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Wed Sep 13 13:53:50 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:53:50 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:54:31 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Wed Sep 13 13:54:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:54:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:55:25 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Wed Sep 13 13:55:36 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:55:36 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:56:32 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Wed Sep 13 13:56:44 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:56:44 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:57:44 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Wed Sep 13 13:57:56 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:57:56 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 13:59:08 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Wed Sep 13 13:59:22 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 13:59:22 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:00:39 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2016. Wed Sep 13 14:00:53 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:00:53 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:02:10 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2017. Wed Sep 13 14:02:22 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:02:22 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:03:40 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2018. Wed Sep 13 14:03:53 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:03:54 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:05:14 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2019. Wed Sep 13 14:05:27 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:05:27 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:07:46 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2020. Wed Sep 13 14:08:12 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:08:12 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:11:09 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2021. Wed Sep 13 14:11:39 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Wed Sep 13 14:11:39 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_28192\\421491971.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Wed Sep 13 14:14:59 2023\n",
      "\n",
      "Done with all years in set. Wed Sep 13 14:15:24 2023\n"
     ]
    }
   ],
   "source": [
    "# Bounded = gpd.read_file(r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Bounded')\n",
    "\n",
    "for item in WSFE_Years:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    YearSet = Bounded[Bounded['year'].between(WSFE_start, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    YearDissolve = YearSet.dissolve(by='Bounded_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\", \"Sett_ID\":\"min\"}, # Though ADM_ID and Sett_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    YearName = ''.join(['Cu', str(item), '_Bounded'])\n",
    "    YearDissolve.to_file(driver='GPKG', filename=os.path.join(Results,'CumulativeSettlements.gpkg'), layer=YearName)\n",
    "    del YearSet, YearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "6385c0d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "SettAreas = gpd.read_file(os.path.join(Results,'CumulativeSettlements.gpkg'), \n",
    "                          layer=''.join(['Cu', str(WSFE_end), '_Bounded']))\n",
    "SettAreas['AREA'+str(WSFE_end)] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "49eaeaeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2020. Wed Sep 13 14:15:28 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:15:34 2023\n",
      "\n",
      "Merging variables from 2020 onto our latest year (2021) via table join. Wed Sep 13 14:15:34 2023\n",
      "\n",
      "Loading cumulative layer for year 2019. Wed Sep 13 14:15:34 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:15:40 2023\n",
      "\n",
      "Merging variables from 2019 onto our latest year (2021) via table join. Wed Sep 13 14:15:40 2023\n",
      "\n",
      "Loading cumulative layer for year 2018. Wed Sep 13 14:15:40 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:15:45 2023\n",
      "\n",
      "Merging variables from 2018 onto our latest year (2021) via table join. Wed Sep 13 14:15:45 2023\n",
      "\n",
      "Loading cumulative layer for year 2017. Wed Sep 13 14:15:45 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:15:51 2023\n",
      "\n",
      "Merging variables from 2017 onto our latest year (2021) via table join. Wed Sep 13 14:15:51 2023\n",
      "\n",
      "Loading cumulative layer for year 2016. Wed Sep 13 14:15:51 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:15:57 2023\n",
      "\n",
      "Merging variables from 2016 onto our latest year (2021) via table join. Wed Sep 13 14:15:57 2023\n",
      "\n",
      "Loading cumulative layer for year 2015. Wed Sep 13 14:15:57 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:03 2023\n",
      "\n",
      "Merging variables from 2015 onto our latest year (2021) via table join. Wed Sep 13 14:16:03 2023\n",
      "\n",
      "Loading cumulative layer for year 2014. Wed Sep 13 14:16:03 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:09 2023\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2021) via table join. Wed Sep 13 14:16:09 2023\n",
      "\n",
      "Loading cumulative layer for year 2013. Wed Sep 13 14:16:09 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:14 2023\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2021) via table join. Wed Sep 13 14:16:14 2023\n",
      "\n",
      "Loading cumulative layer for year 2012. Wed Sep 13 14:16:14 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:19 2023\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2021) via table join. Wed Sep 13 14:16:19 2023\n",
      "\n",
      "Loading cumulative layer for year 2011. Wed Sep 13 14:16:19 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:24 2023\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2021) via table join. Wed Sep 13 14:16:24 2023\n",
      "\n",
      "Loading cumulative layer for year 2010. Wed Sep 13 14:16:24 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:28 2023\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2021) via table join. Wed Sep 13 14:16:28 2023\n",
      "\n",
      "Loading cumulative layer for year 2009. Wed Sep 13 14:16:28 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:30 2023\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2021) via table join. Wed Sep 13 14:16:30 2023\n",
      "\n",
      "Loading cumulative layer for year 2008. Wed Sep 13 14:16:30 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:32 2023\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2021) via table join. Wed Sep 13 14:16:32 2023\n",
      "\n",
      "Loading cumulative layer for year 2007. Wed Sep 13 14:16:32 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:34 2023\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2021) via table join. Wed Sep 13 14:16:34 2023\n",
      "\n",
      "Loading cumulative layer for year 2006. Wed Sep 13 14:16:34 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:35 2023\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2021) via table join. Wed Sep 13 14:16:35 2023\n",
      "\n",
      "Loading cumulative layer for year 2005. Wed Sep 13 14:16:35 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:37 2023\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2021) via table join. Wed Sep 13 14:16:37 2023\n",
      "\n",
      "Loading cumulative layer for year 2004. Wed Sep 13 14:16:37 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:39 2023\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2021) via table join. Wed Sep 13 14:16:39 2023\n",
      "\n",
      "Loading cumulative layer for year 2003. Wed Sep 13 14:16:39 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:40 2023\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2021) via table join. Wed Sep 13 14:16:40 2023\n",
      "\n",
      "Loading cumulative layer for year 2002. Wed Sep 13 14:16:41 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:42 2023\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2021) via table join. Wed Sep 13 14:16:42 2023\n",
      "\n",
      "Loading cumulative layer for year 2001. Wed Sep 13 14:16:42 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:44 2023\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2021) via table join. Wed Sep 13 14:16:44 2023\n",
      "\n",
      "Loading cumulative layer for year 2000. Wed Sep 13 14:16:44 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:46 2023\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2021) via table join. Wed Sep 13 14:16:46 2023\n",
      "\n",
      "Loading cumulative layer for year 1999. Wed Sep 13 14:16:46 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:47 2023\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2021) via table join. Wed Sep 13 14:16:47 2023\n",
      "\n",
      "Loading cumulative layer for year 1998. Wed Sep 13 14:16:47 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:49 2023\n",
      "\n",
      "Merging variables from 1998 onto our latest year (2021) via table join. Wed Sep 13 14:16:49 2023\n",
      "\n",
      "Loading cumulative layer for year 1997. Wed Sep 13 14:16:49 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:51 2023\n",
      "\n",
      "Merging variables from 1997 onto our latest year (2021) via table join. Wed Sep 13 14:16:51 2023\n",
      "\n",
      "Loading cumulative layer for year 1996. Wed Sep 13 14:16:51 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:53 2023\n",
      "\n",
      "Merging variables from 1996 onto our latest year (2021) via table join. Wed Sep 13 14:16:53 2023\n",
      "\n",
      "Loading cumulative layer for year 1995. Wed Sep 13 14:16:53 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:55 2023\n",
      "\n",
      "Merging variables from 1995 onto our latest year (2021) via table join. Wed Sep 13 14:16:55 2023\n",
      "\n",
      "Loading cumulative layer for year 1994. Wed Sep 13 14:16:55 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:56 2023\n",
      "\n",
      "Merging variables from 1994 onto our latest year (2021) via table join. Wed Sep 13 14:16:56 2023\n",
      "\n",
      "Loading cumulative layer for year 1993. Wed Sep 13 14:16:56 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:16:58 2023\n",
      "\n",
      "Merging variables from 1993 onto our latest year (2021) via table join. Wed Sep 13 14:16:58 2023\n",
      "\n",
      "Loading cumulative layer for year 1992. Wed Sep 13 14:16:58 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:00 2023\n",
      "\n",
      "Merging variables from 1992 onto our latest year (2021) via table join. Wed Sep 13 14:17:00 2023\n",
      "\n",
      "Loading cumulative layer for year 1991. Wed Sep 13 14:17:00 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:02 2023\n",
      "\n",
      "Merging variables from 1991 onto our latest year (2021) via table join. Wed Sep 13 14:17:02 2023\n",
      "\n",
      "Loading cumulative layer for year 1990. Wed Sep 13 14:17:02 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:04 2023\n",
      "\n",
      "Merging variables from 1990 onto our latest year (2021) via table join. Wed Sep 13 14:17:04 2023\n",
      "\n",
      "Loading cumulative layer for year 1989. Wed Sep 13 14:17:04 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:05 2023\n",
      "\n",
      "Merging variables from 1989 onto our latest year (2021) via table join. Wed Sep 13 14:17:05 2023\n",
      "\n",
      "Loading cumulative layer for year 1988. Wed Sep 13 14:17:05 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:07 2023\n",
      "\n",
      "Merging variables from 1988 onto our latest year (2021) via table join. Wed Sep 13 14:17:07 2023\n",
      "\n",
      "Loading cumulative layer for year 1987. Wed Sep 13 14:17:07 2023\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:08 2023\n",
      "\n",
      "Merging variables from 1987 onto our latest year (2021) via table join. Wed Sep 13 14:17:08 2023\n",
      "\n",
      "Loading cumulative layer for year 1986. Wed Sep 13 14:17:08 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:10 2023\n",
      "\n",
      "Merging variables from 1986 onto our latest year (2021) via table join. Wed Sep 13 14:17:10 2023\n",
      "\n",
      "Loading cumulative layer for year 1985. Wed Sep 13 14:17:10 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Wed Sep 13 14:17:11 2023\n",
      "\n",
      "Merging variables from 1985 onto our latest year (2021) via table join. Wed Sep 13 14:17:11 2023\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Wed Sep 13 14:17:11 2023\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 19095 entries, 0 to 19094\n",
      "Data columns (total 41 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Bounded_ID  19095 non-null  int64  \n",
      " 1   year        19095 non-null  int64  \n",
      " 2   ADM_ID      19095 non-null  int64  \n",
      " 3   Sett_ID     19095 non-null  int64  \n",
      " 4   AREA2021    19095 non-null  float64\n",
      " 5   AREA2020    18331 non-null  float64\n",
      " 6   AREA2019    17970 non-null  float64\n",
      " 7   AREA2018    17556 non-null  float64\n",
      " 8   AREA2017    17340 non-null  float64\n",
      " 9   AREA2016    17122 non-null  float64\n",
      " 10  AREA2015    16968 non-null  float64\n",
      " 11  AREA2014    16420 non-null  float64\n",
      " 12  AREA2013    14576 non-null  float64\n",
      " 13  AREA2012    13957 non-null  float64\n",
      " 14  AREA2011    11829 non-null  float64\n",
      " 15  AREA2010    11802 non-null  float64\n",
      " 16  AREA2009    4926 non-null   float64\n",
      " 17  AREA2008    4915 non-null   float64\n",
      " 18  AREA2007    4910 non-null   float64\n",
      " 19  AREA2006    4901 non-null   float64\n",
      " 20  AREA2005    4894 non-null   float64\n",
      " 21  AREA2004    4886 non-null   float64\n",
      " 22  AREA2003    4870 non-null   float64\n",
      " 23  AREA2002    4860 non-null   float64\n",
      " 24  AREA2001    4851 non-null   float64\n",
      " 25  AREA2000    4841 non-null   float64\n",
      " 26  AREA1999    4228 non-null   float64\n",
      " 27  AREA1998    4211 non-null   float64\n",
      " 28  AREA1997    4184 non-null   float64\n",
      " 29  AREA1996    4170 non-null   float64\n",
      " 30  AREA1995    4101 non-null   float64\n",
      " 31  AREA1994    4059 non-null   float64\n",
      " 32  AREA1993    4035 non-null   float64\n",
      " 33  AREA1992    4013 non-null   float64\n",
      " 34  AREA1991    4000 non-null   float64\n",
      " 35  AREA1990    3982 non-null   float64\n",
      " 36  AREA1989    2614 non-null   float64\n",
      " 37  AREA1988    2591 non-null   float64\n",
      " 38  AREA1987    2547 non-null   float64\n",
      " 39  AREA1986    2476 non-null   float64\n",
      " 40  AREA1985    2232 non-null   float64\n",
      "dtypes: float64(37), int64(4)\n",
      "memory usage: 6.1 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "for item in Reversed_WSFE_Years:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(os.path.join(Results,'CumulativeSettlements.gpkg'), \n",
    "                              layer=''.join(['Cu', str(item), '_Bounded']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['AREA', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Bounded_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, WSFE_end, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Bounded_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(Results, 'Area_Bounded.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "e7574cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "447ecaf8",
   "metadata": {},
   "source": [
    "### 5.4 One settlement geofile to rule them all. ...and in the Sett_ID bind them.\n",
    "The annualized values can be stored as distinct non-spatial dataframes. Their Sett_IDs will be used to join onto this geoversion with place names for the summary stats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "80315077",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   18552 non-null  int64   \n",
      " 1   ADM_ID    18552 non-null  int64   \n",
      " 2   geometry  18552 non-null  geometry\n",
      "dtypes: geometry(1), int64(2)\n",
      "memory usage: 434.9 KB\n",
      "None\n",
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]]\n"
     ]
    }
   ],
   "source": [
    "Settlements = gpd.read_file(os.path.join(Results,'CumulativeSettlements.gpkg'), \n",
    "                           layer=''.join(['Cu', str(WSFE_end), '_Boundless']))[['Sett_ID', 'ADM_ID', 'geometry']]\n",
    "print(Settlements.info())\n",
    "print(Settlements.crs)\n",
    "Settlements.to_file(driver='GPKG', \n",
    "                       filename=os.path.join(Results,'SETTLEMENTS.gpkg'), \n",
    "                       layer='SETTLEMENTS_equalarea')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "d12c8f85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   18552 non-null  int64   \n",
      " 1   ADM_ID    18552 non-null  int64   \n",
      " 2   geometry  18552 non-null  geometry\n",
      "dtypes: geometry(1), int64(2)\n",
      "memory usage: 434.9 KB\n",
      "None\n",
      "epsg:4326\n"
     ]
    }
   ],
   "source": [
    "# Saving all the final products as WGS84.\n",
    "Settlements_WGS = Settlements.to_crs(4326) \n",
    "print(Settlements_WGS.info())\n",
    "print(Settlements_WGS.crs)\n",
    "Settlements_WGS.to_file(driver='GPKG', \n",
    "                       filename=os.path.join(Results,'SETTLEMENTS.gpkg'), \n",
    "                       layer='SETTLEMENTS')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb8856d7",
   "metadata": {},
   "source": [
    "### 5.5 Buffer the area of the Boundless dataset's latest year to mask raster data in later sections.\n",
    "The Bounded dataset would also be fine for our purposes here. The buffer is dissolved to a single feature to be used for its total extents, which are identical between Bounded & Boundless datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "e8bb2910",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Wed Sep 13 16:03:28 2023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\miniconda3\\envs\\geo\\lib\\site-packages\\geopandas\\geodataframe.py:1472: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  super().__setitem__(key, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished buffer layer creation. Wed Sep 13 18:34:08 2023\n",
      "Saved to file. Wed Sep 13 18:34:14 2023\n"
     ]
    }
   ],
   "source": [
    "# Create buffer layer(s) to use as maximum distance for Near joins.\n",
    "\n",
    "# Population buffer: 2km\n",
    "Distance = 2000 # The Africa Albers projection is in meters. Saving in this projection to use in later sections.\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = Settlements[['Sett_ID', 'geometry']]\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "\n",
    "BufferFileName1 = ''.join(['Buff', str(Distance), 'm_', str(WSFE_end)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=os.path.join(Intermediate,'Catchment.gpkg'), layer=BufferFileName1)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "f1556d01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Wed Sep 13 18:34:14 2023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\miniconda3\\envs\\geo\\lib\\site-packages\\geopandas\\geodataframe.py:1472: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  super().__setitem__(key, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished buffer layer creation. Wed Sep 13 18:39:20 2023\n",
      "Saved to file. Wed Sep 13 18:39:28 2023\n"
     ]
    }
   ],
   "source": [
    "# Nighttime Lights buffer: 250m\n",
    "Distance = 250\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = Settlements[['Sett_ID', 'geometry']]\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "\n",
    "BufferFileName2 = ''.join(['Buff', str(Distance), 'm_', str(WSFE_end)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=os.path.join(Intermediate,'Catchment.gpkg'), layer=BufferFileName2)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbef5b7f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e33ce84a",
   "metadata": {},
   "source": [
    "## 6. PLACE NAMES\n",
    "Join urban place names from UCDB, Africapolis, and GeoNames onto the settlement vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b456bf69",
   "metadata": {},
   "source": [
    "### 6.1 Load placename datasets, filter, and project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "49d66193",
   "metadata": {},
   "outputs": [],
   "source": [
    "# WSFE_end = 2021\n",
    "# WSFE_start = 1985\n",
    "# WSFE_Years = ListFromRange(WSFE_start, WSFE_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "612f815f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   18552 non-null  int64   \n",
      " 1   ADM_ID    18552 non-null  int64   \n",
      " 2   geometry  18552 non-null  geometry\n",
      " 3   AREA2021  18552 non-null  float64 \n",
      "dtypes: float64(1), geometry(1), int64(2)\n",
      "memory usage: 579.9 KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13135 entries, 0 to 13134\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   UCDB_Name  13135 non-null  object  \n",
      " 1   geometry   13135 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 205.4+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 7720 entries, 0 to 7719\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   Afpl_Name  7720 non-null   object  \n",
      " 1   geometry   7720 non-null   geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 120.8+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 199390 entries, 0 to 199389\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   GeoName   199390 non-null  object  \n",
      " 1   geometry  199390 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 3.0+ MB\n",
      "None None None None\n"
     ]
    }
   ],
   "source": [
    "# Anytime we use a spatial join or work with area, \n",
    "# my preference is to keep it in a planar, equal area, meters projection. So we'll load as the Africa Albers.\n",
    "Settlements = gpd.read_file(os.path.join(Results, 'SETTLEMENTS.gpkg'), layer='SETTLEMENTS_equalarea')\n",
    "Settlements['AREA'+str(WSFE_end)] = Settlements['geometry'].area / 10**6\n",
    "\n",
    "# Load, pull name field, rename, and reproject to match the catchments CRS.\n",
    "UCDB = gpd.read_file(os.path.join(Source, 'PlaceName', 'GHS_STAT_UCDB2015MT_GLOBE_R2019A_V1_2.gpkg'), \n",
    "                     layer=0)[['UC_NM_MN', 'geometry']].rename(\n",
    "    columns={\"UC_NM_MN\": \"UCDB_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "Africapolis = gpd.read_file(os.path.join(Source, 'PlaceName', 'AFRICAPOLIS2020.shp'))[['agglosName', 'geometry']].rename(\n",
    "    columns={\"agglosName\": \"Afpl_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "GeoNames = gpd.read_file(os.path.join(Source, 'PlaceName', 'GeoNames.gpkg'), \n",
    "                         layer=0)[['GeoName', 'geometry']].to_crs(\"ESRI:102022\")\n",
    "\n",
    "print(Settlements.info(), UCDB.info(), Africapolis.info(), GeoNames.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e934567",
   "metadata": {},
   "source": [
    "### 6.2 Join placenames onto settlements geodataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "106f2719",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We wrap it in pd.DataFrame() since the sjoin() is the last time we need the geometry.\n",
    "\n",
    "GeoNames = pd.DataFrame(gpd.sjoin_nearest(GeoNames, Settlements, \n",
    "                             how='left', distance_col=\"distGN\", max_distance=250, \n",
    "                             lsuffix=\"G3\", rsuffix=\"GN\")).drop(columns='geometry')\n",
    "Africapolis = pd.DataFrame(gpd.sjoin_nearest(Africapolis, Settlements, \n",
    "                             how='left', distance_col=\"distAF\", max_distance=250,\n",
    "                             lsuffix=\"G3\", rsuffix=\"Af\")).drop(columns='geometry')\n",
    "UCDB = pd.DataFrame(gpd.sjoin_nearest(UCDB, Settlements, \n",
    "                             how='left', distance_col=\"distUC\", max_distance=250,\n",
    "                             lsuffix=\"G3\", rsuffix=\"UC\")).drop(columns='geometry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "2e2f0ee0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 199390 entries, 0 to 199389\n",
      "Data columns (total 6 columns):\n",
      " #   Column    Non-Null Count   Dtype  \n",
      "---  ------    --------------   -----  \n",
      " 0   GeoName   199390 non-null  object \n",
      " 1   index_GN  277 non-null     float64\n",
      " 2   Sett_ID   277 non-null     float64\n",
      " 3   ADM_ID    277 non-null     float64\n",
      " 4   AREA2021  277 non-null     float64\n",
      " 5   distGN    277 non-null     float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 10.6+ MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 7720 entries, 0 to 7719\n",
      "Data columns (total 6 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Afpl_Name  7720 non-null   object \n",
      " 1   index_Af   0 non-null      float64\n",
      " 2   Sett_ID    0 non-null      float64\n",
      " 3   ADM_ID     0 non-null      float64\n",
      " 4   AREA2021   0 non-null      float64\n",
      " 5   distAF     0 non-null      float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 422.2+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13890 entries, 0 to 13134\n",
      "Data columns (total 6 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   UCDB_Name  13890 non-null  object \n",
      " 1   index_UC   781 non-null    float64\n",
      " 2   Sett_ID    781 non-null    float64\n",
      " 3   ADM_ID     781 non-null    float64\n",
      " 4   AREA2021   781 non-null    float64\n",
      " 5   distUC     781 non-null    float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 759.6+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(GeoNames.info())\n",
    "print(Africapolis.info())\n",
    "print(UCDB.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "96e1a75a",
   "metadata": {},
   "outputs": [],
   "source": [
    "alldatasets = [pd.DataFrame(Settlements).drop(columns='geometry'),\n",
    "               Africapolis[['Sett_ID', 'Afpl_Name', 'distAF']], \n",
    "               GeoNames[['Sett_ID', 'GeoName', 'distGN']],\n",
    "               UCDB[['Sett_ID', 'UCDB_Name', 'distUC']]]\n",
    "\n",
    "SettlementsNamed = reduce(lambda left,right: pd.merge(left,right,on=['Sett_ID'], how='left'), alldatasets)\n",
    "SettlementsNamed[['Afpl_Name', 'GeoName', 'UCDB_Name']] = SettlementsNamed[['Afpl_Name', 'GeoName', 'UCDB_Name']].fillna('UNK')\n",
    "\n",
    "# Replace NaN values with a countable distance.\n",
    "SettlementsNamed[['distAF', 'distGN', 'distUC']] = SettlementsNamed[['distAF', 'distGN', 'distUC']].fillna(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "3af429c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18575 entries, 0 to 18574\n",
      "Data columns (total 9 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    18575 non-null  int64  \n",
      " 1   ADM_ID     18575 non-null  int64  \n",
      " 2   AREA2021   18575 non-null  float64\n",
      " 3   Afpl_Name  18575 non-null  object \n",
      " 4   distAF     18575 non-null  float64\n",
      " 5   GeoName    18575 non-null  object \n",
      " 6   distGN     18575 non-null  float64\n",
      " 7   UCDB_Name  18575 non-null  object \n",
      " 8   distUC     18575 non-null  float64\n",
      "dtypes: float64(4), int64(2), object(3)\n",
      "memory usage: 1.4+ MB\n",
      "None\n",
      "       Sett_ID  ADM_ID  AREA2021 Afpl_Name  distAF GeoName  distGN UCDB_Name  \\\n",
      "15937   135240     141  0.034431       UNK    -1.0     UNK    -1.0       UNK   \n",
      "15848   133939      26  0.032136       UNK    -1.0     UNK    -1.0       UNK   \n",
      "5111     21703     212  0.000765       UNK    -1.0     UNK    -1.0       UNK   \n",
      "6241     29106      46  0.002295       UNK    -1.0     UNK    -1.0       UNK   \n",
      "17269   153255     193  0.242550       UNK    -1.0     UNK    -1.0       UNK   \n",
      "3047     11082      61  0.369563       UNK    -1.0     UNK    -1.0  Damascus   \n",
      "8156     40936      45  0.035196       UNK    -1.0     UNK    -1.0       UNK   \n",
      "10690    57272     175  0.004591       UNK    -1.0     UNK    -1.0       UNK   \n",
      "10902    58382      99  0.002295       UNK    -1.0     UNK    -1.0       UNK   \n",
      "7575     37442     118  0.021424       UNK    -1.0     UNK    -1.0       UNK   \n",
      "\n",
      "       distUC  \n",
      "15937    -1.0  \n",
      "15848    -1.0  \n",
      "5111     -1.0  \n",
      "6241     -1.0  \n",
      "17269    -1.0  \n",
      "3047      0.0  \n",
      "8156     -1.0  \n",
      "10690    -1.0  \n",
      "10902    -1.0  \n",
      "7575     -1.0  \n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info())\n",
    "print(SettlementsNamed.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8f92fab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "del UCDB, Africapolis, GeoNames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2fc7cd",
   "metadata": {},
   "source": [
    "The near joins should have prevented duplication of rows, but if df1 intersects with two features in df2, it creates a new row. Two of our placenames sources are polygons, so there may be instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "4f28f065",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3205</th>\n",
       "      <td>11822</td>\n",
       "      <td>44</td>\n",
       "      <td>18.081043</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Maḑāyā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3206</th>\n",
       "      <td>11822</td>\n",
       "      <td>44</td>\n",
       "      <td>18.081043</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Az Zabadānī</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4862</th>\n",
       "      <td>20308</td>\n",
       "      <td>50</td>\n",
       "      <td>9.228363</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Baniyas</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4863</th>\n",
       "      <td>20308</td>\n",
       "      <td>50</td>\n",
       "      <td>9.228363</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Tartus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5042</th>\n",
       "      <td>21342</td>\n",
       "      <td>210</td>\n",
       "      <td>3.762963</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Şāfītā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5043</th>\n",
       "      <td>21342</td>\n",
       "      <td>210</td>\n",
       "      <td>3.762963</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Ra’s al Khashūfah</td>\n",
       "      <td>37.970870</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5471</th>\n",
       "      <td>24050</td>\n",
       "      <td>104</td>\n",
       "      <td>27.997268</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5472</th>\n",
       "      <td>24050</td>\n",
       "      <td>104</td>\n",
       "      <td>27.997268</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Hinādī</td>\n",
       "      <td>14.481241</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5866</th>\n",
       "      <td>26727</td>\n",
       "      <td>50</td>\n",
       "      <td>11.317198</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Quţaylibīyah</td>\n",
       "      <td>60.441147</td>\n",
       "      <td>Jablah</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5867</th>\n",
       "      <td>26727</td>\n",
       "      <td>50</td>\n",
       "      <td>11.317198</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Jablah</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Jablah</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7747</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7748</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Kafr Baţnā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7749</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Jaramānā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7750</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>‘Irbīn</td>\n",
       "      <td>21.321031</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7751</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Ḩarastā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7752</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Dūmā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7753</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7754</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Dayr al ‘Aşāfīr</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7755</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Dārayyā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7756</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Babīlā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7757</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>An Nashābīyah</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7758</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Mulayḩah</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7759</th>\n",
       "      <td>38534</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Ḩajar al Aswad</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8543</th>\n",
       "      <td>43444</td>\n",
       "      <td>23</td>\n",
       "      <td>14.768748</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Yabrūd</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8544</th>\n",
       "      <td>43444</td>\n",
       "      <td>23</td>\n",
       "      <td>14.768748</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>An Nabk</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10801</th>\n",
       "      <td>57837</td>\n",
       "      <td>27</td>\n",
       "      <td>11.605656</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Tallbīsah</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10802</th>\n",
       "      <td>57837</td>\n",
       "      <td>27</td>\n",
       "      <td>11.605656</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Ar Rastan</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11068</th>\n",
       "      <td>59420</td>\n",
       "      <td>107</td>\n",
       "      <td>4.518157</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Talldaww</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11069</th>\n",
       "      <td>59420</td>\n",
       "      <td>107</td>\n",
       "      <td>4.518157</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Kafr Lāhā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12266</th>\n",
       "      <td>75655</td>\n",
       "      <td>43</td>\n",
       "      <td>47.433375</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12267</th>\n",
       "      <td>75655</td>\n",
       "      <td>43</td>\n",
       "      <td>47.433375</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Ad Dānā</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12775</th>\n",
       "      <td>81242</td>\n",
       "      <td>158</td>\n",
       "      <td>0.797277</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Ma‘baţlī</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12776</th>\n",
       "      <td>81242</td>\n",
       "      <td>158</td>\n",
       "      <td>0.797277</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Qenterê</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13517</th>\n",
       "      <td>96393</td>\n",
       "      <td>110</td>\n",
       "      <td>115.537029</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>80.444231</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13518</th>\n",
       "      <td>96393</td>\n",
       "      <td>110</td>\n",
       "      <td>115.537029</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID    AREA2021 Afpl_Name  distAF            GeoName  \\\n",
       "3205     11822      44   18.081043       UNK    -1.0             Maḑāyā   \n",
       "3206     11822      44   18.081043       UNK    -1.0        Az Zabadānī   \n",
       "4862     20308      50    9.228363       UNK    -1.0                UNK   \n",
       "4863     20308      50    9.228363       UNK    -1.0                UNK   \n",
       "5042     21342     210    3.762963       UNK    -1.0             Şāfītā   \n",
       "5043     21342     210    3.762963       UNK    -1.0  Ra’s al Khashūfah   \n",
       "5471     24050     104   27.997268       UNK    -1.0            Latakia   \n",
       "5472     24050     104   27.997268       UNK    -1.0          Al Hinādī   \n",
       "5866     26727      50   11.317198       UNK    -1.0    Al Quţaylibīyah   \n",
       "5867     26727      50   11.317198       UNK    -1.0             Jablah   \n",
       "7747     38534      28  281.043874       UNK    -1.0            Şaḩnāyā   \n",
       "7748     38534      28  281.043874       UNK    -1.0         Kafr Baţnā   \n",
       "7749     38534      28  281.043874       UNK    -1.0           Jaramānā   \n",
       "7750     38534      28  281.043874       UNK    -1.0             ‘Irbīn   \n",
       "7751     38534      28  281.043874       UNK    -1.0            Ḩarastā   \n",
       "7752     38534      28  281.043874       UNK    -1.0               Dūmā   \n",
       "7753     38534      28  281.043874       UNK    -1.0           Damascus   \n",
       "7754     38534      28  281.043874       UNK    -1.0    Dayr al ‘Aşāfīr   \n",
       "7755     38534      28  281.043874       UNK    -1.0            Dārayyā   \n",
       "7756     38534      28  281.043874       UNK    -1.0             Babīlā   \n",
       "7757     38534      28  281.043874       UNK    -1.0      An Nashābīyah   \n",
       "7758     38534      28  281.043874       UNK    -1.0        Al Mulayḩah   \n",
       "7759     38534      28  281.043874       UNK    -1.0  Al Ḩajar al Aswad   \n",
       "8543     43444      23   14.768748       UNK    -1.0             Yabrūd   \n",
       "8544     43444      23   14.768748       UNK    -1.0            An Nabk   \n",
       "10801    57837      27   11.605656       UNK    -1.0          Tallbīsah   \n",
       "10802    57837      27   11.605656       UNK    -1.0          Ar Rastan   \n",
       "11068    59420     107    4.518157       UNK    -1.0           Talldaww   \n",
       "11069    59420     107    4.518157       UNK    -1.0          Kafr Lāhā   \n",
       "12266    75655      43   47.433375       UNK    -1.0             Batabo   \n",
       "12267    75655      43   47.433375       UNK    -1.0            Ad Dānā   \n",
       "12775    81242     158    0.797277       UNK    -1.0        Al Ma‘baţlī   \n",
       "12776    81242     158    0.797277       UNK    -1.0            Qenterê   \n",
       "13517    96393     110  115.537029       UNK    -1.0           Ḩuraytān   \n",
       "13518    96393     110  115.537029       UNK    -1.0             Aleppo   \n",
       "\n",
       "          distGN UCDB_Name  distUC  \n",
       "3205    0.000000       UNK    -1.0  \n",
       "3206    0.000000       UNK    -1.0  \n",
       "4862   -1.000000   Baniyas     0.0  \n",
       "4863   -1.000000    Tartus     0.0  \n",
       "5042    0.000000       UNK    -1.0  \n",
       "5043   37.970870       UNK    -1.0  \n",
       "5471    0.000000   Latakia     0.0  \n",
       "5472   14.481241   Latakia     0.0  \n",
       "5866   60.441147    Jablah     0.0  \n",
       "5867    0.000000    Jablah     0.0  \n",
       "7747    0.000000  Damascus     0.0  \n",
       "7748    0.000000  Damascus     0.0  \n",
       "7749    0.000000  Damascus     0.0  \n",
       "7750   21.321031  Damascus     0.0  \n",
       "7751    0.000000  Damascus     0.0  \n",
       "7752    0.000000  Damascus     0.0  \n",
       "7753    0.000000  Damascus     0.0  \n",
       "7754    0.000000  Damascus     0.0  \n",
       "7755    0.000000  Damascus     0.0  \n",
       "7756    0.000000  Damascus     0.0  \n",
       "7757    0.000000  Damascus     0.0  \n",
       "7758    0.000000  Damascus     0.0  \n",
       "7759    0.000000  Damascus     0.0  \n",
       "8543    0.000000       UNK    -1.0  \n",
       "8544    0.000000       UNK    -1.0  \n",
       "10801   0.000000       UNK    -1.0  \n",
       "10802   0.000000       UNK    -1.0  \n",
       "11068   0.000000       UNK    -1.0  \n",
       "11069   0.000000       UNK    -1.0  \n",
       "12266   0.000000       UNK    -1.0  \n",
       "12267   0.000000       UNK    -1.0  \n",
       "12775   0.000000       UNK    -1.0  \n",
       "12776   0.000000       UNK    -1.0  \n",
       "13517  80.444231    Aleppo     0.0  \n",
       "13518   0.000000    Aleppo     0.0  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed[SettlementsNamed.duplicated('Sett_ID', keep=False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "42d1b4b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18574\n",
      "Data columns (total 9 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    18552 non-null  int64  \n",
      " 1   ADM_ID     18552 non-null  int64  \n",
      " 2   AREA2021   18552 non-null  float64\n",
      " 3   Afpl_Name  18552 non-null  object \n",
      " 4   distAF     18552 non-null  float64\n",
      " 5   GeoName    18552 non-null  object \n",
      " 6   distGN     18552 non-null  float64\n",
      " 7   UCDB_Name  18552 non-null  object \n",
      " 8   distUC     18552 non-null  float64\n",
      "dtypes: float64(4), int64(2), object(3)\n",
      "memory usage: 1.4+ MB\n"
     ]
    }
   ],
   "source": [
    "SettlementsNamed.drop_duplicates(subset=['Sett_ID'], inplace=True, keep='first')\n",
    "SettlementsNamed.info() # Range of entries should be the same as original Settlements file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a16201",
   "metadata": {},
   "source": [
    "### 6.3 Reduce to single name column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ad4c8be2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine which source has a name geometrically closest to the settlement.\n",
    "# Since we switched NaN values to -1 earlier, we also resolved what happens in the event of a tie, \n",
    "# i.e. when more than one source is 0.0 meters from the settlement. It will take the value from the first column.\n",
    "SettlementsNamed['SettName'] = \"UNK\"\n",
    "SettlementsNamed['closest'] = SettlementsNamed[['distAF', 'distGN', 'distUC']].idxmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f6551664",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "      <th>SettName</th>\n",
       "      <th>closest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9793</th>\n",
       "      <td>52034</td>\n",
       "      <td>16</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3459</th>\n",
       "      <td>12905</td>\n",
       "      <td>197</td>\n",
       "      <td>0.045143</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7436</th>\n",
       "      <td>36432</td>\n",
       "      <td>176</td>\n",
       "      <td>0.160680</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12097</th>\n",
       "      <td>73868</td>\n",
       "      <td>62</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12953</th>\n",
       "      <td>83313</td>\n",
       "      <td>121</td>\n",
       "      <td>0.090287</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5434</th>\n",
       "      <td>23785</td>\n",
       "      <td>204</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13892</th>\n",
       "      <td>100934</td>\n",
       "      <td>129</td>\n",
       "      <td>0.005356</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4846</th>\n",
       "      <td>20219</td>\n",
       "      <td>268</td>\n",
       "      <td>0.003826</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12049</th>\n",
       "      <td>73382</td>\n",
       "      <td>223</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8949</th>\n",
       "      <td>47325</td>\n",
       "      <td>115</td>\n",
       "      <td>0.009947</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5402</th>\n",
       "      <td>23528</td>\n",
       "      <td>15</td>\n",
       "      <td>0.006886</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11899</th>\n",
       "      <td>71811</td>\n",
       "      <td>161</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1692</th>\n",
       "      <td>6256</td>\n",
       "      <td>13</td>\n",
       "      <td>0.022954</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12976</th>\n",
       "      <td>83547</td>\n",
       "      <td>206</td>\n",
       "      <td>0.124718</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13915</th>\n",
       "      <td>101207</td>\n",
       "      <td>110</td>\n",
       "      <td>0.181338</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16741</th>\n",
       "      <td>146887</td>\n",
       "      <td>29</td>\n",
       "      <td>0.368033</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8033</th>\n",
       "      <td>40169</td>\n",
       "      <td>93</td>\n",
       "      <td>0.058151</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3134</th>\n",
       "      <td>11497</td>\n",
       "      <td>197</td>\n",
       "      <td>0.002295</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2144</th>\n",
       "      <td>7775</td>\n",
       "      <td>13</td>\n",
       "      <td>0.003061</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12302</th>\n",
       "      <td>76085</td>\n",
       "      <td>62</td>\n",
       "      <td>0.030606</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID  AREA2021 Afpl_Name  distAF GeoName  distGN UCDB_Name  \\\n",
       "9793     52034      16  0.000765       UNK    -1.0     UNK    -1.0       UNK   \n",
       "3459     12905     197  0.045143       UNK    -1.0     UNK    -1.0       UNK   \n",
       "7436     36432     176  0.160680       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12097    73868      62  0.001530       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12953    83313     121  0.090287       UNK    -1.0     UNK    -1.0       UNK   \n",
       "5434     23785     204  0.000765       UNK    -1.0     UNK    -1.0       UNK   \n",
       "13892   100934     129  0.005356       UNK    -1.0     UNK    -1.0    Aleppo   \n",
       "4846     20219     268  0.003826       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12049    73382     223  0.001530       UNK    -1.0     UNK    -1.0       UNK   \n",
       "8949     47325     115  0.009947       UNK    -1.0     UNK    -1.0       UNK   \n",
       "5402     23528      15  0.006886       UNK    -1.0     UNK    -1.0       UNK   \n",
       "11899    71811     161  0.001530       UNK    -1.0     UNK    -1.0       UNK   \n",
       "1692      6256      13  0.022954       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12976    83547     206  0.124718       UNK    -1.0     UNK    -1.0       UNK   \n",
       "13915   101207     110  0.181338       UNK    -1.0     UNK    -1.0       UNK   \n",
       "16741   146887      29  0.368033       UNK    -1.0     UNK    -1.0       UNK   \n",
       "8033     40169      93  0.058151       UNK    -1.0     UNK    -1.0       UNK   \n",
       "3134     11497     197  0.002295       UNK    -1.0     UNK    -1.0       UNK   \n",
       "2144      7775      13  0.003061       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12302    76085      62  0.030606       UNK    -1.0     UNK    -1.0       UNK   \n",
       "\n",
       "       distUC SettName closest  \n",
       "9793     -1.0      UNK  distAF  \n",
       "3459     -1.0      UNK  distAF  \n",
       "7436     -1.0      UNK  distAF  \n",
       "12097    -1.0      UNK  distAF  \n",
       "12953    -1.0      UNK  distAF  \n",
       "5434     -1.0      UNK  distAF  \n",
       "13892     0.0      UNK  distUC  \n",
       "4846     -1.0      UNK  distAF  \n",
       "12049    -1.0      UNK  distAF  \n",
       "8949     -1.0      UNK  distAF  \n",
       "5402     -1.0      UNK  distAF  \n",
       "11899    -1.0      UNK  distAF  \n",
       "1692     -1.0      UNK  distAF  \n",
       "12976    -1.0      UNK  distAF  \n",
       "13915    -1.0      UNK  distAF  \n",
       "16741    -1.0      UNK  distAF  \n",
       "8033     -1.0      UNK  distAF  \n",
       "3134     -1.0      UNK  distAF  \n",
       "2144     -1.0      UNK  distAF  \n",
       "12302    -1.0      UNK  distAF  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "21fcb748",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single name column where non-named settlements are \"UNK\" but all others use one of the three name sources.\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distAF\", \n",
    "    'SettName'] = SettlementsNamed['Afpl_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distUC\", \n",
    "    'SettName'] = SettlementsNamed['UCDB_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distGN\", \n",
    "    'SettName'] = SettlementsNamed['GeoName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b754859a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "      <th>SettName</th>\n",
       "      <th>closest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12161</th>\n",
       "      <td>74561</td>\n",
       "      <td>223</td>\n",
       "      <td>0.035962</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12771</th>\n",
       "      <td>81223</td>\n",
       "      <td>121</td>\n",
       "      <td>0.005356</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3940</th>\n",
       "      <td>15245</td>\n",
       "      <td>101</td>\n",
       "      <td>0.890624</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Karīmah</td>\n",
       "      <td>0.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Al Karīmah</td>\n",
       "      <td>distGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1225</th>\n",
       "      <td>4613</td>\n",
       "      <td>125</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1032</th>\n",
       "      <td>3907</td>\n",
       "      <td>170</td>\n",
       "      <td>0.002295</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6865</th>\n",
       "      <td>32880</td>\n",
       "      <td>39</td>\n",
       "      <td>0.162210</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17772</th>\n",
       "      <td>155355</td>\n",
       "      <td>126</td>\n",
       "      <td>0.087226</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12356</th>\n",
       "      <td>76657</td>\n",
       "      <td>225</td>\n",
       "      <td>0.002295</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13519</th>\n",
       "      <td>96403</td>\n",
       "      <td>35</td>\n",
       "      <td>0.003061</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4386</th>\n",
       "      <td>17283</td>\n",
       "      <td>260</td>\n",
       "      <td>0.006121</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6003</th>\n",
       "      <td>27633</td>\n",
       "      <td>140</td>\n",
       "      <td>0.020659</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6702</th>\n",
       "      <td>31925</td>\n",
       "      <td>39</td>\n",
       "      <td>0.006121</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12051</th>\n",
       "      <td>73390</td>\n",
       "      <td>43</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6698</th>\n",
       "      <td>31914</td>\n",
       "      <td>39</td>\n",
       "      <td>5.896940</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10627</th>\n",
       "      <td>56870</td>\n",
       "      <td>182</td>\n",
       "      <td>0.006121</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11722</th>\n",
       "      <td>69319</td>\n",
       "      <td>143</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14981</th>\n",
       "      <td>115163</td>\n",
       "      <td>123</td>\n",
       "      <td>0.161445</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6159</th>\n",
       "      <td>28636</td>\n",
       "      <td>46</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7812</th>\n",
       "      <td>38889</td>\n",
       "      <td>70</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18197</th>\n",
       "      <td>156301</td>\n",
       "      <td>14</td>\n",
       "      <td>0.003826</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID  AREA2021 Afpl_Name  distAF     GeoName  distGN  \\\n",
       "12161    74561     223  0.035962       UNK    -1.0         UNK    -1.0   \n",
       "12771    81223     121  0.005356       UNK    -1.0         UNK    -1.0   \n",
       "3940     15245     101  0.890624       UNK    -1.0  Al Karīmah     0.0   \n",
       "1225      4613     125  0.001530       UNK    -1.0         UNK    -1.0   \n",
       "1032      3907     170  0.002295       UNK    -1.0         UNK    -1.0   \n",
       "6865     32880      39  0.162210       UNK    -1.0         UNK    -1.0   \n",
       "17772   155355     126  0.087226       UNK    -1.0         UNK    -1.0   \n",
       "12356    76657     225  0.002295       UNK    -1.0         UNK    -1.0   \n",
       "13519    96403      35  0.003061       UNK    -1.0         UNK    -1.0   \n",
       "4386     17283     260  0.006121       UNK    -1.0         UNK    -1.0   \n",
       "6003     27633     140  0.020659       UNK    -1.0         UNK    -1.0   \n",
       "6702     31925      39  0.006121       UNK    -1.0         UNK    -1.0   \n",
       "12051    73390      43  0.000765       UNK    -1.0         UNK    -1.0   \n",
       "6698     31914      39  5.896940       UNK    -1.0         UNK    -1.0   \n",
       "10627    56870     182  0.006121       UNK    -1.0         UNK    -1.0   \n",
       "11722    69319     143  0.000765       UNK    -1.0         UNK    -1.0   \n",
       "14981   115163     123  0.161445       UNK    -1.0         UNK    -1.0   \n",
       "6159     28636      46  0.000765       UNK    -1.0         UNK    -1.0   \n",
       "7812     38889      70  0.001530       UNK    -1.0         UNK    -1.0   \n",
       "18197   156301      14  0.003826       UNK    -1.0         UNK    -1.0   \n",
       "\n",
       "      UCDB_Name  distUC    SettName closest  \n",
       "12161       UNK    -1.0         UNK  distAF  \n",
       "12771       UNK    -1.0         UNK  distAF  \n",
       "3940        UNK    -1.0  Al Karīmah  distGN  \n",
       "1225        UNK    -1.0         UNK  distAF  \n",
       "1032        UNK    -1.0         UNK  distAF  \n",
       "6865        UNK    -1.0         UNK  distAF  \n",
       "17772       UNK    -1.0         UNK  distAF  \n",
       "12356       UNK    -1.0         UNK  distAF  \n",
       "13519       UNK    -1.0         UNK  distAF  \n",
       "4386        UNK    -1.0         UNK  distAF  \n",
       "6003        UNK    -1.0         UNK  distAF  \n",
       "6702        UNK    -1.0         UNK  distAF  \n",
       "12051       UNK    -1.0         UNK  distAF  \n",
       "6698        UNK    -1.0         UNK  distAF  \n",
       "10627       UNK    -1.0         UNK  distAF  \n",
       "11722       UNK    -1.0         UNK  distAF  \n",
       "14981       UNK    -1.0         UNK  distAF  \n",
       "6159        UNK    -1.0         UNK  distAF  \n",
       "7812        UNK    -1.0         UNK  distAF  \n",
       "18197       UNK    -1.0         UNK  distAF  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "80c57cc5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "      <th>SettName</th>\n",
       "      <th>closest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16802</th>\n",
       "      <td>147611</td>\n",
       "      <td>11</td>\n",
       "      <td>0.101764</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Al-Hasakah</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Al-Hasakah</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15216</th>\n",
       "      <td>121705</td>\n",
       "      <td>177</td>\n",
       "      <td>0.003826</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Manbij</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Manbij</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16080</th>\n",
       "      <td>138061</td>\n",
       "      <td>68</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Deir Ez Zor</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Deir Ez Zor</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16756</th>\n",
       "      <td>147274</td>\n",
       "      <td>11</td>\n",
       "      <td>0.229542</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Al-Hasakah</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Al-Hasakah</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16595</th>\n",
       "      <td>143516</td>\n",
       "      <td>20</td>\n",
       "      <td>1.068902</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Al Qurayya</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Al Qurayya</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2976</th>\n",
       "      <td>10766</td>\n",
       "      <td>61</td>\n",
       "      <td>0.377980</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10262</th>\n",
       "      <td>54694</td>\n",
       "      <td>182</td>\n",
       "      <td>0.656491</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Tremseh</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Tremseh</td>\n",
       "      <td>distGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1991</th>\n",
       "      <td>7271</td>\n",
       "      <td>213</td>\n",
       "      <td>4.597731</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Sa‘sa‘</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Sa‘sa‘</td>\n",
       "      <td>distGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13405</th>\n",
       "      <td>94742</td>\n",
       "      <td>129</td>\n",
       "      <td>0.013007</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16316</th>\n",
       "      <td>141741</td>\n",
       "      <td>209</td>\n",
       "      <td>0.002295</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Ceylanpınar</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Ceylanpınar</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13388</th>\n",
       "      <td>94522</td>\n",
       "      <td>129</td>\n",
       "      <td>0.462145</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8833</th>\n",
       "      <td>46701</td>\n",
       "      <td>115</td>\n",
       "      <td>0.352730</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Homs</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Homs</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6563</th>\n",
       "      <td>31063</td>\n",
       "      <td>58</td>\n",
       "      <td>7.463949</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Buşrá ash Shām</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Buşrá ash Shām</td>\n",
       "      <td>distGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13682</th>\n",
       "      <td>98411</td>\n",
       "      <td>129</td>\n",
       "      <td>0.001530</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7726</th>\n",
       "      <td>38419</td>\n",
       "      <td>153</td>\n",
       "      <td>0.019129</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9751</th>\n",
       "      <td>51800</td>\n",
       "      <td>115</td>\n",
       "      <td>0.010712</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Homs</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Homs</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9856</th>\n",
       "      <td>52398</td>\n",
       "      <td>115</td>\n",
       "      <td>0.022954</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Homs</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Homs</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16623</th>\n",
       "      <td>143567</td>\n",
       "      <td>41</td>\n",
       "      <td>0.051264</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Al Qurayya</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Al Qurayya</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3290</th>\n",
       "      <td>12179</td>\n",
       "      <td>169</td>\n",
       "      <td>0.000765</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>distUC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10729</th>\n",
       "      <td>57472</td>\n",
       "      <td>189</td>\n",
       "      <td>0.302231</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>‘Awaj</td>\n",
       "      <td>125.436638</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>‘Awaj</td>\n",
       "      <td>distGN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID  AREA2021 Afpl_Name  distAF         GeoName  \\\n",
       "16802   147611      11  0.101764       UNK    -1.0             UNK   \n",
       "15216   121705     177  0.003826       UNK    -1.0             UNK   \n",
       "16080   138061      68  0.001530       UNK    -1.0             UNK   \n",
       "16756   147274      11  0.229542       UNK    -1.0             UNK   \n",
       "16595   143516      20  1.068902       UNK    -1.0             UNK   \n",
       "2976     10766      61  0.377980       UNK    -1.0             UNK   \n",
       "10262    54694     182  0.656491       UNK    -1.0         Tremseh   \n",
       "1991      7271     213  4.597731       UNK    -1.0          Sa‘sa‘   \n",
       "13405    94742     129  0.013007       UNK    -1.0             UNK   \n",
       "16316   141741     209  0.002295       UNK    -1.0             UNK   \n",
       "13388    94522     129  0.462145       UNK    -1.0             UNK   \n",
       "8833     46701     115  0.352730       UNK    -1.0             UNK   \n",
       "6563     31063      58  7.463949       UNK    -1.0  Buşrá ash Shām   \n",
       "13682    98411     129  0.001530       UNK    -1.0             UNK   \n",
       "7726     38419     153  0.019129       UNK    -1.0             UNK   \n",
       "9751     51800     115  0.010712       UNK    -1.0             UNK   \n",
       "9856     52398     115  0.022954       UNK    -1.0             UNK   \n",
       "16623   143567      41  0.051264       UNK    -1.0             UNK   \n",
       "3290     12179     169  0.000765       UNK    -1.0             UNK   \n",
       "10729    57472     189  0.302231       UNK    -1.0           ‘Awaj   \n",
       "\n",
       "           distGN    UCDB_Name  distUC        SettName closest  \n",
       "16802   -1.000000   Al-Hasakah     0.0      Al-Hasakah  distUC  \n",
       "15216   -1.000000       Manbij     0.0          Manbij  distUC  \n",
       "16080   -1.000000  Deir Ez Zor     0.0     Deir Ez Zor  distUC  \n",
       "16756   -1.000000   Al-Hasakah     0.0      Al-Hasakah  distUC  \n",
       "16595   -1.000000   Al Qurayya     0.0      Al Qurayya  distUC  \n",
       "2976    -1.000000     Damascus     0.0        Damascus  distUC  \n",
       "10262    0.000000          UNK    -1.0         Tremseh  distGN  \n",
       "1991     0.000000          UNK    -1.0          Sa‘sa‘  distGN  \n",
       "13405   -1.000000       Aleppo     0.0          Aleppo  distUC  \n",
       "16316   -1.000000  Ceylanpınar     0.0     Ceylanpınar  distUC  \n",
       "13388   -1.000000       Aleppo     0.0          Aleppo  distUC  \n",
       "8833    -1.000000         Homs     0.0            Homs  distUC  \n",
       "6563     0.000000          UNK    -1.0  Buşrá ash Shām  distGN  \n",
       "13682   -1.000000       Aleppo     0.0          Aleppo  distUC  \n",
       "7726    -1.000000     Damascus     0.0        Damascus  distUC  \n",
       "9751    -1.000000         Homs     0.0            Homs  distUC  \n",
       "9856    -1.000000         Homs     0.0            Homs  distUC  \n",
       "16623   -1.000000   Al Qurayya     0.0      Al Qurayya  distUC  \n",
       "3290    -1.000000     Damascus     0.0        Damascus  distUC  \n",
       "10729  125.436638          UNK    -1.0           ‘Awaj  distGN  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed[SettlementsNamed['SettName'] != 'UNK'].sample(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c748e23",
   "metadata": {},
   "source": [
    "### 6.4 Make sure place name is unique by stripping smaller localities of duplicated names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "a0ec2f07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of named settlements: 1004\n",
      "Number of named settlements where name is duplicated at least once: 754\n"
     ]
    }
   ],
   "source": [
    "Dupes = SettlementsNamed[ \n",
    "    (SettlementsNamed['SettName'] != 'UNK') & \n",
    "    (SettlementsNamed.duplicated('SettName', keep=False)) ] # keep=False is necessary to retain *all* duplicates, not just first or last in each group.\n",
    "\n",
    "print(\"Number of named settlements: %s\" % SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False])\n",
    "print(\"Number of named settlements where name is duplicated at least once: %s\" % len(Dupes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "030d7d5a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Sett_ID  ADM_ID   AREA2021 Afpl_Name  distAF    GeoName  distGN  \\\n",
      "15649   130670     264   0.306056       UNK    -1.0        UNK    -1.0   \n",
      "16595   143516      20   1.068902       UNK    -1.0        UNK    -1.0   \n",
      "16772   147384      11   2.609130       UNK    -1.0        UNK    -1.0   \n",
      "15528   126749      17   1.182143       UNK    -1.0        UNK    -1.0   \n",
      "13558    96879      43  16.290613       UNK    -1.0        UNK    -1.0   \n",
      "15830   133699      26  24.873198       UNK    -1.0  Ar Raqqah     0.0   \n",
      "13390    94530      35   0.074219       UNK    -1.0        UNK    -1.0   \n",
      "7079     34022      39   0.078810       UNK    -1.0        UNK    -1.0   \n",
      "4862     20308      50   9.228363       UNK    -1.0        UNK    -1.0   \n",
      "14647   107695      18   0.035962       UNK    -1.0        UNK    -1.0   \n",
      "16352   142065     209   0.115536       UNK    -1.0        UNK    -1.0   \n",
      "2956     10657     201   1.179082       UNK    -1.0        UNK    -1.0   \n",
      "16004   137704      68   0.342783       UNK    -1.0        UNK    -1.0   \n",
      "11265    60466     182   0.226482       UNK    -1.0        UNK    -1.0   \n",
      "10567    56483      99   3.770614       UNK    -1.0        UNK    -1.0   \n",
      "9542     50649     115  61.194432       UNK    -1.0       Homs     0.0   \n",
      "12078    73662     117  15.764961       UNK    -1.0      Idlib     0.0   \n",
      "5335     23138     119   0.075749       UNK    -1.0        UNK    -1.0   \n",
      "15710   131612       8   0.049734       UNK    -1.0        UNK    -1.0   \n",
      "5471     24050     104  27.997268       UNK    -1.0    Latakia     0.0   \n",
      "14952   114572     177  14.663924       UNK    -1.0     Manbij     0.0   \n",
      "17189   152924     200   0.779678       UNK    -1.0        UNK    -1.0   \n",
      "5072     21506     260   0.328245       UNK    -1.0        UNK    -1.0   \n",
      "17629   154670     269   0.205823       UNK    -1.0        UNK    -1.0   \n",
      "10087    53670     246   4.709442       UNK    -1.0      Şūrān     0.0   \n",
      "\n",
      "         UCDB_Name  distUC     SettName closest  \n",
      "15649     Akçakale     0.0     Akçakale  distUC  \n",
      "16595   Al Qurayya     0.0   Al Qurayya  distUC  \n",
      "16772   Al-Hasakah     0.0   Al-Hasakah  distUC  \n",
      "15528    Al-Tabqah     0.0    Al-Tabqah  distUC  \n",
      "13558       Aleppo     0.0       Aleppo  distUC  \n",
      "15830    Ar Raqqah     0.0    Ar Raqqah  distGN  \n",
      "13390   As Safirah     0.0   As Safirah  distUC  \n",
      "7079    As Suwayda     0.0   As Suwayda  distUC  \n",
      "4862       Baniyas     0.0      Baniyas  distUC  \n",
      "14647        Bza`a     0.0        Bza`a  distUC  \n",
      "16352  Ceylanpınar     0.0  Ceylanpınar  distUC  \n",
      "2956      Damascus     0.0     Damascus  distUC  \n",
      "16004  Deir Ez Zor     0.0  Deir Ez Zor  distUC  \n",
      "11265      Halfaya     0.0      Halfaya  distUC  \n",
      "10567         Hama     0.0         Hama  distUC  \n",
      "9542          Homs     0.0         Homs  distGN  \n",
      "12078        Idlib     0.0        Idlib  distGN  \n",
      "5335        Jablah     0.0       Jablah  distUC  \n",
      "15710       Kobani     0.0       Kobani  distUC  \n",
      "5471       Latakia     0.0      Latakia  distGN  \n",
      "14952       Manbij     0.0       Manbij  distGN  \n",
      "17189     Qamishli     0.0     Qamishli  distUC  \n",
      "5072        Tartus     0.0       Tartus  distUC  \n",
      "17629    Yaaroubia     0.0    Yaaroubia  distUC  \n",
      "10087          UNK    -1.0        Şūrān  distGN  \n"
     ]
    }
   ],
   "source": [
    "Largest = Dupes.loc[Dupes.groupby([\"SettName\"])[\"AREA\"+str(WSFE_end)].idxmax()]\n",
    "print(Largest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "82bd26e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter to settlements which have a duplicated name and are not the largest of those with that name, then replace with UNK.\n",
    "SettlementsNamed.loc[(~SettlementsNamed.Sett_ID.isin(Largest.Sett_ID)) \n",
    "                     & (SettlementsNamed.Sett_ID.isin(Dupes.Sett_ID)), \n",
    "                     'SettName'] = 'UNK'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "28bddc71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of named settlements: 275\n",
      "Number of named settlements where name is duplicated at least once: 0\n"
     ]
    }
   ],
   "source": [
    "# Second number should now be zero.\n",
    "\n",
    "print(\"Number of named settlements: %s\" % SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False])\n",
    "print(\"Number of named settlements where name is duplicated at least once: %s\" % len(SettlementsNamed[ \n",
    "    (SettlementsNamed['SettName'] != 'UNK') & \n",
    "    (SettlementsNamed.duplicated('SettName', keep=False)) ]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "885dd231",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18574\n",
      "Data columns (total 11 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    18552 non-null  int64  \n",
      " 1   ADM_ID     18552 non-null  int64  \n",
      " 2   AREA2021   18552 non-null  float64\n",
      " 3   Afpl_Name  18552 non-null  object \n",
      " 4   distAF     18552 non-null  float64\n",
      " 5   GeoName    18552 non-null  object \n",
      " 6   distGN     18552 non-null  float64\n",
      " 7   UCDB_Name  18552 non-null  object \n",
      " 8   distUC     18552 non-null  float64\n",
      " 9   SettName   18552 non-null  object \n",
      " 10  closest    18552 non-null  object \n",
      "dtypes: float64(4), int64(2), object(5)\n",
      "memory usage: 2.2+ MB\n",
      "None        Sett_ID  ADM_ID   AREA2021 Afpl_Name  distAF          GeoName  \\\n",
      "1991      7271     213   4.597731       UNK    -1.0           Sa‘sa‘   \n",
      "15290   123287     242   0.016068       UNK    -1.0       As Sukhnah   \n",
      "16012   137724     181   0.008417       UNK    -1.0        Mūḩ Ḩasan   \n",
      "974       3721      90   2.888407       UNK    -1.0        Ghabāghib   \n",
      "2966     10708      61  55.778766       UNK    -1.0         Qadsayyā   \n",
      "11823    71046      32   1.721567       UNK    -1.0          Armanāz   \n",
      "933       3613     170   1.925859       UNK    -1.0          Buq‘ātā   \n",
      "16004   137704      68   0.342783       UNK    -1.0              UNK   \n",
      "16388   142586       3   0.813345       UNK    -1.0              UNK   \n",
      "7873     39222      21   8.272703       UNK    -1.0      Ar Ruḩaybah   \n",
      "3783     14492     226   1.246414       UNK    -1.0         Sirghāyā   \n",
      "4862     20308      50   9.228363       UNK    -1.0              UNK   \n",
      "11606    66477      38   2.545624       UNK    -1.0  As Suqaylibīyah   \n",
      "12775    81242     158   0.797277       UNK    -1.0      Al Ma‘baţlī   \n",
      "2245      8158     145   4.231229       UNK    -1.0     Khān Arnabah   \n",
      "6705     31944     171   1.839399       UNK    -1.0     Al Mushannaf   \n",
      "14576   107002     167   2.655039       UNK    -1.0            Māri‘   \n",
      "8586     44106     159   0.182104       UNK    -1.0          Ma‘lūlā   \n",
      "14337   104893      25   1.232642       UNK    -1.0          Ar Rā‘ī   \n",
      "12469    77824     263   1.283141       UNK    -1.0         Taftanāz   \n",
      "\n",
      "          distGN    UCDB_Name  distUC         SettName closest  \n",
      "1991    0.000000          UNK    -1.0           Sa‘sa‘  distGN  \n",
      "15290  34.196069          UNK    -1.0       As Sukhnah  distGN  \n",
      "16012   0.000000          UNK    -1.0        Mūḩ Ḩasan  distGN  \n",
      "974     0.000000          UNK    -1.0        Ghabāghib  distGN  \n",
      "2966    0.000000     Damascus     0.0         Qadsayyā  distGN  \n",
      "11823   0.000000          UNK    -1.0          Armanāz  distGN  \n",
      "933     0.000000          UNK    -1.0          Buq‘ātā  distGN  \n",
      "16004  -1.000000  Deir Ez Zor     0.0      Deir Ez Zor  distUC  \n",
      "16388  -1.000000    Abu Kamal     0.0        Abu Kamal  distUC  \n",
      "7873   12.500065          UNK    -1.0      Ar Ruḩaybah  distGN  \n",
      "3783    0.000000          UNK    -1.0         Sirghāyā  distGN  \n",
      "4862   -1.000000      Baniyas     0.0          Baniyas  distUC  \n",
      "11606   0.000000          UNK    -1.0  As Suqaylibīyah  distGN  \n",
      "12775   0.000000          UNK    -1.0      Al Ma‘baţlī  distGN  \n",
      "2245    0.000000          UNK    -1.0     Khān Arnabah  distGN  \n",
      "6705    0.000000          UNK    -1.0     Al Mushannaf  distGN  \n",
      "14576   0.000000          UNK    -1.0            Māri‘  distGN  \n",
      "8586   51.149179          UNK    -1.0          Ma‘lūlā  distGN  \n",
      "14337   0.000000          UNK    -1.0          Ar Rā‘ī  distGN  \n",
      "12469   0.000000          UNK    -1.0         Taftanāz  distGN  \n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info(), SettlementsNamed[SettlementsNamed['SettName'] != \"UNK\"].sample(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "0d202603",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop extra columns and save to file.\n",
    "# SettlementsNamed = SettlementsNamed[['Sett_ID', 'SettName']]\n",
    "SettlementsNamed = SettlementsNamed[['Sett_ID', 'SettName', 'GeoName', 'UCDB_Name']]\n",
    "SettlementsNamed.to_csv(r'Results/PlaceNames.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "db1e4aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettlementsNamed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c1c3d6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8d7b6d",
   "metadata": {},
   "source": [
    "## 7. CREATE FRAGMENTATION INDEX\n",
    "We are determining what percentage of a settlement's area lies outside of its administrative zone each year.\n",
    "The index is a range of 0 to 100, i.e. the percent of the settlement area which is fragmented.\n",
    "\n",
    "For each Sett_ID:\n",
    "((Area of Boundless settlement - Area of largest Bounded settlement feature) / Area of Boundless settlement) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa7537f",
   "metadata": {},
   "source": [
    "### 7.1 Load boundless and bounded cumulative settlements and clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7b9bdf01",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. Thu Sep 14 16:57:36 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 41 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  18552 non-null  int64  \n",
      " 1   Sett_ID     18552 non-null  int64  \n",
      " 2   year        18552 non-null  int64  \n",
      " 3   ADM_ID      18552 non-null  int64  \n",
      " 4   AREA2021    18552 non-null  float64\n",
      " 5   AREA2020    17807 non-null  float64\n",
      " 6   AREA2019    17460 non-null  float64\n",
      " 7   AREA2018    17058 non-null  float64\n",
      " 8   AREA2017    16845 non-null  float64\n",
      " 9   AREA2016    16629 non-null  float64\n",
      " 10  AREA2015    16479 non-null  float64\n",
      " 11  AREA2014    15938 non-null  float64\n",
      " 12  AREA2013    14137 non-null  float64\n",
      " 13  AREA2012    13533 non-null  float64\n",
      " 14  AREA2011    11442 non-null  float64\n",
      " 15  AREA2010    11415 non-null  float64\n",
      " 16  AREA2009    4755 non-null   float64\n",
      " 17  AREA2008    4744 non-null   float64\n",
      " 18  AREA2007    4739 non-null   float64\n",
      " 19  AREA2006    4730 non-null   float64\n",
      " 20  AREA2005    4723 non-null   float64\n",
      " 21  AREA2004    4715 non-null   float64\n",
      " 22  AREA2003    4699 non-null   float64\n",
      " 23  AREA2002    4689 non-null   float64\n",
      " 24  AREA2001    4680 non-null   float64\n",
      " 25  AREA2000    4670 non-null   float64\n",
      " 26  AREA1999    4075 non-null   float64\n",
      " 27  AREA1998    4058 non-null   float64\n",
      " 28  AREA1997    4031 non-null   float64\n",
      " 29  AREA1996    4017 non-null   float64\n",
      " 30  AREA1995    3949 non-null   float64\n",
      " 31  AREA1994    3907 non-null   float64\n",
      " 32  AREA1993    3885 non-null   float64\n",
      " 33  AREA1992    3864 non-null   float64\n",
      " 34  AREA1991    3853 non-null   float64\n",
      " 35  AREA1990    3835 non-null   float64\n",
      " 36  AREA1989    2507 non-null   float64\n",
      " 37  AREA1988    2484 non-null   float64\n",
      " 38  AREA1987    2441 non-null   float64\n",
      " 39  AREA1986    2372 non-null   float64\n",
      " 40  AREA1985    2139 non-null   float64\n",
      "dtypes: float64(37), int64(4)\n",
      "memory usage: 5.8 MB\n",
      "None\n",
      "Loaded Bounded dataset, which will factor into the fragmentation calculation. Thu Sep 14 16:57:37 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 19095 entries, 0 to 19094\n",
      "Data columns (total 42 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  19095 non-null  int64  \n",
      " 1   Bounded_ID  19095 non-null  int64  \n",
      " 2   year        19095 non-null  int64  \n",
      " 3   ADM_ID      19095 non-null  int64  \n",
      " 4   Sett_ID     19095 non-null  int64  \n",
      " 5   AREA2021    19095 non-null  float64\n",
      " 6   AREA2020    18331 non-null  float64\n",
      " 7   AREA2019    17970 non-null  float64\n",
      " 8   AREA2018    17556 non-null  float64\n",
      " 9   AREA2017    17340 non-null  float64\n",
      " 10  AREA2016    17122 non-null  float64\n",
      " 11  AREA2015    16968 non-null  float64\n",
      " 12  AREA2014    16420 non-null  float64\n",
      " 13  AREA2013    14576 non-null  float64\n",
      " 14  AREA2012    13957 non-null  float64\n",
      " 15  AREA2011    11829 non-null  float64\n",
      " 16  AREA2010    11802 non-null  float64\n",
      " 17  AREA2009    4926 non-null   float64\n",
      " 18  AREA2008    4915 non-null   float64\n",
      " 19  AREA2007    4910 non-null   float64\n",
      " 20  AREA2006    4901 non-null   float64\n",
      " 21  AREA2005    4894 non-null   float64\n",
      " 22  AREA2004    4886 non-null   float64\n",
      " 23  AREA2003    4870 non-null   float64\n",
      " 24  AREA2002    4860 non-null   float64\n",
      " 25  AREA2001    4851 non-null   float64\n",
      " 26  AREA2000    4841 non-null   float64\n",
      " 27  AREA1999    4228 non-null   float64\n",
      " 28  AREA1998    4211 non-null   float64\n",
      " 29  AREA1997    4184 non-null   float64\n",
      " 30  AREA1996    4170 non-null   float64\n",
      " 31  AREA1995    4101 non-null   float64\n",
      " 32  AREA1994    4059 non-null   float64\n",
      " 33  AREA1993    4035 non-null   float64\n",
      " 34  AREA1992    4013 non-null   float64\n",
      " 35  AREA1991    4000 non-null   float64\n",
      " 36  AREA1990    3982 non-null   float64\n",
      " 37  AREA1989    2614 non-null   float64\n",
      " 38  AREA1988    2591 non-null   float64\n",
      " 39  AREA1987    2547 non-null   float64\n",
      " 40  AREA1986    2476 non-null   float64\n",
      " 41  AREA1985    2232 non-null   float64\n",
      "dtypes: float64(37), int64(5)\n",
      "memory usage: 6.1 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "BoundlessAreas = pd.read_csv(os.path.join(Results, 'Area.csv'))\n",
    "print('Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. %s' \n",
    "      % time.ctime())\n",
    "print(BoundlessAreas.info())\n",
    "\n",
    "BoundedAreas = pd.read_csv(os.path.join(Results, 'Area_Bounded.csv'))\n",
    "print('Loaded Bounded dataset, which will factor into the fragmentation calculation. %s' % time.ctime())\n",
    "print(BoundedAreas.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "bf60fbef",
   "metadata": {},
   "outputs": [],
   "source": [
    "BoundlessAreas = BoundlessAreas.loc[:, ~BoundlessAreas.columns.str.contains('Unnamed')]\n",
    "BoundedAreas = BoundedAreas.loc[:, ~BoundedAreas.columns.str.contains('Unnamed')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "261495a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 19094\n",
      "Data columns (total 41 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Bounded_ID  18552 non-null  int64  \n",
      " 1   year        18552 non-null  int64  \n",
      " 2   ADM_ID      18552 non-null  int64  \n",
      " 3   Sett_ID     18552 non-null  int64  \n",
      " 4   AREA2021    18552 non-null  float64\n",
      " 5   AREA2020    17806 non-null  float64\n",
      " 6   AREA2019    17458 non-null  float64\n",
      " 7   AREA2018    17055 non-null  float64\n",
      " 8   AREA2017    16842 non-null  float64\n",
      " 9   AREA2016    16626 non-null  float64\n",
      " 10  AREA2015    16476 non-null  float64\n",
      " 11  AREA2014    15935 non-null  float64\n",
      " 12  AREA2013    14130 non-null  float64\n",
      " 13  AREA2012    13528 non-null  float64\n",
      " 14  AREA2011    11434 non-null  float64\n",
      " 15  AREA2010    11407 non-null  float64\n",
      " 16  AREA2009    4746 non-null   float64\n",
      " 17  AREA2008    4735 non-null   float64\n",
      " 18  AREA2007    4730 non-null   float64\n",
      " 19  AREA2006    4721 non-null   float64\n",
      " 20  AREA2005    4714 non-null   float64\n",
      " 21  AREA2004    4706 non-null   float64\n",
      " 22  AREA2003    4690 non-null   float64\n",
      " 23  AREA2002    4680 non-null   float64\n",
      " 24  AREA2001    4671 non-null   float64\n",
      " 25  AREA2000    4661 non-null   float64\n",
      " 26  AREA1999    4065 non-null   float64\n",
      " 27  AREA1998    4048 non-null   float64\n",
      " 28  AREA1997    4021 non-null   float64\n",
      " 29  AREA1996    4007 non-null   float64\n",
      " 30  AREA1995    3939 non-null   float64\n",
      " 31  AREA1994    3897 non-null   float64\n",
      " 32  AREA1993    3875 non-null   float64\n",
      " 33  AREA1992    3854 non-null   float64\n",
      " 34  AREA1991    3843 non-null   float64\n",
      " 35  AREA1990    3825 non-null   float64\n",
      " 36  AREA1989    2501 non-null   float64\n",
      " 37  AREA1988    2478 non-null   float64\n",
      " 38  AREA1987    2435 non-null   float64\n",
      " 39  AREA1986    2365 non-null   float64\n",
      " 40  AREA1985    2132 non-null   float64\n",
      "dtypes: float64(37), int64(4)\n",
      "memory usage: 5.9 MB\n",
      "None\n",
      "Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. Thu Sep 14 16:57:41 2023\n",
      "Renamed columns to avoid duplication during merge, and dropped unnecessary columns. Thu Sep 14 16:57:41 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 78 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   Sett_ID      18552 non-null  int64  \n",
      " 1   year         18552 non-null  int64  \n",
      " 2   ADM_ID       18552 non-null  int64  \n",
      " 3   AREA2021     18552 non-null  float64\n",
      " 4   AREA2020     17807 non-null  float64\n",
      " 5   AREA2019     17460 non-null  float64\n",
      " 6   AREA2018     17058 non-null  float64\n",
      " 7   AREA2017     16845 non-null  float64\n",
      " 8   AREA2016     16629 non-null  float64\n",
      " 9   AREA2015     16479 non-null  float64\n",
      " 10  AREA2014     15938 non-null  float64\n",
      " 11  AREA2013     14137 non-null  float64\n",
      " 12  AREA2012     13533 non-null  float64\n",
      " 13  AREA2011     11442 non-null  float64\n",
      " 14  AREA2010     11415 non-null  float64\n",
      " 15  AREA2009     4755 non-null   float64\n",
      " 16  AREA2008     4744 non-null   float64\n",
      " 17  AREA2007     4739 non-null   float64\n",
      " 18  AREA2006     4730 non-null   float64\n",
      " 19  AREA2005     4723 non-null   float64\n",
      " 20  AREA2004     4715 non-null   float64\n",
      " 21  AREA2003     4699 non-null   float64\n",
      " 22  AREA2002     4689 non-null   float64\n",
      " 23  AREA2001     4680 non-null   float64\n",
      " 24  AREA2000     4670 non-null   float64\n",
      " 25  AREA1999     4075 non-null   float64\n",
      " 26  AREA1998     4058 non-null   float64\n",
      " 27  AREA1997     4031 non-null   float64\n",
      " 28  AREA1996     4017 non-null   float64\n",
      " 29  AREA1995     3949 non-null   float64\n",
      " 30  AREA1994     3907 non-null   float64\n",
      " 31  AREA1993     3885 non-null   float64\n",
      " 32  AREA1992     3864 non-null   float64\n",
      " 33  AREA1991     3853 non-null   float64\n",
      " 34  AREA1990     3835 non-null   float64\n",
      " 35  AREA1989     2507 non-null   float64\n",
      " 36  AREA1988     2484 non-null   float64\n",
      " 37  AREA1987     2441 non-null   float64\n",
      " 38  AREA1986     2372 non-null   float64\n",
      " 39  AREA1985     2139 non-null   float64\n",
      " 40  Bounded_ID   18552 non-null  int64  \n",
      " 41  Largest2021  18552 non-null  float64\n",
      " 42  Largest2020  17806 non-null  float64\n",
      " 43  Largest2019  17458 non-null  float64\n",
      " 44  Largest2018  17055 non-null  float64\n",
      " 45  Largest2017  16842 non-null  float64\n",
      " 46  Largest2016  16626 non-null  float64\n",
      " 47  Largest2015  16476 non-null  float64\n",
      " 48  Largest2014  15935 non-null  float64\n",
      " 49  Largest2013  14130 non-null  float64\n",
      " 50  Largest2012  13528 non-null  float64\n",
      " 51  Largest2011  11434 non-null  float64\n",
      " 52  Largest2010  11407 non-null  float64\n",
      " 53  Largest2009  4746 non-null   float64\n",
      " 54  Largest2008  4735 non-null   float64\n",
      " 55  Largest2007  4730 non-null   float64\n",
      " 56  Largest2006  4721 non-null   float64\n",
      " 57  Largest2005  4714 non-null   float64\n",
      " 58  Largest2004  4706 non-null   float64\n",
      " 59  Largest2003  4690 non-null   float64\n",
      " 60  Largest2002  4680 non-null   float64\n",
      " 61  Largest2001  4671 non-null   float64\n",
      " 62  Largest2000  4661 non-null   float64\n",
      " 63  Largest1999  4065 non-null   float64\n",
      " 64  Largest1998  4048 non-null   float64\n",
      " 65  Largest1997  4021 non-null   float64\n",
      " 66  Largest1996  4007 non-null   float64\n",
      " 67  Largest1995  3939 non-null   float64\n",
      " 68  Largest1994  3897 non-null   float64\n",
      " 69  Largest1993  3875 non-null   float64\n",
      " 70  Largest1992  3854 non-null   float64\n",
      " 71  Largest1991  3843 non-null   float64\n",
      " 72  Largest1990  3825 non-null   float64\n",
      " 73  Largest1989  2501 non-null   float64\n",
      " 74  Largest1988  2478 non-null   float64\n",
      " 75  Largest1987  2435 non-null   float64\n",
      " 76  Largest1986  2365 non-null   float64\n",
      " 77  Largest1985  2132 non-null   float64\n",
      "dtypes: float64(74), int64(4)\n",
      "memory usage: 11.2 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "LargestFragments = BoundedAreas.loc[BoundedAreas.groupby([\"Sett_ID\"])[\"AREA\"+str(WSFE_end)].idxmax()] \n",
    "print(LargestFragments.info())\n",
    "print(\"Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. %s\" % time.ctime())\n",
    "LargestFragments.columns = LargestFragments.columns.str.replace('AREA', 'Largest')\n",
    "LargestFragments = LargestFragments.drop(columns=['year', 'ADM_ID'])\n",
    "print(\"Renamed columns to avoid duplication during merge, and dropped unnecessary columns. %s\" % time.ctime())\n",
    "FragIndices = BoundlessAreas.merge(LargestFragments, how='left', on='Sett_ID')\n",
    "print(FragIndices.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "d7a828bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "del BoundlessAreas, BoundedAreas, LargestFragments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "964d4c7b",
   "metadata": {},
   "source": [
    "### 7.2 Merge and run fragmentation calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "b17bcaeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created names for Year 1985's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1985. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1986's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1986. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1987's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1987. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1988's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1988. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1989's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1989. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1990's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1990. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1991's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1991. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1992's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1992. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1993's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1993. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1994's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1994. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1995's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1995. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1996's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1996. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1997's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1997. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1998's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1998. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 1999's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 1999. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2000's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2000. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2001's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2001. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2002's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2002. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2003's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2003. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2004's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2004. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2005's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2005. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2006's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2006. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2007's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2007. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2008's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2008. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2009's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2009. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2010's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2010. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2011's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2011. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2012's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2012. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2013's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2013. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2014's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2014. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2015's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2015. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2016's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2016. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2017's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2017. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2018's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2018. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2019's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2019. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2020's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2020. Thu Sep 14 16:59:22 2023\n",
      "Created names for Year 2021's variables and temporary objects. Thu Sep 14 16:59:22 2023\n",
      "Calculated fragmentation index for year 2021. Thu Sep 14 16:59:22 2023\n",
      "Completed fragmentation index calculations for all years. Thu Sep 14 16:59:22 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 41 columns):\n",
      " #   Column      Non-Null Count  Dtype\n",
      "---  ------      --------------  -----\n",
      " 0   Sett_ID     18552 non-null  int64\n",
      " 1   year        18552 non-null  int64\n",
      " 2   ADM_ID      18552 non-null  int64\n",
      " 3   Bounded_ID  18552 non-null  int64\n",
      " 4   Frag1985    18552 non-null  int32\n",
      " 5   Frag1986    18552 non-null  int32\n",
      " 6   Frag1987    18552 non-null  int32\n",
      " 7   Frag1988    18552 non-null  int32\n",
      " 8   Frag1989    18552 non-null  int32\n",
      " 9   Frag1990    18552 non-null  int32\n",
      " 10  Frag1991    18552 non-null  int32\n",
      " 11  Frag1992    18552 non-null  int32\n",
      " 12  Frag1993    18552 non-null  int32\n",
      " 13  Frag1994    18552 non-null  int32\n",
      " 14  Frag1995    18552 non-null  int32\n",
      " 15  Frag1996    18552 non-null  int32\n",
      " 16  Frag1997    18552 non-null  int32\n",
      " 17  Frag1998    18552 non-null  int32\n",
      " 18  Frag1999    18552 non-null  int32\n",
      " 19  Frag2000    18552 non-null  int32\n",
      " 20  Frag2001    18552 non-null  int32\n",
      " 21  Frag2002    18552 non-null  int32\n",
      " 22  Frag2003    18552 non-null  int32\n",
      " 23  Frag2004    18552 non-null  int32\n",
      " 24  Frag2005    18552 non-null  int32\n",
      " 25  Frag2006    18552 non-null  int32\n",
      " 26  Frag2007    18552 non-null  int32\n",
      " 27  Frag2008    18552 non-null  int32\n",
      " 28  Frag2009    18552 non-null  int32\n",
      " 29  Frag2010    18552 non-null  int32\n",
      " 30  Frag2011    18552 non-null  int32\n",
      " 31  Frag2012    18552 non-null  int32\n",
      " 32  Frag2013    18552 non-null  int32\n",
      " 33  Frag2014    18552 non-null  int32\n",
      " 34  Frag2015    18552 non-null  int32\n",
      " 35  Frag2016    18552 non-null  int32\n",
      " 36  Frag2017    18552 non-null  int32\n",
      " 37  Frag2018    18552 non-null  int32\n",
      " 38  Frag2019    18552 non-null  int32\n",
      " 39  Frag2020    18552 non-null  int32\n",
      " 40  Frag2021    18552 non-null  int32\n",
      "dtypes: int32(37), int64(4)\n",
      "memory usage: 3.3 MB\n",
      "None\n",
      "       Sett_ID  year  ADM_ID  Bounded_ID  Frag1985  Frag1986  Frag1987  \\\n",
      "7742     38534  2021      28       39211        54        54        54   \n",
      "2966     10708  2021      61       10886        89        84        80   \n",
      "5933     27175  2021      15       27741         0         0        75   \n",
      "1821      6679  2021      13        6761         0         0         0   \n",
      "3417     12738  2012      61       12983         0         0         0   \n",
      "10824    58042  2021     128       59073         0         0         0   \n",
      "1810      6635  2021     145        6715         0         0         0   \n",
      "3458     12905  2021     197       13157         0         0         0   \n",
      "9345     49624  2021      95       50481         0         0         0   \n",
      "18425   156819  2015      14      158423         0         0         0   \n",
      "3821     14686  2021     218       14938         0         0         0   \n",
      "2607      9357  2021     145        9499         0         0         0   \n",
      "12824    82095  2021       6       83487         0         0         0   \n",
      "5016     21189  2021      60       21643         0         0         0   \n",
      "5764     26101  2019      78       26630         0         0         0   \n",
      "7411     36280  2021     153       36896         0         0         0   \n",
      "10374    55439  2014      99       56382         0         0         0   \n",
      "7856     39222  2021      21       39930        37        37        37   \n",
      "14116   103247  2021      18      104895         0         0         0   \n",
      "10783    57837  2021      27       58858        37        37        37   \n",
      "\n",
      "       Frag1988  Frag1989  Frag1990  ...  Frag2012  Frag2013  Frag2014  \\\n",
      "7742         53        53        58  ...        71        72        73   \n",
      "2966         78        78        79  ...        68        67        62   \n",
      "5933         75        80        71  ...        64        62        58   \n",
      "1821          0         0         0  ...        38        40        49   \n",
      "3417          0         0         0  ...        50        50        50   \n",
      "10824         0         0         0  ...         0         0         0   \n",
      "1810          0         0         0  ...        50        20        37   \n",
      "3458          0         0         0  ...        55        55        47   \n",
      "9345          0         0         0  ...         0         0         0   \n",
      "18425         0         0         0  ...         0         0         0   \n",
      "3821          0         0         0  ...         0         0         0   \n",
      "2607          0         0         0  ...        49        50        50   \n",
      "12824         0         0         0  ...        55        54        59   \n",
      "5016          0         0         0  ...        36        45        47   \n",
      "5764          0         0         0  ...        33        33        33   \n",
      "7411          0         0         0  ...        54        54        51   \n",
      "10374         0         0         0  ...         0         0        49   \n",
      "7856         37        37        38  ...        47        47        47   \n",
      "14116         0         0         0  ...        45        52        48   \n",
      "10783        37        37        36  ...        45        45        47   \n",
      "\n",
      "       Frag2015  Frag2016  Frag2017  Frag2018  Frag2019  Frag2020  Frag2021  \n",
      "7742         73        73        73        73        73        74        74  \n",
      "2966         62        62        61        61        61        61        60  \n",
      "5933         58        58        57        57        56        55        55  \n",
      "1821         49        49        50        50        50        50        52  \n",
      "3417         50        50        50        50        50        50        50  \n",
      "10824         0         0         0         0        66        66        50  \n",
      "1810         38        38        38        38        38        46        49  \n",
      "3458         47        47        47        47        47        47        49  \n",
      "9345         66        66        66        49        49        57        49  \n",
      "18425        49        49        49        49        49        49        49  \n",
      "3821          0         0         0         0         0         0        49  \n",
      "2607         50        50        50        50        49        49        49  \n",
      "12824        58        58        57        52        51        50        49  \n",
      "5016         47        49        49        50        51        50        49  \n",
      "5764         33        33        33        33        49        49        49  \n",
      "7411         51        49        49        49        49        49        49  \n",
      "10374        49        49        49        49        49        49        49  \n",
      "7856         48        48        48        48        48        48        48  \n",
      "14116        48        46        48        44        43        43        48  \n",
      "10783        47        47        47        47        48        48        48  \n",
      "\n",
      "[20 rows x 41 columns]\n"
     ]
    }
   ],
   "source": [
    "for item in WSFE_Years:\n",
    "    YY = str(item) # 4-digit year\n",
    "    AreaYY = ''.join([\"AREA\", YY]) # The Boundless area variable name\n",
    "    LargestYY = ''.join(['Largest', YY]) # The Bounded largest area variable name\n",
    "    FragYY = ''.join([\"Frag\", YY]) # Name for the fragmentation index variable\n",
    "    print(\"Created names for Year %s's variables and temporary objects. %s\" % (item, time.ctime()))\n",
    "    \n",
    "    FragIndices[FragYY] = ((FragIndices[AreaYY] - FragIndices[LargestYY]) / FragIndices[AreaYY]) * 100\n",
    "    FragIndices[FragYY] = (FragIndices[FragYY].fillna(0).replace([np.inf, -np.inf], 0)).astype('int')\n",
    "    print(\"Calculated fragmentation index for year %s. %s\" % (item, time.ctime()))\n",
    "\n",
    "# Remove unnecessary columns.\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('Largest')]\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('AREA')]\n",
    "\n",
    "print('Completed fragmentation index calculations for all years. %s' % time.ctime())\n",
    "print(FragIndices.info())\n",
    "print(FragIndices.sort_values(by='Frag'+str(WSFE_end), ascending=False).head(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "5972dd80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Thu Sep 14 16:59:33 2023\n"
     ]
    }
   ],
   "source": [
    "FragIndices = FragIndices.drop(columns=['year', 'ADM_ID'])\n",
    "FragIndices.to_csv(os.path.join(Results, 'FragIndex.csv'))\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "df90dce8",
   "metadata": {},
   "outputs": [],
   "source": [
    "del FragIndices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec36f039",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585db18c",
   "metadata": {},
   "source": [
    "## 8. PREPARE YEARLY DATASETS: POPULATION\n",
    "Can use this as a template for other annualized rasters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d8bd172",
   "metadata": {},
   "source": [
    "### 8.1 Reproject and reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "596ae7f5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2000_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2001_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2002_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2003_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2004_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2005_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2006_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2007_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2008_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2009_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2010_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2011_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2012_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2013_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2014_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2015_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2016_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2017_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2018_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2019_UNadj.tif',\n",
       " 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\Syria\\\\Source\\\\Population\\\\syr_ppp_2020_UNadj.tif']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PopList = glob.glob(os.path.join(Source, 'Population', \"*.tif\"))\n",
    "Settlements = gpd.read_file(os.path.join(Results, 'SETTLEMENTS.gpkg'), layer='SETTLEMENTS')\n",
    "PopList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "a6d1b759",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe to merge with: \n",
      "        Sett_ID\n",
      "0            3\n",
      "1            7\n",
      "2           14\n",
      "3           17\n",
      "4           21\n",
      "...        ...\n",
      "18547   157067\n",
      "18548   157070\n",
      "18549   157072\n",
      "18550   157073\n",
      "18551   157079\n",
      "\n",
      "[18552 rows x 1 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:00:20 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2000_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:00:21 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2000_UNadj.tif: POP2000\n",
      "       Sett_ID   POP2000\n",
      "123        425  3.420516\n",
      "4282     16700       NaN\n",
      "17412   153894       NaN\n",
      "16159   138628  4.391536\n",
      "12337    76672       NaN\n",
      "Loading with rasterio. Thu Sep 14 17:02:47 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2001_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:02:49 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2001_UNadj.tif: POP2001\n",
      "       Sett_ID    POP2000    POP2001\n",
      "7717     38401   6.698985   7.912901\n",
      "15625   130642  12.055336  15.780809\n",
      "5320     23079  40.148026  51.264168\n",
      "13122    86622   3.496887   2.770306\n",
      "4860     20298        NaN        NaN\n",
      "Loading with rasterio. Thu Sep 14 17:05:04 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2002_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:05:06 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2002_UNadj.tif: POP2002\n",
      "       Sett_ID   POP2000   POP2001   POP2002\n",
      "14          36  3.478003  2.812438  3.164444\n",
      "11251    60492  4.102612  2.619807  2.314653\n",
      "13321    93893       NaN       NaN       NaN\n",
      "6978     33544  7.704494  9.823883  8.568492\n",
      "6254     29237       NaN       NaN       NaN\n",
      "Loading with rasterio. Thu Sep 14 17:07:30 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2003_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:07:32 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2003_UNadj.tif: POP2003\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003\n",
      "11450    62743        NaN        NaN        NaN        NaN\n",
      "18092   156129   2.167151   2.377575   1.814329   1.988452\n",
      "14920   114163   0.776562   0.421325   0.711695   0.807899\n",
      "14291   104658   3.055732   5.361570   5.086761   5.164720\n",
      "8758     46378  48.093071  53.429695  52.693428  55.590687\n",
      "Loading with rasterio. Thu Sep 14 17:09:48 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2004_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:09:49 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2004_UNadj.tif: POP2004\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003   POP2004\n",
      "14796   110460   1.164354   1.245336   1.274323   1.269829   1.05112\n",
      "15414   125261        NaN        NaN        NaN        NaN       NaN\n",
      "11418    61636        NaN        NaN        NaN        NaN       NaN\n",
      "10566    56570        NaN        NaN        NaN        NaN       NaN\n",
      "12550    79188  22.768398  22.164394  21.817678  22.580170  39.14909\n",
      "Loading with rasterio. Thu Sep 14 17:11:29 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2005_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:11:31 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2005_UNadj.tif: POP2005\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003     POP2004  \\\n",
      "1789      6558   1.449197   1.374317   1.489159   1.553145    1.575607   \n",
      "979       3731        NaN        NaN        NaN        NaN         NaN   \n",
      "1288      4804  95.020416  98.412247  85.695885  88.510696  102.401375   \n",
      "1658      6134        NaN        NaN        NaN        NaN         NaN   \n",
      "15435   125663        NaN        NaN        NaN        NaN         NaN   \n",
      "\n",
      "        POP2005  \n",
      "1789     1.5688  \n",
      "979         NaN  \n",
      "1288   111.2323  \n",
      "1658        NaN  \n",
      "15435       NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:13:14 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2006_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:13:15 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2006_UNadj.tif: POP2006\n",
      "       Sett_ID    POP2000   POP2001    POP2002   POP2003    POP2004   POP2005  \\\n",
      "7068     33990        NaN       NaN        NaN       NaN        NaN       NaN   \n",
      "5465     24036        NaN       NaN        NaN       NaN        NaN       NaN   \n",
      "6120     28434   0.888350  0.831182   0.827499  0.889847   0.943039  0.916586   \n",
      "12068    73750  16.416367  9.200320  10.290077  9.727873  11.937431  9.366594   \n",
      "3150     11586   7.823553  7.006561   6.483796  6.575887   6.313289  6.584577   \n",
      "\n",
      "        POP2006  \n",
      "7068        NaN  \n",
      "5465        NaN  \n",
      "6120   0.930995  \n",
      "12068  9.764940  \n",
      "3150   6.158329  \n",
      "Loading with rasterio. Thu Sep 14 17:14:26 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2007_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:14:27 2023\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2007_UNadj.tif: POP2007\n",
      "      Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "3652    13848   0.684297   0.614603   0.726203   0.717191   0.727754   \n",
      "2292     8322        NaN        NaN        NaN        NaN        NaN   \n",
      "6454    30477  40.995060  48.956299  48.238609  53.317711  43.049911   \n",
      "6476    30597   1.003719   0.940220   0.988451   1.104733   1.014861   \n",
      "9491    50458   0.896038   0.708521   0.750977   0.714063   1.230495   \n",
      "\n",
      "        POP2005    POP2006    POP2007  \n",
      "3652   0.641799   0.598872   0.653812  \n",
      "2292        NaN        NaN        NaN  \n",
      "6454  56.415146  52.577904  42.857689  \n",
      "6476   1.211061   1.256148   1.149581  \n",
      "9491   1.209414   1.379240   1.259499  \n",
      "Loading with rasterio. Thu Sep 14 17:15:16 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2008_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:15:16 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2008_UNadj.tif: POP2008\n",
      "       Sett_ID   POP2000   POP2001   POP2002   POP2003   POP2004   POP2005  \\\n",
      "11942    72514  9.201828  8.638971  9.271885  9.160055  9.100514  9.822687   \n",
      "2347      8506  0.508597  0.373817  0.366647  0.400267  0.423555  0.441208   \n",
      "14317   104911  1.055667  1.181777  1.304319  1.386728  1.491501  1.546957   \n",
      "9118     48389       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "3645     13822  9.263098  6.906010  6.484000  6.074114  6.406008  6.406133   \n",
      "\n",
      "         POP2006    POP2007    POP2008  \n",
      "11942  10.567913  10.276720  10.809091  \n",
      "2347    0.391765   0.623900   0.579907  \n",
      "14317   1.560741   1.596630   1.559436  \n",
      "9118         NaN        NaN        NaN  \n",
      "3645    6.397079   7.391023   7.625306  \n",
      "Loading with rasterio. Thu Sep 14 17:16:07 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2009_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:16:08 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2009_UNadj.tif: POP2009\n",
      "       Sett_ID   POP2000   POP2001   POP2002   POP2003   POP2004   POP2005  \\\n",
      "1978      7213  2.390918  2.648486  2.216614  2.458052  2.754282  2.459796   \n",
      "13152    87618  2.287377  1.869686  1.759210  1.737484  1.914313  2.211252   \n",
      "14971   115501       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "53         186       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "4652     18921  0.629693  0.471073  0.541284  0.497678  0.552246  0.545277   \n",
      "\n",
      "        POP2006   POP2007   POP2008   POP2009  \n",
      "1978   2.518260  3.202233  4.474163  3.569447  \n",
      "13152  2.134184  2.532154  2.404018  1.965572  \n",
      "14971       NaN       NaN       NaN       NaN  \n",
      "53          NaN       NaN       NaN       NaN  \n",
      "4652   0.499867  0.544728  0.540730  0.674219  \n",
      "Loading with rasterio. Thu Sep 14 17:16:59 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2010_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:17:00 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2010_UNadj.tif: POP2010\n",
      "       Sett_ID   POP2000   POP2001   POP2002   POP2003   POP2004   POP2005  \\\n",
      "15923   135406       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "372       1418       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "174        668  3.220158  2.592202  2.743902  2.801880  3.046392  3.093926   \n",
      "9520     50602  2.412763  4.492281  4.635955  4.394717  4.315004  4.966748   \n",
      "10325    55146       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "\n",
      "        POP2006   POP2007   POP2008   POP2009   POP2010  \n",
      "15923       NaN       NaN       NaN       NaN       NaN  \n",
      "372         NaN       NaN       NaN       NaN       NaN  \n",
      "174    2.701532  3.738181  2.934770  3.749576  3.804858  \n",
      "9520   5.299660  5.537779  5.514761  5.237820  4.738075  \n",
      "10325       NaN       NaN       NaN       NaN       NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:17:50 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2011_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:17:51 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2011_UNadj.tif: POP2011\n",
      "       Sett_ID   POP2000   POP2001   POP2002   POP2003   POP2004   POP2005  \\\n",
      "137        497  4.386341  3.175554  2.886236  3.101026  3.167019  3.110486   \n",
      "12751    81225       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "3493     13072       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "12136    74530       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "10231    54597       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "\n",
      "       POP2006   POP2007   POP2008   POP2009   POP2010   POP2011  \n",
      "137    3.77929  4.192003  3.992988  5.044788  5.154848  4.711612  \n",
      "12751      NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "3493       NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "12136      NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "10231      NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:18:42 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2012_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:18:43 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2012_UNadj.tif: POP2012\n",
      "       Sett_ID     POP2000      POP2001      POP2002      POP2003  \\\n",
      "15897   135042   49.331619    57.380371    61.326504    60.059319   \n",
      "336       1288  793.904297  1098.907471  1173.243896  1198.385010   \n",
      "13418    95218  678.731323   444.188660   450.163696   419.311218   \n",
      "10495    56227   40.928555    49.899059    45.260906    48.281780   \n",
      "1393      5145         NaN          NaN          NaN          NaN   \n",
      "\n",
      "           POP2004      POP2005      POP2006      POP2007      POP2008  \\\n",
      "15897    52.918274    60.694511    64.391342    59.969315    65.269089   \n",
      "336    1079.894043  1192.056641  1264.006836  1183.993896  1390.473877   \n",
      "13418   424.296478   429.466003   425.197662   465.102753   611.448853   \n",
      "10495    38.846710    48.613869    63.648289    58.732922    51.944645   \n",
      "1393           NaN          NaN          NaN          NaN          NaN   \n",
      "\n",
      "           POP2009      POP2010      POP2011      POP2012  \n",
      "15897    71.573425    67.599258    56.074665    60.264374  \n",
      "336    1419.328247  1105.816650  1301.119019  1983.820068  \n",
      "13418   613.834412   583.100403   611.518433   588.081421  \n",
      "10495    29.580219    37.143650    37.246887    43.067764  \n",
      "1393           NaN          NaN          NaN          NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:19:33 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2013_UNadj.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:19:33 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2013_UNadj.tif: POP2013\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "18338   156649   3.088691   2.833685   2.750787   4.517529   4.578218   \n",
      "1249      4678        NaN        NaN        NaN        NaN        NaN   \n",
      "15530   127101  21.688093  22.016289  20.894442  22.799839  22.740576   \n",
      "17372   153758   0.146723   0.116561   0.105576   0.175090   0.170308   \n",
      "13798   100027   9.406130  11.291383  38.007156  38.423813  28.804192   \n",
      "\n",
      "         POP2005    POP2006    POP2007    POP2008    POP2009    POP2010  \\\n",
      "18338   4.774724   4.700241   4.769453   4.511438   3.942056   4.581893   \n",
      "1249         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "15530  24.281986  24.359793  24.835474  23.982281  26.782263  23.533983   \n",
      "17372   0.175562   0.193849   0.188386   0.212040   0.183686   0.197198   \n",
      "13798  25.729469  44.317337  38.680466  37.992828  28.506578  22.338524   \n",
      "\n",
      "         POP2011    POP2012    POP2013  \n",
      "18338   4.589625   4.912981   3.583951  \n",
      "1249         NaN        NaN        NaN  \n",
      "15530  15.454952  21.448307  17.954754  \n",
      "17372   0.169755   0.178952   0.101438  \n",
      "13798  21.948586  39.783920  45.495930  \n",
      "Loading with rasterio. Thu Sep 14 17:20:22 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2014_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:20:23 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2014_UNadj.tif: POP2014\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003     POP2004  \\\n",
      "13540    96929  10.158822   9.346326  10.535967  10.862626   10.018578   \n",
      "10797    57900  59.404835  98.416801  93.824097  99.438690  101.769867   \n",
      "14302   104770   0.433287   0.293317   0.291148   0.439143    0.327354   \n",
      "17023   152388   6.136724   5.138113   4.682093   5.074855    5.536795   \n",
      "16859   148565   0.239830   0.173641   0.204682   0.203284    0.191541   \n",
      "\n",
      "          POP2005     POP2006     POP2007     POP2008    POP2009     POP2010  \\\n",
      "13540   15.564333   16.600609   16.860449   19.037447  20.368797   17.702837   \n",
      "10797  107.854156  113.469337  111.179535  135.265411  94.503418  114.425453   \n",
      "14302    0.348707    0.340506    0.409932    0.593597   0.695365    0.615966   \n",
      "17023    5.798828    5.879605    5.238898    6.060028   5.819877    4.461157   \n",
      "16859    0.235826    0.231352    0.213416    0.263578   0.302181    0.279041   \n",
      "\n",
      "          POP2011     POP2012     POP2013    POP2014  \n",
      "13540   16.840160    7.026689    7.858826   6.121370  \n",
      "10797  123.163284  114.040749  100.465843  71.033150  \n",
      "14302    0.683798    1.097775    0.772951   0.667730  \n",
      "17023    5.331698    3.976610    3.365012   2.672637  \n",
      "16859    0.234609    0.508595    0.505479   0.516624  \n",
      "Loading with rasterio. Thu Sep 14 17:21:11 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2015_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:21:12 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2015_UNadj.tif: POP2015\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "11296    60730   3.627711   3.042057   2.962526   2.960005   3.302741   \n",
      "17618   154795   0.708531   0.755129   0.568840   0.614405   0.612060   \n",
      "17443   154000  30.744564  45.804752  52.321243  53.296883  53.209671   \n",
      "10203    54400   3.492282   2.931734   2.959597   2.735279   2.758635   \n",
      "5784     26259   0.700177   0.624643   0.669626   0.672340   0.657917   \n",
      "\n",
      "         POP2005    POP2006    POP2007    POP2008    POP2009    POP2010  \\\n",
      "11296   3.331166   3.003090   3.369392   2.841878   5.198206   5.603109   \n",
      "17618   0.587124   0.583607   0.589942   0.656829   0.910639   0.839745   \n",
      "17443  59.224274  61.043266  54.475037  53.297100  99.107338  73.696495   \n",
      "10203   3.301407   3.538189   3.101385   3.473396   3.973423   3.613223   \n",
      "5784    0.672023   0.665499   0.665756   0.708804   0.770349   0.688661   \n",
      "\n",
      "         POP2011    POP2012    POP2013    POP2014    POP2015  \n",
      "11296   6.079038   4.385268   4.463673   2.924192   2.118608  \n",
      "17618   1.376256   1.905481   2.038804   1.410923   1.409983  \n",
      "17443  46.381989  30.369335  23.832424  20.571987  16.954189  \n",
      "10203   7.148264   6.418905   5.486819   5.003192   4.805415  \n",
      "5784    0.738357   0.617143   0.556377   0.788930   1.283346  \n",
      "Loading with rasterio. Thu Sep 14 17:21:59 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2016_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:22:00 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2016_UNadj.tif: POP2016\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "8511     43129        NaN        NaN        NaN        NaN        NaN   \n",
      "10910    58579   2.200424   1.966337   1.847371   1.676279   1.733087   \n",
      "3680     13958        NaN        NaN        NaN        NaN        NaN   \n",
      "11299    60748  39.505722  44.712513  48.549683  47.069683  59.698757   \n",
      "1087      4117        NaN        NaN        NaN        NaN        NaN   \n",
      "\n",
      "         POP2005    POP2006    POP2007    POP2008    POP2009    POP2010  \\\n",
      "8511         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "10910   1.922460   1.913516   2.311408   2.658217   2.658201   2.958239   \n",
      "3680         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "11299  59.325951  66.853043  68.603340  64.732033  67.512329  94.478180   \n",
      "1087         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "\n",
      "         POP2011    POP2012    POP2013    POP2014    POP2015    POP2016  \n",
      "8511         NaN        NaN        NaN        NaN        NaN        NaN  \n",
      "10910   2.899952   2.416216   3.083122   2.229974   1.714684   1.712218  \n",
      "3680         NaN        NaN        NaN        NaN        NaN        NaN  \n",
      "11299  64.562714  40.673164  36.995895  31.026966  23.618664  21.996696  \n",
      "1087         NaN        NaN        NaN        NaN        NaN        NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:22:54 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2017_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:22:55 2023\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2017_UNadj.tif: POP2017\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "8635     45272  10.993009  19.153950  18.753742  18.560871  18.870796   \n",
      "9933     52961   0.516831   0.421406   0.464261   0.482957   0.504045   \n",
      "12476    78325   1.794587   1.524969   1.454607   1.667410   1.600494   \n",
      "14760   109581   0.054352   0.054390   0.057397   0.047176   0.057171   \n",
      "13102    85987  46.873096  70.289780  73.410362  75.025879  78.724762   \n",
      "\n",
      "         POP2005    POP2006    POP2007    POP2008    POP2009    POP2010  \\\n",
      "8635   21.219223  22.664143  20.658342  20.691639  23.581322  26.268219   \n",
      "9933    0.528743   0.558412   0.518287   0.507905   0.516004   0.494172   \n",
      "12476   1.755992   1.717225   1.854907   1.904724   2.399550   1.987571   \n",
      "14760   0.056860   0.055723   0.058462   0.065138   0.074405   0.071015   \n",
      "13102  78.843262  76.738426  68.282387  67.158470  75.767212  70.663712   \n",
      "\n",
      "         POP2011    POP2012    POP2013    POP2014    POP2015    POP2016  \\\n",
      "8635   17.341702  19.927353  15.343497  16.143383  16.277987  13.333458   \n",
      "9933    0.480147   0.332269   0.247506   0.212775   0.198646   0.220085   \n",
      "12476   1.963239   0.834416   0.880830   0.688478   0.490818   0.660567   \n",
      "14760   0.690734   0.787922   0.323939   0.283353   0.326035   0.207082   \n",
      "13102  71.346924  61.256424  43.675900  36.418236  32.263042  32.264256   \n",
      "\n",
      "         POP2017  \n",
      "8635   12.731419  \n",
      "9933    0.205480  \n",
      "12476   0.597106  \n",
      "14760   0.205538  \n",
      "13102  33.014767  \n",
      "Loading with rasterio. Thu Sep 14 17:23:45 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2018_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:23:46 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2018_UNadj.tif: POP2018\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003     POP2004  \\\n",
      "10310    55070        NaN        NaN        NaN        NaN         NaN   \n",
      "10445    55881        NaN        NaN        NaN        NaN         NaN   \n",
      "6930     33234   9.478598   6.013293   5.826735   6.382409    6.016468   \n",
      "8633     45244  59.513462  95.973854  93.238075  89.529503  103.190651   \n",
      "15403   125134   1.688707   1.272093   1.134462   1.334369    1.587446   \n",
      "\n",
      "          POP2005     POP2006     POP2007     POP2008     POP2009     POP2010  \\\n",
      "10310         NaN         NaN         NaN         NaN         NaN         NaN   \n",
      "10445         NaN         NaN         NaN         NaN         NaN         NaN   \n",
      "6930     6.624771    6.920751    7.390131    7.124354    5.015418    4.872450   \n",
      "8633   157.595322  181.089340  150.461334  178.679230  217.021271  230.313202   \n",
      "15403    1.680387    1.730018    1.763581    2.151929    1.918723    1.387380   \n",
      "\n",
      "          POP2011     POP2012    POP2013    POP2014    POP2015    POP2016  \\\n",
      "10310         NaN         NaN        NaN        NaN        NaN        NaN   \n",
      "10445         NaN         NaN        NaN        NaN        NaN        NaN   \n",
      "6930     6.622455    6.688788   8.356930   7.076992   6.074819   6.131983   \n",
      "8633   164.642792  154.572784  98.932419  86.135544  77.338791  69.738800   \n",
      "15403    1.819178    1.318620   1.102470   0.817834   0.681520   0.876265   \n",
      "\n",
      "         POP2017    POP2018  \n",
      "10310        NaN        NaN  \n",
      "10445        NaN        NaN  \n",
      "6930    6.223697   5.037113  \n",
      "8633   75.750656  61.935432  \n",
      "15403   0.880871   0.644587  \n",
      "Loading with rasterio. Thu Sep 14 17:24:35 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2019_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:24:36 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2019_UNadj.tif: POP2019\n",
      "       Sett_ID    POP2000    POP2001    POP2002    POP2003    POP2004  \\\n",
      "5832     26513   3.652889   2.382399   2.370961   2.515371   2.442817   \n",
      "2729      9773        NaN        NaN        NaN        NaN        NaN   \n",
      "16215   140244   8.161581  16.587242  17.546251  20.189127  20.605408   \n",
      "9462     50292  40.824879  38.733276  35.955631  36.879005  31.660023   \n",
      "7500     36931        NaN        NaN        NaN        NaN        NaN   \n",
      "\n",
      "         POP2005    POP2006    POP2007    POP2008  ...    POP2010    POP2011  \\\n",
      "5832    2.719762   2.949389   3.231804   3.059117  ...   6.178075   6.020821   \n",
      "2729         NaN        NaN        NaN        NaN  ...        NaN        NaN   \n",
      "16215  21.724758  24.645281  20.964468  24.519215  ...  17.143303  18.378643   \n",
      "9462   34.882164  41.749630  42.886925  44.163734  ...  42.462200  37.235443   \n",
      "7500         NaN        NaN        NaN        NaN  ...        NaN        NaN   \n",
      "\n",
      "         POP2012    POP2013    POP2014    POP2015    POP2016    POP2017  \\\n",
      "5832    4.076410   4.261151   3.459879   3.248724   3.413695   3.091566   \n",
      "2729         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "16215  18.202675  24.311390  29.132851  26.971024  15.551362  15.100351   \n",
      "9462   42.940605  37.181736  32.667858  32.473579  30.709955  31.486887   \n",
      "7500         NaN        NaN        NaN        NaN        NaN        NaN   \n",
      "\n",
      "         POP2018    POP2019  \n",
      "5832    2.573633   2.834559  \n",
      "2729         NaN        NaN  \n",
      "16215  26.316660  16.892525  \n",
      "9462   30.402571  30.042007  \n",
      "7500         NaN        NaN  \n",
      "\n",
      "[5 rows x 21 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:25:28 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2020_UNadj.tif\n",
      "| 0.00, 0.00, 35.72|\n",
      "| 0.00,-0.00, 37.32|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " ...\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]\n",
      " [-99999. -99999. -99999. ... -99999. -99999. -99999.]]\n",
      "Zonal statistics. Thu Sep 14 17:25:28 2023\n",
      "sum stat output field for Q:\\GIS\\povertyequity\\urban_growth\\Syria\\Source\\Population\\syr_ppp_2020_UNadj.tif: POP2020\n",
      "       Sett_ID   POP2000   POP2001   POP2002   POP2003   POP2004   POP2005  \\\n",
      "9293     49372       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "917       3547       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "1898      6966       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "3239     11978  1.070162  0.699859  0.760245  0.973958  0.978317  0.961451   \n",
      "15642   131339       NaN       NaN       NaN       NaN       NaN       NaN   \n",
      "\n",
      "        POP2006   POP2007   POP2008  ...   POP2011   POP2012   POP2013  \\\n",
      "9293        NaN       NaN       NaN  ...       NaN       NaN       NaN   \n",
      "917         NaN       NaN       NaN  ...       NaN       NaN       NaN   \n",
      "1898        NaN       NaN       NaN  ...       NaN       NaN       NaN   \n",
      "3239   0.870706  0.904211  1.028639  ...  1.168633  1.927553  1.828728   \n",
      "15642       NaN       NaN       NaN  ...       NaN       NaN       NaN   \n",
      "\n",
      "        POP2014   POP2015   POP2016   POP2017   POP2018   POP2019   POP2020  \n",
      "9293        NaN       NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "917         NaN       NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "1898        NaN       NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "3239   1.360822  1.177198  1.188499  1.137118  0.822773  1.154144  0.905934  \n",
      "15642       NaN       NaN       NaN       NaN       NaN       NaN       NaN  \n",
      "\n",
      "[5 rows x 22 columns]\n",
      "Final dataframe: \n",
      "        Sett_ID  POP2000  POP2001  POP2002  POP2003  POP2004  POP2005  POP2006  \\\n",
      "10564    56560      NaN      NaN      NaN      NaN      NaN      NaN      NaN   \n",
      "4382     17271      NaN      NaN      NaN      NaN      NaN      NaN      NaN   \n",
      "18124   156206      NaN      NaN      NaN      NaN      NaN      NaN      NaN   \n",
      "1746      6419      NaN      NaN      NaN      NaN      NaN      NaN      NaN   \n",
      "13316    93762      NaN      NaN      NaN      NaN      NaN      NaN      NaN   \n",
      "\n",
      "       POP2007  POP2008  ...  POP2011  POP2012  POP2013  POP2014  POP2015  \\\n",
      "10564      NaN      NaN  ...      NaN      NaN      NaN      NaN      NaN   \n",
      "4382       NaN      NaN  ...      NaN      NaN      NaN      NaN      NaN   \n",
      "18124      NaN      NaN  ...      NaN      NaN      NaN      NaN      NaN   \n",
      "1746       NaN      NaN  ...      NaN      NaN      NaN      NaN      NaN   \n",
      "13316      NaN      NaN  ...      NaN      NaN      NaN      NaN      NaN   \n",
      "\n",
      "       POP2016  POP2017  POP2018  POP2019  POP2020  \n",
      "10564      NaN      NaN      NaN      NaN      NaN  \n",
      "4382       NaN      NaN      NaN      NaN      NaN  \n",
      "18124      NaN      NaN      NaN      NaN      NaN  \n",
      "1746       NaN      NaN      NaN      NaN      NaN  \n",
      "13316      NaN      NaN      NaN      NaN      NaN  \n",
      "\n",
      "[5 rows x 22 columns]\n"
     ]
    }
   ],
   "source": [
    "BatchZonal(RasterFileList= PopList, Zones = Settlements, \n",
    "           KeepFields = ['Sett_ID'], \n",
    "           RasterDirectory = os.path.join(Source, 'Population'), \n",
    "           OutPath = os.path.join(Results, 'Population.csv'), \n",
    "           Statistics=['sum'], \n",
    "           NoDataVal = -99999, \n",
    "           Prefix = 'POP',\n",
    "           Suffix = '', \n",
    "           DropStatName = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47a37640",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e49e4cf",
   "metadata": {},
   "source": [
    "## 9. PREPARE YEARLY DATASETS: NIGHTTIME LIGHTS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4667dd97",
   "metadata": {},
   "source": [
    "### 9.1 Reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements. The two NTL sources have already been reprojected in a separate script, and cropped to Central & Western Africa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "904fc3d6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_1999_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2000_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2001_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2002_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2003_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2004_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2005_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2006_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2007_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2008_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2009_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2010_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2011_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2012_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2013_avg.tif'] ['Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2012_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2013_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2014_avg.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2015_avg.tif'] ['Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_1999_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2000_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2001_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2002_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2003_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2004_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2005_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2006_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2007_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2008_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2009_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2010_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2011_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2012_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\D_2013_cfc.tif'] ['Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2012_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2013_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2014_cfc.tif', 'Q:\\\\GIS\\\\povertyequity\\\\urban_growth\\\\NighttimeLights_VIIRS_DMSP\\\\Temp\\\\V_2015_cfc.tif']\n"
     ]
    }
   ],
   "source": [
    "D_avg = glob.glob(os.path.join(NTL, \"*D*avg.tif\"))\n",
    "V_avg = glob.glob(os.path.join(NTL, \"*V*avg.tif\"))\n",
    "D_cfc = glob.glob(os.path.join(NTL, \"*D*cfc.tif\"))\n",
    "V_cfc = glob.glob(os.path.join(NTL, \"*V*cfc.tif\"))\n",
    "Settlements = gpd.read_file(os.path.join(Results, 'SETTLEMENTS.gpkg'), layer='SETTLEMENTS')\n",
    "\n",
    "print(D_avg, V_avg, D_cfc, V_cfc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94b57e5e",
   "metadata": {},
   "source": [
    "#### Nighttime lights from two different sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "94a3aa57",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe to merge with: \n",
      "        Sett_ID\n",
      "0            3\n",
      "1            7\n",
      "2           14\n",
      "3           17\n",
      "4           21\n",
      "...        ...\n",
      "18547   157067\n",
      "18548   157070\n",
      "18549   157072\n",
      "18550   157073\n",
      "18551   157079\n",
      "\n",
      "[18552 rows x 1 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:48:02 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_1999_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " ...\n",
      " [6.270092  6.047826  6.1272793 ... 6.0887446 6.471717  6.471717 ]\n",
      " [6.413352  5.6817465 5.7555556 ... 6.1291924 6.332485  6.332485 ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]]\n",
      "Zonal statistics. Thu Sep 14 17:48:10 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_1999_avg.tif: NTL_Dstd1999\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "6974     33519           NaN           NaN            NaN              NaN   \n",
      "368       1410           NaN           NaN            NaN              NaN   \n",
      "5221     22403           NaN           NaN            NaN              NaN   \n",
      "15053   117926           NaN           NaN            NaN              NaN   \n",
      "18476   156927           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  \n",
      "6974            NaN  \n",
      "368             NaN  \n",
      "5221            NaN  \n",
      "15053           NaN  \n",
      "18476           NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:49:00 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2000_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " ...\n",
      " [2.5344827 2.7583334 2.7333333 ... 2.4491525 2.5916667 2.5916667]\n",
      " [2.775     2.7711864 2.7166667 ... 2.5847456 2.5       2.5      ]\n",
      " [2.7459016 2.819672  2.7419355 ... 2.5847456 2.5169492 2.5169492]]\n",
      "Zonal statistics. Thu Sep 14 17:49:08 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2000_avg.tif: NTL_Dstd2000\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "5968     27407           NaN           NaN            NaN              NaN   \n",
      "15042   117489           NaN           NaN            NaN              NaN   \n",
      "13753    99407           NaN           NaN            NaN              NaN   \n",
      "18098   156140           NaN           NaN            NaN              NaN   \n",
      "17431   153967           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "5968            NaN           NaN           NaN            NaN   \n",
      "15042           NaN           NaN           NaN            NaN   \n",
      "13753           NaN           NaN           NaN            NaN   \n",
      "18098           NaN           NaN           NaN            NaN   \n",
      "17431           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  NTL_Dstd2000  \n",
      "5968               NaN           NaN  \n",
      "15042              NaN           NaN  \n",
      "13753              NaN           NaN  \n",
      "18098              NaN           NaN  \n",
      "17431              NaN           NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:49:57 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2001_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]\n",
      " ...\n",
      " [2.6666667 2.6078432 2.6153846 ... 2.8461537 2.6960785 2.6960785]\n",
      " [2.7058823 2.59      2.6764705 ... 2.8365386 2.7156863 2.7156863]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]]\n",
      "Zonal statistics. Thu Sep 14 17:50:05 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2001_avg.tif: NTL_Dstd2001\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "11384    61232           NaN           NaN            NaN              NaN   \n",
      "1320      4920           NaN           NaN            NaN              NaN   \n",
      "13334    94006           NaN           NaN            NaN              NaN   \n",
      "10296    54974           NaN           NaN            NaN              NaN   \n",
      "12598    79583           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "11384           NaN           NaN           NaN            NaN   \n",
      "1320            NaN           NaN           NaN            NaN   \n",
      "13334           NaN           NaN           NaN            NaN   \n",
      "10296           NaN           NaN           NaN            NaN   \n",
      "12598           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  NTL_Dstd2000  NTL_Dsum2001  NTL_Dmax2001  \\\n",
      "11384              NaN           NaN           NaN           NaN   \n",
      "1320               NaN           NaN           NaN           NaN   \n",
      "13334              NaN           NaN           NaN           NaN   \n",
      "10296              NaN           NaN           NaN           NaN   \n",
      "12598              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2001  NTL_Dmedian2001  NTL_Dstd2001  \n",
      "11384            NaN              NaN           NaN  \n",
      "1320             NaN              NaN           NaN  \n",
      "13334            NaN              NaN           NaN  \n",
      "10296            NaN              NaN           NaN  \n",
      "12598            NaN              NaN           NaN  \n",
      "Loading with rasterio. Thu Sep 14 17:50:53 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2002_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[6.25      6.25      6.25      ... 6.25      6.25      6.25     ]\n",
      " [6.25      6.25      6.25      ... 6.25      6.25      6.25     ]\n",
      " [6.25      6.25      6.25      ... 6.25      6.25      6.25     ]\n",
      " ...\n",
      " [2.4615386 2.381579  2.391892  ... 2.3289473 2.381579  2.381579 ]\n",
      " [2.4871795 2.381579  2.368421  ... 2.3589745 2.3589745 2.3589745]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]]\n",
      "Zonal statistics. Thu Sep 14 17:51:01 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2002_avg.tif: NTL_Dstd2002\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "3878     14967           NaN           NaN            NaN              NaN   \n",
      "29          83           NaN           NaN            NaN              NaN   \n",
      "11472    63207           NaN           NaN            NaN              NaN   \n",
      "17381   153797           NaN           NaN            NaN              NaN   \n",
      "11867    71711           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "3878            NaN           NaN           NaN            NaN   \n",
      "29              NaN           NaN           NaN            NaN   \n",
      "11472           NaN           NaN           NaN            NaN   \n",
      "17381           NaN           NaN           NaN            NaN   \n",
      "11867           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2001  NTL_Dmax2001  NTL_Dmean2001  \\\n",
      "3878               NaN  ...           NaN           NaN            NaN   \n",
      "29                 NaN  ...           NaN           NaN            NaN   \n",
      "11472              NaN  ...           NaN           NaN            NaN   \n",
      "17381              NaN  ...           NaN           NaN            NaN   \n",
      "11867              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2001  NTL_Dstd2001  NTL_Dsum2002  NTL_Dmax2002  \\\n",
      "3878               NaN           NaN           NaN           NaN   \n",
      "29                 NaN           NaN           NaN           NaN   \n",
      "11472              NaN           NaN           NaN           NaN   \n",
      "17381              NaN           NaN           NaN           NaN   \n",
      "11867              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2002  NTL_Dmedian2002  NTL_Dstd2002  \n",
      "3878             NaN              NaN           NaN  \n",
      "29               NaN              NaN           NaN  \n",
      "11472            NaN              NaN           NaN  \n",
      "17381            NaN              NaN           NaN  \n",
      "11867            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 21 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:51:48 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2003_avg.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n",
      "Zonal statistics. Thu Sep 14 17:51:56 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2003_avg.tif: NTL_Dstd2003\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "4468     17790           NaN           NaN            NaN              NaN   \n",
      "15267   123287           NaN           NaN            NaN              NaN   \n",
      "15502   126642           NaN           NaN            NaN              NaN   \n",
      "7265     35264           NaN           NaN            NaN              NaN   \n",
      "3046     11070           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "4468            NaN           NaN           NaN            NaN   \n",
      "15267           NaN           NaN           NaN            NaN   \n",
      "15502           NaN           NaN           NaN            NaN   \n",
      "7265            NaN           NaN           NaN            NaN   \n",
      "3046            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2002  NTL_Dmax2002  NTL_Dmean2002  \\\n",
      "4468               NaN  ...           NaN           NaN            NaN   \n",
      "15267              NaN  ...           NaN           NaN            NaN   \n",
      "15502              NaN  ...           NaN           NaN            NaN   \n",
      "7265               NaN  ...           NaN           NaN            NaN   \n",
      "3046               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2002  NTL_Dstd2002  NTL_Dsum2003  NTL_Dmax2003  \\\n",
      "4468               NaN           NaN           NaN           NaN   \n",
      "15267              NaN           NaN           NaN           NaN   \n",
      "15502              NaN           NaN           NaN           NaN   \n",
      "7265               NaN           NaN           NaN           NaN   \n",
      "3046               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2003  NTL_Dmedian2003  NTL_Dstd2003  \n",
      "4468             NaN              NaN           NaN  \n",
      "15267            NaN              NaN           NaN  \n",
      "15502            NaN              NaN           NaN  \n",
      "7265             NaN              NaN           NaN  \n",
      "3046             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 26 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:52:44 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2004_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[2.607143  2.642857  2.6785715 ... 2.3846154 2.607143  2.607143 ]\n",
      " [2.607143  2.4230769 2.642857  ... 2.3846154 2.7142856 2.607143 ]\n",
      " [5.035714  4.714286  4.7115383 ... 4.892857  4.8642855 4.9857144]\n",
      " ...\n",
      " [2.9166665 2.9285715 4.        ... 3.        3.        2.9166665]\n",
      " [2.8333335 2.5       2.5       ... 2.9166665 3.        2.8333335]\n",
      " [2.8333335 2.5       2.5       ... 2.357143  2.9166665 2.8333335]]\n",
      "Zonal statistics. Thu Sep 14 17:52:52 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2004_avg.tif: NTL_Dstd2004\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "314       1217           NaN           NaN            NaN              NaN   \n",
      "12988    84362           NaN           NaN            NaN              NaN   \n",
      "1849      6795           NaN           NaN            NaN              NaN   \n",
      "1346      4986           NaN           NaN            NaN              NaN   \n",
      "12098    74071           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "314             NaN           NaN           NaN            NaN   \n",
      "12988           NaN           NaN           NaN            NaN   \n",
      "1849            NaN           NaN           NaN            NaN   \n",
      "1346            NaN           NaN           NaN            NaN   \n",
      "12098           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2003  NTL_Dmax2003  NTL_Dmean2003  \\\n",
      "314                NaN  ...           NaN           NaN            NaN   \n",
      "12988              NaN  ...           NaN           NaN            NaN   \n",
      "1849               NaN  ...           NaN           NaN            NaN   \n",
      "1346               NaN  ...           NaN           NaN            NaN   \n",
      "12098              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2003  NTL_Dstd2003  NTL_Dsum2004  NTL_Dmax2004  \\\n",
      "314                NaN           NaN           NaN           NaN   \n",
      "12988              NaN           NaN           NaN           NaN   \n",
      "1849               NaN           NaN           NaN           NaN   \n",
      "1346               NaN           NaN           NaN           NaN   \n",
      "12098              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2004  NTL_Dmedian2004  NTL_Dstd2004  \n",
      "314              NaN              NaN           NaN  \n",
      "12988            NaN              NaN           NaN  \n",
      "1849             NaN              NaN           NaN  \n",
      "1346             NaN              NaN           NaN  \n",
      "12098            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 31 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:53:39 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2005_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [3.22      3.22      3.24      ... 2.9791667 3.16      3.14     ]\n",
      " [5.781739  5.781739  5.8452177 ... 5.630435  5.6494565 5.6834784]\n",
      " ...\n",
      " [3.1666665 3.1666665 3.5       ... 2.5       2.75      3.25     ]\n",
      " [3.25      3.25      3.3333335 ... 3.4166665 3.4166665 3.4166665]\n",
      " [3.9166667 3.9166667 3.3333335 ... 3.8333335 4.        3.9166667]]\n",
      "Zonal statistics. Thu Sep 14 17:53:47 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2005_avg.tif: NTL_Dstd2005\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "16541   143451           NaN           NaN            NaN              NaN   \n",
      "7429     36406           NaN           NaN            NaN              NaN   \n",
      "11895    72025           NaN           NaN            NaN              NaN   \n",
      "18176   156305           NaN           NaN            NaN              NaN   \n",
      "8148     40997           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "16541           NaN           NaN           NaN            NaN   \n",
      "7429            NaN           NaN           NaN            NaN   \n",
      "11895           NaN           NaN           NaN            NaN   \n",
      "18176           NaN           NaN           NaN            NaN   \n",
      "8148            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2004  NTL_Dmax2004  NTL_Dmean2004  \\\n",
      "16541              NaN  ...           NaN           NaN            NaN   \n",
      "7429               NaN  ...           NaN           NaN            NaN   \n",
      "11895              NaN  ...           NaN           NaN            NaN   \n",
      "18176              NaN  ...           NaN           NaN            NaN   \n",
      "8148               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2004  NTL_Dstd2004  NTL_Dsum2005  NTL_Dmax2005  \\\n",
      "16541              NaN           NaN           NaN           NaN   \n",
      "7429               NaN           NaN           NaN           NaN   \n",
      "11895              NaN           NaN           NaN           NaN   \n",
      "18176              NaN           NaN           NaN           NaN   \n",
      "8148               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2005  NTL_Dmedian2005  NTL_Dstd2005  \n",
      "16541            NaN              NaN           NaN  \n",
      "7429             NaN              NaN           NaN  \n",
      "11895            NaN              NaN           NaN  \n",
      "18176            NaN              NaN           NaN  \n",
      "8148             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 36 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:54:34 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2006_avg.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [2.5714285 2.642857  2.7142856 ... 2.7142856 3.4       2.5714285]\n",
      " [4.5714283 4.8690476 4.6875    ... 4.8142853 5.5       4.5714283]\n",
      " ...\n",
      " [2.4       2.6       2.6       ... 2.375     2.275     2.4      ]\n",
      " [2.6       2.5       2.7       ... 2.5       2.5       2.6      ]\n",
      " [3.        2.8       3.        ... 2.6       2.9       3.       ]]\n",
      "Zonal statistics. Thu Sep 14 17:54:42 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2006_avg.tif: NTL_Dstd2006\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "16896   149079           NaN           NaN            NaN              NaN   \n",
      "13730    99136           NaN           NaN            NaN              NaN   \n",
      "9630     51241      6.223214      6.223214       6.223214         6.223214   \n",
      "6504     30813      8.808655      8.808655       8.808655         8.808655   \n",
      "5965     27387           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "16896           NaN           NaN           NaN            NaN   \n",
      "13730           NaN           NaN           NaN            NaN   \n",
      "9630            0.0      6.303455      6.303455       6.303455   \n",
      "6504            0.0      8.210000      8.210000       8.210000   \n",
      "5965            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2005  NTL_Dmax2005  NTL_Dmean2005  \\\n",
      "16896              NaN  ...           NaN           NaN            NaN   \n",
      "13730              NaN  ...           NaN           NaN            NaN   \n",
      "9630          6.303455  ...      5.911462      5.911462       5.911462   \n",
      "6504          8.210000  ...      7.756649      7.756649       7.756649   \n",
      "5965               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2005  NTL_Dstd2005  NTL_Dsum2006  NTL_Dmax2006  \\\n",
      "16896              NaN           NaN           NaN           NaN   \n",
      "13730              NaN           NaN           NaN           NaN   \n",
      "9630          5.911462           0.0      7.375862      7.375862   \n",
      "6504          7.756649           0.0      8.940228      8.940228   \n",
      "5965               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2006  NTL_Dmedian2006  NTL_Dstd2006  \n",
      "16896            NaN              NaN           NaN  \n",
      "13730            NaN              NaN           NaN  \n",
      "9630        7.375862         7.375862           0.0  \n",
      "6504        8.940228         8.940228           0.0  \n",
      "5965             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 41 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:55:30 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2007_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.        0.        0.        ... 0.        0.        0.       ]\n",
      " [1.8541666 1.8125    1.8333334 ... 1.875     1.8541666 1.8541666]\n",
      " [4.3894234 3.9625    3.9833336 ... 4.4134617 4.407051  4.3894234]\n",
      " ...\n",
      " [2.3333335 2.6666665 2.0833335 ... 1.65      2.3333335 2.3333335]\n",
      " [1.4       2.0833335 2.0833335 ... 1.4       1.4       1.4      ]\n",
      " [2.0833333 2.4166665 1.8333334 ... 1.65      1.65      2.0833333]]\n",
      "Zonal statistics. Thu Sep 14 17:55:36 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2007_avg.tif: NTL_Dstd2007\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "4320     16884           NaN           NaN            NaN              NaN   \n",
      "6176     28775      4.745465      4.745465       4.745465         4.745465   \n",
      "789       3006           NaN           NaN            NaN              NaN   \n",
      "1688      6240           NaN           NaN            NaN              NaN   \n",
      "18347   156667     14.853659     14.853659      14.853659        14.853659   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "4320            NaN           NaN           NaN            NaN   \n",
      "6176            0.0      5.021661      5.021661       5.021661   \n",
      "789             NaN           NaN           NaN            NaN   \n",
      "1688            NaN           NaN           NaN            NaN   \n",
      "18347           0.0      8.207265      8.207265       8.207265   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2006  NTL_Dmax2006  NTL_Dmean2006  \\\n",
      "4320               NaN  ...           NaN           NaN            NaN   \n",
      "6176          5.021661  ...      4.470696      4.470696       4.470696   \n",
      "789                NaN  ...           NaN           NaN            NaN   \n",
      "1688               NaN  ...           NaN           NaN            NaN   \n",
      "18347         8.207265  ...     27.908833     27.908833      27.908833   \n",
      "\n",
      "       NTL_Dmedian2006  NTL_Dstd2006  NTL_Dsum2007  NTL_Dmax2007  \\\n",
      "4320               NaN           NaN           NaN           NaN   \n",
      "6176          4.470696           0.0      4.644831      4.644831   \n",
      "789                NaN           NaN           NaN           NaN   \n",
      "1688               NaN           NaN           NaN           NaN   \n",
      "18347        27.908833           0.0     33.617500     33.617500   \n",
      "\n",
      "       NTL_Dmean2007  NTL_Dmedian2007  NTL_Dstd2007  \n",
      "4320             NaN              NaN           NaN  \n",
      "6176        4.644831         4.644831           0.0  \n",
      "789              NaN              NaN           NaN  \n",
      "1688             NaN              NaN           NaN  \n",
      "18347      33.617500        33.617500           0.0  \n",
      "\n",
      "[5 rows x 46 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:56:22 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2008_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [3. 3. 3. ... 2. 2. 3.]\n",
      " [2. 3. 3. ... 2. 2. 2.]\n",
      " [2. 4. 4. ... 2. 2. 2.]]\n",
      "Zonal statistics. Thu Sep 14 17:56:31 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2008_avg.tif: NTL_Dstd2008\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "427       1603           NaN           NaN            NaN              NaN   \n",
      "11960    72674           NaN           NaN            NaN              NaN   \n",
      "9515     50586      4.238234      4.238234       4.238234         4.238234   \n",
      "11858    71600           NaN           NaN            NaN              NaN   \n",
      "12402    77317           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "427             NaN           NaN           NaN            NaN   \n",
      "11960           NaN           NaN           NaN            NaN   \n",
      "9515            0.0      4.240385      4.240385       4.240385   \n",
      "11858           NaN           NaN           NaN            NaN   \n",
      "12402           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2007  NTL_Dmax2007  NTL_Dmean2007  \\\n",
      "427                NaN  ...           NaN           NaN            NaN   \n",
      "11960              NaN  ...           NaN           NaN            NaN   \n",
      "9515          4.240385  ...      4.296627      4.296627       4.296627   \n",
      "11858              NaN  ...           NaN           NaN            NaN   \n",
      "12402              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2007  NTL_Dstd2007  NTL_Dsum2008  NTL_Dmax2008  \\\n",
      "427                NaN           NaN           NaN           NaN   \n",
      "11960              NaN           NaN           NaN           NaN   \n",
      "9515          4.296627           0.0      4.317073      4.317073   \n",
      "11858              NaN           NaN           NaN           NaN   \n",
      "12402              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2008  NTL_Dmedian2008  NTL_Dstd2008  \n",
      "427              NaN              NaN           NaN  \n",
      "11960            NaN              NaN           NaN  \n",
      "9515        4.317073         4.317073           0.0  \n",
      "11858            NaN              NaN           NaN  \n",
      "12402            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 51 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:57:15 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2009_avg.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n",
      "Zonal statistics. Thu Sep 14 17:57:23 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2009_avg.tif: NTL_Dstd2009\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "9137     48520           NaN           NaN            NaN              NaN   \n",
      "11607    67013           NaN           NaN            NaN              NaN   \n",
      "9539     50728           NaN           NaN            NaN              NaN   \n",
      "6906     33079           NaN           NaN            NaN              NaN   \n",
      "9378     49840           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "9137            NaN           NaN           NaN            NaN   \n",
      "11607           NaN           NaN           NaN            NaN   \n",
      "9539            NaN           NaN           NaN            NaN   \n",
      "6906            NaN           NaN           NaN            NaN   \n",
      "9378            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2008  NTL_Dmax2008  NTL_Dmean2008  \\\n",
      "9137               NaN  ...           NaN           NaN            NaN   \n",
      "11607              NaN  ...           NaN           NaN            NaN   \n",
      "9539               NaN  ...           NaN           NaN            NaN   \n",
      "6906               NaN  ...           NaN           NaN            NaN   \n",
      "9378               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2008  NTL_Dstd2008  NTL_Dsum2009  NTL_Dmax2009  \\\n",
      "9137               NaN           NaN           NaN           NaN   \n",
      "11607              NaN           NaN           NaN           NaN   \n",
      "9539               NaN           NaN           NaN           NaN   \n",
      "6906               NaN           NaN           NaN           NaN   \n",
      "9378               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2009  NTL_Dmedian2009  NTL_Dstd2009  \n",
      "9137             NaN              NaN           NaN  \n",
      "11607            NaN              NaN           NaN  \n",
      "9539             NaN              NaN           NaN  \n",
      "6906             NaN              NaN           NaN  \n",
      "9378             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 56 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:58:10 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2010_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [5. 5. 5. ... 5. 5. 5.]\n",
      " [9. 9. 9. ... 5. 5. 5.]\n",
      " [9. 9. 9. ... 9. 9. 9.]]\n",
      "Zonal statistics. Thu Sep 14 17:58:18 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2010_avg.tif: NTL_Dstd2010\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "2855     10215           NaN           NaN            NaN              NaN   \n",
      "5064     21476           NaN           NaN            NaN              NaN   \n",
      "18444   156862           NaN           NaN            NaN              NaN   \n",
      "12395    77257           NaN           NaN            NaN              NaN   \n",
      "15069   118381           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "2855            NaN           NaN           NaN            NaN   \n",
      "5064            NaN           NaN           NaN            NaN   \n",
      "18444           NaN           NaN           NaN            NaN   \n",
      "12395           NaN           NaN           NaN            NaN   \n",
      "15069           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2009  NTL_Dmax2009  NTL_Dmean2009  \\\n",
      "2855               NaN  ...           NaN           NaN            NaN   \n",
      "5064               NaN  ...           NaN           NaN            NaN   \n",
      "18444              NaN  ...           NaN           NaN            NaN   \n",
      "12395              NaN  ...           NaN           NaN            NaN   \n",
      "15069              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2009  NTL_Dstd2009  NTL_Dsum2010  NTL_Dmax2010  \\\n",
      "2855               NaN           NaN           NaN           NaN   \n",
      "5064               NaN           NaN           NaN           NaN   \n",
      "18444              NaN           NaN           NaN           NaN   \n",
      "12395              NaN           NaN           NaN           NaN   \n",
      "15069              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2010  NTL_Dmedian2010  NTL_Dstd2010  \n",
      "2855             NaN              NaN           NaN  \n",
      "5064             NaN              NaN           NaN  \n",
      "18444            NaN              NaN           NaN  \n",
      "12395            NaN              NaN           NaN  \n",
      "15069            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 61 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:59:04 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2011_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[6 6 6 ... 5 5 5]\n",
      " [5 5 4 ... 4 5 5]\n",
      " [4 5 5 ... 4 4 4]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "Zonal statistics. Thu Sep 14 17:59:07 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2011_avg.tif: NTL_Dstd2011\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "11549    65498           NaN           NaN            NaN              NaN   \n",
      "5132     21831           NaN           NaN            NaN              NaN   \n",
      "9815     52281           NaN           NaN            NaN              NaN   \n",
      "4503     17998           NaN           NaN            NaN              NaN   \n",
      "1641      6074           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "11549           NaN           NaN           NaN            NaN   \n",
      "5132            NaN           NaN           NaN            NaN   \n",
      "9815            NaN           NaN           NaN            NaN   \n",
      "4503            NaN           NaN           NaN            NaN   \n",
      "1641            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2010  NTL_Dmax2010  NTL_Dmean2010  \\\n",
      "11549              NaN  ...           NaN           NaN            NaN   \n",
      "5132               NaN  ...           NaN           NaN            NaN   \n",
      "9815               NaN  ...           NaN           NaN            NaN   \n",
      "4503               NaN  ...           NaN           NaN            NaN   \n",
      "1641               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2010  NTL_Dstd2010  NTL_Dsum2011  NTL_Dmax2011  \\\n",
      "11549              NaN           NaN           NaN           NaN   \n",
      "5132               NaN           NaN           NaN           NaN   \n",
      "9815               NaN           NaN           NaN           NaN   \n",
      "4503               NaN           NaN           NaN           NaN   \n",
      "1641               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2011  NTL_Dmedian2011  NTL_Dstd2011  \n",
      "11549            NaN              NaN           NaN  \n",
      "5132             NaN              NaN           NaN  \n",
      "9815             NaN              NaN           NaN  \n",
      "4503             NaN              NaN           NaN  \n",
      "1641             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 66 columns]\n",
      "Loading with rasterio. Thu Sep 14 17:59:55 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2012_avg.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[8.   7.75 7.75 ... 8.   8.   8.  ]\n",
      " [7.75 7.75 7.75 ... 8.   8.25 8.25]\n",
      " [8.25 7.75 7.25 ... 8.25 8.25 8.25]\n",
      " ...\n",
      " [0.   0.   0.   ... 0.   0.   0.  ]\n",
      " [0.   0.   0.   ... 0.   0.   0.  ]\n",
      " [0.   0.   0.   ... 0.   0.   0.  ]]\n",
      "Zonal statistics. Thu Sep 14 18:00:02 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2012_avg.tif: NTL_Dstd2012\n",
      "       Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "6408     30205           NaN           NaN            NaN              NaN   \n",
      "3303     12277           NaN           NaN            NaN              NaN   \n",
      "11997    73010           NaN           NaN            NaN              NaN   \n",
      "16267   141569           NaN           NaN            NaN              NaN   \n",
      "15840   134186           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "6408            NaN           NaN           NaN            NaN   \n",
      "3303            NaN           NaN           NaN            NaN   \n",
      "11997           NaN           NaN           NaN            NaN   \n",
      "16267           NaN           NaN           NaN            NaN   \n",
      "15840           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2011  NTL_Dmax2011  NTL_Dmean2011  \\\n",
      "6408               NaN  ...           NaN           NaN            NaN   \n",
      "3303               NaN  ...           NaN           NaN            NaN   \n",
      "11997              NaN  ...           NaN           NaN            NaN   \n",
      "16267              NaN  ...           NaN           NaN            NaN   \n",
      "15840              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2011  NTL_Dstd2011  NTL_Dsum2012  NTL_Dmax2012  \\\n",
      "6408               NaN           NaN           NaN           NaN   \n",
      "3303               NaN           NaN           NaN           NaN   \n",
      "11997              NaN           NaN           NaN           NaN   \n",
      "16267              NaN           NaN           NaN           NaN   \n",
      "15840              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2012  NTL_Dmedian2012  NTL_Dstd2012  \n",
      "6408             NaN              NaN           NaN  \n",
      "3303             NaN              NaN           NaN  \n",
      "11997            NaN              NaN           NaN  \n",
      "16267            NaN              NaN           NaN  \n",
      "15840            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 71 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:00:48 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2013_avg.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[5.5652175 5.9583335 6.5833335 ... 6.125     6.1666665 6.1666665]\n",
      " [5.        6.1666665 6.4583335 ... 5.7391305 6.130435  6.130435 ]\n",
      " [5.5       5.095238  5.        ... 5.826087  6.173913  6.173913 ]\n",
      " ...\n",
      " [6.        6.6666665 6.6666665 ... 4.6666665 4.6666665 4.6666665]\n",
      " [6.        6.6666665 6.6666665 ... 4.6666665 4.6666665 4.6666665]\n",
      " [0.        0.        0.        ... 0.        0.        0.       ]]\n",
      "Zonal statistics. Thu Sep 14 18:00:56 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2013_avg.tif: NTL_Dstd2013\n",
      "      Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "7158    34478           NaN           NaN            NaN              NaN   \n",
      "8804    46657           NaN           NaN            NaN              NaN   \n",
      "8838    46805           NaN           NaN            NaN              NaN   \n",
      "2928    10544           NaN           NaN            NaN              NaN   \n",
      "1617     5974           NaN           NaN            NaN              NaN   \n",
      "\n",
      "      NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "7158           NaN           NaN           NaN            NaN   \n",
      "8804           NaN           NaN           NaN            NaN   \n",
      "8838           NaN           NaN           NaN            NaN   \n",
      "2928           NaN           NaN           NaN            NaN   \n",
      "1617           NaN           NaN           NaN            NaN   \n",
      "\n",
      "      NTL_Dmedian2000  ...  NTL_Dsum2012  NTL_Dmax2012  NTL_Dmean2012  \\\n",
      "7158              NaN  ...           NaN           NaN            NaN   \n",
      "8804              NaN  ...           NaN           NaN            NaN   \n",
      "8838              NaN  ...           NaN           NaN            NaN   \n",
      "2928              NaN  ...           NaN           NaN            NaN   \n",
      "1617              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "      NTL_Dmedian2012  NTL_Dstd2012  NTL_Dsum2013  NTL_Dmax2013  \\\n",
      "7158              NaN           NaN           NaN           NaN   \n",
      "8804              NaN           NaN           NaN           NaN   \n",
      "8838              NaN           NaN           NaN           NaN   \n",
      "2928              NaN           NaN           NaN           NaN   \n",
      "1617              NaN           NaN           NaN           NaN   \n",
      "\n",
      "      NTL_Dmean2013  NTL_Dmedian2013  NTL_Dstd2013  \n",
      "7158            NaN              NaN           NaN  \n",
      "8804            NaN              NaN           NaN  \n",
      "8838            NaN              NaN           NaN  \n",
      "2928            NaN              NaN           NaN  \n",
      "1617            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 76 columns]\n",
      "Final dataframe: \n",
      "        Sett_ID  NTL_Dsum1999  NTL_Dmax1999  NTL_Dmean1999  NTL_Dmedian1999  \\\n",
      "9559     50829           NaN           NaN            NaN              NaN   \n",
      "8533     43580           NaN           NaN            NaN              NaN   \n",
      "4498     17979           NaN           NaN            NaN              NaN   \n",
      "7073     34020           NaN           NaN            NaN              NaN   \n",
      "14743   108827           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Dstd1999  NTL_Dsum2000  NTL_Dmax2000  NTL_Dmean2000  \\\n",
      "9559            NaN           NaN           NaN            NaN   \n",
      "8533            NaN           NaN           NaN            NaN   \n",
      "4498            NaN           NaN           NaN            NaN   \n",
      "7073            NaN           NaN           NaN            NaN   \n",
      "14743           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2000  ...  NTL_Dsum2012  NTL_Dmax2012  NTL_Dmean2012  \\\n",
      "9559               NaN  ...           NaN           NaN            NaN   \n",
      "8533               NaN  ...           NaN           NaN            NaN   \n",
      "4498               NaN  ...           NaN           NaN            NaN   \n",
      "7073               NaN  ...           NaN           NaN            NaN   \n",
      "14743              NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Dmedian2012  NTL_Dstd2012  NTL_Dsum2013  NTL_Dmax2013  \\\n",
      "9559               NaN           NaN           NaN           NaN   \n",
      "8533               NaN           NaN           NaN           NaN   \n",
      "4498               NaN           NaN           NaN           NaN   \n",
      "7073               NaN           NaN           NaN           NaN   \n",
      "14743              NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Dmean2013  NTL_Dmedian2013  NTL_Dstd2013  \n",
      "9559             NaN              NaN           NaN  \n",
      "8533             NaN              NaN           NaN  \n",
      "4498             NaN              NaN           NaN  \n",
      "7073             NaN              NaN           NaN  \n",
      "14743            NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 76 columns]\n",
      "Dataframe to merge with: \n",
      "        Sett_ID\n",
      "0            3\n",
      "1            7\n",
      "2           14\n",
      "3           17\n",
      "4           21\n",
      "...        ...\n",
      "18547   157067\n",
      "18548   157070\n",
      "18549   157072\n",
      "18550   157073\n",
      "18551   157079\n",
      "\n",
      "[18552 rows x 1 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:01:45 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2012_avg.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0.9718189  0.8348664  0.97964    ... 0.99186504 0.98909014 0.9666358 ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " ...\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.7874452  0.77528137 0.7197873 ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]]\n",
      "Zonal statistics. Thu Sep 14 18:02:08 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2012_avg.tif: NTL_Vstd2012\n",
      "       Sett_ID  NTL_Vsum2012  NTL_Vmax2012  NTL_Vmean2012  NTL_Vmedian2012  \\\n",
      "17120   152755           NaN           NaN            NaN              NaN   \n",
      "3809     14629           NaN           NaN            NaN              NaN   \n",
      "7253     35185           NaN           NaN            NaN              NaN   \n",
      "15035   117358           NaN           NaN            NaN              NaN   \n",
      "1260      4721           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Vstd2012  \n",
      "17120           NaN  \n",
      "3809            NaN  \n",
      "7253            NaN  \n",
      "15035           NaN  \n",
      "1260            NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:02:56 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2013_avg.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " ...\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.9139684   0.9352511\n",
      "   0.90836704]\n",
      " [ 0.          0.         -1.5        ...  0.          0.\n",
      "   0.        ]]\n",
      "Zonal statistics. Thu Sep 14 18:03:21 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2013_avg.tif: NTL_Vstd2013\n",
      "       Sett_ID  NTL_Vsum2012  NTL_Vmax2012  NTL_Vmean2012  NTL_Vmedian2012  \\\n",
      "4895     20442      3.742971      2.067995       1.871485         1.871485   \n",
      "9177     48751           NaN           NaN            NaN              NaN   \n",
      "17493   154177           NaN           NaN            NaN              NaN   \n",
      "14123   103311      4.367990      2.116349       1.091997         0.839253   \n",
      "11768    70745           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Vstd2012  NTL_Vsum2013  NTL_Vmax2013  NTL_Vmean2013  \\\n",
      "4895       0.196509      2.944382      1.574636       1.472191   \n",
      "9177            NaN           NaN           NaN            NaN   \n",
      "17493           NaN           NaN           NaN            NaN   \n",
      "14123      0.601377      1.100094      0.628651       0.275023   \n",
      "11768           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Vmedian2013  NTL_Vstd2013  \n",
      "4895          1.472191      0.102445  \n",
      "9177               NaN           NaN  \n",
      "17493              NaN           NaN  \n",
      "14123         0.235721      0.280584  \n",
      "11768              NaN           NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:04:09 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2014_avg.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[1.1100459  1.1288927  1.127595   ... 1.1011702  1.1024839  1.087924  ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " ...\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.77174467 0.795612   0.7914716 ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]]\n",
      "Zonal statistics. Thu Sep 14 18:04:32 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2014_avg.tif: NTL_Vstd2014\n",
      "       Sett_ID  NTL_Vsum2012  NTL_Vmax2012  NTL_Vmean2012  NTL_Vmedian2012  \\\n",
      "11023    59258      2.083235      2.083235       2.083235         2.083235   \n",
      "833       3208           NaN           NaN            NaN              NaN   \n",
      "7067     33976           NaN           NaN            NaN              NaN   \n",
      "16317   141939           NaN           NaN            NaN              NaN   \n",
      "232        922      1.004412      1.004412       1.004412         1.004412   \n",
      "\n",
      "       NTL_Vstd2012  NTL_Vsum2013  NTL_Vmax2013  NTL_Vmean2013  \\\n",
      "11023           0.0      1.854394      1.854394       1.854394   \n",
      "833             NaN           NaN           NaN            NaN   \n",
      "7067            NaN           NaN           NaN            NaN   \n",
      "16317           NaN           NaN           NaN            NaN   \n",
      "232             0.0      0.825966      0.825966       0.825966   \n",
      "\n",
      "       NTL_Vmedian2013  NTL_Vstd2013  NTL_Vsum2014  NTL_Vmax2014  \\\n",
      "11023         1.854394           0.0      1.156010      1.156010   \n",
      "833                NaN           NaN           NaN           NaN   \n",
      "7067               NaN           NaN           NaN           NaN   \n",
      "16317              NaN           NaN           NaN           NaN   \n",
      "232           0.825966           0.0      0.400378      0.400378   \n",
      "\n",
      "       NTL_Vmean2014  NTL_Vmedian2014  NTL_Vstd2014  \n",
      "11023       1.156010         1.156010           0.0  \n",
      "833              NaN              NaN           NaN  \n",
      "7067             NaN              NaN           NaN  \n",
      "16317            NaN              NaN           NaN  \n",
      "232         0.400378         0.400378           0.0  \n",
      "Loading with rasterio. Thu Sep 14 18:05:19 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2015_avg.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[1.0836837  1.083844   1.0713766  ... 1.075265   1.0795301  0.9049964 ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " ...\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.86478096 0.8409789  0.8316555 ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]]\n",
      "Zonal statistics. Thu Sep 14 18:05:49 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2015_avg.tif: NTL_Vstd2015\n",
      "       Sett_ID  NTL_Vsum2012  NTL_Vmax2012  NTL_Vmean2012  NTL_Vmedian2012  \\\n",
      "9745     51870           NaN           NaN            NaN              NaN   \n",
      "15902   135166     119.83107     17.882324      13.314563        13.740604   \n",
      "8827     46760      17.51733      3.939332       3.503466         3.549689   \n",
      "12846    82356           NaN           NaN            NaN              NaN   \n",
      "9527     50671           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Vstd2012  NTL_Vsum2013  NTL_Vmax2013  NTL_Vmean2013  \\\n",
      "9745            NaN           NaN           NaN            NaN   \n",
      "15902      2.906797     48.547955      6.720458       5.394217   \n",
      "8827       0.438526     16.034124      3.845594       3.206825   \n",
      "12846           NaN           NaN           NaN            NaN   \n",
      "9527            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Vmedian2013  ...  NTL_Vsum2014  NTL_Vmax2014  NTL_Vmean2014  \\\n",
      "9745               NaN  ...           NaN           NaN            NaN   \n",
      "15902         5.322951  ...     16.976601      2.318966       1.886289   \n",
      "8827          2.973457  ...     12.028257      3.382969       2.405651   \n",
      "12846              NaN  ...           NaN           NaN            NaN   \n",
      "9527               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Vmedian2014  NTL_Vstd2014  NTL_Vsum2015  NTL_Vmax2015  \\\n",
      "9745               NaN           NaN           NaN           NaN   \n",
      "15902         1.915276      0.290658     10.087254      1.584491   \n",
      "8827          2.180706      0.552960      6.362129      1.657359   \n",
      "12846              NaN           NaN           NaN           NaN   \n",
      "9527               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Vmean2015  NTL_Vmedian2015  NTL_Vstd2015  \n",
      "9745             NaN              NaN           NaN  \n",
      "15902       1.120806         1.203101      0.453915  \n",
      "8827        1.272426         1.190109      0.259632  \n",
      "12846            NaN              NaN           NaN  \n",
      "9527             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 21 columns]\n",
      "Final dataframe: \n",
      "        Sett_ID  NTL_Vsum2012  NTL_Vmax2012  NTL_Vmean2012  NTL_Vmedian2012  \\\n",
      "1828      6705           NaN           NaN            NaN              NaN   \n",
      "18028   156005           NaN           NaN            NaN              NaN   \n",
      "7979     40003    324.869019     23.696671      17.098369        18.230017   \n",
      "11536    65291      7.798949      2.890060       2.599650         2.587439   \n",
      "5860     26705           NaN           NaN            NaN              NaN   \n",
      "\n",
      "       NTL_Vstd2012  NTL_Vsum2013  NTL_Vmax2013  NTL_Vmean2013  \\\n",
      "1828            NaN           NaN           NaN            NaN   \n",
      "18028           NaN           NaN           NaN            NaN   \n",
      "7979       4.276195    264.167114     21.123024      13.903532   \n",
      "11536      0.232295      0.000000      0.000000       0.000000   \n",
      "5860            NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Vmedian2013  ...  NTL_Vsum2014  NTL_Vmax2014  NTL_Vmean2014  \\\n",
      "1828               NaN  ...           NaN           NaN            NaN   \n",
      "18028              NaN  ...           NaN           NaN            NaN   \n",
      "7979         14.260806  ...    165.613205     11.692944       8.716484   \n",
      "11536         0.000000  ...      1.976990      0.951001       0.658997   \n",
      "5860               NaN  ...           NaN           NaN            NaN   \n",
      "\n",
      "       NTL_Vmedian2014  NTL_Vstd2014  NTL_Vsum2015  NTL_Vmax2015  \\\n",
      "1828               NaN           NaN           NaN           NaN   \n",
      "18028              NaN           NaN           NaN           NaN   \n",
      "7979          8.725273      1.600201    110.231735      8.436835   \n",
      "11536         0.553395      0.209097      0.000000      0.000000   \n",
      "5860               NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTL_Vmean2015  NTL_Vmedian2015  NTL_Vstd2015  \n",
      "1828             NaN              NaN           NaN  \n",
      "18028            NaN              NaN           NaN  \n",
      "7979         5.80167         5.984446      1.234909  \n",
      "11536        0.00000         0.000000      0.000000  \n",
      "5860             NaN              NaN           NaN  \n",
      "\n",
      "[5 rows x 21 columns]\n"
     ]
    }
   ],
   "source": [
    "BatchZonal(RasterFileList= D_avg, Zones = Settlements, \n",
    "           KeepFields = ['Sett_ID'], \n",
    "           RasterDirectory = NTL, \n",
    "           OutPath = os.path.join(Results, 'NTL_DMSP.csv'), \n",
    "           Statistics=['sum', 'max', 'mean', 'median', 'std'], \n",
    "           NoDataVal = -99999, # For NTL, the actual NoData is a \"soft NaN\": 3.40282e+38. Just using pop's here.\n",
    "           Prefix = 'NTL_D',\n",
    "           Suffix = '', \n",
    "           DropStatName = False)\n",
    "\n",
    "BatchZonal(RasterFileList= V_avg, Zones = Settlements, \n",
    "           KeepFields = ['Sett_ID'], \n",
    "           RasterDirectory = NTL, \n",
    "           OutPath = os.path.join(Results, 'NTL_VIIRS.csv'), \n",
    "           Statistics=['sum', 'max', 'mean', 'median', 'std'], \n",
    "           NoDataVal = -99999,\n",
    "           Prefix = 'NTL_V',\n",
    "           Suffix = '', \n",
    "           DropStatName = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc1a2b5",
   "metadata": {},
   "source": [
    "#### Cloud-free coverage (confidence metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "53b2a8c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe to merge with: \n",
      "        Sett_ID\n",
      "0            3\n",
      "1            7\n",
      "2           14\n",
      "3           17\n",
      "4           21\n",
      "...        ...\n",
      "18547   157067\n",
      "18548   157070\n",
      "18549   157072\n",
      "18550   157073\n",
      "18551   157079\n",
      "\n",
      "[18552 rows x 1 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:06:35 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_1999_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " ...\n",
      " [44 44 44 ... 44 44 44]\n",
      " [43 43 44 ... 42 42 42]\n",
      " [ 0  0  0 ...  0  0  0]]\n",
      "Zonal statistics. Thu Sep 14 18:06:39 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_1999_cfc.tif: NTL_Dstd1999_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "14925   114356               NaN                   0                NaN   \n",
      "4130     16046              42.0                   1               42.0   \n",
      "14239   104291               NaN                   0                NaN   \n",
      "7829     39073               NaN                   0                NaN   \n",
      "1539      5698               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  \n",
      "14925                  NaN               NaN  \n",
      "4130                  42.0               0.0  \n",
      "14239                  NaN               NaN  \n",
      "7829                   NaN               NaN  \n",
      "1539                   NaN               NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:07:22 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2000_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " ...\n",
      " [32 32 32 ... 32 32 32]\n",
      " [32 31 32 ... 32 32 32]\n",
      " [32 32 32 ... 32 32 32]]\n",
      "Zonal statistics. Thu Sep 14 18:07:26 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2000_cfc.tif: NTL_Dstd2000_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "6811     32583               NaN                   0                NaN   \n",
      "616       2323               NaN                   0                NaN   \n",
      "630       2369               NaN                   0                NaN   \n",
      "1299      4840               NaN                   0                NaN   \n",
      "15003   116528               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "6811                   NaN               NaN               NaN   \n",
      "616                    NaN               NaN               NaN   \n",
      "630                    NaN               NaN               NaN   \n",
      "1299                   NaN               NaN               NaN   \n",
      "15003                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  \\\n",
      "6811                    0                NaN                  NaN   \n",
      "616                     0                NaN                  NaN   \n",
      "630                     0                NaN                  NaN   \n",
      "1299                    0                NaN                  NaN   \n",
      "15003                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2000_cfc  \n",
      "6811                NaN  \n",
      "616                 NaN  \n",
      "630                 NaN  \n",
      "1299                NaN  \n",
      "15003               NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:08:09 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2001_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " [ 0  0  0 ...  0  0  0]\n",
      " ...\n",
      " [30 30 31 ... 30 30 30]\n",
      " [30 30 30 ... 30 30 30]\n",
      " [ 0  0  0 ...  0  0  0]]\n",
      "Zonal statistics. Thu Sep 14 18:08:13 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2001_cfc.tif: NTL_Dstd2001_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "8949     47455               NaN                   0                NaN   \n",
      "16223   140570               NaN                   0                NaN   \n",
      "2656      9537               NaN                   0                NaN   \n",
      "4869     20343               NaN                   0                NaN   \n",
      "17776   155419               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "8949                   NaN               NaN               NaN   \n",
      "16223                  NaN               NaN               NaN   \n",
      "2656                   NaN               NaN               NaN   \n",
      "4869                   NaN               NaN               NaN   \n",
      "17776                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  \\\n",
      "8949                    0                NaN                  NaN   \n",
      "16223                   0                NaN                  NaN   \n",
      "2656                    0                NaN                  NaN   \n",
      "4869                    0                NaN                  NaN   \n",
      "17776                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2000_cfc  NTL_Dsum2001_cfc  NTL_Dcount2001_cfc  \\\n",
      "8949                NaN               NaN                   0   \n",
      "16223               NaN               NaN                   0   \n",
      "2656                NaN               NaN                   0   \n",
      "4869                NaN               NaN                   0   \n",
      "17776               NaN               NaN                   0   \n",
      "\n",
      "       NTL_Dmean2001_cfc  NTL_Dmedian2001_cfc  NTL_Dstd2001_cfc  \n",
      "8949                 NaN                  NaN               NaN  \n",
      "16223                NaN                  NaN               NaN  \n",
      "2656                 NaN                  NaN               NaN  \n",
      "4869                 NaN                  NaN               NaN  \n",
      "17776                NaN                  NaN               NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:08:56 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2002_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 1  1  1 ...  1  1  1]\n",
      " [ 1  1  1 ...  1  1  1]\n",
      " [ 1  1  1 ...  1  1  1]\n",
      " ...\n",
      " [22 22 21 ... 22 22 22]\n",
      " [22 22 22 ... 22 22 22]\n",
      " [ 0  0  0 ...  0  0  0]]\n",
      "Zonal statistics. Thu Sep 14 18:09:01 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2002_cfc.tif: NTL_Dstd2002_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "11023    59258               NaN                   0                NaN   \n",
      "16897   149102               NaN                   0                NaN   \n",
      "9459     50272               NaN                   0                NaN   \n",
      "8443     42754               NaN                   0                NaN   \n",
      "3541     13281              41.0                   1               41.0   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "11023                  NaN               NaN               NaN   \n",
      "16897                  NaN               NaN               NaN   \n",
      "9459                   NaN               NaN               NaN   \n",
      "8443                   NaN               NaN               NaN   \n",
      "3541                  41.0               0.0              59.0   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "11023                   0                NaN                  NaN  ...   \n",
      "16897                   0                NaN                  NaN  ...   \n",
      "9459                    0                NaN                  NaN  ...   \n",
      "8443                    0                NaN                  NaN  ...   \n",
      "3541                    1               59.0                 59.0  ...   \n",
      "\n",
      "       NTL_Dsum2001_cfc  NTL_Dcount2001_cfc  NTL_Dmean2001_cfc  \\\n",
      "11023               NaN                   0                NaN   \n",
      "16897               NaN                   0                NaN   \n",
      "9459                NaN                   0                NaN   \n",
      "8443                NaN                   0                NaN   \n",
      "3541               54.0                   1               54.0   \n",
      "\n",
      "       NTL_Dmedian2001_cfc  NTL_Dstd2001_cfc  NTL_Dsum2002_cfc  \\\n",
      "11023                  NaN               NaN               NaN   \n",
      "16897                  NaN               NaN               NaN   \n",
      "9459                   NaN               NaN               NaN   \n",
      "8443                   NaN               NaN               NaN   \n",
      "3541                  54.0               0.0              47.0   \n",
      "\n",
      "       NTL_Dcount2002_cfc  NTL_Dmean2002_cfc  NTL_Dmedian2002_cfc  \\\n",
      "11023                   0                NaN                  NaN   \n",
      "16897                   0                NaN                  NaN   \n",
      "9459                    0                NaN                  NaN   \n",
      "8443                    0                NaN                  NaN   \n",
      "3541                    1               47.0                 47.0   \n",
      "\n",
      "       NTL_Dstd2002_cfc  \n",
      "11023               NaN  \n",
      "16897               NaN  \n",
      "9459                NaN  \n",
      "8443                NaN  \n",
      "3541                0.0  \n",
      "\n",
      "[5 rows x 21 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:09:45 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2003_cfc.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "Zonal statistics. Thu Sep 14 18:09:49 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2003_cfc.tif: NTL_Dstd2003_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "12937    83373               NaN                   0                NaN   \n",
      "7200     34801               NaN                   0                NaN   \n",
      "11924    72335               NaN                   0                NaN   \n",
      "7219     34920               NaN                   0                NaN   \n",
      "3640     13803               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "12937                  NaN               NaN               NaN   \n",
      "7200                   NaN               NaN               NaN   \n",
      "11924                  NaN               NaN               NaN   \n",
      "7219                   NaN               NaN               NaN   \n",
      "3640                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "12937                   0                NaN                  NaN  ...   \n",
      "7200                    0                NaN                  NaN  ...   \n",
      "11924                   0                NaN                  NaN  ...   \n",
      "7219                    0                NaN                  NaN  ...   \n",
      "3640                    0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2002_cfc  NTL_Dcount2002_cfc  NTL_Dmean2002_cfc  \\\n",
      "12937               NaN                   0                NaN   \n",
      "7200                NaN                   0                NaN   \n",
      "11924               NaN                   0                NaN   \n",
      "7219                NaN                   0                NaN   \n",
      "3640                NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2002_cfc  NTL_Dstd2002_cfc  NTL_Dsum2003_cfc  \\\n",
      "12937                  NaN               NaN               NaN   \n",
      "7200                   NaN               NaN               NaN   \n",
      "11924                  NaN               NaN               NaN   \n",
      "7219                   NaN               NaN               NaN   \n",
      "3640                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2003_cfc  NTL_Dmean2003_cfc  NTL_Dmedian2003_cfc  \\\n",
      "12937                   0                NaN                  NaN   \n",
      "7200                    0                NaN                  NaN   \n",
      "11924                   0                NaN                  NaN   \n",
      "7219                    0                NaN                  NaN   \n",
      "3640                    0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2003_cfc  \n",
      "12937               NaN  \n",
      "7200                NaN  \n",
      "11924               NaN  \n",
      "7219                NaN  \n",
      "3640                NaN  \n",
      "\n",
      "[5 rows x 26 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:10:32 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2004_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 8  8  8 ...  8  8  8]\n",
      " [ 8  8  8 ...  8  8  8]\n",
      " [13 13 13 ... 13 13 13]\n",
      " ...\n",
      " [ 4  4  4 ...  4  4  4]\n",
      " [ 4  5  5 ...  4  4  4]\n",
      " [ 4  5  5 ...  4  4  4]]\n",
      "Zonal statistics. Thu Sep 14 18:10:36 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2004_cfc.tif: NTL_Dstd2004_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "610       2314               NaN                   0                NaN   \n",
      "16320   141992              42.0                   1               42.0   \n",
      "11121    59799               NaN                   0                NaN   \n",
      "250        985               NaN                   0                NaN   \n",
      "460       1743               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "610                    NaN               NaN               NaN   \n",
      "16320                 42.0               0.0              50.0   \n",
      "11121                  NaN               NaN               NaN   \n",
      "250                    NaN               NaN               NaN   \n",
      "460                    NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "610                     0                NaN                  NaN  ...   \n",
      "16320                   1               50.0                 50.0  ...   \n",
      "11121                   0                NaN                  NaN  ...   \n",
      "250                     0                NaN                  NaN  ...   \n",
      "460                     0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2003_cfc  NTL_Dcount2003_cfc  NTL_Dmean2003_cfc  \\\n",
      "610                 NaN                   0                NaN   \n",
      "16320              48.0                   1               48.0   \n",
      "11121               NaN                   0                NaN   \n",
      "250                 NaN                   0                NaN   \n",
      "460                 NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2003_cfc  NTL_Dstd2003_cfc  NTL_Dsum2004_cfc  \\\n",
      "610                    NaN               NaN               NaN   \n",
      "16320                 48.0               0.0              55.0   \n",
      "11121                  NaN               NaN               NaN   \n",
      "250                    NaN               NaN               NaN   \n",
      "460                    NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2004_cfc  NTL_Dmean2004_cfc  NTL_Dmedian2004_cfc  \\\n",
      "610                     0                NaN                  NaN   \n",
      "16320                   1               55.0                 55.0   \n",
      "11121                   0                NaN                  NaN   \n",
      "250                     0                NaN                  NaN   \n",
      "460                     0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2004_cfc  \n",
      "610                 NaN  \n",
      "16320               0.0  \n",
      "11121               NaN  \n",
      "250                 NaN  \n",
      "460                 NaN  \n",
      "\n",
      "[5 rows x 31 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:11:20 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2005_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0  0  0 ...  0  0  0]\n",
      " [13 13 13 ... 13 13 13]\n",
      " [24 24 24 ... 24 24 24]\n",
      " ...\n",
      " [ 3  3  2 ...  3  3  3]\n",
      " [ 3  3  3 ...  3  3  3]\n",
      " [ 3  3  3 ...  3  3  3]]\n",
      "Zonal statistics. Thu Sep 14 18:11:24 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2005_cfc.tif: NTL_Dstd2005_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "4324     16899               NaN                   0                NaN   \n",
      "11932    72412               NaN                   0                NaN   \n",
      "17444   154002               NaN                   0                NaN   \n",
      "12275    76010               NaN                   0                NaN   \n",
      "7947     39792               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "4324                   NaN               NaN               NaN   \n",
      "11932                  NaN               NaN               NaN   \n",
      "17444                  NaN               NaN               NaN   \n",
      "12275                  NaN               NaN               NaN   \n",
      "7947                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "4324                    0                NaN                  NaN  ...   \n",
      "11932                   0                NaN                  NaN  ...   \n",
      "17444                   0                NaN                  NaN  ...   \n",
      "12275                   0                NaN                  NaN  ...   \n",
      "7947                    0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2004_cfc  NTL_Dcount2004_cfc  NTL_Dmean2004_cfc  \\\n",
      "4324                NaN                   0                NaN   \n",
      "11932               NaN                   0                NaN   \n",
      "17444               NaN                   0                NaN   \n",
      "12275               NaN                   0                NaN   \n",
      "7947                NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2004_cfc  NTL_Dstd2004_cfc  NTL_Dsum2005_cfc  \\\n",
      "4324                   NaN               NaN               NaN   \n",
      "11932                  NaN               NaN               NaN   \n",
      "17444                  NaN               NaN               NaN   \n",
      "12275                  NaN               NaN               NaN   \n",
      "7947                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2005_cfc  NTL_Dmean2005_cfc  NTL_Dmedian2005_cfc  \\\n",
      "4324                    0                NaN                  NaN   \n",
      "11932                   0                NaN                  NaN   \n",
      "17444                   0                NaN                  NaN   \n",
      "12275                   0                NaN                  NaN   \n",
      "7947                    0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2005_cfc  \n",
      "4324                NaN  \n",
      "11932               NaN  \n",
      "17444               NaN  \n",
      "12275               NaN  \n",
      "7947                NaN  \n",
      "\n",
      "[5 rows x 36 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:12:07 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2006_cfc.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0 0 0 ... 0 0 0]\n",
      " [5 5 5 ... 5 5 5]\n",
      " [8 8 8 ... 8 8 8]\n",
      " ...\n",
      " [5 5 5 ... 5 5 5]\n",
      " [5 5 5 ... 5 5 5]\n",
      " [5 5 5 ... 5 5 5]]\n",
      "Zonal statistics. Thu Sep 14 18:12:12 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2006_cfc.tif: NTL_Dstd2006_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "7741     38533               NaN                   0                NaN   \n",
      "12276    76021               NaN                   0                NaN   \n",
      "1720      6348               NaN                   0                NaN   \n",
      "12165    74884               NaN                   0                NaN   \n",
      "4169     16240               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "7741                   NaN               NaN               NaN   \n",
      "12276                  NaN               NaN               NaN   \n",
      "1720                   NaN               NaN               NaN   \n",
      "12165                  NaN               NaN               NaN   \n",
      "4169                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "7741                    0                NaN                  NaN  ...   \n",
      "12276                   0                NaN                  NaN  ...   \n",
      "1720                    0                NaN                  NaN  ...   \n",
      "12165                   0                NaN                  NaN  ...   \n",
      "4169                    0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2005_cfc  NTL_Dcount2005_cfc  NTL_Dmean2005_cfc  \\\n",
      "7741                NaN                   0                NaN   \n",
      "12276               NaN                   0                NaN   \n",
      "1720                NaN                   0                NaN   \n",
      "12165               NaN                   0                NaN   \n",
      "4169                NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2005_cfc  NTL_Dstd2005_cfc  NTL_Dsum2006_cfc  \\\n",
      "7741                   NaN               NaN               NaN   \n",
      "12276                  NaN               NaN               NaN   \n",
      "1720                   NaN               NaN               NaN   \n",
      "12165                  NaN               NaN               NaN   \n",
      "4169                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2006_cfc  NTL_Dmean2006_cfc  NTL_Dmedian2006_cfc  \\\n",
      "7741                    0                NaN                  NaN   \n",
      "12276                   0                NaN                  NaN   \n",
      "1720                    0                NaN                  NaN   \n",
      "12165                   0                NaN                  NaN   \n",
      "4169                    0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2006_cfc  \n",
      "7741                NaN  \n",
      "12276               NaN  \n",
      "1720                NaN  \n",
      "12165               NaN  \n",
      "4169                NaN  \n",
      "\n",
      "[5 rows x 41 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:12:55 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2007_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 0  0  0 ...  0  0  0]\n",
      " [12 12 12 ... 12 12 12]\n",
      " [19 19 19 ... 19 19 19]\n",
      " ...\n",
      " [ 4  4  4 ...  4  4  4]\n",
      " [ 4  4  4 ...  4  4  4]\n",
      " [ 4  4  4 ...  4  4  4]]\n",
      "Zonal statistics. Thu Sep 14 18:12:59 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2007_cfc.tif: NTL_Dstd2007_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "9509     50559               NaN                   0                NaN   \n",
      "3002     10891               NaN                   0                NaN   \n",
      "17824   155532               NaN                   0                NaN   \n",
      "15076   118589               NaN                   0                NaN   \n",
      "2957     10666               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "9509                   NaN               NaN               NaN   \n",
      "3002                   NaN               NaN               NaN   \n",
      "17824                  NaN               NaN               NaN   \n",
      "15076                  NaN               NaN               NaN   \n",
      "2957                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "9509                    0                NaN                  NaN  ...   \n",
      "3002                    0                NaN                  NaN  ...   \n",
      "17824                   0                NaN                  NaN  ...   \n",
      "15076                   0                NaN                  NaN  ...   \n",
      "2957                    0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2006_cfc  NTL_Dcount2006_cfc  NTL_Dmean2006_cfc  \\\n",
      "9509                NaN                   0                NaN   \n",
      "3002                NaN                   0                NaN   \n",
      "17824               NaN                   0                NaN   \n",
      "15076               NaN                   0                NaN   \n",
      "2957                NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2006_cfc  NTL_Dstd2006_cfc  NTL_Dsum2007_cfc  \\\n",
      "9509                   NaN               NaN               NaN   \n",
      "3002                   NaN               NaN               NaN   \n",
      "17824                  NaN               NaN               NaN   \n",
      "15076                  NaN               NaN               NaN   \n",
      "2957                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2007_cfc  NTL_Dmean2007_cfc  NTL_Dmedian2007_cfc  \\\n",
      "9509                    0                NaN                  NaN   \n",
      "3002                    0                NaN                  NaN   \n",
      "17824                   0                NaN                  NaN   \n",
      "15076                   0                NaN                  NaN   \n",
      "2957                    0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2007_cfc  \n",
      "9509                NaN  \n",
      "3002                NaN  \n",
      "17824               NaN  \n",
      "15076               NaN  \n",
      "2957                NaN  \n",
      "\n",
      "[5 rows x 46 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:13:42 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2008_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]]\n",
      "Zonal statistics. Thu Sep 14 18:13:47 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2008_cfc.tif: NTL_Dstd2008_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "13679    98587               NaN                   0                NaN   \n",
      "3052     11095               NaN                   0                NaN   \n",
      "4270     16670               NaN                   0                NaN   \n",
      "9892     52710               NaN                   0                NaN   \n",
      "14548   106971               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "13679                  NaN               NaN               NaN   \n",
      "3052                   NaN               NaN               NaN   \n",
      "4270                   NaN               NaN               NaN   \n",
      "9892                   NaN               NaN               NaN   \n",
      "14548                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "13679                   0                NaN                  NaN  ...   \n",
      "3052                    0                NaN                  NaN  ...   \n",
      "4270                    0                NaN                  NaN  ...   \n",
      "9892                    0                NaN                  NaN  ...   \n",
      "14548                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2007_cfc  NTL_Dcount2007_cfc  NTL_Dmean2007_cfc  \\\n",
      "13679               NaN                   0                NaN   \n",
      "3052                NaN                   0                NaN   \n",
      "4270                NaN                   0                NaN   \n",
      "9892                NaN                   0                NaN   \n",
      "14548               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2007_cfc  NTL_Dstd2007_cfc  NTL_Dsum2008_cfc  \\\n",
      "13679                  NaN               NaN               NaN   \n",
      "3052                   NaN               NaN               NaN   \n",
      "4270                   NaN               NaN               NaN   \n",
      "9892                   NaN               NaN               NaN   \n",
      "14548                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2008_cfc  NTL_Dmean2008_cfc  NTL_Dmedian2008_cfc  \\\n",
      "13679                   0                NaN                  NaN   \n",
      "3052                    0                NaN                  NaN   \n",
      "4270                    0                NaN                  NaN   \n",
      "9892                    0                NaN                  NaN   \n",
      "14548                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2008_cfc  \n",
      "13679               NaN  \n",
      "3052                NaN  \n",
      "4270                NaN  \n",
      "9892                NaN  \n",
      "14548               NaN  \n",
      "\n",
      "[5 rows x 51 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:14:30 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2009_cfc.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "Zonal statistics. Thu Sep 14 18:14:34 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2009_cfc.tif: NTL_Dstd2009_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "11985    72878               NaN                   0                NaN   \n",
      "9296     49381               NaN                   0                NaN   \n",
      "11384    61232               NaN                   0                NaN   \n",
      "5239     22546               NaN                   0                NaN   \n",
      "3028     10996               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "11985                  NaN               NaN               NaN   \n",
      "9296                   NaN               NaN               NaN   \n",
      "11384                  NaN               NaN               NaN   \n",
      "5239                   NaN               NaN               NaN   \n",
      "3028                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "11985                   0                NaN                  NaN  ...   \n",
      "9296                    0                NaN                  NaN  ...   \n",
      "11384                   0                NaN                  NaN  ...   \n",
      "5239                    0                NaN                  NaN  ...   \n",
      "3028                    0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2008_cfc  NTL_Dcount2008_cfc  NTL_Dmean2008_cfc  \\\n",
      "11985               NaN                   0                NaN   \n",
      "9296                NaN                   0                NaN   \n",
      "11384               NaN                   0                NaN   \n",
      "5239                NaN                   0                NaN   \n",
      "3028                NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2008_cfc  NTL_Dstd2008_cfc  NTL_Dsum2009_cfc  \\\n",
      "11985                  NaN               NaN               NaN   \n",
      "9296                   NaN               NaN               NaN   \n",
      "11384                  NaN               NaN               NaN   \n",
      "5239                   NaN               NaN               NaN   \n",
      "3028                   NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2009_cfc  NTL_Dmean2009_cfc  NTL_Dmedian2009_cfc  \\\n",
      "11985                   0                NaN                  NaN   \n",
      "9296                    0                NaN                  NaN   \n",
      "11384                   0                NaN                  NaN   \n",
      "5239                    0                NaN                  NaN   \n",
      "3028                    0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2009_cfc  \n",
      "11985               NaN  \n",
      "9296                NaN  \n",
      "11384               NaN  \n",
      "5239                NaN  \n",
      "3028                NaN  \n",
      "\n",
      "[5 rows x 56 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:15:17 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2010_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]\n",
      " [1 1 1 ... 1 1 1]]\n",
      "Zonal statistics. Thu Sep 14 18:15:20 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2010_cfc.tif: NTL_Dstd2010_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "11774    70810               NaN                   0                NaN   \n",
      "10493    56222               NaN                   0                NaN   \n",
      "8101     40737               NaN                   0                NaN   \n",
      "10379    55475               NaN                   0                NaN   \n",
      "10733    57600               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "11774                  NaN               NaN               NaN   \n",
      "10493                  NaN               NaN               NaN   \n",
      "8101                   NaN               NaN               NaN   \n",
      "10379                  NaN               NaN               NaN   \n",
      "10733                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "11774                   0                NaN                  NaN  ...   \n",
      "10493                   0                NaN                  NaN  ...   \n",
      "8101                    0                NaN                  NaN  ...   \n",
      "10379                   0                NaN                  NaN  ...   \n",
      "10733                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2009_cfc  NTL_Dcount2009_cfc  NTL_Dmean2009_cfc  \\\n",
      "11774               NaN                   0                NaN   \n",
      "10493               NaN                   0                NaN   \n",
      "8101                NaN                   0                NaN   \n",
      "10379               NaN                   0                NaN   \n",
      "10733               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2009_cfc  NTL_Dstd2009_cfc  NTL_Dsum2010_cfc  \\\n",
      "11774                  NaN               NaN               NaN   \n",
      "10493                  NaN               NaN               NaN   \n",
      "8101                   NaN               NaN               NaN   \n",
      "10379                  NaN               NaN               NaN   \n",
      "10733                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2010_cfc  NTL_Dmean2010_cfc  NTL_Dmedian2010_cfc  \\\n",
      "11774                   0                NaN                  NaN   \n",
      "10493                   0                NaN                  NaN   \n",
      "8101                    0                NaN                  NaN   \n",
      "10379                   0                NaN                  NaN   \n",
      "10733                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2010_cfc  \n",
      "11774               NaN  \n",
      "10493               NaN  \n",
      "8101                NaN  \n",
      "10379               NaN  \n",
      "10733               NaN  \n",
      "\n",
      "[5 rows x 61 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:16:03 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2011_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[8 8 8 ... 8 8 8]\n",
      " [8 8 8 ... 8 8 8]\n",
      " [8 8 8 ... 8 8 8]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "Zonal statistics. Thu Sep 14 18:16:07 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2011_cfc.tif: NTL_Dstd2011_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "7619     37770               NaN                   0                NaN   \n",
      "10520    56365               NaN                   0                NaN   \n",
      "5980     27502               NaN                   0                NaN   \n",
      "11976    72772               NaN                   0                NaN   \n",
      "16568   143508               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "7619                   NaN               NaN               NaN   \n",
      "10520                  NaN               NaN               NaN   \n",
      "5980                   NaN               NaN               NaN   \n",
      "11976                  NaN               NaN               NaN   \n",
      "16568                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "7619                    0                NaN                  NaN  ...   \n",
      "10520                   0                NaN                  NaN  ...   \n",
      "5980                    0                NaN                  NaN  ...   \n",
      "11976                   0                NaN                  NaN  ...   \n",
      "16568                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2010_cfc  NTL_Dcount2010_cfc  NTL_Dmean2010_cfc  \\\n",
      "7619                NaN                   0                NaN   \n",
      "10520               NaN                   0                NaN   \n",
      "5980                NaN                   0                NaN   \n",
      "11976               NaN                   0                NaN   \n",
      "16568               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2010_cfc  NTL_Dstd2010_cfc  NTL_Dsum2011_cfc  \\\n",
      "7619                   NaN               NaN               NaN   \n",
      "10520                  NaN               NaN               NaN   \n",
      "5980                   NaN               NaN               NaN   \n",
      "11976                  NaN               NaN               NaN   \n",
      "16568                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2011_cfc  NTL_Dmean2011_cfc  NTL_Dmedian2011_cfc  \\\n",
      "7619                    0                NaN                  NaN   \n",
      "10520                   0                NaN                  NaN   \n",
      "5980                    0                NaN                  NaN   \n",
      "11976                   0                NaN                  NaN   \n",
      "16568                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2011_cfc  \n",
      "7619                NaN  \n",
      "10520               NaN  \n",
      "5980                NaN  \n",
      "11976               NaN  \n",
      "16568               NaN  \n",
      "\n",
      "[5 rows x 66 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:16:50 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2012_cfc.tif\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[4 4 4 ... 4 4 4]\n",
      " [4 4 4 ... 4 4 4]\n",
      " [4 4 4 ... 4 4 4]\n",
      " ...\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n",
      "Zonal statistics. Thu Sep 14 18:16:54 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2012_cfc.tif: NTL_Dstd2012_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "10484    56175               NaN                   0                NaN   \n",
      "7811     38993               NaN                   0                NaN   \n",
      "16925   149603               NaN                   0                NaN   \n",
      "14870   112773               NaN                   0                NaN   \n",
      "14760   109581               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "10484                  NaN               NaN               NaN   \n",
      "7811                   NaN               NaN               NaN   \n",
      "16925                  NaN               NaN               NaN   \n",
      "14870                  NaN               NaN               NaN   \n",
      "14760                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "10484                   0                NaN                  NaN  ...   \n",
      "7811                    0                NaN                  NaN  ...   \n",
      "16925                   0                NaN                  NaN  ...   \n",
      "14870                   0                NaN                  NaN  ...   \n",
      "14760                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2011_cfc  NTL_Dcount2011_cfc  NTL_Dmean2011_cfc  \\\n",
      "10484               NaN                   0                NaN   \n",
      "7811                NaN                   0                NaN   \n",
      "16925               NaN                   0                NaN   \n",
      "14870               NaN                   0                NaN   \n",
      "14760               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2011_cfc  NTL_Dstd2011_cfc  NTL_Dsum2012_cfc  \\\n",
      "10484                  NaN               NaN               NaN   \n",
      "7811                   NaN               NaN               NaN   \n",
      "16925                  NaN               NaN               NaN   \n",
      "14870                  NaN               NaN               NaN   \n",
      "14760                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2012_cfc  NTL_Dmean2012_cfc  NTL_Dmedian2012_cfc  \\\n",
      "10484                   0                NaN                  NaN   \n",
      "7811                    0                NaN                  NaN   \n",
      "16925                   0                NaN                  NaN   \n",
      "14870                   0                NaN                  NaN   \n",
      "14760                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2012_cfc  \n",
      "10484               NaN  \n",
      "7811                NaN  \n",
      "16925               NaN  \n",
      "14870               NaN  \n",
      "14760               NaN  \n",
      "\n",
      "[5 rows x 71 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:17:37 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2013_cfc.tif\n",
      "| 0.01, 0.00,-180.00|\n",
      "| 0.00,-0.01, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[24 24 24 ... 24 24 24]\n",
      " [24 24 24 ... 24 24 24]\n",
      " [24 24 24 ... 24 24 24]\n",
      " ...\n",
      " [ 3  3  3 ...  3  3  3]\n",
      " [ 3  3  3 ...  3  3  3]\n",
      " [ 0  0  0 ...  0  0  0]]\n",
      "Zonal statistics. Thu Sep 14 18:17:41 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\D_2013_cfc.tif: NTL_Dstd2013_cfc\n",
      "       Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "7842     39158               NaN                   0                NaN   \n",
      "12711    80738               NaN                   0                NaN   \n",
      "4413     17450               NaN                   0                NaN   \n",
      "13271    90635               NaN                   0                NaN   \n",
      "17104   152693               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "7842                   NaN               NaN               NaN   \n",
      "12711                  NaN               NaN               NaN   \n",
      "4413                   NaN               NaN               NaN   \n",
      "13271                  NaN               NaN               NaN   \n",
      "17104                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "7842                    0                NaN                  NaN  ...   \n",
      "12711                   0                NaN                  NaN  ...   \n",
      "4413                    0                NaN                  NaN  ...   \n",
      "13271                   0                NaN                  NaN  ...   \n",
      "17104                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2012_cfc  NTL_Dcount2012_cfc  NTL_Dmean2012_cfc  \\\n",
      "7842                NaN                   0                NaN   \n",
      "12711               NaN                   0                NaN   \n",
      "4413                NaN                   0                NaN   \n",
      "13271               NaN                   0                NaN   \n",
      "17104               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2012_cfc  NTL_Dstd2012_cfc  NTL_Dsum2013_cfc  \\\n",
      "7842                   NaN               NaN               NaN   \n",
      "12711                  NaN               NaN               NaN   \n",
      "4413                   NaN               NaN               NaN   \n",
      "13271                  NaN               NaN               NaN   \n",
      "17104                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2013_cfc  NTL_Dmean2013_cfc  NTL_Dmedian2013_cfc  \\\n",
      "7842                    0                NaN                  NaN   \n",
      "12711                   0                NaN                  NaN   \n",
      "4413                    0                NaN                  NaN   \n",
      "13271                   0                NaN                  NaN   \n",
      "17104                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2013_cfc  \n",
      "7842                NaN  \n",
      "12711               NaN  \n",
      "4413                NaN  \n",
      "13271               NaN  \n",
      "17104               NaN  \n",
      "\n",
      "[5 rows x 76 columns]\n",
      "Final dataframe: \n",
      "        Sett_ID  NTL_Dsum1999_cfc  NTL_Dcount1999_cfc  NTL_Dmean1999_cfc  \\\n",
      "11080    59587               NaN                   0                NaN   \n",
      "10963    58905               NaN                   0                NaN   \n",
      "6503     30790               NaN                   0                NaN   \n",
      "3902     15070               NaN                   0                NaN   \n",
      "15944   137604               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian1999_cfc  NTL_Dstd1999_cfc  NTL_Dsum2000_cfc  \\\n",
      "11080                  NaN               NaN               NaN   \n",
      "10963                  NaN               NaN               NaN   \n",
      "6503                   NaN               NaN               NaN   \n",
      "3902                   NaN               NaN               NaN   \n",
      "15944                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2000_cfc  NTL_Dmean2000_cfc  NTL_Dmedian2000_cfc  ...  \\\n",
      "11080                   0                NaN                  NaN  ...   \n",
      "10963                   0                NaN                  NaN  ...   \n",
      "6503                    0                NaN                  NaN  ...   \n",
      "3902                    0                NaN                  NaN  ...   \n",
      "15944                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Dsum2012_cfc  NTL_Dcount2012_cfc  NTL_Dmean2012_cfc  \\\n",
      "11080               NaN                   0                NaN   \n",
      "10963               NaN                   0                NaN   \n",
      "6503                NaN                   0                NaN   \n",
      "3902                NaN                   0                NaN   \n",
      "15944               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Dmedian2012_cfc  NTL_Dstd2012_cfc  NTL_Dsum2013_cfc  \\\n",
      "11080                  NaN               NaN               NaN   \n",
      "10963                  NaN               NaN               NaN   \n",
      "6503                   NaN               NaN               NaN   \n",
      "3902                   NaN               NaN               NaN   \n",
      "15944                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Dcount2013_cfc  NTL_Dmean2013_cfc  NTL_Dmedian2013_cfc  \\\n",
      "11080                   0                NaN                  NaN   \n",
      "10963                   0                NaN                  NaN   \n",
      "6503                    0                NaN                  NaN   \n",
      "3902                    0                NaN                  NaN   \n",
      "15944                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Dstd2013_cfc  \n",
      "11080               NaN  \n",
      "10963               NaN  \n",
      "6503                NaN  \n",
      "3902                NaN  \n",
      "15944               NaN  \n",
      "\n",
      "[5 rows x 76 columns]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe to merge with: \n",
      "        Sett_ID\n",
      "0            3\n",
      "1            7\n",
      "2           14\n",
      "3           17\n",
      "4           21\n",
      "...        ...\n",
      "18547   157067\n",
      "18548   157070\n",
      "18549   157072\n",
      "18550   157073\n",
      "18551   157079\n",
      "\n",
      "[18552 rows x 1 columns]\n",
      "Loading with rasterio. Thu Sep 14 18:18:24 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2012_cfc.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[42 40 40 ... 42 42 28]\n",
      " [40 40 40 ... 41 41 27]\n",
      " [40 40 40 ... 41 41 27]\n",
      " ...\n",
      " [58 61 63 ... 61 58 54]\n",
      " [59 64 66 ... 59 61 55]\n",
      " [ 0  0  0 ...  0  0  0]]\n",
      "Zonal statistics. Thu Sep 14 18:18:40 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2012_cfc.tif: NTL_Vstd2012_cfc\n",
      "       Sett_ID  NTL_Vsum2012_cfc  NTL_Vcount2012_cfc  NTL_Vmean2012_cfc  \\\n",
      "9074     48169               NaN                   0                NaN   \n",
      "12564    79308               NaN                   0                NaN   \n",
      "9365     49718               NaN                   0                NaN   \n",
      "9680     51532               NaN                   0                NaN   \n",
      "13898   101261              94.0                   1               94.0   \n",
      "\n",
      "       NTL_Vmedian2012_cfc  NTL_Vstd2012_cfc  \n",
      "9074                   NaN               NaN  \n",
      "12564                  NaN               NaN  \n",
      "9365                   NaN               NaN  \n",
      "9680                   NaN               NaN  \n",
      "13898                 94.0               0.0  \n",
      "Loading with rasterio. Thu Sep 14 18:19:24 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2013_cfc.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[ 75  75  76 ...  74  75  55]\n",
      " [ 74  75  75 ...  74  74  54]\n",
      " [ 74  74  74 ...  74  74  54]\n",
      " ...\n",
      " [107 106 105 ... 106 108 102]\n",
      " [114 115 108 ... 108 112 109]\n",
      " [  0   0   0 ...   0   0   0]]\n",
      "Zonal statistics. Thu Sep 14 18:19:42 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2013_cfc.tif: NTL_Vstd2013_cfc\n",
      "       Sett_ID  NTL_Vsum2012_cfc  NTL_Vcount2012_cfc  NTL_Vmean2012_cfc  \\\n",
      "17901   155701              87.0                   1               87.0   \n",
      "14960   115187               NaN                   0                NaN   \n",
      "12857    82493               NaN                   0                NaN   \n",
      "8643     45358               NaN                   0                NaN   \n",
      "14379   105475               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Vmedian2012_cfc  NTL_Vstd2012_cfc  NTL_Vsum2013_cfc  \\\n",
      "17901                 87.0               0.0             127.0   \n",
      "14960                  NaN               NaN               NaN   \n",
      "12857                  NaN               NaN               NaN   \n",
      "8643                   NaN               NaN               NaN   \n",
      "14379                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Vcount2013_cfc  NTL_Vmean2013_cfc  NTL_Vmedian2013_cfc  \\\n",
      "17901                   1              127.0                127.0   \n",
      "14960                   0                NaN                  NaN   \n",
      "12857                   0                NaN                  NaN   \n",
      "8643                    0                NaN                  NaN   \n",
      "14379                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Vstd2013_cfc  \n",
      "17901               0.0  \n",
      "14960               NaN  \n",
      "12857               NaN  \n",
      "8643                NaN  \n",
      "14379               NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:20:25 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2014_cfc.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[252 253 252 ... 249 249 131]\n",
      " [253 254 252 ... 251 252 133]\n",
      " [249 248 247 ... 247 248 131]\n",
      " ...\n",
      " [124 130 131 ... 124 124 119]\n",
      " [132 142 142 ... 126 129 127]\n",
      " [  0   0   0 ...   0   0   0]]\n",
      "Zonal statistics. Thu Sep 14 18:20:42 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2014_cfc.tif: NTL_Vstd2014_cfc\n",
      "       Sett_ID  NTL_Vsum2012_cfc  NTL_Vcount2012_cfc  NTL_Vmean2012_cfc  \\\n",
      "4048     15737             274.0                   3          91.333333   \n",
      "6800     32541             124.0                   1         124.000000   \n",
      "5924     27111              87.0                   1          87.000000   \n",
      "16189   138775               NaN                   0                NaN   \n",
      "13443    95561               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Vmedian2012_cfc  NTL_Vstd2012_cfc  NTL_Vsum2013_cfc  \\\n",
      "4048                  90.0          2.624669             394.0   \n",
      "6800                 124.0          0.000000             176.0   \n",
      "5924                  87.0          0.000000             127.0   \n",
      "16189                  NaN               NaN               NaN   \n",
      "13443                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Vcount2013_cfc  NTL_Vmean2013_cfc  NTL_Vmedian2013_cfc  \\\n",
      "4048                    3         131.333333                132.0   \n",
      "6800                    1         176.000000                176.0   \n",
      "5924                    1         127.000000                127.0   \n",
      "16189                   0                NaN                  NaN   \n",
      "13443                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Vstd2013_cfc  NTL_Vsum2014_cfc  NTL_Vcount2014_cfc  \\\n",
      "4048           1.699673             511.0                   3   \n",
      "6800           0.000000             202.0                   1   \n",
      "5924           0.000000             177.0                   1   \n",
      "16189               NaN               NaN                   0   \n",
      "13443               NaN               NaN                   0   \n",
      "\n",
      "       NTL_Vmean2014_cfc  NTL_Vmedian2014_cfc  NTL_Vstd2014_cfc  \n",
      "4048          170.333333                171.0          0.942809  \n",
      "6800          202.000000                202.0          0.000000  \n",
      "5924          177.000000                177.0          0.000000  \n",
      "16189                NaN                  NaN               NaN  \n",
      "13443                NaN                  NaN               NaN  \n",
      "Loading with rasterio. Thu Sep 14 18:21:26 2023\n",
      "Raster: Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2015_cfc.tif\n",
      "| 0.00, 0.00,-180.00|\n",
      "| 0.00,-0.00, 75.00|\n",
      "| 0.00, 0.00, 1.00|\n",
      "[[270 267 268 ... 271 271 148]\n",
      " [269 266 265 ... 269 268 148]\n",
      " [267 263 264 ... 268 267 146]\n",
      " ...\n",
      " [ 94  99  97 ...  95  93  87]\n",
      " [ 93  98  94 ...  91  91  86]\n",
      " [  0   0   0 ...   0   0   0]]\n",
      "Zonal statistics. Thu Sep 14 18:21:43 2023\n",
      "std stat output field for Q:\\GIS\\povertyequity\\urban_growth\\NighttimeLights_VIIRS_DMSP\\Temp\\V_2015_cfc.tif: NTL_Vstd2015_cfc\n",
      "       Sett_ID  NTL_Vsum2012_cfc  NTL_Vcount2012_cfc  NTL_Vmean2012_cfc  \\\n",
      "6471     30554               NaN                   0                NaN   \n",
      "1192      4488               NaN                   0                NaN   \n",
      "7151     34427               NaN                   0                NaN   \n",
      "1272      4752               NaN                   0                NaN   \n",
      "16443   143226               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Vmedian2012_cfc  NTL_Vstd2012_cfc  NTL_Vsum2013_cfc  \\\n",
      "6471                   NaN               NaN               NaN   \n",
      "1192                   NaN               NaN               NaN   \n",
      "7151                   NaN               NaN               NaN   \n",
      "1272                   NaN               NaN               NaN   \n",
      "16443                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Vcount2013_cfc  NTL_Vmean2013_cfc  NTL_Vmedian2013_cfc  ...  \\\n",
      "6471                    0                NaN                  NaN  ...   \n",
      "1192                    0                NaN                  NaN  ...   \n",
      "7151                    0                NaN                  NaN  ...   \n",
      "1272                    0                NaN                  NaN  ...   \n",
      "16443                   0                NaN                  NaN  ...   \n",
      "\n",
      "       NTL_Vsum2014_cfc  NTL_Vcount2014_cfc  NTL_Vmean2014_cfc  \\\n",
      "6471                NaN                   0                NaN   \n",
      "1192                NaN                   0                NaN   \n",
      "7151                NaN                   0                NaN   \n",
      "1272                NaN                   0                NaN   \n",
      "16443               NaN                   0                NaN   \n",
      "\n",
      "       NTL_Vmedian2014_cfc  NTL_Vstd2014_cfc  NTL_Vsum2015_cfc  \\\n",
      "6471                   NaN               NaN               NaN   \n",
      "1192                   NaN               NaN               NaN   \n",
      "7151                   NaN               NaN               NaN   \n",
      "1272                   NaN               NaN               NaN   \n",
      "16443                  NaN               NaN               NaN   \n",
      "\n",
      "       NTL_Vcount2015_cfc  NTL_Vmean2015_cfc  NTL_Vmedian2015_cfc  \\\n",
      "6471                    0                NaN                  NaN   \n",
      "1192                    0                NaN                  NaN   \n",
      "7151                    0                NaN                  NaN   \n",
      "1272                    0                NaN                  NaN   \n",
      "16443                   0                NaN                  NaN   \n",
      "\n",
      "       NTL_Vstd2015_cfc  \n",
      "6471                NaN  \n",
      "1192                NaN  \n",
      "7151                NaN  \n",
      "1272                NaN  \n",
      "16443               NaN  \n",
      "\n",
      "[5 rows x 21 columns]\n",
      "Final dataframe: \n",
      "        Sett_ID  NTL_Vsum2012_cfc  NTL_Vcount2012_cfc  NTL_Vmean2012_cfc  \\\n",
      "16076   138155             888.0                   9          98.666667   \n",
      "6475     30580               NaN                   0                NaN   \n",
      "18130   156218               NaN                   0                NaN   \n",
      "13137    87098               NaN                   0                NaN   \n",
      "14685   108243              89.0                   1          89.000000   \n",
      "\n",
      "       NTL_Vmedian2012_cfc  NTL_Vstd2012_cfc  NTL_Vsum2013_cfc  \\\n",
      "16076                 99.0          1.699673            1301.0   \n",
      "6475                   NaN               NaN               NaN   \n",
      "18130                  NaN               NaN               NaN   \n",
      "13137                  NaN               NaN               NaN   \n",
      "14685                 89.0          0.000000             136.0   \n",
      "\n",
      "       NTL_Vcount2013_cfc  NTL_Vmean2013_cfc  NTL_Vmedian2013_cfc  ...  \\\n",
      "16076                   9         144.555556                144.0  ...   \n",
      "6475                    0                NaN                  NaN  ...   \n",
      "18130                   0                NaN                  NaN  ...   \n",
      "13137                   0                NaN                  NaN  ...   \n",
      "14685                   1         136.000000                136.0  ...   \n",
      "\n",
      "       NTL_Vsum2014_cfc  NTL_Vcount2014_cfc  NTL_Vmean2014_cfc  \\\n",
      "16076            1762.0                   9         195.777778   \n",
      "6475                NaN                   0                NaN   \n",
      "18130               NaN                   0                NaN   \n",
      "13137               NaN                   0                NaN   \n",
      "14685             185.0                   1         185.000000   \n",
      "\n",
      "       NTL_Vmedian2014_cfc  NTL_Vstd2014_cfc  NTL_Vsum2015_cfc  \\\n",
      "16076                196.0          2.148787            1593.0   \n",
      "6475                   NaN               NaN               NaN   \n",
      "18130                  NaN               NaN               NaN   \n",
      "13137                  NaN               NaN               NaN   \n",
      "14685                185.0          0.000000             181.0   \n",
      "\n",
      "       NTL_Vcount2015_cfc  NTL_Vmean2015_cfc  NTL_Vmedian2015_cfc  \\\n",
      "16076                   9              177.0                178.0   \n",
      "6475                    0                NaN                  NaN   \n",
      "18130                   0                NaN                  NaN   \n",
      "13137                   0                NaN                  NaN   \n",
      "14685                   1              181.0                181.0   \n",
      "\n",
      "       NTL_Vstd2015_cfc  \n",
      "16076          4.136558  \n",
      "6475                NaN  \n",
      "18130               NaN  \n",
      "13137               NaN  \n",
      "14685          0.000000  \n",
      "\n",
      "[5 rows x 21 columns]\n"
     ]
    }
   ],
   "source": [
    "BatchZonal(RasterFileList= D_cfc, Zones = Settlements, \n",
    "           KeepFields = ['Sett_ID'], \n",
    "           RasterDirectory = NTL, \n",
    "           OutPath = os.path.join(Results, 'NTL_DMSPcfc.csv'), \n",
    "           Statistics=['sum', 'count', 'mean', 'median', 'std'], \n",
    "           NoDataVal = -99999, \n",
    "           Prefix = 'NTL_D',\n",
    "           Suffix = '_cfc', \n",
    "           DropStatName = False)\n",
    "\n",
    "BatchZonal(RasterFileList= V_cfc, Zones = Settlements, \n",
    "           KeepFields = ['Sett_ID'], \n",
    "           RasterDirectory = NTL, \n",
    "           OutPath = os.path.join(Results, 'NTL_VIIRScfc.csv'), \n",
    "           Statistics=['sum', 'count', 'mean', 'median', 'std'], \n",
    "           NoDataVal = -99999,\n",
    "           Prefix = 'NTL_V',\n",
    "           Suffix = '_cfc', \n",
    "           DropStatName = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b80ab1eb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d3dbfb",
   "metadata": {},
   "source": [
    "## 10. FLOOD EXPOSURE BY RETURN PERIOD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07bb0569",
   "metadata": {},
   "source": [
    "### 10.1 Calculate Expected Annual Depth (EAD) using exceedance probabilities of every flood return period."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69ccc6d",
   "metadata": {},
   "source": [
    "##### Flood layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c15030b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "FloodFolder = os.path.join(Source, 'Flood')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d39a6aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "InRasters = os.listdir(FloodFolder)\n",
    "InRasters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86401134",
   "metadata": {},
   "outputs": [],
   "source": [
    "Exceedances = []\n",
    "    \n",
    "for Raster in InRasters:\n",
    "    InPath = os.path.join(SourceFolder, Raster)\n",
    "    RP = re.sub('\\D', '', Raster)[1:] # Get the return period\n",
    "    NewFileName = Raster.replace('.tif', '_EXC.tif')\n",
    "    OutPath = os.path.join(FloodFolder, NewFileName)\n",
    "    \n",
    "    Calc = \"(1/\" + RP + \")*A\"\n",
    "\n",
    "    calcShell(A=InPath, OutFile=OutPath, Calculation = Calc)\n",
    "    Exceedances = Exceedances + [NewFileName]\n",
    "    \n",
    "print('Done with list. New flood set: %s' % Exceedances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e4a35a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gdal_calc doesn't always take well to adding together a large number of files, so we'll do it in 2 batches.\n",
    "\n",
    "Calc = 'A+B+C+D+E'\n",
    "OutName = 'Batch1.tif'\n",
    "\n",
    "A = os.path.join(FloodFolder, Exceedances[0])\n",
    "B = os.path.join(FloodFolder, Exceedances[1])\n",
    "C = os.path.join(FloodFolder, Exceedances[2])\n",
    "D = os.path.join(FloodFolder, Exceedances[3])\n",
    "E = os.path.join(FloodFolder, Exceedances[4])\n",
    "\n",
    "calcShell(A=A, B=B, C=C, D=D, E=E,\n",
    "          OutFile = os.path.join(FloodFolder, OutName), \n",
    "          Calculation = Calc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5302d45",
   "metadata": {},
   "outputs": [],
   "source": [
    "Calc = 'A+B+C+D+E+F'\n",
    "OutName = 'FU_ExpectedAnnualDepth.tif'\n",
    "\n",
    "A = os.path.join(FloodFolder, 'Batch1.tif')\n",
    "B = os.path.join(FloodFolder, Exceedances[5])\n",
    "C = os.path.join(FloodFolder, Exceedances[6])\n",
    "D = os.path.join(FloodFolder, Exceedances[7])\n",
    "E = os.path.join(FloodFolder, Exceedances[8])\n",
    "F = os.path.join(FloodFolder, Exceedances[9])\n",
    "\n",
    "calcShell(A=A, B=B, C=C, D=D, E=E, F=F,\n",
    "          OutFile = os.path.join(FloodFolder, OutName), \n",
    "          Calculation = Calc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0692b6",
   "metadata": {},
   "source": [
    "### 10.2 Reclassify and resample flood data and buildup data in preparation for the impact calculation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aba2c398",
   "metadata": {},
   "source": [
    "##### Reclassify flood as a binary: flooded / not-flooded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c88d7b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "InPath = os.path.join(FloodFolder, OutName)\n",
    "OutPath = os.path.join(FloodFolder, 'FU_EAD_reclassed.tif')\n",
    "\n",
    "[xsize, ysize, geotransform, geoproj, Z] = readRaster(InPath)\n",
    "\n",
    "Z[Z<0.15] = 0 # Not-flooded category. This includes no data cells.\n",
    "Z[Z>=0.15] = 1 # Flooded category. This includes permanent water bodies.\n",
    "\n",
    "writeRaster(OutPath,geotransform,geoproj,Z)\n",
    "InPath = OutPath = None\n",
    "\n",
    "print('Finished reclassifying. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f04af5e4",
   "metadata": {},
   "source": [
    "##### Buildup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eebf3f5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "WSFE = 'WSFE_equalarea.tif'\n",
    "WSFEPath = os.path.join(Workspace, 'Buildup', WSFE)\n",
    "OutPath = os.path.join(FloodFolder, WSFE.replace('equalarea.tif', 'simplified.tif'))\n",
    "\n",
    "[xsize, ysize, geotransform, geoproj, Z] = readRaster(WSFEPath)\n",
    "\n",
    "np.putmask(Z, Z>0, Z-1984) # All years now converted to at most 2 digits: 1-31. (All non-buildup = 0)\n",
    "\n",
    "writeRaster(OutPath,geotransform,geoproj,Z)\n",
    "\n",
    "print('\\nSimplified buildup file: %s' % OutPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "374df178",
   "metadata": {},
   "source": [
    "##### Resample flood to match buildup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c548886c",
   "metadata": {},
   "outputs": [],
   "source": [
    "WSFEPath = os.path.join(FloodFolder, 'WSFE_simplified.tif') \n",
    "\n",
    "RasterPath = os.path.join(FloodFolder, 'FU_EAD_reclassed.tif')\n",
    "OutPath = os.path.join(FloodFolder, 'FU_EAD_resampled.tif')\n",
    "resampleRaster(RasterPath, WSFEPath, OutPath)\n",
    "    \n",
    "print('Resampled to match WSFE. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dad4e00",
   "metadata": {},
   "source": [
    "### 10.3 Mask out built areas that were not flooded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6026a3ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# WSFEPath = os.path.join(FloodFolder, 'WSFE_simplified.tif') \n",
    "    \n",
    "InPath = os.path.join(FloodFolder, 'FU_EAD_resampled.tif')\n",
    "OutPath = os.path.join(FloodFolder, 'FU_EAD_WSFEimpact.tif')\n",
    "\n",
    "calcShell(A=WSFEPath, B=InPath, OutFile=OutPath, Calculation=\"A*B\", OutType=\" --type=Byte\")\n",
    "    \n",
    "print('Done. Only built-up cells that have been flooded remain.. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a4250f",
   "metadata": {},
   "source": [
    "### 10.4 Join with Settlements via concatenation\n",
    "Using the serial method, combine settlement IDs with 1) WSFE year cells and 2) the flooded-only WSFE year cells under each scenario."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da10ec8",
   "metadata": {},
   "source": [
    "##### Rasterize the settlements we've created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "417cce08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# WSFEPath = os.path.join(FloodFolder, 'WSFE_simplified.tif') \n",
    "OutSett = os.path.join(Results, ''.join(['Settlements', str(WSFE_end), '_rasterized.tif'])\n",
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')[['Sett_ID', 'geometry']]\n",
    "\n",
    "len_WSFE = 2 # We already know that the WSFE years are reclassified to 1-31, i.e. a max of 2 digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e175146",
   "metadata": {},
   "outputs": [],
   "source": [
    "ShapeToRaster(Shapefile=Settlements, ValueVar=\"Sett_ID\", MetaRasterPath=WSFEPath, OutFilePath=OutSett, NewDType = 'uint32')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "468957a0",
   "metadata": {},
   "source": [
    "##### Calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e20c3b0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "Calc = \"(A*\" + str(10**len_WSFE) + \")+B\" \n",
    "\n",
    "FloodImpactPath = os.path.join(FloodFolder, 'FU_EAD_WSFEimpact.tif')\n",
    "FloodSerialPath = os.path.join(FloodFolder, 'FU_Settlements_serial.tif')\n",
    "\n",
    "#WSFEPath = os.path.join(FloodFolder, 'WSFE_simplified.tif')\n",
    "WSFESerialPath = os.path.join(FloodFolder, 'WSFE_Settlements_serial.tif')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a26c15ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "calcShell(A=OutSett, B=FloodImpactPath, OutFile=FloodSerialPath, Calculation=Calc)\n",
    "calcShell(A=OutSett, B=WSFEPath, OutFile=WSFESerialPath, Calculation=Calc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4465cf86",
   "metadata": {},
   "source": [
    "### 10.5 Vector math to split raster strings into Settlement and WSFE year assignments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "376f7306",
   "metadata": {},
   "source": [
    "##### Vectorize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "981aa267",
   "metadata": {},
   "outputs": [],
   "source": [
    "FloodVec = 'FloodedBuildup.shp' # Was having write issues when putting both in the same gpkg, so we're settling for .shp.\n",
    "FloodVecPath = os.path.join(FloodFolder, FloodVec)\n",
    "BuildVec = 'AllBuildup.shp' \n",
    "BuildVecPath = os.path.join(FloodFolder, BuildVec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3160f5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "RasterToShapefile(InRasterPath=FloodSerialPath, OutFilePath=FloodVecPath, \n",
    "                  OutName='', VariableName='gridcode')\n",
    "RasterToShapefile(InRasterPath=WSFESerialPath, OutFilePath=BuildVecPath, \n",
    "                  OutName='', VariableName='gridcode')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df522b41",
   "metadata": {},
   "source": [
    "##### Split string into separate fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "132451f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "Sett_rio = rasterio.open(os.path.join(Results, ''.join(['Settlements', str(WSFE_end), '_rasterized.tif']))).read(1)\n",
    "len_Sett = len(str(Sett_rio.max()))\n",
    "Sett_rio = None\n",
    "\n",
    "Fill = len_Sett + 2 # Add the digits stored as len_WSFE # or just write +2 since we already know the length of reclassed WSFE.\n",
    "\n",
    "OutPackage = os.path.join(FloodFolder, 'FloodedSettlements.gpkg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a472a100",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load newly created vectorized datasets.\n",
    "for File in [FloodVec, BuildVec]:\n",
    "    InObject = gpd.read_file(os.path.join(FloodFolder, File)).to_crs(\"ESRI:102022\")\n",
    "    print(InObject.info(), '\\n\\n', InObject.sample(10), '\\n\\n', InObject.crs, '\\n\\n', InObject['gridcode'].max())\n",
    "    \n",
    "    InObject['gridstring'] = InObject['gridcode'].astype(str).str.zfill(Fill)\n",
    "\n",
    "    InObject['Sett_ID'] = InObject['gridstring'].str[:-2].astype(int) # Remove the last digits to get the Sett ID portion.\n",
    "    InObject['year'] = InObject['gridstring'].str[-2:].astype(int) # Keep only the last digits to get the year portion.\n",
    "    InObject['year'] = np.where(InObject['year'] > 0, InObject['year'] + 1984, InObject['year']) # Reclass back to year value.\n",
    "    \n",
    "    print('%s Serial split by year of buildup and Sett ID.\\n\\n' % time.ctime(), InObject.sample(10))\n",
    "    \n",
    "    # Remove features where year or settlement = 0.\n",
    "    print(\"%s Before: %s\\n\" % (File, InObject.shape))\n",
    "    InObject = InObject.loc[(InObject[\"year\"] >1984) & (InObject[\"year\"] < 2016) & (InObject[\"Sett_ID\"] != 0)] \n",
    "    print(\"%s After: %s\\n\" % (File, InObject.shape))\n",
    "\n",
    "    # Save intermediate file.\n",
    "    InObject.to_file(driver='GPKG', filename=OutPackage, layer=File.replace('.shp', ''))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8758a744",
   "metadata": {},
   "source": [
    "### 10.6 Group by settlement and count cells for each year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0d39571",
   "metadata": {},
   "source": [
    "##### Flooded buildings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61edaeb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')\n",
    "AllSummaries = pd.DataFrame(Settlements).drop(columns='geometry')[['Sett_ID']]\n",
    "Settlements = None\n",
    "\n",
    "ValObject = pd.DataFrame(gpd.read_file(OutPackage, layer='FloodedBuildup'))[['Sett_ID', 'year']]\n",
    "\n",
    "print(AllSummaries.info(), '\\n', AllSummaries.sample(10), '\\n', ValObject.info(), '\\n', ValObject.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b62294e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "for BuiltYear in EligibleYears:\n",
    "    GroupedVals = ValObject[\n",
    "        ValObject['year']<=BuiltYear].groupby(\n",
    "        'Sett_ID', as_index=False)\n",
    "    \n",
    "    VariableName = ''.join(['FLDct_', str(BuiltYear)])\n",
    "    \n",
    "    AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'year': VariableName}), how = 'left', on='Sett_ID')\n",
    "\n",
    "    print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (BuiltYear, time.ctime()))\n",
    "\n",
    "    # Save in-progress results\n",
    "    AllSummaries.to_csv(os.path.join(FloodFolder, 'FloodedCellCount.csv'))\n",
    "    print(AllSummaries.sort_values(by=AllSummaries.columns[1], ascending=False).head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8379ac1",
   "metadata": {},
   "source": [
    "##### All buildings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9662e0fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')\n",
    "AllSummaries = pd.DataFrame(Settlements).drop(columns='geometry')[['Sett_ID']]\n",
    "Settlements = None\n",
    "AllSummaries\n",
    "\n",
    "ValObject = pd.DataFrame(gpd.read_file(OutPackage, layer='AllBuildup'))[['Sett_ID', 'year']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff1c580d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for BuiltYear in EligibleYears:\n",
    "    GroupedVals = ValObject[\n",
    "        ValObject['year']<=BuiltYear].groupby(\n",
    "        'Sett_ID', as_index=False)\n",
    "    \n",
    "    VariableName = ''.join(['BLDct_', str(BuiltYear)])\n",
    "    \n",
    "    AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'year': VariableName}), how = 'left', on='Sett_ID')\n",
    "\n",
    "    print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (BuiltYear, time.ctime()))\n",
    "\n",
    "    # Save in-progress results\n",
    "    AllSummaries.to_csv(os.path.join(FloodFolder, 'BuiltCellCount.csv'))\n",
    "    print(AllSummaries.sort_values(by=AllSummaries.columns[1], ascending=False).head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "821f81df",
   "metadata": {},
   "source": [
    "### 10.7 Calculate area and percent flooded and save to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7cf63d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "BuiltArea = pd.read_csv(os.path.join(FloodFolder, 'BuiltCellCount.csv'))\n",
    "Flood = pd.read_csv(os.path.join(FloodFolder, 'FloodedCellCount.csv'))\n",
    "Areas = pd.read_csv(os.path.join(Results, 'Areas.csv'))\n",
    "\n",
    "for Dataset in [BuiltArea, Flood, Areas]:\n",
    "    if 'Unnamed: 0' in Dataset.columns:\n",
    "        Dataset.drop(columns='Unnamed: 0', inplace=True)\n",
    "    else:\n",
    "        pass\n",
    "    print(Dataset.info())\n",
    "\n",
    "Stats = reduce(lambda  left,right: pd.merge(left,right,on=['Sett_ID'],\n",
    "                                            how='outer'), [BuiltArea, Flood, Areas])\n",
    "print(Stats.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77bc579b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick spot-checking. Number of flood cells should always be less than or equal to number of built area cells.\n",
    "Check1 = (Stats['FLDct_2007'] > Stats['BLDct_2007']).sum()\n",
    "Check2 = (Stats['FLDct_2000'] > Stats['BLDct_2000']).sum()\n",
    "Check3 = (Stats['FLDct_2011'] > Stats['BLDct_2011']).sum()\n",
    "print(Check1, Check2, Check3) # All should be zero. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68bea3a9",
   "metadata": {},
   "source": [
    "##### Percent flooded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76e622c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['FLDct_', str(year)])\n",
    "    DenomVar = ''.join(['BLDct_', str(year)])\n",
    "    NewVar = ''.join(['FLDpc', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "        Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "    else:\n",
    "        pass\n",
    "Stats.sort_values(by='FLDpc2005', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6311fea",
   "metadata": {},
   "source": [
    "##### Area flooded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "390bf845",
   "metadata": {},
   "outputs": [],
   "source": [
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['FLDpc', str(year)])\n",
    "    DenomVar = ''.join(['AREA', str(year)])\n",
    "    NewVar = ''.join(['FLDarea', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "        Stats[NewVar] = Stats[RawVar] * Stats[DenomVar]\n",
    "    else:\n",
    "        pass\n",
    "Stats.sort_values(by='FLDarea2005', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d519e096",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|FLDpc|FLDarea')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16b5856a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to file\n",
    "Stats.to_csv(os.path.join(Results, 'Flood.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7429f6cd",
   "metadata": {},
   "source": [
    "## 11. GROWTH STATISTICS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f179e1f8",
   "metadata": {},
   "source": [
    "### 11.1 Load and prep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06e45388",
   "metadata": {},
   "outputs": [],
   "source": [
    "WSFE_end = 2021\n",
    "WSFE_start = 1985\n",
    "WSFE_Years = ListFromRange(WSFE_start, WSFE_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "e1cedfe1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 4 columns):\n",
      " #   Column     Non-Null Count  Dtype \n",
      "---  ------     --------------  ----- \n",
      " 0   Sett_ID    18552 non-null  int64 \n",
      " 1   SettName   18552 non-null  object\n",
      " 2   GeoName    18552 non-null  object\n",
      " 3   UCDB_Name  18552 non-null  object\n",
      "dtypes: int64(1), object(3)\n",
      "memory usage: 579.9+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 39 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   18552 non-null  int64  \n",
      " 1   ADM_ID    18552 non-null  int64  \n",
      " 2   AREA2021  18552 non-null  float64\n",
      " 3   AREA2020  17807 non-null  float64\n",
      " 4   AREA2019  17460 non-null  float64\n",
      " 5   AREA2018  17058 non-null  float64\n",
      " 6   AREA2017  16845 non-null  float64\n",
      " 7   AREA2016  16629 non-null  float64\n",
      " 8   AREA2015  16479 non-null  float64\n",
      " 9   AREA2014  15938 non-null  float64\n",
      " 10  AREA2013  14137 non-null  float64\n",
      " 11  AREA2012  13533 non-null  float64\n",
      " 12  AREA2011  11442 non-null  float64\n",
      " 13  AREA2010  11415 non-null  float64\n",
      " 14  AREA2009  4755 non-null   float64\n",
      " 15  AREA2008  4744 non-null   float64\n",
      " 16  AREA2007  4739 non-null   float64\n",
      " 17  AREA2006  4730 non-null   float64\n",
      " 18  AREA2005  4723 non-null   float64\n",
      " 19  AREA2004  4715 non-null   float64\n",
      " 20  AREA2003  4699 non-null   float64\n",
      " 21  AREA2002  4689 non-null   float64\n",
      " 22  AREA2001  4680 non-null   float64\n",
      " 23  AREA2000  4670 non-null   float64\n",
      " 24  AREA1999  4075 non-null   float64\n",
      " 25  AREA1998  4058 non-null   float64\n",
      " 26  AREA1997  4031 non-null   float64\n",
      " 27  AREA1996  4017 non-null   float64\n",
      " 28  AREA1995  3949 non-null   float64\n",
      " 29  AREA1994  3907 non-null   float64\n",
      " 30  AREA1993  3885 non-null   float64\n",
      " 31  AREA1992  3864 non-null   float64\n",
      " 32  AREA1991  3853 non-null   float64\n",
      " 33  AREA1990  3835 non-null   float64\n",
      " 34  AREA1989  2507 non-null   float64\n",
      " 35  AREA1988  2484 non-null   float64\n",
      " 36  AREA1987  2441 non-null   float64\n",
      " 37  AREA1986  2372 non-null   float64\n",
      " 38  AREA1985  2139 non-null   float64\n",
      "dtypes: float64(37), int64(2)\n",
      "memory usage: 5.5 MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 22 columns):\n",
      " #   Column   Non-Null Count  Dtype  \n",
      "---  ------   --------------  -----  \n",
      " 0   Sett_ID  18552 non-null  int64  \n",
      " 1   POP2000  12047 non-null  float64\n",
      " 2   POP2001  12047 non-null  float64\n",
      " 3   POP2002  12047 non-null  float64\n",
      " 4   POP2003  12047 non-null  float64\n",
      " 5   POP2004  12047 non-null  float64\n",
      " 6   POP2005  12047 non-null  float64\n",
      " 7   POP2006  12047 non-null  float64\n",
      " 8   POP2007  12047 non-null  float64\n",
      " 9   POP2008  12047 non-null  float64\n",
      " 10  POP2009  12047 non-null  float64\n",
      " 11  POP2010  12047 non-null  float64\n",
      " 12  POP2011  12047 non-null  float64\n",
      " 13  POP2012  12047 non-null  float64\n",
      " 14  POP2013  12047 non-null  float64\n",
      " 15  POP2014  12047 non-null  float64\n",
      " 16  POP2015  12047 non-null  float64\n",
      " 17  POP2016  12047 non-null  float64\n",
      " 18  POP2017  12047 non-null  float64\n",
      " 19  POP2018  12047 non-null  float64\n",
      " 20  POP2019  12047 non-null  float64\n",
      " 21  POP2020  12047 non-null  float64\n",
      "dtypes: float64(21), int64(1)\n",
      "memory usage: 3.1 MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 76 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   Sett_ID          18552 non-null  int64  \n",
      " 1   NTL_Dsum1999     1399 non-null   float64\n",
      " 2   NTL_Dmax1999     1399 non-null   float64\n",
      " 3   NTL_Dmean1999    1399 non-null   float64\n",
      " 4   NTL_Dmedian1999  1399 non-null   float64\n",
      " 5   NTL_Dstd1999     1399 non-null   float64\n",
      " 6   NTL_Dsum2000     1399 non-null   float64\n",
      " 7   NTL_Dmax2000     1399 non-null   float64\n",
      " 8   NTL_Dmean2000    1399 non-null   float64\n",
      " 9   NTL_Dmedian2000  1399 non-null   float64\n",
      " 10  NTL_Dstd2000     1399 non-null   float64\n",
      " 11  NTL_Dsum2001     1399 non-null   float64\n",
      " 12  NTL_Dmax2001     1399 non-null   float64\n",
      " 13  NTL_Dmean2001    1399 non-null   float64\n",
      " 14  NTL_Dmedian2001  1399 non-null   float64\n",
      " 15  NTL_Dstd2001     1399 non-null   float64\n",
      " 16  NTL_Dsum2002     1399 non-null   float64\n",
      " 17  NTL_Dmax2002     1399 non-null   float64\n",
      " 18  NTL_Dmean2002    1399 non-null   float64\n",
      " 19  NTL_Dmedian2002  1399 non-null   float64\n",
      " 20  NTL_Dstd2002     1399 non-null   float64\n",
      " 21  NTL_Dsum2003     1399 non-null   float64\n",
      " 22  NTL_Dmax2003     1399 non-null   float64\n",
      " 23  NTL_Dmean2003    1399 non-null   float64\n",
      " 24  NTL_Dmedian2003  1399 non-null   float64\n",
      " 25  NTL_Dstd2003     1399 non-null   float64\n",
      " 26  NTL_Dsum2004     1399 non-null   float64\n",
      " 27  NTL_Dmax2004     1399 non-null   float64\n",
      " 28  NTL_Dmean2004    1399 non-null   float64\n",
      " 29  NTL_Dmedian2004  1399 non-null   float64\n",
      " 30  NTL_Dstd2004     1399 non-null   float64\n",
      " 31  NTL_Dsum2005     1399 non-null   float64\n",
      " 32  NTL_Dmax2005     1399 non-null   float64\n",
      " 33  NTL_Dmean2005    1399 non-null   float64\n",
      " 34  NTL_Dmedian2005  1399 non-null   float64\n",
      " 35  NTL_Dstd2005     1399 non-null   float64\n",
      " 36  NTL_Dsum2006     1399 non-null   float64\n",
      " 37  NTL_Dmax2006     1399 non-null   float64\n",
      " 38  NTL_Dmean2006    1399 non-null   float64\n",
      " 39  NTL_Dmedian2006  1399 non-null   float64\n",
      " 40  NTL_Dstd2006     1399 non-null   float64\n",
      " 41  NTL_Dsum2007     1399 non-null   float64\n",
      " 42  NTL_Dmax2007     1399 non-null   float64\n",
      " 43  NTL_Dmean2007    1399 non-null   float64\n",
      " 44  NTL_Dmedian2007  1399 non-null   float64\n",
      " 45  NTL_Dstd2007     1399 non-null   float64\n",
      " 46  NTL_Dsum2008     1399 non-null   float64\n",
      " 47  NTL_Dmax2008     1399 non-null   float64\n",
      " 48  NTL_Dmean2008    1399 non-null   float64\n",
      " 49  NTL_Dmedian2008  1399 non-null   float64\n",
      " 50  NTL_Dstd2008     1399 non-null   float64\n",
      " 51  NTL_Dsum2009     1448 non-null   float64\n",
      " 52  NTL_Dmax2009     1448 non-null   float64\n",
      " 53  NTL_Dmean2009    1448 non-null   float64\n",
      " 54  NTL_Dmedian2009  1448 non-null   float64\n",
      " 55  NTL_Dstd2009     1448 non-null   float64\n",
      " 56  NTL_Dsum2010     1448 non-null   float64\n",
      " 57  NTL_Dmax2010     1448 non-null   float64\n",
      " 58  NTL_Dmean2010    1448 non-null   float64\n",
      " 59  NTL_Dmedian2010  1448 non-null   float64\n",
      " 60  NTL_Dstd2010     1448 non-null   float64\n",
      " 61  NTL_Dsum2011     1398 non-null   float64\n",
      " 62  NTL_Dmax2011     1398 non-null   float64\n",
      " 63  NTL_Dmean2011    1398 non-null   float64\n",
      " 64  NTL_Dmedian2011  1398 non-null   float64\n",
      " 65  NTL_Dstd2011     1398 non-null   float64\n",
      " 66  NTL_Dsum2012     1399 non-null   float64\n",
      " 67  NTL_Dmax2012     1399 non-null   float64\n",
      " 68  NTL_Dmean2012    1399 non-null   float64\n",
      " 69  NTL_Dmedian2012  1399 non-null   float64\n",
      " 70  NTL_Dstd2012     1399 non-null   float64\n",
      " 71  NTL_Dsum2013     1399 non-null   float64\n",
      " 72  NTL_Dmax2013     1399 non-null   float64\n",
      " 73  NTL_Dmean2013    1399 non-null   float64\n",
      " 74  NTL_Dmedian2013  1399 non-null   float64\n",
      " 75  NTL_Dstd2013     1399 non-null   float64\n",
      "dtypes: float64(75), int64(1)\n",
      "memory usage: 10.8 MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 18552 entries, 0 to 18551\n",
      "Data columns (total 21 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   Sett_ID          18552 non-null  int64  \n",
      " 1   NTL_Vsum2012     3252 non-null   float64\n",
      " 2   NTL_Vmax2012     3252 non-null   float64\n",
      " 3   NTL_Vmean2012    3252 non-null   float64\n",
      " 4   NTL_Vmedian2012  3252 non-null   float64\n",
      " 5   NTL_Vstd2012     3252 non-null   float64\n",
      " 6   NTL_Vsum2013     3252 non-null   float64\n",
      " 7   NTL_Vmax2013     3252 non-null   float64\n",
      " 8   NTL_Vmean2013    3252 non-null   float64\n",
      " 9   NTL_Vmedian2013  3252 non-null   float64\n",
      " 10  NTL_Vstd2013     3252 non-null   float64\n",
      " 11  NTL_Vsum2014     3252 non-null   float64\n",
      " 12  NTL_Vmax2014     3252 non-null   float64\n",
      " 13  NTL_Vmean2014    3252 non-null   float64\n",
      " 14  NTL_Vmedian2014  3252 non-null   float64\n",
      " 15  NTL_Vstd2014     3252 non-null   float64\n",
      " 16  NTL_Vsum2015     3252 non-null   float64\n",
      " 17  NTL_Vmax2015     3252 non-null   float64\n",
      " 18  NTL_Vmean2015    3252 non-null   float64\n",
      " 19  NTL_Vmedian2015  3252 non-null   float64\n",
      " 20  NTL_Vstd2015     3252 non-null   float64\n",
      "dtypes: float64(20), int64(1)\n",
      "memory usage: 3.0 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "PlaceNames = pd.read_csv(os.path.join(Results, 'PlaceNames.csv'))\n",
    "Areas = pd.read_csv(os.path.join(Results, 'Area.csv'))\n",
    "Population = pd.read_csv(os.path.join(Results, 'Population.csv'))\n",
    "DMSP = pd.read_csv(os.path.join(Results, 'NTL_DMSP.csv'))\n",
    "VNL = pd.read_csv(os.path.join(Results, 'NTL_VIIRS.csv'))\n",
    "#Flood = pd.read_csv(os.path.join(Results, 'Flood.csv'))\n",
    "\n",
    "RawValues = [PlaceNames, Areas, Population, DMSP, VNL]\n",
    "\n",
    "for Dataset in RawValues:\n",
    "    if 'Unnamed: 0' in Dataset.columns:\n",
    "        Dataset.drop(columns='Unnamed: 0', inplace=True)\n",
    "    else:\n",
    "        pass\n",
    "    if 'year' in Dataset.columns:\n",
    "        Dataset.drop(columns='year', inplace=True)\n",
    "    else:\n",
    "        pass\n",
    "    print(Dataset.info(verbose=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "07909650",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>AREA2020</th>\n",
       "      <th>AREA2019</th>\n",
       "      <th>AREA2018</th>\n",
       "      <th>AREA2017</th>\n",
       "      <th>...</th>\n",
       "      <th>NTL_Vsum2014</th>\n",
       "      <th>NTL_Vmax2014</th>\n",
       "      <th>NTL_Vmean2014</th>\n",
       "      <th>NTL_Vmedian2014</th>\n",
       "      <th>NTL_Vstd2014</th>\n",
       "      <th>NTL_Vsum2015</th>\n",
       "      <th>NTL_Vmax2015</th>\n",
       "      <th>NTL_Vmean2015</th>\n",
       "      <th>NTL_Vmedian2015</th>\n",
       "      <th>NTL_Vstd2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11049</th>\n",
       "      <td>59420</td>\n",
       "      <td>Talldaww</td>\n",
       "      <td>Talldaww</td>\n",
       "      <td>UNK</td>\n",
       "      <td>107</td>\n",
       "      <td>4.518157</td>\n",
       "      <td>4.209040</td>\n",
       "      <td>4.105746</td>\n",
       "      <td>3.994800</td>\n",
       "      <td>3.971846</td>\n",
       "      <td>...</td>\n",
       "      <td>50.489422</td>\n",
       "      <td>4.321935</td>\n",
       "      <td>2.019577</td>\n",
       "      <td>1.872939</td>\n",
       "      <td>0.912116</td>\n",
       "      <td>27.809113</td>\n",
       "      <td>2.145282</td>\n",
       "      <td>1.112365</td>\n",
       "      <td>1.028654</td>\n",
       "      <td>0.427772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1758</th>\n",
       "      <td>6457</td>\n",
       "      <td>‘Ein Qunīya</td>\n",
       "      <td>‘Ein Qunīya</td>\n",
       "      <td>UNK</td>\n",
       "      <td>170</td>\n",
       "      <td>0.270860</td>\n",
       "      <td>0.263208</td>\n",
       "      <td>0.257087</td>\n",
       "      <td>0.257087</td>\n",
       "      <td>0.253262</td>\n",
       "      <td>...</td>\n",
       "      <td>22.408588</td>\n",
       "      <td>13.588633</td>\n",
       "      <td>11.204294</td>\n",
       "      <td>11.204294</td>\n",
       "      <td>2.384338</td>\n",
       "      <td>26.076504</td>\n",
       "      <td>16.042847</td>\n",
       "      <td>13.038252</td>\n",
       "      <td>13.038252</td>\n",
       "      <td>3.004595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6388</th>\n",
       "      <td>30082</td>\n",
       "      <td>Şalkhad</td>\n",
       "      <td>Şalkhad</td>\n",
       "      <td>UNK</td>\n",
       "      <td>222</td>\n",
       "      <td>4.533459</td>\n",
       "      <td>4.449294</td>\n",
       "      <td>4.429400</td>\n",
       "      <td>4.388083</td>\n",
       "      <td>4.361303</td>\n",
       "      <td>...</td>\n",
       "      <td>80.896942</td>\n",
       "      <td>6.295742</td>\n",
       "      <td>3.370706</td>\n",
       "      <td>3.180894</td>\n",
       "      <td>1.623381</td>\n",
       "      <td>48.982742</td>\n",
       "      <td>3.665474</td>\n",
       "      <td>2.040948</td>\n",
       "      <td>2.092541</td>\n",
       "      <td>0.956662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5812</th>\n",
       "      <td>26407</td>\n",
       "      <td>Ḩarf al Musaytirah</td>\n",
       "      <td>Ḩarf al Musaytirah</td>\n",
       "      <td>UNK</td>\n",
       "      <td>108</td>\n",
       "      <td>0.508819</td>\n",
       "      <td>0.447607</td>\n",
       "      <td>0.439191</td>\n",
       "      <td>0.428479</td>\n",
       "      <td>0.423888</td>\n",
       "      <td>...</td>\n",
       "      <td>13.791924</td>\n",
       "      <td>5.109694</td>\n",
       "      <td>4.597308</td>\n",
       "      <td>4.344055</td>\n",
       "      <td>0.362319</td>\n",
       "      <td>7.653924</td>\n",
       "      <td>2.685563</td>\n",
       "      <td>2.551308</td>\n",
       "      <td>2.663573</td>\n",
       "      <td>0.174547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4720</th>\n",
       "      <td>19348</td>\n",
       "      <td>Al Qamşīyah</td>\n",
       "      <td>Al Qamşīyah</td>\n",
       "      <td>UNK</td>\n",
       "      <td>202</td>\n",
       "      <td>1.111750</td>\n",
       "      <td>0.941123</td>\n",
       "      <td>0.917404</td>\n",
       "      <td>0.890624</td>\n",
       "      <td>0.889094</td>\n",
       "      <td>...</td>\n",
       "      <td>14.041906</td>\n",
       "      <td>2.151807</td>\n",
       "      <td>1.404191</td>\n",
       "      <td>1.343488</td>\n",
       "      <td>0.344194</td>\n",
       "      <td>8.071981</td>\n",
       "      <td>1.105958</td>\n",
       "      <td>0.807198</td>\n",
       "      <td>0.803568</td>\n",
       "      <td>0.146231</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 158 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID            SettName             GeoName UCDB_Name  ADM_ID  \\\n",
       "11049    59420            Talldaww            Talldaww       UNK     107   \n",
       "1758      6457         ‘Ein Qunīya         ‘Ein Qunīya       UNK     170   \n",
       "6388     30082             Şalkhad             Şalkhad       UNK     222   \n",
       "5812     26407  Ḩarf al Musaytirah  Ḩarf al Musaytirah       UNK     108   \n",
       "4720     19348         Al Qamşīyah         Al Qamşīyah       UNK     202   \n",
       "\n",
       "       AREA2021  AREA2020  AREA2019  AREA2018  AREA2017  ...  NTL_Vsum2014  \\\n",
       "11049  4.518157  4.209040  4.105746  3.994800  3.971846  ...     50.489422   \n",
       "1758   0.270860  0.263208  0.257087  0.257087  0.253262  ...     22.408588   \n",
       "6388   4.533459  4.449294  4.429400  4.388083  4.361303  ...     80.896942   \n",
       "5812   0.508819  0.447607  0.439191  0.428479  0.423888  ...     13.791924   \n",
       "4720   1.111750  0.941123  0.917404  0.890624  0.889094  ...     14.041906   \n",
       "\n",
       "       NTL_Vmax2014  NTL_Vmean2014  NTL_Vmedian2014  NTL_Vstd2014  \\\n",
       "11049      4.321935       2.019577         1.872939      0.912116   \n",
       "1758      13.588633      11.204294        11.204294      2.384338   \n",
       "6388       6.295742       3.370706         3.180894      1.623381   \n",
       "5812       5.109694       4.597308         4.344055      0.362319   \n",
       "4720       2.151807       1.404191         1.343488      0.344194   \n",
       "\n",
       "       NTL_Vsum2015  NTL_Vmax2015  NTL_Vmean2015  NTL_Vmedian2015  \\\n",
       "11049     27.809113      2.145282       1.112365         1.028654   \n",
       "1758      26.076504     16.042847      13.038252        13.038252   \n",
       "6388      48.982742      3.665474       2.040948         2.092541   \n",
       "5812       7.653924      2.685563       2.551308         2.663573   \n",
       "4720       8.071981      1.105958       0.807198         0.803568   \n",
       "\n",
       "       NTL_Vstd2015  \n",
       "11049      0.427772  \n",
       "1758       3.004595  \n",
       "6388       0.956662  \n",
       "5812       0.174547  \n",
       "4720       0.146231  \n",
       "\n",
       "[5 rows x 158 columns]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "AllStats = reduce(lambda  left,right: pd.merge(left,right,on=['Sett_ID'],\n",
    "                                            how='outer'), RawValues)\n",
    "AllStats.to_csv(os.path.join(Results, 'AllStats.csv'))\n",
    "\n",
    "AllStats[AllStats.SettName!='UNK'].sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224573a1",
   "metadata": {},
   "source": [
    "### 11.2 Change over time of raw variables\n",
    "pch = percent change"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39152aff",
   "metadata": {},
   "source": [
    "#### Population change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "1307b9dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Population, how = 'outer', on='Sett_ID')\n",
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['POP', str(year)])\n",
    "    LagVar = ''.join(['POP', str(year-1)])\n",
    "    NewVar = ''.join(['POPpch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "e720c495",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'POPpch2001', 'POPpch2002', 'POPpch2003',\n",
       "       'POPpch2004', 'POPpch2005', 'POPpch2006', 'POPpch2007', 'POPpch2008',\n",
       "       'POPpch2009', 'POPpch2010', 'POPpch2011', 'POPpch2012', 'POPpch2013',\n",
       "       'POPpch2014', 'POPpch2015', 'POPpch2016', 'POPpch2017', 'POPpch2018',\n",
       "       'POPpch2019', 'POPpch2020'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|pch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "f93e099b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>AREA2020</th>\n",
       "      <th>AREA2019</th>\n",
       "      <th>AREA2018</th>\n",
       "      <th>AREA2017</th>\n",
       "      <th>...</th>\n",
       "      <th>POPpch2011</th>\n",
       "      <th>POPpch2012</th>\n",
       "      <th>POPpch2013</th>\n",
       "      <th>POPpch2014</th>\n",
       "      <th>POPpch2015</th>\n",
       "      <th>POPpch2016</th>\n",
       "      <th>POPpch2017</th>\n",
       "      <th>POPpch2018</th>\n",
       "      <th>POPpch2019</th>\n",
       "      <th>POPpch2020</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7742</th>\n",
       "      <td>38534</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>275.274712</td>\n",
       "      <td>273.803346</td>\n",
       "      <td>270.629542</td>\n",
       "      <td>268.027298</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.031286</td>\n",
       "      <td>0.023206</td>\n",
       "      <td>-0.064291</td>\n",
       "      <td>-0.037347</td>\n",
       "      <td>-0.035103</td>\n",
       "      <td>-0.043332</td>\n",
       "      <td>-0.027361</td>\n",
       "      <td>0.001813</td>\n",
       "      <td>-0.004629</td>\n",
       "      <td>0.040336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13495</th>\n",
       "      <td>96393</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>110</td>\n",
       "      <td>115.537029</td>\n",
       "      <td>113.038844</td>\n",
       "      <td>112.337975</td>\n",
       "      <td>111.384609</td>\n",
       "      <td>110.751838</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.095489</td>\n",
       "      <td>0.155745</td>\n",
       "      <td>-0.130523</td>\n",
       "      <td>-0.025649</td>\n",
       "      <td>-0.056103</td>\n",
       "      <td>-0.074796</td>\n",
       "      <td>-0.040655</td>\n",
       "      <td>0.036965</td>\n",
       "      <td>-0.052175</td>\n",
       "      <td>0.101168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9524</th>\n",
       "      <td>50649</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>115</td>\n",
       "      <td>61.194432</td>\n",
       "      <td>58.507258</td>\n",
       "      <td>57.903562</td>\n",
       "      <td>57.279972</td>\n",
       "      <td>56.778040</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.095587</td>\n",
       "      <td>0.095903</td>\n",
       "      <td>-0.127497</td>\n",
       "      <td>-0.023739</td>\n",
       "      <td>-0.052680</td>\n",
       "      <td>-0.036555</td>\n",
       "      <td>-0.075504</td>\n",
       "      <td>0.045738</td>\n",
       "      <td>-0.046187</td>\n",
       "      <td>0.078855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2966</th>\n",
       "      <td>10708</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>61</td>\n",
       "      <td>55.778766</td>\n",
       "      <td>55.175835</td>\n",
       "      <td>54.813923</td>\n",
       "      <td>54.318112</td>\n",
       "      <td>53.143621</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.006455</td>\n",
       "      <td>-0.021466</td>\n",
       "      <td>-0.008715</td>\n",
       "      <td>-0.037205</td>\n",
       "      <td>-0.055950</td>\n",
       "      <td>-0.005616</td>\n",
       "      <td>-0.062011</td>\n",
       "      <td>-0.004596</td>\n",
       "      <td>-0.005218</td>\n",
       "      <td>0.011233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10840</th>\n",
       "      <td>58135</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Hama</td>\n",
       "      <td>99</td>\n",
       "      <td>48.053139</td>\n",
       "      <td>45.659013</td>\n",
       "      <td>44.727072</td>\n",
       "      <td>43.723972</td>\n",
       "      <td>43.055239</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.109395</td>\n",
       "      <td>0.131973</td>\n",
       "      <td>-0.199106</td>\n",
       "      <td>-0.019397</td>\n",
       "      <td>-0.054380</td>\n",
       "      <td>-0.028810</td>\n",
       "      <td>-0.075182</td>\n",
       "      <td>0.039631</td>\n",
       "      <td>-0.043457</td>\n",
       "      <td>0.069917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12246</th>\n",
       "      <td>75655</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>UNK</td>\n",
       "      <td>43</td>\n",
       "      <td>47.433375</td>\n",
       "      <td>42.164615</td>\n",
       "      <td>39.378737</td>\n",
       "      <td>37.328925</td>\n",
       "      <td>36.593625</td>\n",
       "      <td>...</td>\n",
       "      <td>0.003911</td>\n",
       "      <td>0.076824</td>\n",
       "      <td>-0.015279</td>\n",
       "      <td>-0.047508</td>\n",
       "      <td>-0.052922</td>\n",
       "      <td>-0.094035</td>\n",
       "      <td>-0.032110</td>\n",
       "      <td>0.066806</td>\n",
       "      <td>-0.047084</td>\n",
       "      <td>0.073159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6744</th>\n",
       "      <td>32195</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As Suwayda</td>\n",
       "      <td>39</td>\n",
       "      <td>30.747184</td>\n",
       "      <td>29.289591</td>\n",
       "      <td>28.877945</td>\n",
       "      <td>28.323218</td>\n",
       "      <td>27.970488</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.142862</td>\n",
       "      <td>0.159841</td>\n",
       "      <td>-0.230049</td>\n",
       "      <td>-0.002613</td>\n",
       "      <td>-0.060484</td>\n",
       "      <td>-0.032212</td>\n",
       "      <td>-0.049829</td>\n",
       "      <td>0.050318</td>\n",
       "      <td>-0.067175</td>\n",
       "      <td>0.081853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5468</th>\n",
       "      <td>24050</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>104</td>\n",
       "      <td>27.997268</td>\n",
       "      <td>27.450192</td>\n",
       "      <td>27.229066</td>\n",
       "      <td>27.001819</td>\n",
       "      <td>26.691937</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.042622</td>\n",
       "      <td>0.078339</td>\n",
       "      <td>-0.103963</td>\n",
       "      <td>0.003850</td>\n",
       "      <td>-0.053288</td>\n",
       "      <td>-0.045033</td>\n",
       "      <td>-0.046994</td>\n",
       "      <td>0.005794</td>\n",
       "      <td>-0.026737</td>\n",
       "      <td>0.056819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15807</th>\n",
       "      <td>133699</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>26</td>\n",
       "      <td>24.873198</td>\n",
       "      <td>23.289356</td>\n",
       "      <td>23.169994</td>\n",
       "      <td>22.967997</td>\n",
       "      <td>22.587722</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.162722</td>\n",
       "      <td>0.234975</td>\n",
       "      <td>-0.228939</td>\n",
       "      <td>0.027231</td>\n",
       "      <td>-0.142488</td>\n",
       "      <td>-0.076009</td>\n",
       "      <td>-0.093134</td>\n",
       "      <td>0.121209</td>\n",
       "      <td>-0.141902</td>\n",
       "      <td>0.173449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3211</th>\n",
       "      <td>11851</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>UNK</td>\n",
       "      <td>153</td>\n",
       "      <td>24.496748</td>\n",
       "      <td>24.205230</td>\n",
       "      <td>24.108822</td>\n",
       "      <td>23.938196</td>\n",
       "      <td>23.769865</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.055476</td>\n",
       "      <td>0.114704</td>\n",
       "      <td>-0.152174</td>\n",
       "      <td>-0.036340</td>\n",
       "      <td>-0.022059</td>\n",
       "      <td>-0.003674</td>\n",
       "      <td>-0.017776</td>\n",
       "      <td>0.001058</td>\n",
       "      <td>0.001656</td>\n",
       "      <td>0.068369</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 178 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID    SettName     GeoName   UCDB_Name  ADM_ID    AREA2021  \\\n",
       "7742     38534     Şaḩnāyā     Şaḩnāyā    Damascus      28  281.043874   \n",
       "13495    96393    Ḩuraytān    Ḩuraytān      Aleppo     110  115.537029   \n",
       "9524     50649        Homs        Homs        Homs     115   61.194432   \n",
       "2966     10708    Qadsayyā    Qadsayyā    Damascus      61   55.778766   \n",
       "10840    58135       Ḩamāh       Ḩamāh        Hama      99   48.053139   \n",
       "12246    75655      Batabo      Batabo         UNK      43   47.433375   \n",
       "6744     32195  As-Suwayda  As-Suwayda  As Suwayda      39   30.747184   \n",
       "5468     24050     Latakia     Latakia     Latakia     104   27.997268   \n",
       "15807   133699   Ar Raqqah   Ar Raqqah   Ar Raqqah      26   24.873198   \n",
       "3211     11851   Al Kiswah   Al Kiswah         UNK     153   24.496748   \n",
       "\n",
       "         AREA2020    AREA2019    AREA2018    AREA2017  ...  POPpch2011  \\\n",
       "7742   275.274712  273.803346  270.629542  268.027298  ...   -0.031286   \n",
       "13495  113.038844  112.337975  111.384609  110.751838  ...   -0.095489   \n",
       "9524    58.507258   57.903562   57.279972   56.778040  ...   -0.095587   \n",
       "2966    55.175835   54.813923   54.318112   53.143621  ...   -0.006455   \n",
       "10840   45.659013   44.727072   43.723972   43.055239  ...   -0.109395   \n",
       "12246   42.164615   39.378737   37.328925   36.593625  ...    0.003911   \n",
       "6744    29.289591   28.877945   28.323218   27.970488  ...   -0.142862   \n",
       "5468    27.450192   27.229066   27.001819   26.691937  ...   -0.042622   \n",
       "15807   23.289356   23.169994   22.967997   22.587722  ...   -0.162722   \n",
       "3211    24.205230   24.108822   23.938196   23.769865  ...   -0.055476   \n",
       "\n",
       "       POPpch2012  POPpch2013  POPpch2014  POPpch2015  POPpch2016  POPpch2017  \\\n",
       "7742     0.023206   -0.064291   -0.037347   -0.035103   -0.043332   -0.027361   \n",
       "13495    0.155745   -0.130523   -0.025649   -0.056103   -0.074796   -0.040655   \n",
       "9524     0.095903   -0.127497   -0.023739   -0.052680   -0.036555   -0.075504   \n",
       "2966    -0.021466   -0.008715   -0.037205   -0.055950   -0.005616   -0.062011   \n",
       "10840    0.131973   -0.199106   -0.019397   -0.054380   -0.028810   -0.075182   \n",
       "12246    0.076824   -0.015279   -0.047508   -0.052922   -0.094035   -0.032110   \n",
       "6744     0.159841   -0.230049   -0.002613   -0.060484   -0.032212   -0.049829   \n",
       "5468     0.078339   -0.103963    0.003850   -0.053288   -0.045033   -0.046994   \n",
       "15807    0.234975   -0.228939    0.027231   -0.142488   -0.076009   -0.093134   \n",
       "3211     0.114704   -0.152174   -0.036340   -0.022059   -0.003674   -0.017776   \n",
       "\n",
       "       POPpch2018  POPpch2019  POPpch2020  \n",
       "7742     0.001813   -0.004629    0.040336  \n",
       "13495    0.036965   -0.052175    0.101168  \n",
       "9524     0.045738   -0.046187    0.078855  \n",
       "2966    -0.004596   -0.005218    0.011233  \n",
       "10840    0.039631   -0.043457    0.069917  \n",
       "12246    0.066806   -0.047084    0.073159  \n",
       "6744     0.050318   -0.067175    0.081853  \n",
       "5468     0.005794   -0.026737    0.056819  \n",
       "15807    0.121209   -0.141902    0.173449  \n",
       "3211     0.001058    0.001656    0.068369  \n",
       "\n",
       "[10 rows x 178 columns]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'PopChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87709ba2",
   "metadata": {},
   "source": [
    "#### Area change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "0deb53d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Areas, how = 'outer', on='Sett_ID')\n",
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['AREA', str(year)])\n",
    "    LagVar = ''.join(['AREA', str(year-1)])\n",
    "    NewVar = ''.join(['AREApch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "1569a1e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'AREApch1986', 'AREApch1987', 'AREApch1988',\n",
       "       'AREApch1989', 'AREApch1990', 'AREApch1991', 'AREApch1992',\n",
       "       'AREApch1993', 'AREApch1994', 'AREApch1995', 'AREApch1996',\n",
       "       'AREApch1997', 'AREApch1998', 'AREApch1999', 'AREApch2000',\n",
       "       'AREApch2001', 'AREApch2002', 'AREApch2003', 'AREApch2004',\n",
       "       'AREApch2005', 'AREApch2006', 'AREApch2007', 'AREApch2008',\n",
       "       'AREApch2009', 'AREApch2010', 'AREApch2011', 'AREApch2012',\n",
       "       'AREApch2013', 'AREApch2014', 'AREApch2015', 'AREApch2016',\n",
       "       'AREApch2017', 'AREApch2018', 'AREApch2019', 'AREApch2020',\n",
       "       'AREApch2021'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|pch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "743cb7ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>AREA2020</th>\n",
       "      <th>AREA2019</th>\n",
       "      <th>AREA2018</th>\n",
       "      <th>AREA2017</th>\n",
       "      <th>...</th>\n",
       "      <th>AREApch2012</th>\n",
       "      <th>AREApch2013</th>\n",
       "      <th>AREApch2014</th>\n",
       "      <th>AREApch2015</th>\n",
       "      <th>AREApch2016</th>\n",
       "      <th>AREApch2017</th>\n",
       "      <th>AREApch2018</th>\n",
       "      <th>AREApch2019</th>\n",
       "      <th>AREApch2020</th>\n",
       "      <th>AREApch2021</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7742</th>\n",
       "      <td>38534</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>275.274712</td>\n",
       "      <td>273.803346</td>\n",
       "      <td>270.629542</td>\n",
       "      <td>268.027298</td>\n",
       "      <td>...</td>\n",
       "      <td>0.107653</td>\n",
       "      <td>0.035557</td>\n",
       "      <td>0.054777</td>\n",
       "      <td>0.027551</td>\n",
       "      <td>0.009488</td>\n",
       "      <td>0.010879</td>\n",
       "      <td>0.009709</td>\n",
       "      <td>0.011727</td>\n",
       "      <td>0.005374</td>\n",
       "      <td>0.020958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13495</th>\n",
       "      <td>96393</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>110</td>\n",
       "      <td>115.537029</td>\n",
       "      <td>113.038844</td>\n",
       "      <td>112.337975</td>\n",
       "      <td>111.384609</td>\n",
       "      <td>110.751838</td>\n",
       "      <td>...</td>\n",
       "      <td>0.052468</td>\n",
       "      <td>0.019745</td>\n",
       "      <td>0.040996</td>\n",
       "      <td>0.014849</td>\n",
       "      <td>0.003445</td>\n",
       "      <td>0.006040</td>\n",
       "      <td>0.005713</td>\n",
       "      <td>0.008559</td>\n",
       "      <td>0.006239</td>\n",
       "      <td>0.022100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9524</th>\n",
       "      <td>50649</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>115</td>\n",
       "      <td>61.194432</td>\n",
       "      <td>58.507258</td>\n",
       "      <td>57.903562</td>\n",
       "      <td>57.279972</td>\n",
       "      <td>56.778040</td>\n",
       "      <td>...</td>\n",
       "      <td>0.039025</td>\n",
       "      <td>0.020639</td>\n",
       "      <td>0.049080</td>\n",
       "      <td>0.013810</td>\n",
       "      <td>0.005801</td>\n",
       "      <td>0.006989</td>\n",
       "      <td>0.008840</td>\n",
       "      <td>0.010887</td>\n",
       "      <td>0.010426</td>\n",
       "      <td>0.045929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2966</th>\n",
       "      <td>10708</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>61</td>\n",
       "      <td>55.778766</td>\n",
       "      <td>55.175835</td>\n",
       "      <td>54.813923</td>\n",
       "      <td>54.318112</td>\n",
       "      <td>53.143621</td>\n",
       "      <td>...</td>\n",
       "      <td>0.092324</td>\n",
       "      <td>0.047622</td>\n",
       "      <td>0.125536</td>\n",
       "      <td>0.039791</td>\n",
       "      <td>0.027261</td>\n",
       "      <td>0.019478</td>\n",
       "      <td>0.022100</td>\n",
       "      <td>0.009128</td>\n",
       "      <td>0.006603</td>\n",
       "      <td>0.010927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10840</th>\n",
       "      <td>58135</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Hama</td>\n",
       "      <td>99</td>\n",
       "      <td>48.053139</td>\n",
       "      <td>45.659013</td>\n",
       "      <td>44.727072</td>\n",
       "      <td>43.723972</td>\n",
       "      <td>43.055239</td>\n",
       "      <td>...</td>\n",
       "      <td>0.052100</td>\n",
       "      <td>0.045541</td>\n",
       "      <td>0.099840</td>\n",
       "      <td>0.016797</td>\n",
       "      <td>0.010025</td>\n",
       "      <td>0.019236</td>\n",
       "      <td>0.015532</td>\n",
       "      <td>0.022942</td>\n",
       "      <td>0.020836</td>\n",
       "      <td>0.052435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12246</th>\n",
       "      <td>75655</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>UNK</td>\n",
       "      <td>43</td>\n",
       "      <td>47.433375</td>\n",
       "      <td>42.164615</td>\n",
       "      <td>39.378737</td>\n",
       "      <td>37.328925</td>\n",
       "      <td>36.593625</td>\n",
       "      <td>...</td>\n",
       "      <td>0.494250</td>\n",
       "      <td>0.199247</td>\n",
       "      <td>2.191315</td>\n",
       "      <td>0.037783</td>\n",
       "      <td>0.004694</td>\n",
       "      <td>0.024901</td>\n",
       "      <td>0.020094</td>\n",
       "      <td>0.054912</td>\n",
       "      <td>0.070746</td>\n",
       "      <td>0.124957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6744</th>\n",
       "      <td>32195</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As Suwayda</td>\n",
       "      <td>39</td>\n",
       "      <td>30.747184</td>\n",
       "      <td>29.289591</td>\n",
       "      <td>28.877945</td>\n",
       "      <td>28.323218</td>\n",
       "      <td>27.970488</td>\n",
       "      <td>...</td>\n",
       "      <td>0.164490</td>\n",
       "      <td>0.072502</td>\n",
       "      <td>0.153695</td>\n",
       "      <td>0.018550</td>\n",
       "      <td>0.008731</td>\n",
       "      <td>0.007635</td>\n",
       "      <td>0.012611</td>\n",
       "      <td>0.019586</td>\n",
       "      <td>0.014255</td>\n",
       "      <td>0.049765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5468</th>\n",
       "      <td>24050</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>104</td>\n",
       "      <td>27.997268</td>\n",
       "      <td>27.450192</td>\n",
       "      <td>27.229066</td>\n",
       "      <td>27.001819</td>\n",
       "      <td>26.691937</td>\n",
       "      <td>...</td>\n",
       "      <td>0.033893</td>\n",
       "      <td>0.018205</td>\n",
       "      <td>0.024678</td>\n",
       "      <td>0.089413</td>\n",
       "      <td>0.011478</td>\n",
       "      <td>0.009871</td>\n",
       "      <td>0.011610</td>\n",
       "      <td>0.008416</td>\n",
       "      <td>0.008121</td>\n",
       "      <td>0.019930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15807</th>\n",
       "      <td>133699</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>26</td>\n",
       "      <td>24.873198</td>\n",
       "      <td>23.289356</td>\n",
       "      <td>23.169994</td>\n",
       "      <td>22.967997</td>\n",
       "      <td>22.587722</td>\n",
       "      <td>...</td>\n",
       "      <td>0.038875</td>\n",
       "      <td>0.031967</td>\n",
       "      <td>0.012719</td>\n",
       "      <td>0.021267</td>\n",
       "      <td>0.016878</td>\n",
       "      <td>0.022939</td>\n",
       "      <td>0.016835</td>\n",
       "      <td>0.008795</td>\n",
       "      <td>0.005152</td>\n",
       "      <td>0.068007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3211</th>\n",
       "      <td>11851</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>UNK</td>\n",
       "      <td>153</td>\n",
       "      <td>24.496748</td>\n",
       "      <td>24.205230</td>\n",
       "      <td>24.108822</td>\n",
       "      <td>23.938196</td>\n",
       "      <td>23.769865</td>\n",
       "      <td>...</td>\n",
       "      <td>0.066751</td>\n",
       "      <td>0.027890</td>\n",
       "      <td>0.047031</td>\n",
       "      <td>0.016678</td>\n",
       "      <td>0.007026</td>\n",
       "      <td>0.008113</td>\n",
       "      <td>0.007082</td>\n",
       "      <td>0.007128</td>\n",
       "      <td>0.003999</td>\n",
       "      <td>0.012044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 214 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID    SettName     GeoName   UCDB_Name  ADM_ID    AREA2021  \\\n",
       "7742     38534     Şaḩnāyā     Şaḩnāyā    Damascus      28  281.043874   \n",
       "13495    96393    Ḩuraytān    Ḩuraytān      Aleppo     110  115.537029   \n",
       "9524     50649        Homs        Homs        Homs     115   61.194432   \n",
       "2966     10708    Qadsayyā    Qadsayyā    Damascus      61   55.778766   \n",
       "10840    58135       Ḩamāh       Ḩamāh        Hama      99   48.053139   \n",
       "12246    75655      Batabo      Batabo         UNK      43   47.433375   \n",
       "6744     32195  As-Suwayda  As-Suwayda  As Suwayda      39   30.747184   \n",
       "5468     24050     Latakia     Latakia     Latakia     104   27.997268   \n",
       "15807   133699   Ar Raqqah   Ar Raqqah   Ar Raqqah      26   24.873198   \n",
       "3211     11851   Al Kiswah   Al Kiswah         UNK     153   24.496748   \n",
       "\n",
       "         AREA2020    AREA2019    AREA2018    AREA2017  ...  AREApch2012  \\\n",
       "7742   275.274712  273.803346  270.629542  268.027298  ...     0.107653   \n",
       "13495  113.038844  112.337975  111.384609  110.751838  ...     0.052468   \n",
       "9524    58.507258   57.903562   57.279972   56.778040  ...     0.039025   \n",
       "2966    55.175835   54.813923   54.318112   53.143621  ...     0.092324   \n",
       "10840   45.659013   44.727072   43.723972   43.055239  ...     0.052100   \n",
       "12246   42.164615   39.378737   37.328925   36.593625  ...     0.494250   \n",
       "6744    29.289591   28.877945   28.323218   27.970488  ...     0.164490   \n",
       "5468    27.450192   27.229066   27.001819   26.691937  ...     0.033893   \n",
       "15807   23.289356   23.169994   22.967997   22.587722  ...     0.038875   \n",
       "3211    24.205230   24.108822   23.938196   23.769865  ...     0.066751   \n",
       "\n",
       "       AREApch2013  AREApch2014  AREApch2015  AREApch2016  AREApch2017  \\\n",
       "7742      0.035557     0.054777     0.027551     0.009488     0.010879   \n",
       "13495     0.019745     0.040996     0.014849     0.003445     0.006040   \n",
       "9524      0.020639     0.049080     0.013810     0.005801     0.006989   \n",
       "2966      0.047622     0.125536     0.039791     0.027261     0.019478   \n",
       "10840     0.045541     0.099840     0.016797     0.010025     0.019236   \n",
       "12246     0.199247     2.191315     0.037783     0.004694     0.024901   \n",
       "6744      0.072502     0.153695     0.018550     0.008731     0.007635   \n",
       "5468      0.018205     0.024678     0.089413     0.011478     0.009871   \n",
       "15807     0.031967     0.012719     0.021267     0.016878     0.022939   \n",
       "3211      0.027890     0.047031     0.016678     0.007026     0.008113   \n",
       "\n",
       "       AREApch2018  AREApch2019  AREApch2020  AREApch2021  \n",
       "7742      0.009709     0.011727     0.005374     0.020958  \n",
       "13495     0.005713     0.008559     0.006239     0.022100  \n",
       "9524      0.008840     0.010887     0.010426     0.045929  \n",
       "2966      0.022100     0.009128     0.006603     0.010927  \n",
       "10840     0.015532     0.022942     0.020836     0.052435  \n",
       "12246     0.020094     0.054912     0.070746     0.124957  \n",
       "6744      0.012611     0.019586     0.014255     0.049765  \n",
       "5468      0.011610     0.008416     0.008121     0.019930  \n",
       "15807     0.016835     0.008795     0.005152     0.068007  \n",
       "3211      0.007082     0.007128     0.003999     0.012044  \n",
       "\n",
       "[10 rows x 214 columns]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'AreaChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eacb6568",
   "metadata": {},
   "source": [
    "#### NTL change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "ff131334",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stats = PlaceNames.copy().merge(VNL, how = 'outer', on='Sett_ID')\n",
    "# Stats = Stats.merge(DMSP, how = 'outer', on='Sett_ID')\n",
    "# Sensors = ['D_', 'V_']\n",
    "# Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "# for year in WSFE_Years:\n",
    "#     for Sensor in Sensors:\n",
    "#         for agg in Methods:\n",
    "#             RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "#             LagVar = ''.join(['NTL', agg, Sensor, str(year-1)])\n",
    "#             NewVar = ''.join(['NTL', agg, '_pch', Sensor, str(year)])\n",
    "#             if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "#                 Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "#             else:\n",
    "#                 pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "1104d70a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName'], dtype='object')"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "# Stats = Stats.loc[:, Stats.columns.str.contains('Sett|pch')]\n",
    "# Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92e2831",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stats.to_csv(os.path.join(Results, 'NTLChange.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76fd27f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stats.drop(columns='SettName', inplace=True)\n",
    "# AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "# AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdafa4cd",
   "metadata": {},
   "source": [
    "#### Flood change: change in area and change in percent of total area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfbea939",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Flood, how = 'outer', on='Sett_ID')\n",
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['FLDarea', str(year)])\n",
    "    LagVar = ''.join(['FLDarea', str(year-1)])\n",
    "    NewVar = ''.join(['FLDareapch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e14ed0c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['FLDpc', str(year)])\n",
    "    LagVar = ''.join(['FLDpc', str(year-1)])\n",
    "    NewVar = ''.join(['FLDpcpch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acdfb8b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|FLDareapch|FLDpcpch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9ebf6f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'FloodChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c165c1e6",
   "metadata": {},
   "source": [
    "#### Update parent spreadsheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "300f7a1c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 214 columns):\n",
      " #    Column           Dtype  \n",
      "---   ------           -----  \n",
      " 0    Sett_ID          int64  \n",
      " 1    SettName         object \n",
      " 2    GeoName          object \n",
      " 3    UCDB_Name        object \n",
      " 4    ADM_ID           int64  \n",
      " 5    AREA2021         float64\n",
      " 6    AREA2020         float64\n",
      " 7    AREA2019         float64\n",
      " 8    AREA2018         float64\n",
      " 9    AREA2017         float64\n",
      " 10   AREA2016         float64\n",
      " 11   AREA2015         float64\n",
      " 12   AREA2014         float64\n",
      " 13   AREA2013         float64\n",
      " 14   AREA2012         float64\n",
      " 15   AREA2011         float64\n",
      " 16   AREA2010         float64\n",
      " 17   AREA2009         float64\n",
      " 18   AREA2008         float64\n",
      " 19   AREA2007         float64\n",
      " 20   AREA2006         float64\n",
      " 21   AREA2005         float64\n",
      " 22   AREA2004         float64\n",
      " 23   AREA2003         float64\n",
      " 24   AREA2002         float64\n",
      " 25   AREA2001         float64\n",
      " 26   AREA2000         float64\n",
      " 27   AREA1999         float64\n",
      " 28   AREA1998         float64\n",
      " 29   AREA1997         float64\n",
      " 30   AREA1996         float64\n",
      " 31   AREA1995         float64\n",
      " 32   AREA1994         float64\n",
      " 33   AREA1993         float64\n",
      " 34   AREA1992         float64\n",
      " 35   AREA1991         float64\n",
      " 36   AREA1990         float64\n",
      " 37   AREA1989         float64\n",
      " 38   AREA1988         float64\n",
      " 39   AREA1987         float64\n",
      " 40   AREA1986         float64\n",
      " 41   AREA1985         float64\n",
      " 42   POP2000          float64\n",
      " 43   POP2001          float64\n",
      " 44   POP2002          float64\n",
      " 45   POP2003          float64\n",
      " 46   POP2004          float64\n",
      " 47   POP2005          float64\n",
      " 48   POP2006          float64\n",
      " 49   POP2007          float64\n",
      " 50   POP2008          float64\n",
      " 51   POP2009          float64\n",
      " 52   POP2010          float64\n",
      " 53   POP2011          float64\n",
      " 54   POP2012          float64\n",
      " 55   POP2013          float64\n",
      " 56   POP2014          float64\n",
      " 57   POP2015          float64\n",
      " 58   POP2016          float64\n",
      " 59   POP2017          float64\n",
      " 60   POP2018          float64\n",
      " 61   POP2019          float64\n",
      " 62   POP2020          float64\n",
      " 63   NTL_Dsum1999     float64\n",
      " 64   NTL_Dmax1999     float64\n",
      " 65   NTL_Dmean1999    float64\n",
      " 66   NTL_Dmedian1999  float64\n",
      " 67   NTL_Dstd1999     float64\n",
      " 68   NTL_Dsum2000     float64\n",
      " 69   NTL_Dmax2000     float64\n",
      " 70   NTL_Dmean2000    float64\n",
      " 71   NTL_Dmedian2000  float64\n",
      " 72   NTL_Dstd2000     float64\n",
      " 73   NTL_Dsum2001     float64\n",
      " 74   NTL_Dmax2001     float64\n",
      " 75   NTL_Dmean2001    float64\n",
      " 76   NTL_Dmedian2001  float64\n",
      " 77   NTL_Dstd2001     float64\n",
      " 78   NTL_Dsum2002     float64\n",
      " 79   NTL_Dmax2002     float64\n",
      " 80   NTL_Dmean2002    float64\n",
      " 81   NTL_Dmedian2002  float64\n",
      " 82   NTL_Dstd2002     float64\n",
      " 83   NTL_Dsum2003     float64\n",
      " 84   NTL_Dmax2003     float64\n",
      " 85   NTL_Dmean2003    float64\n",
      " 86   NTL_Dmedian2003  float64\n",
      " 87   NTL_Dstd2003     float64\n",
      " 88   NTL_Dsum2004     float64\n",
      " 89   NTL_Dmax2004     float64\n",
      " 90   NTL_Dmean2004    float64\n",
      " 91   NTL_Dmedian2004  float64\n",
      " 92   NTL_Dstd2004     float64\n",
      " 93   NTL_Dsum2005     float64\n",
      " 94   NTL_Dmax2005     float64\n",
      " 95   NTL_Dmean2005    float64\n",
      " 96   NTL_Dmedian2005  float64\n",
      " 97   NTL_Dstd2005     float64\n",
      " 98   NTL_Dsum2006     float64\n",
      " 99   NTL_Dmax2006     float64\n",
      " 100  NTL_Dmean2006    float64\n",
      " 101  NTL_Dmedian2006  float64\n",
      " 102  NTL_Dstd2006     float64\n",
      " 103  NTL_Dsum2007     float64\n",
      " 104  NTL_Dmax2007     float64\n",
      " 105  NTL_Dmean2007    float64\n",
      " 106  NTL_Dmedian2007  float64\n",
      " 107  NTL_Dstd2007     float64\n",
      " 108  NTL_Dsum2008     float64\n",
      " 109  NTL_Dmax2008     float64\n",
      " 110  NTL_Dmean2008    float64\n",
      " 111  NTL_Dmedian2008  float64\n",
      " 112  NTL_Dstd2008     float64\n",
      " 113  NTL_Dsum2009     float64\n",
      " 114  NTL_Dmax2009     float64\n",
      " 115  NTL_Dmean2009    float64\n",
      " 116  NTL_Dmedian2009  float64\n",
      " 117  NTL_Dstd2009     float64\n",
      " 118  NTL_Dsum2010     float64\n",
      " 119  NTL_Dmax2010     float64\n",
      " 120  NTL_Dmean2010    float64\n",
      " 121  NTL_Dmedian2010  float64\n",
      " 122  NTL_Dstd2010     float64\n",
      " 123  NTL_Dsum2011     float64\n",
      " 124  NTL_Dmax2011     float64\n",
      " 125  NTL_Dmean2011    float64\n",
      " 126  NTL_Dmedian2011  float64\n",
      " 127  NTL_Dstd2011     float64\n",
      " 128  NTL_Dsum2012     float64\n",
      " 129  NTL_Dmax2012     float64\n",
      " 130  NTL_Dmean2012    float64\n",
      " 131  NTL_Dmedian2012  float64\n",
      " 132  NTL_Dstd2012     float64\n",
      " 133  NTL_Dsum2013     float64\n",
      " 134  NTL_Dmax2013     float64\n",
      " 135  NTL_Dmean2013    float64\n",
      " 136  NTL_Dmedian2013  float64\n",
      " 137  NTL_Dstd2013     float64\n",
      " 138  NTL_Vsum2012     float64\n",
      " 139  NTL_Vmax2012     float64\n",
      " 140  NTL_Vmean2012    float64\n",
      " 141  NTL_Vmedian2012  float64\n",
      " 142  NTL_Vstd2012     float64\n",
      " 143  NTL_Vsum2013     float64\n",
      " 144  NTL_Vmax2013     float64\n",
      " 145  NTL_Vmean2013    float64\n",
      " 146  NTL_Vmedian2013  float64\n",
      " 147  NTL_Vstd2013     float64\n",
      " 148  NTL_Vsum2014     float64\n",
      " 149  NTL_Vmax2014     float64\n",
      " 150  NTL_Vmean2014    float64\n",
      " 151  NTL_Vmedian2014  float64\n",
      " 152  NTL_Vstd2014     float64\n",
      " 153  NTL_Vsum2015     float64\n",
      " 154  NTL_Vmax2015     float64\n",
      " 155  NTL_Vmean2015    float64\n",
      " 156  NTL_Vmedian2015  float64\n",
      " 157  NTL_Vstd2015     float64\n",
      " 158  POPpch2001       float64\n",
      " 159  POPpch2002       float64\n",
      " 160  POPpch2003       float64\n",
      " 161  POPpch2004       float64\n",
      " 162  POPpch2005       float64\n",
      " 163  POPpch2006       float64\n",
      " 164  POPpch2007       float64\n",
      " 165  POPpch2008       float64\n",
      " 166  POPpch2009       float64\n",
      " 167  POPpch2010       float64\n",
      " 168  POPpch2011       float64\n",
      " 169  POPpch2012       float64\n",
      " 170  POPpch2013       float64\n",
      " 171  POPpch2014       float64\n",
      " 172  POPpch2015       float64\n",
      " 173  POPpch2016       float64\n",
      " 174  POPpch2017       float64\n",
      " 175  POPpch2018       float64\n",
      " 176  POPpch2019       float64\n",
      " 177  POPpch2020       float64\n",
      " 178  AREApch1986      float64\n",
      " 179  AREApch1987      float64\n",
      " 180  AREApch1988      float64\n",
      " 181  AREApch1989      float64\n",
      " 182  AREApch1990      float64\n",
      " 183  AREApch1991      float64\n",
      " 184  AREApch1992      float64\n",
      " 185  AREApch1993      float64\n",
      " 186  AREApch1994      float64\n",
      " 187  AREApch1995      float64\n",
      " 188  AREApch1996      float64\n",
      " 189  AREApch1997      float64\n",
      " 190  AREApch1998      float64\n",
      " 191  AREApch1999      float64\n",
      " 192  AREApch2000      float64\n",
      " 193  AREApch2001      float64\n",
      " 194  AREApch2002      float64\n",
      " 195  AREApch2003      float64\n",
      " 196  AREApch2004      float64\n",
      " 197  AREApch2005      float64\n",
      " 198  AREApch2006      float64\n",
      " 199  AREApch2007      float64\n",
      " 200  AREApch2008      float64\n",
      " 201  AREApch2009      float64\n",
      " 202  AREApch2010      float64\n",
      " 203  AREApch2011      float64\n",
      " 204  AREApch2012      float64\n",
      " 205  AREApch2013      float64\n",
      " 206  AREApch2014      float64\n",
      " 207  AREApch2015      float64\n",
      " 208  AREApch2016      float64\n",
      " 209  AREApch2017      float64\n",
      " 210  AREApch2018      float64\n",
      " 211  AREApch2019      float64\n",
      " 212  AREApch2020      float64\n",
      " 213  AREApch2021      float64\n",
      "dtypes: float64(209), int64(2), object(3)\n",
      "memory usage: 30.4+ MB\n"
     ]
    }
   ],
   "source": [
    "AllStats.info(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "cef4598a",
   "metadata": {},
   "outputs": [],
   "source": [
    "AllStats.to_csv(os.path.join(Results, 'AllStats.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d074141",
   "metadata": {},
   "source": [
    "### 11.3 Densities\n",
    "POPden = people per square kilometer\n",
    "<br>NTL...den = nighttime light luminosity per square kilometer\n",
    "<br>NTL...pop = nighttime light luminosity per capita"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b08da5d",
   "metadata": {},
   "source": [
    "#### Population Density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "501f3dfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Population, how = 'outer', on='Sett_ID')\n",
    "Stats = Stats.merge(Areas, how='left', on='Sett_ID')\n",
    "\n",
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['POP', str(year)])\n",
    "    DenomVar = ''.join(['AREA', str(year)])\n",
    "    NewVar = ''.join(['POPden', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "        Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "20848ba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change in density\n",
    "for year in EligibleYears:\n",
    "    RawVar = ''.join(['POPden', str(year)])\n",
    "    LagVar = ''.join(['POPden', str(year-1)])\n",
    "    NewVar = ''.join(['POPdenpch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "f3edb9a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'GeoName', 'UCDB_Name', 'POPden2000',\n",
       "       'POPden2001', 'POPden2002', 'POPden2003', 'POPden2004', 'POPden2005',\n",
       "       'POPden2006', 'POPden2007', 'POPden2008', 'POPden2009', 'POPden2010',\n",
       "       'POPden2011', 'POPden2012', 'POPden2013', 'POPden2014', 'POPden2015',\n",
       "       'POPden2016', 'POPden2017', 'POPden2018', 'POPden2019', 'POPden2020',\n",
       "       'POPdenpch2001', 'POPdenpch2002', 'POPdenpch2003', 'POPdenpch2004',\n",
       "       'POPdenpch2005', 'POPdenpch2006', 'POPdenpch2007', 'POPdenpch2008',\n",
       "       'POPdenpch2009', 'POPdenpch2010', 'POPdenpch2011', 'POPdenpch2012',\n",
       "       'POPdenpch2013', 'POPdenpch2014', 'POPdenpch2015', 'POPdenpch2016',\n",
       "       'POPdenpch2017', 'POPdenpch2018', 'POPdenpch2019', 'POPdenpch2020'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, ~Stats.columns.str.contains('AREA|POP1|POP2|ct|year|ADM')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "08b74d6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>GeoName_x</th>\n",
       "      <th>UCDB_Name_x</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2021</th>\n",
       "      <th>AREA2020</th>\n",
       "      <th>AREA2019</th>\n",
       "      <th>AREA2018</th>\n",
       "      <th>AREA2017</th>\n",
       "      <th>...</th>\n",
       "      <th>POPdenpch2011</th>\n",
       "      <th>POPdenpch2012</th>\n",
       "      <th>POPdenpch2013</th>\n",
       "      <th>POPdenpch2014</th>\n",
       "      <th>POPdenpch2015</th>\n",
       "      <th>POPdenpch2016</th>\n",
       "      <th>POPdenpch2017</th>\n",
       "      <th>POPdenpch2018</th>\n",
       "      <th>POPdenpch2019</th>\n",
       "      <th>POPdenpch2020</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7742</th>\n",
       "      <td>38534</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Şaḩnāyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>28</td>\n",
       "      <td>281.043874</td>\n",
       "      <td>275.274712</td>\n",
       "      <td>273.803346</td>\n",
       "      <td>270.629542</td>\n",
       "      <td>268.027298</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.038499</td>\n",
       "      <td>-0.076240</td>\n",
       "      <td>-0.096420</td>\n",
       "      <td>-0.087340</td>\n",
       "      <td>-0.060974</td>\n",
       "      <td>-0.052324</td>\n",
       "      <td>-0.037829</td>\n",
       "      <td>-0.007820</td>\n",
       "      <td>-0.016166</td>\n",
       "      <td>0.034776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13495</th>\n",
       "      <td>96393</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Ḩuraytān</td>\n",
       "      <td>Aleppo</td>\n",
       "      <td>110</td>\n",
       "      <td>115.537029</td>\n",
       "      <td>113.038844</td>\n",
       "      <td>112.337975</td>\n",
       "      <td>111.384609</td>\n",
       "      <td>110.751838</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.096340</td>\n",
       "      <td>0.098128</td>\n",
       "      <td>-0.147359</td>\n",
       "      <td>-0.064020</td>\n",
       "      <td>-0.069914</td>\n",
       "      <td>-0.077973</td>\n",
       "      <td>-0.046414</td>\n",
       "      <td>0.031074</td>\n",
       "      <td>-0.060219</td>\n",
       "      <td>0.094340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9524</th>\n",
       "      <td>50649</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>Homs</td>\n",
       "      <td>115</td>\n",
       "      <td>61.194432</td>\n",
       "      <td>58.507258</td>\n",
       "      <td>57.903562</td>\n",
       "      <td>57.279972</td>\n",
       "      <td>56.778040</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.100613</td>\n",
       "      <td>0.054742</td>\n",
       "      <td>-0.145141</td>\n",
       "      <td>-0.069413</td>\n",
       "      <td>-0.065584</td>\n",
       "      <td>-0.042112</td>\n",
       "      <td>-0.081920</td>\n",
       "      <td>0.036575</td>\n",
       "      <td>-0.056459</td>\n",
       "      <td>0.067723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2966</th>\n",
       "      <td>10708</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Qadsayyā</td>\n",
       "      <td>Damascus</td>\n",
       "      <td>61</td>\n",
       "      <td>55.778766</td>\n",
       "      <td>55.175835</td>\n",
       "      <td>54.813923</td>\n",
       "      <td>54.318112</td>\n",
       "      <td>53.143621</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.045077</td>\n",
       "      <td>-0.104173</td>\n",
       "      <td>-0.053776</td>\n",
       "      <td>-0.144590</td>\n",
       "      <td>-0.092078</td>\n",
       "      <td>-0.032005</td>\n",
       "      <td>-0.079932</td>\n",
       "      <td>-0.026119</td>\n",
       "      <td>-0.014216</td>\n",
       "      <td>0.004600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10840</th>\n",
       "      <td>58135</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Ḩamāh</td>\n",
       "      <td>Hama</td>\n",
       "      <td>99</td>\n",
       "      <td>48.053139</td>\n",
       "      <td>45.659013</td>\n",
       "      <td>44.727072</td>\n",
       "      <td>43.723972</td>\n",
       "      <td>43.055239</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.112722</td>\n",
       "      <td>0.075918</td>\n",
       "      <td>-0.233991</td>\n",
       "      <td>-0.108413</td>\n",
       "      <td>-0.070002</td>\n",
       "      <td>-0.038450</td>\n",
       "      <td>-0.092636</td>\n",
       "      <td>0.023730</td>\n",
       "      <td>-0.064909</td>\n",
       "      <td>0.048079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12246</th>\n",
       "      <td>75655</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>Batabo</td>\n",
       "      <td>UNK</td>\n",
       "      <td>43</td>\n",
       "      <td>47.433375</td>\n",
       "      <td>42.164615</td>\n",
       "      <td>39.378737</td>\n",
       "      <td>37.328925</td>\n",
       "      <td>36.593625</td>\n",
       "      <td>...</td>\n",
       "      <td>0.003911</td>\n",
       "      <td>-0.279355</td>\n",
       "      <td>-0.178884</td>\n",
       "      <td>-0.701536</td>\n",
       "      <td>-0.087403</td>\n",
       "      <td>-0.098267</td>\n",
       "      <td>-0.055627</td>\n",
       "      <td>0.045792</td>\n",
       "      <td>-0.096687</td>\n",
       "      <td>0.002254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6744</th>\n",
       "      <td>32195</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As-Suwayda</td>\n",
       "      <td>As Suwayda</td>\n",
       "      <td>39</td>\n",
       "      <td>30.747184</td>\n",
       "      <td>29.289591</td>\n",
       "      <td>28.877945</td>\n",
       "      <td>28.323218</td>\n",
       "      <td>27.970488</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.144296</td>\n",
       "      <td>-0.003993</td>\n",
       "      <td>-0.282099</td>\n",
       "      <td>-0.135485</td>\n",
       "      <td>-0.077595</td>\n",
       "      <td>-0.040589</td>\n",
       "      <td>-0.057029</td>\n",
       "      <td>0.037237</td>\n",
       "      <td>-0.085094</td>\n",
       "      <td>0.066648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5468</th>\n",
       "      <td>24050</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>Latakia</td>\n",
       "      <td>104</td>\n",
       "      <td>27.997268</td>\n",
       "      <td>27.450192</td>\n",
       "      <td>27.229066</td>\n",
       "      <td>27.001819</td>\n",
       "      <td>26.691937</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.054514</td>\n",
       "      <td>0.042989</td>\n",
       "      <td>-0.119983</td>\n",
       "      <td>-0.020326</td>\n",
       "      <td>-0.130989</td>\n",
       "      <td>-0.055870</td>\n",
       "      <td>-0.056309</td>\n",
       "      <td>-0.005749</td>\n",
       "      <td>-0.034860</td>\n",
       "      <td>0.048306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15807</th>\n",
       "      <td>133699</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>Ar Raqqah</td>\n",
       "      <td>26</td>\n",
       "      <td>24.873198</td>\n",
       "      <td>23.289356</td>\n",
       "      <td>23.169994</td>\n",
       "      <td>22.967997</td>\n",
       "      <td>22.587722</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.166516</td>\n",
       "      <td>0.188762</td>\n",
       "      <td>-0.252824</td>\n",
       "      <td>0.014330</td>\n",
       "      <td>-0.160346</td>\n",
       "      <td>-0.091346</td>\n",
       "      <td>-0.113470</td>\n",
       "      <td>0.102645</td>\n",
       "      <td>-0.149383</td>\n",
       "      <td>0.167435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3211</th>\n",
       "      <td>11851</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>Al Kiswah</td>\n",
       "      <td>UNK</td>\n",
       "      <td>153</td>\n",
       "      <td>24.496748</td>\n",
       "      <td>24.205230</td>\n",
       "      <td>24.108822</td>\n",
       "      <td>23.938196</td>\n",
       "      <td>23.769865</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.060087</td>\n",
       "      <td>0.044952</td>\n",
       "      <td>-0.175179</td>\n",
       "      <td>-0.079626</td>\n",
       "      <td>-0.038102</td>\n",
       "      <td>-0.010625</td>\n",
       "      <td>-0.025680</td>\n",
       "      <td>-0.005981</td>\n",
       "      <td>-0.005433</td>\n",
       "      <td>0.064113</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 257 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID    SettName   GeoName_x UCDB_Name_x  ADM_ID    AREA2021  \\\n",
       "7742     38534     Şaḩnāyā     Şaḩnāyā    Damascus      28  281.043874   \n",
       "13495    96393    Ḩuraytān    Ḩuraytān      Aleppo     110  115.537029   \n",
       "9524     50649        Homs        Homs        Homs     115   61.194432   \n",
       "2966     10708    Qadsayyā    Qadsayyā    Damascus      61   55.778766   \n",
       "10840    58135       Ḩamāh       Ḩamāh        Hama      99   48.053139   \n",
       "12246    75655      Batabo      Batabo         UNK      43   47.433375   \n",
       "6744     32195  As-Suwayda  As-Suwayda  As Suwayda      39   30.747184   \n",
       "5468     24050     Latakia     Latakia     Latakia     104   27.997268   \n",
       "15807   133699   Ar Raqqah   Ar Raqqah   Ar Raqqah      26   24.873198   \n",
       "3211     11851   Al Kiswah   Al Kiswah         UNK     153   24.496748   \n",
       "\n",
       "         AREA2020    AREA2019    AREA2018    AREA2017  ...  POPdenpch2011  \\\n",
       "7742   275.274712  273.803346  270.629542  268.027298  ...      -0.038499   \n",
       "13495  113.038844  112.337975  111.384609  110.751838  ...      -0.096340   \n",
       "9524    58.507258   57.903562   57.279972   56.778040  ...      -0.100613   \n",
       "2966    55.175835   54.813923   54.318112   53.143621  ...      -0.045077   \n",
       "10840   45.659013   44.727072   43.723972   43.055239  ...      -0.112722   \n",
       "12246   42.164615   39.378737   37.328925   36.593625  ...       0.003911   \n",
       "6744    29.289591   28.877945   28.323218   27.970488  ...      -0.144296   \n",
       "5468    27.450192   27.229066   27.001819   26.691937  ...      -0.054514   \n",
       "15807   23.289356   23.169994   22.967997   22.587722  ...      -0.166516   \n",
       "3211    24.205230   24.108822   23.938196   23.769865  ...      -0.060087   \n",
       "\n",
       "       POPdenpch2012  POPdenpch2013  POPdenpch2014  POPdenpch2015  \\\n",
       "7742       -0.076240      -0.096420      -0.087340      -0.060974   \n",
       "13495       0.098128      -0.147359      -0.064020      -0.069914   \n",
       "9524        0.054742      -0.145141      -0.069413      -0.065584   \n",
       "2966       -0.104173      -0.053776      -0.144590      -0.092078   \n",
       "10840       0.075918      -0.233991      -0.108413      -0.070002   \n",
       "12246      -0.279355      -0.178884      -0.701536      -0.087403   \n",
       "6744       -0.003993      -0.282099      -0.135485      -0.077595   \n",
       "5468        0.042989      -0.119983      -0.020326      -0.130989   \n",
       "15807       0.188762      -0.252824       0.014330      -0.160346   \n",
       "3211        0.044952      -0.175179      -0.079626      -0.038102   \n",
       "\n",
       "       POPdenpch2016  POPdenpch2017  POPdenpch2018  POPdenpch2019  \\\n",
       "7742       -0.052324      -0.037829      -0.007820      -0.016166   \n",
       "13495      -0.077973      -0.046414       0.031074      -0.060219   \n",
       "9524       -0.042112      -0.081920       0.036575      -0.056459   \n",
       "2966       -0.032005      -0.079932      -0.026119      -0.014216   \n",
       "10840      -0.038450      -0.092636       0.023730      -0.064909   \n",
       "12246      -0.098267      -0.055627       0.045792      -0.096687   \n",
       "6744       -0.040589      -0.057029       0.037237      -0.085094   \n",
       "5468       -0.055870      -0.056309      -0.005749      -0.034860   \n",
       "15807      -0.091346      -0.113470       0.102645      -0.149383   \n",
       "3211       -0.010625      -0.025680      -0.005981      -0.005433   \n",
       "\n",
       "       POPdenpch2020  \n",
       "7742        0.034776  \n",
       "13495       0.094340  \n",
       "9524        0.067723  \n",
       "2966        0.004600  \n",
       "10840       0.048079  \n",
       "12246       0.002254  \n",
       "6744        0.066648  \n",
       "5468        0.048306  \n",
       "15807       0.167435  \n",
       "3211        0.064113  \n",
       "\n",
       "[10 rows x 257 columns]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'PopDensity.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b9e3a3",
   "metadata": {},
   "source": [
    "#### Nighttime Lights Density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ef877b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stats = PlaceNames.copy().merge(NTL, how = 'outer', on='Sett_ID')\n",
    "# Stats = Stats.merge(Areas, how='left', on='Sett_ID')\n",
    "# Sensors = ['D_', 'V_']\n",
    "# Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "# for year in EligibleYears:\n",
    "#     for Sensor in Sensors:\n",
    "#         for agg in Methods:\n",
    "#             RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "#             DenomVar = ''.join(['AREA', str(year)])\n",
    "#             NewVar = ''.join(['NTL', agg, '_den', Sensor, str(year)])\n",
    "#             if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "#                 Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "#             else:\n",
    "#                 pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "606d6807",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Change in density\n",
    "# for year in EligibleYears:\n",
    "#     for Sensor in Sensors:\n",
    "#         for agg in Methods:\n",
    "#             RawVar = ''.join(['NTL', agg, '_den', Sensor, str(year)])\n",
    "#             LagVar = ''.join(['NTL', agg, '_den', Sensor, str(year-1)])\n",
    "#             NewVar = ''.join(['NTL', agg, '_denpch', Sensor, str(year)])\n",
    "#             if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "#                 Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "#             else:\n",
    "#                 pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "d18b4f08",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'GeoName',\n",
       " 'UCDB_Name',\n",
       " 'POPden2000',\n",
       " 'POPden2001',\n",
       " 'POPden2002',\n",
       " 'POPden2003',\n",
       " 'POPden2004',\n",
       " 'POPden2005',\n",
       " 'POPden2006',\n",
       " 'POPden2007',\n",
       " 'POPden2008',\n",
       " 'POPden2009',\n",
       " 'POPden2010',\n",
       " 'POPden2011',\n",
       " 'POPden2012',\n",
       " 'POPden2013',\n",
       " 'POPden2014',\n",
       " 'POPden2015',\n",
       " 'POPden2016',\n",
       " 'POPden2017',\n",
       " 'POPden2018',\n",
       " 'POPden2019',\n",
       " 'POPden2020',\n",
       " 'POPdenpch2001',\n",
       " 'POPdenpch2002',\n",
       " 'POPdenpch2003',\n",
       " 'POPdenpch2004',\n",
       " 'POPdenpch2005',\n",
       " 'POPdenpch2006',\n",
       " 'POPdenpch2007',\n",
       " 'POPdenpch2008',\n",
       " 'POPdenpch2009',\n",
       " 'POPdenpch2010',\n",
       " 'POPdenpch2011',\n",
       " 'POPdenpch2012',\n",
       " 'POPdenpch2013',\n",
       " 'POPdenpch2014',\n",
       " 'POPdenpch2015',\n",
       " 'POPdenpch2016',\n",
       " 'POPdenpch2017',\n",
       " 'POPdenpch2018',\n",
       " 'POPdenpch2019',\n",
       " 'POPdenpch2020']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9ca39bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|den')]\n",
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "720e5a3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'NTLDensity.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8af7bc6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e4e01e37",
   "metadata": {},
   "source": [
    "#### Nighttime Lights per Capita"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74429c3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(NTL, how = 'outer', on='Sett_ID')\n",
    "Stats = Stats.merge(Population, how='left', on='Sett_ID')\n",
    "Sensors = ['D_', 'V_']\n",
    "Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "for year in EligibleYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "            DenomVar = ''.join(['POP', str(year)])\n",
    "            NewVar = ''.join(['NTL', agg, '_pop', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "                Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c950974",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change in density\n",
    "for year in EligibleYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, '_pop', Sensor, str(year)])\n",
    "            LagVar = ''.join(['NTL', agg, '_pop', Sensor, str(year-1)])\n",
    "            NewVar = ''.join(['NTL', agg, '_poppch', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "                Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d68f409e",
   "metadata": {},
   "outputs": [],
   "source": [
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2dc4c4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|_pop')]\n",
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc5e9320",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'NTLperCapita.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3be3e9c7",
   "metadata": {},
   "source": [
    "#### Update parent spreadsheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "7028c4ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 257 columns):\n",
      " #    Column           Dtype  \n",
      "---   ------           -----  \n",
      " 0    Sett_ID          int64  \n",
      " 1    SettName         object \n",
      " 2    GeoName_x        object \n",
      " 3    UCDB_Name_x      object \n",
      " 4    ADM_ID           int64  \n",
      " 5    AREA2021         float64\n",
      " 6    AREA2020         float64\n",
      " 7    AREA2019         float64\n",
      " 8    AREA2018         float64\n",
      " 9    AREA2017         float64\n",
      " 10   AREA2016         float64\n",
      " 11   AREA2015         float64\n",
      " 12   AREA2014         float64\n",
      " 13   AREA2013         float64\n",
      " 14   AREA2012         float64\n",
      " 15   AREA2011         float64\n",
      " 16   AREA2010         float64\n",
      " 17   AREA2009         float64\n",
      " 18   AREA2008         float64\n",
      " 19   AREA2007         float64\n",
      " 20   AREA2006         float64\n",
      " 21   AREA2005         float64\n",
      " 22   AREA2004         float64\n",
      " 23   AREA2003         float64\n",
      " 24   AREA2002         float64\n",
      " 25   AREA2001         float64\n",
      " 26   AREA2000         float64\n",
      " 27   AREA1999         float64\n",
      " 28   AREA1998         float64\n",
      " 29   AREA1997         float64\n",
      " 30   AREA1996         float64\n",
      " 31   AREA1995         float64\n",
      " 32   AREA1994         float64\n",
      " 33   AREA1993         float64\n",
      " 34   AREA1992         float64\n",
      " 35   AREA1991         float64\n",
      " 36   AREA1990         float64\n",
      " 37   AREA1989         float64\n",
      " 38   AREA1988         float64\n",
      " 39   AREA1987         float64\n",
      " 40   AREA1986         float64\n",
      " 41   AREA1985         float64\n",
      " 42   POP2000          float64\n",
      " 43   POP2001          float64\n",
      " 44   POP2002          float64\n",
      " 45   POP2003          float64\n",
      " 46   POP2004          float64\n",
      " 47   POP2005          float64\n",
      " 48   POP2006          float64\n",
      " 49   POP2007          float64\n",
      " 50   POP2008          float64\n",
      " 51   POP2009          float64\n",
      " 52   POP2010          float64\n",
      " 53   POP2011          float64\n",
      " 54   POP2012          float64\n",
      " 55   POP2013          float64\n",
      " 56   POP2014          float64\n",
      " 57   POP2015          float64\n",
      " 58   POP2016          float64\n",
      " 59   POP2017          float64\n",
      " 60   POP2018          float64\n",
      " 61   POP2019          float64\n",
      " 62   POP2020          float64\n",
      " 63   NTL_Dsum1999     float64\n",
      " 64   NTL_Dmax1999     float64\n",
      " 65   NTL_Dmean1999    float64\n",
      " 66   NTL_Dmedian1999  float64\n",
      " 67   NTL_Dstd1999     float64\n",
      " 68   NTL_Dsum2000     float64\n",
      " 69   NTL_Dmax2000     float64\n",
      " 70   NTL_Dmean2000    float64\n",
      " 71   NTL_Dmedian2000  float64\n",
      " 72   NTL_Dstd2000     float64\n",
      " 73   NTL_Dsum2001     float64\n",
      " 74   NTL_Dmax2001     float64\n",
      " 75   NTL_Dmean2001    float64\n",
      " 76   NTL_Dmedian2001  float64\n",
      " 77   NTL_Dstd2001     float64\n",
      " 78   NTL_Dsum2002     float64\n",
      " 79   NTL_Dmax2002     float64\n",
      " 80   NTL_Dmean2002    float64\n",
      " 81   NTL_Dmedian2002  float64\n",
      " 82   NTL_Dstd2002     float64\n",
      " 83   NTL_Dsum2003     float64\n",
      " 84   NTL_Dmax2003     float64\n",
      " 85   NTL_Dmean2003    float64\n",
      " 86   NTL_Dmedian2003  float64\n",
      " 87   NTL_Dstd2003     float64\n",
      " 88   NTL_Dsum2004     float64\n",
      " 89   NTL_Dmax2004     float64\n",
      " 90   NTL_Dmean2004    float64\n",
      " 91   NTL_Dmedian2004  float64\n",
      " 92   NTL_Dstd2004     float64\n",
      " 93   NTL_Dsum2005     float64\n",
      " 94   NTL_Dmax2005     float64\n",
      " 95   NTL_Dmean2005    float64\n",
      " 96   NTL_Dmedian2005  float64\n",
      " 97   NTL_Dstd2005     float64\n",
      " 98   NTL_Dsum2006     float64\n",
      " 99   NTL_Dmax2006     float64\n",
      " 100  NTL_Dmean2006    float64\n",
      " 101  NTL_Dmedian2006  float64\n",
      " 102  NTL_Dstd2006     float64\n",
      " 103  NTL_Dsum2007     float64\n",
      " 104  NTL_Dmax2007     float64\n",
      " 105  NTL_Dmean2007    float64\n",
      " 106  NTL_Dmedian2007  float64\n",
      " 107  NTL_Dstd2007     float64\n",
      " 108  NTL_Dsum2008     float64\n",
      " 109  NTL_Dmax2008     float64\n",
      " 110  NTL_Dmean2008    float64\n",
      " 111  NTL_Dmedian2008  float64\n",
      " 112  NTL_Dstd2008     float64\n",
      " 113  NTL_Dsum2009     float64\n",
      " 114  NTL_Dmax2009     float64\n",
      " 115  NTL_Dmean2009    float64\n",
      " 116  NTL_Dmedian2009  float64\n",
      " 117  NTL_Dstd2009     float64\n",
      " 118  NTL_Dsum2010     float64\n",
      " 119  NTL_Dmax2010     float64\n",
      " 120  NTL_Dmean2010    float64\n",
      " 121  NTL_Dmedian2010  float64\n",
      " 122  NTL_Dstd2010     float64\n",
      " 123  NTL_Dsum2011     float64\n",
      " 124  NTL_Dmax2011     float64\n",
      " 125  NTL_Dmean2011    float64\n",
      " 126  NTL_Dmedian2011  float64\n",
      " 127  NTL_Dstd2011     float64\n",
      " 128  NTL_Dsum2012     float64\n",
      " 129  NTL_Dmax2012     float64\n",
      " 130  NTL_Dmean2012    float64\n",
      " 131  NTL_Dmedian2012  float64\n",
      " 132  NTL_Dstd2012     float64\n",
      " 133  NTL_Dsum2013     float64\n",
      " 134  NTL_Dmax2013     float64\n",
      " 135  NTL_Dmean2013    float64\n",
      " 136  NTL_Dmedian2013  float64\n",
      " 137  NTL_Dstd2013     float64\n",
      " 138  NTL_Vsum2012     float64\n",
      " 139  NTL_Vmax2012     float64\n",
      " 140  NTL_Vmean2012    float64\n",
      " 141  NTL_Vmedian2012  float64\n",
      " 142  NTL_Vstd2012     float64\n",
      " 143  NTL_Vsum2013     float64\n",
      " 144  NTL_Vmax2013     float64\n",
      " 145  NTL_Vmean2013    float64\n",
      " 146  NTL_Vmedian2013  float64\n",
      " 147  NTL_Vstd2013     float64\n",
      " 148  NTL_Vsum2014     float64\n",
      " 149  NTL_Vmax2014     float64\n",
      " 150  NTL_Vmean2014    float64\n",
      " 151  NTL_Vmedian2014  float64\n",
      " 152  NTL_Vstd2014     float64\n",
      " 153  NTL_Vsum2015     float64\n",
      " 154  NTL_Vmax2015     float64\n",
      " 155  NTL_Vmean2015    float64\n",
      " 156  NTL_Vmedian2015  float64\n",
      " 157  NTL_Vstd2015     float64\n",
      " 158  POPpch2001       float64\n",
      " 159  POPpch2002       float64\n",
      " 160  POPpch2003       float64\n",
      " 161  POPpch2004       float64\n",
      " 162  POPpch2005       float64\n",
      " 163  POPpch2006       float64\n",
      " 164  POPpch2007       float64\n",
      " 165  POPpch2008       float64\n",
      " 166  POPpch2009       float64\n",
      " 167  POPpch2010       float64\n",
      " 168  POPpch2011       float64\n",
      " 169  POPpch2012       float64\n",
      " 170  POPpch2013       float64\n",
      " 171  POPpch2014       float64\n",
      " 172  POPpch2015       float64\n",
      " 173  POPpch2016       float64\n",
      " 174  POPpch2017       float64\n",
      " 175  POPpch2018       float64\n",
      " 176  POPpch2019       float64\n",
      " 177  POPpch2020       float64\n",
      " 178  AREApch1986      float64\n",
      " 179  AREApch1987      float64\n",
      " 180  AREApch1988      float64\n",
      " 181  AREApch1989      float64\n",
      " 182  AREApch1990      float64\n",
      " 183  AREApch1991      float64\n",
      " 184  AREApch1992      float64\n",
      " 185  AREApch1993      float64\n",
      " 186  AREApch1994      float64\n",
      " 187  AREApch1995      float64\n",
      " 188  AREApch1996      float64\n",
      " 189  AREApch1997      float64\n",
      " 190  AREApch1998      float64\n",
      " 191  AREApch1999      float64\n",
      " 192  AREApch2000      float64\n",
      " 193  AREApch2001      float64\n",
      " 194  AREApch2002      float64\n",
      " 195  AREApch2003      float64\n",
      " 196  AREApch2004      float64\n",
      " 197  AREApch2005      float64\n",
      " 198  AREApch2006      float64\n",
      " 199  AREApch2007      float64\n",
      " 200  AREApch2008      float64\n",
      " 201  AREApch2009      float64\n",
      " 202  AREApch2010      float64\n",
      " 203  AREApch2011      float64\n",
      " 204  AREApch2012      float64\n",
      " 205  AREApch2013      float64\n",
      " 206  AREApch2014      float64\n",
      " 207  AREApch2015      float64\n",
      " 208  AREApch2016      float64\n",
      " 209  AREApch2017      float64\n",
      " 210  AREApch2018      float64\n",
      " 211  AREApch2019      float64\n",
      " 212  AREApch2020      float64\n",
      " 213  AREApch2021      float64\n",
      " 214  GeoName_y        object \n",
      " 215  UCDB_Name_y      object \n",
      " 216  POPden2000       float64\n",
      " 217  POPden2001       float64\n",
      " 218  POPden2002       float64\n",
      " 219  POPden2003       float64\n",
      " 220  POPden2004       float64\n",
      " 221  POPden2005       float64\n",
      " 222  POPden2006       float64\n",
      " 223  POPden2007       float64\n",
      " 224  POPden2008       float64\n",
      " 225  POPden2009       float64\n",
      " 226  POPden2010       float64\n",
      " 227  POPden2011       float64\n",
      " 228  POPden2012       float64\n",
      " 229  POPden2013       float64\n",
      " 230  POPden2014       float64\n",
      " 231  POPden2015       float64\n",
      " 232  POPden2016       float64\n",
      " 233  POPden2017       float64\n",
      " 234  POPden2018       float64\n",
      " 235  POPden2019       float64\n",
      " 236  POPden2020       float64\n",
      " 237  POPdenpch2001    float64\n",
      " 238  POPdenpch2002    float64\n",
      " 239  POPdenpch2003    float64\n",
      " 240  POPdenpch2004    float64\n",
      " 241  POPdenpch2005    float64\n",
      " 242  POPdenpch2006    float64\n",
      " 243  POPdenpch2007    float64\n",
      " 244  POPdenpch2008    float64\n",
      " 245  POPdenpch2009    float64\n",
      " 246  POPdenpch2010    float64\n",
      " 247  POPdenpch2011    float64\n",
      " 248  POPdenpch2012    float64\n",
      " 249  POPdenpch2013    float64\n",
      " 250  POPdenpch2014    float64\n",
      " 251  POPdenpch2015    float64\n",
      " 252  POPdenpch2016    float64\n",
      " 253  POPdenpch2017    float64\n",
      " 254  POPdenpch2018    float64\n",
      " 255  POPdenpch2019    float64\n",
      " 256  POPdenpch2020    float64\n",
      "dtypes: float64(250), int64(2), object(5)\n",
      "memory usage: 36.5+ MB\n"
     ]
    }
   ],
   "source": [
    "AllStats.info(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "45c78bba",
   "metadata": {},
   "outputs": [],
   "source": [
    "AllStats.to_csv(os.path.join(Results, 'AllStats.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fe10b2",
   "metadata": {},
   "source": [
    "### 11.4 Urban Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "5067fcda",
   "metadata": {},
   "outputs": [],
   "source": [
    "for year in EligibleYears:\n",
    "    PopVar = ''.join(['POP', str(year)])\n",
    "    DenVar = ''.join(['POPden', str(year)])\n",
    "    NewVar = ''.join(['UrbType', str(year)])\n",
    "    if ((PopVar in AllStats.columns) and (DenVar in AllStats.columns)):\n",
    "        AllStats[NewVar] = 'LD'\n",
    "        AllStats.loc[(AllStats[PopVar] >= 5000) & (AllStats[DenVar] >= 300), NewVar] = 'SDurban'\n",
    "        AllStats.loc[(AllStats[PopVar] >= 50000) & (AllStats[DenVar] >= 1500), NewVar] = 'HDurban'\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "793e52a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18552 entries, 0 to 18551\n",
      "Data columns (total 278 columns):\n",
      " #    Column           Dtype  \n",
      "---   ------           -----  \n",
      " 0    Sett_ID          int64  \n",
      " 1    SettName         object \n",
      " 2    GeoName_x        object \n",
      " 3    UCDB_Name_x      object \n",
      " 4    ADM_ID           int64  \n",
      " 5    AREA2021         float64\n",
      " 6    AREA2020         float64\n",
      " 7    AREA2019         float64\n",
      " 8    AREA2018         float64\n",
      " 9    AREA2017         float64\n",
      " 10   AREA2016         float64\n",
      " 11   AREA2015         float64\n",
      " 12   AREA2014         float64\n",
      " 13   AREA2013         float64\n",
      " 14   AREA2012         float64\n",
      " 15   AREA2011         float64\n",
      " 16   AREA2010         float64\n",
      " 17   AREA2009         float64\n",
      " 18   AREA2008         float64\n",
      " 19   AREA2007         float64\n",
      " 20   AREA2006         float64\n",
      " 21   AREA2005         float64\n",
      " 22   AREA2004         float64\n",
      " 23   AREA2003         float64\n",
      " 24   AREA2002         float64\n",
      " 25   AREA2001         float64\n",
      " 26   AREA2000         float64\n",
      " 27   AREA1999         float64\n",
      " 28   AREA1998         float64\n",
      " 29   AREA1997         float64\n",
      " 30   AREA1996         float64\n",
      " 31   AREA1995         float64\n",
      " 32   AREA1994         float64\n",
      " 33   AREA1993         float64\n",
      " 34   AREA1992         float64\n",
      " 35   AREA1991         float64\n",
      " 36   AREA1990         float64\n",
      " 37   AREA1989         float64\n",
      " 38   AREA1988         float64\n",
      " 39   AREA1987         float64\n",
      " 40   AREA1986         float64\n",
      " 41   AREA1985         float64\n",
      " 42   POP2000          float64\n",
      " 43   POP2001          float64\n",
      " 44   POP2002          float64\n",
      " 45   POP2003          float64\n",
      " 46   POP2004          float64\n",
      " 47   POP2005          float64\n",
      " 48   POP2006          float64\n",
      " 49   POP2007          float64\n",
      " 50   POP2008          float64\n",
      " 51   POP2009          float64\n",
      " 52   POP2010          float64\n",
      " 53   POP2011          float64\n",
      " 54   POP2012          float64\n",
      " 55   POP2013          float64\n",
      " 56   POP2014          float64\n",
      " 57   POP2015          float64\n",
      " 58   POP2016          float64\n",
      " 59   POP2017          float64\n",
      " 60   POP2018          float64\n",
      " 61   POP2019          float64\n",
      " 62   POP2020          float64\n",
      " 63   NTL_Dsum1999     float64\n",
      " 64   NTL_Dmax1999     float64\n",
      " 65   NTL_Dmean1999    float64\n",
      " 66   NTL_Dmedian1999  float64\n",
      " 67   NTL_Dstd1999     float64\n",
      " 68   NTL_Dsum2000     float64\n",
      " 69   NTL_Dmax2000     float64\n",
      " 70   NTL_Dmean2000    float64\n",
      " 71   NTL_Dmedian2000  float64\n",
      " 72   NTL_Dstd2000     float64\n",
      " 73   NTL_Dsum2001     float64\n",
      " 74   NTL_Dmax2001     float64\n",
      " 75   NTL_Dmean2001    float64\n",
      " 76   NTL_Dmedian2001  float64\n",
      " 77   NTL_Dstd2001     float64\n",
      " 78   NTL_Dsum2002     float64\n",
      " 79   NTL_Dmax2002     float64\n",
      " 80   NTL_Dmean2002    float64\n",
      " 81   NTL_Dmedian2002  float64\n",
      " 82   NTL_Dstd2002     float64\n",
      " 83   NTL_Dsum2003     float64\n",
      " 84   NTL_Dmax2003     float64\n",
      " 85   NTL_Dmean2003    float64\n",
      " 86   NTL_Dmedian2003  float64\n",
      " 87   NTL_Dstd2003     float64\n",
      " 88   NTL_Dsum2004     float64\n",
      " 89   NTL_Dmax2004     float64\n",
      " 90   NTL_Dmean2004    float64\n",
      " 91   NTL_Dmedian2004  float64\n",
      " 92   NTL_Dstd2004     float64\n",
      " 93   NTL_Dsum2005     float64\n",
      " 94   NTL_Dmax2005     float64\n",
      " 95   NTL_Dmean2005    float64\n",
      " 96   NTL_Dmedian2005  float64\n",
      " 97   NTL_Dstd2005     float64\n",
      " 98   NTL_Dsum2006     float64\n",
      " 99   NTL_Dmax2006     float64\n",
      " 100  NTL_Dmean2006    float64\n",
      " 101  NTL_Dmedian2006  float64\n",
      " 102  NTL_Dstd2006     float64\n",
      " 103  NTL_Dsum2007     float64\n",
      " 104  NTL_Dmax2007     float64\n",
      " 105  NTL_Dmean2007    float64\n",
      " 106  NTL_Dmedian2007  float64\n",
      " 107  NTL_Dstd2007     float64\n",
      " 108  NTL_Dsum2008     float64\n",
      " 109  NTL_Dmax2008     float64\n",
      " 110  NTL_Dmean2008    float64\n",
      " 111  NTL_Dmedian2008  float64\n",
      " 112  NTL_Dstd2008     float64\n",
      " 113  NTL_Dsum2009     float64\n",
      " 114  NTL_Dmax2009     float64\n",
      " 115  NTL_Dmean2009    float64\n",
      " 116  NTL_Dmedian2009  float64\n",
      " 117  NTL_Dstd2009     float64\n",
      " 118  NTL_Dsum2010     float64\n",
      " 119  NTL_Dmax2010     float64\n",
      " 120  NTL_Dmean2010    float64\n",
      " 121  NTL_Dmedian2010  float64\n",
      " 122  NTL_Dstd2010     float64\n",
      " 123  NTL_Dsum2011     float64\n",
      " 124  NTL_Dmax2011     float64\n",
      " 125  NTL_Dmean2011    float64\n",
      " 126  NTL_Dmedian2011  float64\n",
      " 127  NTL_Dstd2011     float64\n",
      " 128  NTL_Dsum2012     float64\n",
      " 129  NTL_Dmax2012     float64\n",
      " 130  NTL_Dmean2012    float64\n",
      " 131  NTL_Dmedian2012  float64\n",
      " 132  NTL_Dstd2012     float64\n",
      " 133  NTL_Dsum2013     float64\n",
      " 134  NTL_Dmax2013     float64\n",
      " 135  NTL_Dmean2013    float64\n",
      " 136  NTL_Dmedian2013  float64\n",
      " 137  NTL_Dstd2013     float64\n",
      " 138  NTL_Vsum2012     float64\n",
      " 139  NTL_Vmax2012     float64\n",
      " 140  NTL_Vmean2012    float64\n",
      " 141  NTL_Vmedian2012  float64\n",
      " 142  NTL_Vstd2012     float64\n",
      " 143  NTL_Vsum2013     float64\n",
      " 144  NTL_Vmax2013     float64\n",
      " 145  NTL_Vmean2013    float64\n",
      " 146  NTL_Vmedian2013  float64\n",
      " 147  NTL_Vstd2013     float64\n",
      " 148  NTL_Vsum2014     float64\n",
      " 149  NTL_Vmax2014     float64\n",
      " 150  NTL_Vmean2014    float64\n",
      " 151  NTL_Vmedian2014  float64\n",
      " 152  NTL_Vstd2014     float64\n",
      " 153  NTL_Vsum2015     float64\n",
      " 154  NTL_Vmax2015     float64\n",
      " 155  NTL_Vmean2015    float64\n",
      " 156  NTL_Vmedian2015  float64\n",
      " 157  NTL_Vstd2015     float64\n",
      " 158  POPpch2001       float64\n",
      " 159  POPpch2002       float64\n",
      " 160  POPpch2003       float64\n",
      " 161  POPpch2004       float64\n",
      " 162  POPpch2005       float64\n",
      " 163  POPpch2006       float64\n",
      " 164  POPpch2007       float64\n",
      " 165  POPpch2008       float64\n",
      " 166  POPpch2009       float64\n",
      " 167  POPpch2010       float64\n",
      " 168  POPpch2011       float64\n",
      " 169  POPpch2012       float64\n",
      " 170  POPpch2013       float64\n",
      " 171  POPpch2014       float64\n",
      " 172  POPpch2015       float64\n",
      " 173  POPpch2016       float64\n",
      " 174  POPpch2017       float64\n",
      " 175  POPpch2018       float64\n",
      " 176  POPpch2019       float64\n",
      " 177  POPpch2020       float64\n",
      " 178  AREApch1986      float64\n",
      " 179  AREApch1987      float64\n",
      " 180  AREApch1988      float64\n",
      " 181  AREApch1989      float64\n",
      " 182  AREApch1990      float64\n",
      " 183  AREApch1991      float64\n",
      " 184  AREApch1992      float64\n",
      " 185  AREApch1993      float64\n",
      " 186  AREApch1994      float64\n",
      " 187  AREApch1995      float64\n",
      " 188  AREApch1996      float64\n",
      " 189  AREApch1997      float64\n",
      " 190  AREApch1998      float64\n",
      " 191  AREApch1999      float64\n",
      " 192  AREApch2000      float64\n",
      " 193  AREApch2001      float64\n",
      " 194  AREApch2002      float64\n",
      " 195  AREApch2003      float64\n",
      " 196  AREApch2004      float64\n",
      " 197  AREApch2005      float64\n",
      " 198  AREApch2006      float64\n",
      " 199  AREApch2007      float64\n",
      " 200  AREApch2008      float64\n",
      " 201  AREApch2009      float64\n",
      " 202  AREApch2010      float64\n",
      " 203  AREApch2011      float64\n",
      " 204  AREApch2012      float64\n",
      " 205  AREApch2013      float64\n",
      " 206  AREApch2014      float64\n",
      " 207  AREApch2015      float64\n",
      " 208  AREApch2016      float64\n",
      " 209  AREApch2017      float64\n",
      " 210  AREApch2018      float64\n",
      " 211  AREApch2019      float64\n",
      " 212  AREApch2020      float64\n",
      " 213  AREApch2021      float64\n",
      " 214  GeoName_y        object \n",
      " 215  UCDB_Name_y      object \n",
      " 216  POPden2000       float64\n",
      " 217  POPden2001       float64\n",
      " 218  POPden2002       float64\n",
      " 219  POPden2003       float64\n",
      " 220  POPden2004       float64\n",
      " 221  POPden2005       float64\n",
      " 222  POPden2006       float64\n",
      " 223  POPden2007       float64\n",
      " 224  POPden2008       float64\n",
      " 225  POPden2009       float64\n",
      " 226  POPden2010       float64\n",
      " 227  POPden2011       float64\n",
      " 228  POPden2012       float64\n",
      " 229  POPden2013       float64\n",
      " 230  POPden2014       float64\n",
      " 231  POPden2015       float64\n",
      " 232  POPden2016       float64\n",
      " 233  POPden2017       float64\n",
      " 234  POPden2018       float64\n",
      " 235  POPden2019       float64\n",
      " 236  POPden2020       float64\n",
      " 237  POPdenpch2001    float64\n",
      " 238  POPdenpch2002    float64\n",
      " 239  POPdenpch2003    float64\n",
      " 240  POPdenpch2004    float64\n",
      " 241  POPdenpch2005    float64\n",
      " 242  POPdenpch2006    float64\n",
      " 243  POPdenpch2007    float64\n",
      " 244  POPdenpch2008    float64\n",
      " 245  POPdenpch2009    float64\n",
      " 246  POPdenpch2010    float64\n",
      " 247  POPdenpch2011    float64\n",
      " 248  POPdenpch2012    float64\n",
      " 249  POPdenpch2013    float64\n",
      " 250  POPdenpch2014    float64\n",
      " 251  POPdenpch2015    float64\n",
      " 252  POPdenpch2016    float64\n",
      " 253  POPdenpch2017    float64\n",
      " 254  POPdenpch2018    float64\n",
      " 255  POPdenpch2019    float64\n",
      " 256  POPdenpch2020    float64\n",
      " 257  UrbType2000      object \n",
      " 258  UrbType2001      object \n",
      " 259  UrbType2002      object \n",
      " 260  UrbType2003      object \n",
      " 261  UrbType2004      object \n",
      " 262  UrbType2005      object \n",
      " 263  UrbType2006      object \n",
      " 264  UrbType2007      object \n",
      " 265  UrbType2008      object \n",
      " 266  UrbType2009      object \n",
      " 267  UrbType2010      object \n",
      " 268  UrbType2011      object \n",
      " 269  UrbType2012      object \n",
      " 270  UrbType2013      object \n",
      " 271  UrbType2014      object \n",
      " 272  UrbType2015      object \n",
      " 273  UrbType2016      object \n",
      " 274  UrbType2017      object \n",
      " 275  UrbType2018      object \n",
      " 276  UrbType2019      object \n",
      " 277  UrbType2020      object \n",
      "dtypes: float64(250), int64(2), object(26)\n",
      "memory usage: 39.5+ MB\n"
     ]
    }
   ],
   "source": [
    "AllStats.info(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "7c694c30",
   "metadata": {},
   "outputs": [],
   "source": [
    "AllStats.to_csv(os.path.join(Results, 'AllStats.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "cb7ef1c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'SettName',\n",
       " 'UrbType2000',\n",
       " 'UrbType2001',\n",
       " 'UrbType2002',\n",
       " 'UrbType2003',\n",
       " 'UrbType2004',\n",
       " 'UrbType2005',\n",
       " 'UrbType2006',\n",
       " 'UrbType2007',\n",
       " 'UrbType2008',\n",
       " 'UrbType2009',\n",
       " 'UrbType2010',\n",
       " 'UrbType2011',\n",
       " 'UrbType2012',\n",
       " 'UrbType2013',\n",
       " 'UrbType2014',\n",
       " 'UrbType2015',\n",
       " 'UrbType2016',\n",
       " 'UrbType2017',\n",
       " 'UrbType2018',\n",
       " 'UrbType2019',\n",
       " 'UrbType2020']"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats = AllStats.loc[:, AllStats.columns.str.contains('Sett|UrbType')]\n",
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "93c48bef",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>UrbType2000</th>\n",
       "      <th>UrbType2001</th>\n",
       "      <th>UrbType2002</th>\n",
       "      <th>UrbType2003</th>\n",
       "      <th>UrbType2004</th>\n",
       "      <th>UrbType2005</th>\n",
       "      <th>UrbType2006</th>\n",
       "      <th>UrbType2007</th>\n",
       "      <th>...</th>\n",
       "      <th>UrbType2011</th>\n",
       "      <th>UrbType2012</th>\n",
       "      <th>UrbType2013</th>\n",
       "      <th>UrbType2014</th>\n",
       "      <th>UrbType2015</th>\n",
       "      <th>UrbType2016</th>\n",
       "      <th>UrbType2017</th>\n",
       "      <th>UrbType2018</th>\n",
       "      <th>UrbType2019</th>\n",
       "      <th>UrbType2020</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14553</th>\n",
       "      <td>107002</td>\n",
       "      <td>Māri‘</td>\n",
       "      <td>LD</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>LD</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>LD</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4953</th>\n",
       "      <td>20808</td>\n",
       "      <td>Khirbat al Ma‘azzah</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15236</th>\n",
       "      <td>122595</td>\n",
       "      <td>Jarābulus</td>\n",
       "      <td>LD</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>LD</td>\n",
       "      <td>LD</td>\n",
       "      <td>LD</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12718</th>\n",
       "      <td>80810</td>\n",
       "      <td>‘Afrīn</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15482</th>\n",
       "      <td>126346</td>\n",
       "      <td>Ath Thawrah</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5040</th>\n",
       "      <td>21342</td>\n",
       "      <td>Şāfītā</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9219</th>\n",
       "      <td>48944</td>\n",
       "      <td>Al Quşayr</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15684</th>\n",
       "      <td>131596</td>\n",
       "      <td>‘Ayn al ‘Arab</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4954</th>\n",
       "      <td>20811</td>\n",
       "      <td>Ţarţūs</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4861</th>\n",
       "      <td>20308</td>\n",
       "      <td>Baniyas</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>...</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "      <td>SDurban</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID             SettName UrbType2000 UrbType2001 UrbType2002  \\\n",
       "14553   107002                Māri‘          LD     SDurban     SDurban   \n",
       "4953     20808  Khirbat al Ma‘azzah     SDurban     SDurban     SDurban   \n",
       "15236   122595            Jarābulus          LD     SDurban     SDurban   \n",
       "12718    80810               ‘Afrīn     SDurban     SDurban     SDurban   \n",
       "15482   126346          Ath Thawrah     SDurban     SDurban     SDurban   \n",
       "5040     21342               Şāfītā     SDurban     SDurban     SDurban   \n",
       "9219     48944            Al Quşayr     SDurban     SDurban     SDurban   \n",
       "15684   131596        ‘Ayn al ‘Arab     SDurban     SDurban     SDurban   \n",
       "4954     20811               Ţarţūs     SDurban     SDurban     SDurban   \n",
       "4861     20308              Baniyas     SDurban     SDurban     SDurban   \n",
       "\n",
       "      UrbType2003 UrbType2004 UrbType2005 UrbType2006 UrbType2007  ...  \\\n",
       "14553     SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "4953      SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "15236     SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "12718     SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "15482     SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "5040      SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "9219      SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "15684     SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "4954      SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "4861      SDurban     SDurban     SDurban     SDurban     SDurban  ...   \n",
       "\n",
       "      UrbType2011 UrbType2012 UrbType2013 UrbType2014 UrbType2015 UrbType2016  \\\n",
       "14553     SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "4953      SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "15236     SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "12718     SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "15482     SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "5040      SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "9219      SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "15684     SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "4954      SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "4861      SDurban     SDurban     SDurban     SDurban     SDurban     SDurban   \n",
       "\n",
       "      UrbType2017 UrbType2018 UrbType2019 UrbType2020  \n",
       "14553          LD     SDurban          LD     SDurban  \n",
       "4953      SDurban     SDurban     SDurban     SDurban  \n",
       "15236          LD          LD          LD     SDurban  \n",
       "12718     SDurban     SDurban     SDurban     SDurban  \n",
       "15482     SDurban     SDurban     SDurban     SDurban  \n",
       "5040      SDurban     SDurban     SDurban     SDurban  \n",
       "9219      SDurban     SDurban     SDurban     SDurban  \n",
       "15684     SDurban     SDurban     SDurban     SDurban  \n",
       "4954      SDurban     SDurban     SDurban     SDurban  \n",
       "4861      SDurban     SDurban     SDurban     SDurban  \n",
       "\n",
       "[10 rows x 23 columns]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats[Stats.SettName!='UNK'].sort_values(by=Stats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "4f6f1d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats.to_csv(os.path.join(Results, 'UrbanType.csv'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}