2019_spd/sunpy-1.0-spd2019-aquiring-data-and-data-classes.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sunpy.map\n",
    "from sunpy.coordinates import frames\n",
    "from sunpy.net import Fido, attrs as a\n",
    "from astropy import units as u\n",
    "from astropy.coordinates import SkyCoord\n",
    "from sunpy import timeseries as ts\n",
    "from sunpy.coordinates import get_body_heliographic_stonyhurst\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (16,8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Aquiring Data and SunPy Data Classes\n",
    "In this example notebook we will go over the basics of using SunPy to access, download, load and analyse solar data. \n",
    "\n",
    "We will first look at how SunPy's FIDO functionality allows us to search and download data using a unified interface, and how we can go about making queries of interest. We will then look at how we can load and analyse timeseries data using the SunPy TimeSeries data class, and similarly use the SunPy Map class to load and analyse image data in a specific coordinate frame.\n",
    "\n",
    "Lets first search for and download data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fido Unified Downloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fido (Federated Internet Data Obtainer) is a unified interface for the search and retrival of solar physics data regardless of the underlying client or webservice through which the data is obtained, e.g. VSO, JSOC, etc. \n",
    "\n",
    "In SunPy 1.0 Fido now makes use of an asynchronous download stream through the module `parfive`\n",
    "\n",
    "### Lets look at how we can search and download some solar physics data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#lets search for some SDO/AIA data over some time\n",
    "results = Fido.search(a.Time('2014/05/15 08:00', '2014/05/15 08:10'), a.Instrument('AIA'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This returns a `UnifiedResponse` object that holds the information about the search "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find out the number of files returned in the search we use the `.file_num` attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.file_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Queries to Fido can be make more specific by using other attributes. Lets narrow the search to only look at one wavelength, with a cadence of 1 minute. This makes use of Astropy units "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = Fido.search(a.Time('2014/05/15 08:00', '2014/05/15 08:10'), a.Instrument('AIA'), \n",
    "                      a.Wavelength(17.1*u.nm), a.vso.Sample(1*u.minute))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can then downloaded the data via Fido's `fetch` function\n",
    "Once the files of interest have been located from the search query via `Fido.search`, they can be downloaded using `Fido.fetch` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aia_files = Fido.fetch(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will download the files to a path defined in your sunpy config file. The function returns a `parfive.Results` object, in this case stored in `aia_files`. We can list `aia_files` to get the file paths of where the data was downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aia_files[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If however you want the files to be downloaded to a certain directory, you can specifiy the path for which you would like the data to be downloaded to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aia_files_specific = Fido.fetch(results, path='~/sunpy/spd/{instrument}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aia_files_specific"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More detailed searchs with Fido\n",
    "Fido searches can include searchs from multiple instruments, wavelengths, times etc. This is achieved through the | opertaior which acts as a logical OR operator.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = Fido.search(a.Time('2014/05/15 08:00', '2014/05/15 08:10'), a.Instrument('AIA')|a.Instrument('SWAP'))\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the results can be indexed to access a subset of the search results\n",
    "results[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = Fido.fetch(results[:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing Data\n",
    "\n",
    "SunPy provides core data type classes that are designed to provide a consistent interface acress data types (lightcurves and images) as well as data sources from numerous instruments and observations. They handle all of the manipulation necessary to read data in from mission-specific files.  The two main datatypes in SunPy are the TimeSeries and Map classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TimeSeries\n",
    "The structure of a TimeSeries consists of times and measurements and the underlying structure is that of a `pandas.DataFrame`. SunPy TimeSeries supports time-series data from a wide range of solar-focussed instruments.\n",
    "\n",
    "The `TimeSeries` class now supersedes the `LightCurve` object which is now deprecated in SunPy 1.0. The main difference is that `TimeSeries` does not download the data, which is now done through `Fido` as described above. \n",
    "\n",
    "Lets look at an example of using TimeSeries to deal with GOES/XRS observations. First we query and download the data using `Fido`, and then load the downloaded file into the `TimeSeries` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_results = Fido.search(a.Time('2013-10-28 00:00','2013-10-28 12:00'), a.Instrument('XRS'))\n",
    "goes_files = Fido.fetch(search_results[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load this downloaded file into `TimeSeries` which will automatically detect the instrument source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goes_lc = ts.TimeSeries(goes_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goes_lc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goes_lc.peek()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the meta information in the TimeSeries object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goes_lc.meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And inspect the associated units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "goes_lc.units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also inspect and manipulate the data within the `TimeSeries` class which is implemented as a `pandas.DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_lc = goes_lc.truncate(\"2013-10-28T03:00\", \"2013-10-28T06:00\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_lc.peek()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "virtualenv_sunpy-dev",
   "language": "python",
   "name": "virtualenv_sunpy-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}