first commit (#16)

* first commit

* Moved variable definitions to appconfig, async connection, init unused in TorQ

* Updated info files with torq and non torq differences, added init function

* tidied code/iexfeed/iex.q

* Defined new tables where iex data gets sent; trade_iex and quote_iex. Difference is that these tables have both a source and tickerplant timestamp.

* Added empty iex tables to database. Updated documentation. Added timer error trap. Removed unnecessary time casting

* Minor documentation changes

* Minor documentation changes

README.md

@@ -1,5 +1,5 @@
 # TorQ-Finance-Starter-Pack
-An example production ready market data capture system, using randomly generated financial data.  This is installed on top of the base TorQ package, and includes a version of [kdb+tick](http://code.kx.com/wsvn/code/kx/kdb+tick).
+An example production ready market data capture system, using randomly generated financial data along with market data pulled from the IEX.  This is installed on top of the base TorQ package, and includes a version of [kdb+tick](http://code.kx.com/wsvn/code/kx/kdb+tick). This work was inspired by [Himanshu Gupta](http://www.enlistq.com/qkdb-api-getting-market-financial-data-iex/).
 
 ## Set Up 
 
@@ -13,6 +13,11 @@ Assuming that the [free 32 bit version of kdb+](http://kx.com/software-download.
 
 For more information on how to configure and get started, go to [this site](https://aquaqanalytics.github.io/TorQ-Finance-Starter-Pack/).  You will need to make some modifications if you wish to send emails from the system. 
 
+## Further Information
+
+The [IEX API](https://iextrading.com/developer/docs/) provides free data that currently does not require authentication to access. Any changes to the API will be reflected in this TorQ pack.
+[IEX Terms of Service](https://iextrading.com/api-exhibit-a/)
+
 ## Updating the Documentation with Mkdocs
 
 To make changes to the documentation website you must simply use this command while in the branch you have made the changes on:
@@ -29,6 +34,14 @@ Then:
 
 Head to the address it gives you to check if your changes have worked. More information about using mkdocs can be found [here](http://www.mkdocs.org/)
 
+## SSL Certificate
+
+The web request that goes to the API provided by IEX goes through HTTPS. If the relevant security certificates are not installed then the data requests will fail. The setup is similar for Windows and Linux systems. To install the SSL ceritificates:
+
+1. Install OpenSSL from [SLProWeb](https://slproweb.com/products/Win32OpenSSL.html). This should be installed as default on most Linux distributions.
+2. Download the [certificate file](https://curl.haxx.se/ca/cacert.pem).
+3. The environment path must then be set in command prompt via ``setx SSL_CA_CERT_FILE C:\path\to\cacert.pem`` for Windows, and using ``export SSL_CA_CERT_FILE=path/to/cacert.pem`` for Linux.
+
 ## Release Notes
 
 - **1.0.1, July 2015**:
@@ -50,3 +63,15 @@ Head to the address it gives you to check if your changes have worked. More info
   * Removed kdb+ tick code
   * Moved KDBBASEPORT assignment to setenv.sh
   * Feed process uses timer library
+- **1.4.0, December 2017**:
+  * Rationalised connections
+  * Added metrics engine
+  * Added vwap subscriber process
+  * Added version dependency requirements
+- **1.5.0, January 2018**:
+  * Added IEX feed
+  * Added usage file for IEX functions
+
+## License Info
+
+Data provided for free by [IEX](https://iextrading.com/developer/)

appconfig/application.txt

@@ -1,2 +1,2 @@
 
-TorQ Finance Starter Pack v 1.4.0
+TorQ Finance Starter Pack v 1.5.0

appconfig/dependency.csv

@@ -1,2 +1,2 @@
 app,version,dependency
-TorQ Finance Starter Pack,1.4.0,TorQ 2.7.0;kdb 3.0.0.0000.00.00
+TorQ Finance Starter Pack,1.5.0,TorQ 2.7.0;kdb 3.4.2016.05.31

appconfig/passwords/accesslist.txt

@@ -1,5 +1,6 @@
 discovery:pass
 feed:pass
+iexfeed:pass
 gateway:pass
 hdb:pass
 housekeeping:pass

appconfig/passwords/iexfeed.txt

@@ -0,0 +1 @@
+iexfeed:pass

appconfig/process.csv

@@ -18,3 +18,4 @@ localhost,{KDBBASEPORT}+15,chainedtp,chainedtp1
 localhost,{KDBBASEPORT}+16,sortslave,sortslave1
 localhost,{KDBBASEPORT}+17,sortslave,sortslave2
 localhost,{KDBBASEPORT}+18,metrics,metrics1
+localhost,{KDBBASEPORT}+19,iexfeed,iexfeed1

appconfig/settings/iexfeed.q

@@ -0,0 +1,21 @@
+// Bespoke Feed config : Finance Starter Pack
+
+\d .proc
+loadprocesscode:1b
+
+\d .servers
+enabled:1b
+CONNECTIONS:enlist `tickerplant         // Feedhandler connects to the tickerplant
+HOPENTIMEOUT:30000
+
+\d .iex
+main_url:"https://api.iextrading.com"
+convert_epoch:{"p"$1970.01.01D+1000000j*x}
+reqtype:`both
+syms:`CAT`DOG
+callback:".u.upd"
+quote_suffix:{[sym] "/1.0/stock/",sym,"/quote"}
+trade_suffix:{[sym]"/1.0/tops/last?symbols=",sym}
+upd:{[t;x] .iex.callbackhandle(.iex.callback;t; value flip delete time from x)}
+timerperiod:0D00:00:02.000
+\d .

code/iexfeed/iex.q

@@ -0,0 +1,62 @@
+\d .iex
+
+main_url:@[value;`main_url;"https://api.iextrading.com"];
+convert_epoch:@[value;`convert_epoch;{{"p"$1970.01.01D+1000000j*x}}];
+reqtype:@[value;`reqtype;`both];
+syms:@[value;`syms;`CAT`DOG];
+callback:@[value;`callback;".u.upd"];
+callbackhandle:@[value;`callbackhandle;0i];
+callbackconnection:@[value;`callbackconnection;`];
+quote_suffix:@[value;`quote_suffix;{{[sym] "/1.0/stock/",sym,"/quote"}}];
+trade_suffix:@[value;`trade_suffix;{{[sym]"/1.0/tops/last?symbols=",sym}}];
+upd:@[value;`upd;{{[t;x].iex.callbackhandle(.iex.callback;t; value flip x)}}];
+timerperiod:@[value;`timerperiod;0D00:00:02.000];
+
+init:{[x]
+   if[`main_url in key x;.iex.main_url:x `main_url];
+   if[`quote_suffix in key x;.iex.quote_suffix:x `quote_suffix];
+   if[`trade_suffix in key x;.iex.trade_suffix:x`trade_suffix];
+   if[`syms in key x;.iex.syms: upper x`syms];
+   if[`reqtype in key x;.iex.reqtype:x`reqtype];
+   if[`callbackconnection in key x;.iex.callbackhandle:neg hopen .iex.callbackconnection:x `callbackconnection];
+   if[`callbackhandle in key x;.iex.callbackhandle:x `callbackhandle];
+   if[`callback in key x;.iex.callback: $[.iex.callbackhandle=0; string @[value;x `callback;{[x;y]x set {[t;x]x}}[x`callback]]; x`callback]];
+   if[`upd in key x; .iex.upd:x[`upd]];
+   .iex.timer:$[not .iex.reqtype in key .iex.timer_dict;'`timer;.iex.timer_dict .iex.reqtype];
+   }
+
+get_data:{[main_url;suffix]
+   .Q.hg `$main_url,suffix
+   }
+
+get_last_trade:{tab:{[syms]
+   / This function can run for multiple securities.
+   syms:$[1<count syms;"," sv string[upper syms];string[upper syms]];
+   / Construct the GET request
+   suffix:.iex.trade_suffix[syms];
+   / Parse json response and put into table. Trade data from https://iextrading.com/developer/
+   data:.j.k .iex.get_data[.iex.main_url;suffix];
+   tab:select sym:`$symbol, price:`float$price, size:`int$size, stop:(count data)#0b, cond:(count data)#`char$(), ex:(count data)#`char$(), srctime:.iex.convert_epoch time from data
+   }[.iex.syms]; .iex.upd[`trade_iex;tab]
+   }
+
+get_quote:{tab:raze {[sym]
+   sym:string[upper sym];
+   suffix:.iex.quote_suffix[sym];
+   / Parse json response and put into table
+   data: enlist .j.k .iex.get_data[.iex.main_url;suffix];
+   select sym:`$symbol, bid: `float$iexBidPrice, ask:`float$iexAskPrice, bsize:`long$iexBidSize, asize:`long$iexAskSize, mode:(count data)#`char$(), ex:(count data)#`char$(), srctime:.iex.convert_epoch latestUpdate from data
+   }'[.iex.syms,()]; .iex.upd[`quote_iex;tab] 
+   }
+
+timer_both:{.iex.get_last_trade[];.iex.get_quote[]}
+timer_dict:`trade`quote`both!(.iex.get_last_trade;.iex.get_quote;timer_both)
+
+timer:{ @[$[not .iex.reqtype in key .iex.timer_dict;
+              {'`$"timer request type not valid: ",string .iex.reqtype};
+              .iex.timer_dict[.iex.reqtype]];
+          [];
+          {.lg.e[`iextimer;"failed to run iex timer function: ",x]}]}
+
+
+\d . 

code/processes/iexfeed.q

@@ -0,0 +1,3 @@
+.servers.startup[]
+.iex.callbackhandle:neg .servers.gethandlebytype[`tickerplant;`any]
+.timer.repeat[.proc.cp[];0Wp;.iex.timerperiod;(`.iex.timer;`);"Publish Feed"];

database.q

@@ -1,2 +1,7 @@
 quote:([]time:`timestamp$(); sym:`g#`symbol$(); bid:`float$(); ask:`float$(); bsize:`long$(); asize:`long$(); mode:`char$(); ex:`char$())
 trade:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$(); stop:`boolean$(); cond:`char$(); ex:`char$())
+quote_iex:([]time:`timestamp$(); sym:`g#`symbol$(); bid:`float$(); ask:`float$(); bsize:`long$(); asize:`long$(); mode:`char$(); ex:`char$(); srctime:`timestamp$())
+trade_iex:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$(); stop:`boolean$(); cond:`char$(); ex:`char$(); srctime:`timestamp$())
+
+
+

docs/iex.md

@@ -0,0 +1,72 @@
+# TorQ-IEX
+
+An example production ready market data capture system, using randomly generated financial data along with live market data from the IEX. This is installed on top of the base TorQ package, and includes a version of [kdb+tick](http://code.kx.com/wsvn/code/kx/kdb+tick). The live market data from the IEX is sent to the ``trade_iex`` and ``quote_iex`` tables within the RDB.
+
+### Example-Usage
+
+When the IEX feed is launched in TorQ, ``appconfig/settings/iexfeed.q`` is ran, followed by ``code/iexfeed/iex.q`` and finally ``code/processes/iexfeed.q``. ``appconfig/settings/iexfeed.q`` can be used to change default variables within the ``.iex`` namespace. These default values are set in ``code/iexfeed/iex.q``, if no values have been assigned in ``appconfig/settings/iexfeed.q``.  The .iex variables that can be changed are:  
+
+* ``.iex.main_url``
+  * Used to decide which API to request data from
+  * Default value is ``"https://api.iextrading.com"``
+* ``.iex.quote_suffix``
+  * Default value is ``{[sym] "/1.0/stock/",sym,"/quote"}``
+* ``.iex.trade_suffix``
+  * Default value is ``{[sym] "/1.0/tops/last?symbols=",sym}``
+* ``.iex.syms``
+  * This decides the instruments to collect data for
+  * Default value is `` `CAT`DOG``
+* ``.iex.reqtype``
+  * This decides the request type to make. Trade data, quote data or both can be requested
+  * Default value is `` `both``
+  * This can be changed to either `` `quote`` or `` `trade``
+* ``.iex.upd``
+  * This is called when either a trade or quote request is made
+  * It is called with two arguements. The first is the table name, either `` `trade`` or `` `quote``, and the second is the corresponding table
+  * The default value is ``{[t;x] .iex.callbackhandle(.iex.callback;t; value flip x)}``
+* ``.iex.callback``
+  * This is called within the default ``.iex.upd`` function
+  * The default value is ``.u.upd``
+  * If the ``.iex.upd`` function is unchanged, the ``.iex.callback`` function is executed within a port decided by ``.iex.callbackhandle``
+  * ``.iex.callbackhandle`` is set in ``code/processes/iexfeed.q``. The discovery service is used to open an asynchronous connection to the tickerplant
+
+#### .iex.init
+
+If you wish to use the standalone ``code/iexfeed/iex.q`` script then the ``.iex.init`` function can be called to change the default vaariables described above. This should only be used if not using the TorQ framework. 
+
+When using ``code/iexfeed/iex.q`` outside of the TorQ framework the port in which the ``.iex.callback`` function will be executed in is not set. Thus the ``.iex.init`` function can be used to set two further variables:
+
+* ``.iex.callbackconnection``
+  * These are the server details ``.iex.callback`` function will be executed in
+  * If not supplied during the init call the message callback will be executed locally `` ` ``
+  * If ``.iex.callbackconnection`` is supplied a connection handle will be opened and all updates will be sent asynchronously 
+* ``.iex.callbackhandle``
+  * This can also be changed to decide which server the ``.iex.callback`` function will be executed in, using the handle to the server itself
+  * Default value is ``0i``
+  * If both ``.iex.callbackconnection`` and ``.iex.callbackhandle`` are set using ``.iex.init``, ``.iex.callbackhandle`` has precidence
+  * Messages should be sent asynchronously
+
+#### Example Function Call
+
+``.iex.init`` should be called with a dictionary. The keys of the dictionary should be the names of the variables or functions that are being set. The values associated with these keys should be the new variable values. An example of using ``.iex.init`` can be seen below.
+
+```
+.iex.init (`syms`callbackhandle`callback`reqtype)!(`cat;5i;".u.upd";`both)
+```
+
+### Raw JSON Data
+
+The following data is an example of trade and quote data that can be pulled down with the json requests inside the iex.q file.
+
+##### Raw Trade Data
+
+Taken from: [most recent Apple trade data](https://api.iextrading.com/1.0/tops/last?symbols=AAPL)
+
+` [{"symbol":"AAPL","price":178.675,"size":100,"time":1516723026747}] `
+
+##### Raw Quote Data
+
+Taken from: [most recent Apple quote data](https://api.iextrading.com/1.0/stock/aapl/quote)
+
+` {"symbol":"AAPL","companyName":"Apple Inc.","primaryExchange":"Nasdaq Global Select","sector":"Technology","calculationPrice":"tops","open":177.28,"openTime":1516717800670,"close":177,"closeTime":1516654800146,"high":178.54,"low":176.83,"latestPrice":178.66,"latestSource":"IEX real time price","latestTime":"10:55:27 AM","latestUpdate":1516722927880,"latestVolume":8796961,"iexRealtimePrice":178.66,"iexRealtimeSize":100,"iexLastUpdated":1516722927880,"delayedPrice":178.15,"delayedPriceTime":1516722031057,"previousClose":177,"change":1.66,"changePercent":0.00938,"iexMarketPercent":0.02402,"iexVolume":211303,"avgTotalVolume":25616973,"iexBidPrice":173.39,"iexBidSize":100,"iexAskPrice":179.4,"iexAskSize":200,"marketCap":908853424960,"peRatio":19.42,"week52High":180.1,"week52Low":119.5,"ytdChange":0.027516544757924123} `
+

mkdocs.yml

@@ -5,6 +5,7 @@ pages:
     - Getting Started: gettingstarted.md
     - Architecture: architecture.md
     - Have a Play: play.md
+    - IEX: iex.md
 copyright: 'Copyright &copy 2016 AquaQ Analytics Limited.  Kx &reg and kdb+ are registered trademarks of Kx Systems Inc.'
 
 theme: 'material'

setenv.sh

@@ -16,3 +16,13 @@ export KDBBASEPORT=6000
 # export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$KDBLIB/l[32|64]
 # e.g. osx:
 # export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$KDBLIB/m[32|64]
+
+touch $KDBLOG/torqsslcert.txt
+if [ -z "${SSL_CA_CERT_FILE}" ]; then
+	mkdir ${PWD}/certs
+	curl -s  https://curl.haxx.se/ca/cacert.pm > ${PWD}/certs/cabundle.pem
+	echo "`date`    The SSL securiity certificate has been downloaded to ${PWD}/certs/cabundle.pem" </dev/null >>$KDBLOG/torqsslcert.txt 
+	export SSL_CA_CERT_FILE=${PWD}/certs/cabundle.pem
+else
+	echo "`date`    The SSL security certificate already exists. If https requests fail it may be because of inappropriate certification." </dev/null >>$KDBLOG/torqsslcert.txt 
+fi

start_torq_demo.bat

@@ -55,6 +55,9 @@ start "compression" q torq.q -load code/processes/compression.q -proctype compre
 REM launch feed
 start "feed" q torq.q -load code/tick/feed.q -proctype feed -procname feed1 -U appconfig/passwords/accesslist.txt -localtime 
 
+REM launch iexfeed
+start "iexfeed" q torq.q -load code/processes/iexfeed.q -proctype iexfeed -procname iexfeed1 -U appconfig/passwords/accesslist.txt -localtime
+
 REM launch sort slave process
 start "sortslave1" q torq.q -load code/processes/wdb.q -proctype sortslave -procname slavesort1 -localtime -g 1 
 start "sortslave2" q torq.q -load code/processes/wdb.q -proctype sortslave -procname slavesort2 -localtime -g 1

start_torq_demo.sh

@@ -66,6 +66,10 @@ nohup q torq.q -load code/processes/compression.q ${KDBSTACKID} -proctype compre
 echo 'Starting feed...'
 nohup q torq.q -load code/tick/feed.q ${KDBSTACKID} -proctype feed -procname feed1 -localtime </dev/null >$KDBLOG/torqfeed.txt 2>&1 &
 
+# Launch iexfeed
+echo 'Starting iexfeed...'
+nohup q torq.q -load code/processes/iexfeed.q ${KDBSTACKID} -proctype iexfeed -procname iexfeed1 -localtime </dev/null >$KDBLOG/torqfeed.txt 2>&1 &
+
 # launch sort slave 1
 echo 'Starting sort slave-1...'
 nohup q torq.q -load code/processes/wdb.q ${KDBSTACKID} -proctype sortslave -procname sortslave1 -localtime -g 1 </dev/null >$KDBLOG/torqsortslave1.txt 2>&1 &

start_torq_demo_osx.sh

@@ -65,5 +65,9 @@ q torq.q -load code/processes/compression.q ${KDBSTACKID} -proctype compression
 echo 'Starting feed...'
 q torq.q -load code/tick/feed.q ${KDBSTACKID} -proctype feed -procname feed1 -localtime </dev/null >$KDBLOG/torqfeed.txt 2>&1 &
 
+# launch iexfeed
+echo 'Starting iexfeed...'
+q torq.q -load code/processes/iexfeed.q ${KDBSTACKID} -proctype iexfeed -procname iexfeed1 -localtime </dev/null >$KDBLOG/torqfeed.txt 2>&1 &
+
 # launch metrics
 q torq.q -load code/processes/metrics.q ${KDBSTACKID} -proctype metrics -procname metrics1 -U appconfig/passwords/accesslist.txt -localtime -g 1 </dev/null >$KDBLOG/torqcompress1.txt 2>&1 &  # compression process

stop_torq_demo.bat

@@ -18,4 +18,4 @@ set KDBBASEPORT=6000
 set PATH=%PATH%;%KDBLIB%\w32
 
 REM to kill it, run this:
-start "kill" q torq.q -load code/processes/kill.q -proctype kill -procname killtick -.servers.CONNECTIONS rdb wdb tickerplant chainedtp hdb gateway housekeeping monitor discovery sort sortslave reporter compression feed metrics
+start "kill" q torq.q -load code/processes/kill.q -proctype kill -procname killtick -.servers.CONNECTIONS rdb wdb tickerplant chainedtp hdb gateway housekeeping monitor discovery sort sortslave reporter compression iexfeed feed metrics

stop_torq_demo.sh

@@ -3,4 +3,4 @@
 
 #kill all torq procs
 echo 'Shutting down TorQ...'
-q torq.q -load code/processes/kill.q -proctype kill -procname killtick -.servers.CONNECTIONS sortslave feed rdb tickerplant chainedtp hdb gateway housekeeping monitor discovery wdb sort reporter compression metrics </dev/null >$KDBLOG/torqkill.txt 2>&1 &
+q torq.q -load code/processes/kill.q -proctype kill -procname killtick -.servers.CONNECTIONS sortslave iexfeed feed rdb tickerplant chainedtp hdb gateway housekeeping monitor discovery wdb sort reporter compression metrics </dev/null >$KDBLOG/torqkill.txt 2>&1 &