design.txt

BENCHMARK:

Between 
With comissions calculated! 
7.25% CAGR, .79 risk to return ratio
Between jan 2 2003 and dec 31 2012, 18% stocks 72% bonds rebalanced every 7 trading days

FEATURES:

* Critical Path:
** WIP

2. feature engineering using linear model
2.0.1 create annualization helper that annualizes comission adjusted returns
2.0.2 create comission adjusted volatility function - pretty sure this was ignored :P create comission adjusted risk to return ratio
2.0.2 create 'benchmark' function that takes in a date range and returns the 18% stocks 72% bonds benchmark portfolio over the period to get CAGR and R2R after comissions
2.0.3 redo linear model with single day trading periods, bull dollar
2.1 daily lags (try multiple days up to 5)
2.2 weekly lags (week 1 and week 1 minus day1 for comparison)
2.3 monthly lags (month 1 and month 1 minus week1, we'll do straight trading day months of 25)
2.4 yearly lags (month 2-12)

** Imminent (Version 3):

2.5 RSI
2.6 IBS
2.7 moving averages
2.7.1 above / below 5 dummy, above / below 25, above / below 252 dummy
2.7.2 is 5 above below 25 dummy, etc
2.7.3 replace dummy's with % above below
(NOTE): no interactions for now
2.8 ARMA of daily's (on its own)
2.9 cross correlate the etfs, previous day oil,reit,etc predict next day's spy?
2.10 ENSEMBLE arma and linear model together, try the max risk min return rule?
tag v3

** Anticipated:
*** add max drawdown to performance
*** add comissions handling
*** expected return between two dates, including dates that we havent' calculated yet for forecasting (on overall performance, or strategy)
*** plot portfolio - like plot index but takes a value and accounts for comissions in absolute terms (overall performance)
*** Add precious metals index
*** Manufacture volatility strategy - should I trade vixx or make my own iron condors?
*** Add real portfolio optimization with correlations
*** calculate true sharpe by adjusting returns and volatility by returns and volatility of the risk free asset
*** ./furnace/strategy.py:64: #TODO: should forecast be similar to asset universe, except it also provides expectations on those assets?
*** add volatility model

* Bells and Whistles
** add in metadata to allow for readable debugging and plots? i..e, be nice if the plot auto-named it as "Total return of Buy-And-Hold Single Asset with historical forecast on SPY"
   could take in metadata from portfolio strategy (buy and hold single asset), forecast (historical forecast) and asset universe (spy). probably also rebalance rule

* Tabled
** panels for different stocks?
** accrue dividends if you create cap gains models separate from dividends.

* DONE
*** 1. add helper function for current baseline - 20% stocks or whever i decided on. that way i can easily compare new strategies cagr/simple sharpe
*** 2. Create portfolio that weights based on proportional simplified sharpes (no correlation) (compare spy & bonds to baseline, then add new assets to reconfirm)
*** 2.1 Use total period cagr and simple sharpe for now.
*** 3. Add real estate index ( go for history) (compare to baseline)
*** 4. Add *currency* index and/or money market (compare to new baseline)
*** 4.1 try just spy, lqd and currency
*** 4.1.1 historical
currency index doesn't grow so it won't ever be held
*** 4.1.2 momentum
did *surprisingly well*. CAGR of 8.6% and a SS that beats my baseline! currencies are very sensitive to momentum. removing bonds or stocks *reduces* the return and SS, so we're on to something here.
downside is currencies lose about 4 years of data, so we're not as confident as we are with stocks and bonds
*** 4.1.3 anti-momentum
did very poorly, another indicator that currencies move in momentum
*** 4.2 try spy, lqd, iyr and currency
*** 4.2.1 historical
see above on currency index
*** 4.2.2 momentum
increased returns vs reits alone, but reits did not add returns or SS to the spy/lqd/uup mix
*** 4.2.3 anti-momentum 
pulled uup's bad anti momentum perf up, but with terrible SS
*** 5. Add commodities index (compare to new baseline)
*** 4.2.1 historical
see above on currencies - commodities also have negative CAGR for the period
*** 4.2.2 momentum
did okay, but GSG and iyr ultimately still hurt us
*** 4.2.3 anti-momentum 
same, but better. GSG seems to perform better with reversion to the mean
*** 5.1 consider above - add with and without any asset that has hurt us so far.
after experimentation, best fit is still spy/lqd/uup with momentum over the period 2007 - 2012. this beats baseline CAGR overall, and baseline SS *for the period*, so not pure apples to apples
*** 6. Create last period (momentum) forecast of CAGR
this is done insofar as i know?
*** 7. Create last period forecast of volatility
this is done insofar as i know?
*** 8. Create auto regression forecast of CAGR (compare to simple momentum)
appeared to do better than simple momentum, higher median return and higher return on different time periods, but is more sensitive to crashes than simple momentum
also was odd in that it found the *opposite* of what simple momentum and literature would imply, for instance, stocks were positively auto-correlated and dollars were negatively auto-correlated
*** 8.1 create hack auto regression - simply regress future month returns against previous month returns.
*** 8.2 look for AR model
if we only look at the monthly time series, similar to the vanilla linear regression, we basically have a sample space of around 80. this is just too weak for a normal ARMA model to pull anything
out of. unlikes vanilla regression, i can't explode the sample space by taking each day's month after/month before as indicators easily. there is probably a way to build an ARMA model this way, 
but it's beyond my skillset.

just to check, i ran an ARMA on daily returns and was able to pull out useful signal. this may be indicative that it will be easier to go with daily rebalancing!
*** 9. Tag as version 2!

version 2.1
2 weeks time box of cleaning things up

1.1 fix performance.py number_of_trades
1.1.1 rename asset universe to asset factory
1. add in commissions calculation for overall performance and redo daily rebalance experiment
1.2 redo daily balance experiment


PRODUCTIVITY:

* ipython has integration with vim
* install emacs, evil and org
* nerd tree should hide pyc
* nerd tree bookmarks
* get vim to point swp and un~ files somewhere else
* git flow

BUGS:

* NDay rebalancing strategies are actually one day off, or they are correct and just need to be properly documented. 25 day strategy is actually 25 trading days before or after *today*, but the strategy is always inclusive of *today*, so it really is a series of 26 points.

* i don't handle splits at all. nor do i have any way to detect them other than manually

* simple_sharpe is using CAGR, when it really ought to be using expected return. expected return can be average return ^ 252 just like volatility is daily volatility adjusted per 252

REFACTORS:

Refactor 'performance' object to take any pandas array and provide CAGR, simple_sharpe and volatility methods that run on that array. Will probably have to redo how period performances are done such that performances can be appended together so long as there are no overlaps in dates?
This refactor can also apply to weathermen - they just need to provide growth and volatility forecasts, simple_sharpe is a metric that combines those that they need to know nothing about.

financial calendar should support time points that themselves can refer back to the financial calendar. for instance, if i had one of it's time points, i should be able to do something like timepoint - 25, and work 25 *financial calendar days* back. right now i have to pass in the period, the time point, and the financial calendar. i'd really like to get to the point where even period can be reasoned about without time point and financial calendar (period should know it's time point). i should be able to take in a period and get the next or previous period (or 5 periods away, like period + 5)

period logic is a little wonky. Can review pandas for periods. Could also make trading periods smart and find the next or previous periods themselves. Right now, a 'yearly' rebalancing rate should define whether it means ~252 trading days ago, or the closest trading date to this calendar date (define closest?). Doing math in trading dates is just easier and would be a good way to limit scope.

add checks on asset classes grabbing dates that the class supports that date. the assets only have so much historical information available to them and i end up going past it on accident a lot. it takes me awhile to realize what's going on, more informative error messages would be helpful

i need some sort of auditing checks rathe rthan just the total return, i need value of assets in the portfolio on a day by day basis for debugging and auditing purposes

redo naming scheme of columns on tables to snake_case, this allows direct access that looks pythonic

split weatherman into cagr and volatility models

* General
** add doctests to public methods
+ maintainability
- large scope

** separate out stuff that isn't tested into modules that can be hit with system tests to generate coverage that way. maybe entirely different project?
+ test coverage
+ quality
- medium scope

** check __iter__ functions to see if we can't refactor their use to inside the object that provides them - then remove __iters__
+ encapsulation
- large scope

** adding a _repr_ function helps ipython
+ debugability
- very large scope

ARCHITECTURE:

* all returns are calculated as 0.01 for 1% return. A stock that is valued at $100 that has a 1% return will be valued at $101

* Trading periods are inclusive of both their begins and ends. Forecasts also assume this behavior. A forecast of 25 trading days ago implies a forecast of performance from 25 trading days ago until today. The reason that the ranges are inclusive of both sides is multifold:
** First, it agrees with intuition. During actual trading, I will sell assets and then buy others all on the same day. The price I sell assets at will be the price the older position ends on, whereas the price I buy new assets on will be the price I begin the new position on.
** Secondly, it makes the pct_diff functions on series easy to use. Values for the entire portfolio agree on the day one position sells and the other buys, giving a consistent value for that day with no gaps in trading. Doing a pct difference on the series gives me a total return of multiple periods glued together. The alternative leaves a one day gap where the value doesn't change but the actual value of the assets being traded does.
To recap, a trading period *buys* on day 0, and *sells* on day N. For example, a one day rebalancing rate would buy today and sell the next day.

NOTES:

* to drop into ipython:

import IPython
IPython.embed()

only run one test at a time, won't work on multicore or multi test mode

* to calculate technical debt:

do a count of todos
divide by adjusted physical lines of code

* to get adjusted physical lines:

run metrics
take out line count for 'class'
take out line count for 'def'

* assertion density:

to count assertions
grep assert `find . -regex ".*\.py" | grep -v test` | wc -l 
that takes out tests. divide that by adjusted line count

* to capture todos

grep TODO -n `find . -regex ".*\.py"` 

-n gets the line number

* to calculate metrics:

find . -regex ".*\.py" | xargs metrics

* to run tests continuously:
py.test -n 4 --looponfail

* for coverage reports, from root of project
py.test -n 4 --cov-config _coverage.rc --cov furnace --cov-report html

* to run a single test
py.test -q -s furnace/tests/test_file.py::ClassName

* code smell:

grep pylint `find . -regex ".*\.py"` | wc -l

divide by 2 since each pylint exception is turned on then off.
divide by adjusted lines

* in ipython:
** ctrl-a moves to begin of line
** ctrl-e moves to end of line

* to get rid of protected member warning in pylint
#pylint: disable=W0212

* to start ipython multicore engine
ipcluster start -n4

* libs learned:
** pandas

* libs to read:
** scikit-learn
** quantlib
** statsmodels
** scipy
** numpy

* interesting metrics to track:

** number of #pylint: lines. 
lower is better, means i'm closer to pylint's suggested style and make fewer exceptions ('code smell'? so long as it's pylint clean) 2.3% at last check, goal is 2%

** number of todo's, number of todo's / logical line count - technical debt
technical debt (todos / adjusted line count) - last check was 4.97%, go for 4%

** test coverage %, test path coverage %, test logical line count / logical line count, #tests / #functions (public functions)
** assertion density - last check was 4.47%, go for 5%

** comment and docstring length / adjusted logical lines
** something for code modularity? average file size or maybe just look for certain file sizes? not just line count but also method or class count?

* principles:
** *REMEMBER*: eliminate waste before improving quality. then you have less work to do!

** cycle:
*** feature push
*** code audit
*** plan triage
*** metrics check and adjust. keep coverage up, etc