todo.txt

##########################################################################################################################



8/3

option class - framework.                         [Done]
futures class - framework.                        [Done]
portfolio class - framework.                      [Done]
black scholes (euro)                              [Done]
computational methods                             [Done]


9/3

moneyness                                         [Done]
securities into options and futures               [Done]
re-testing classes after changes.                 [Done]
exercise option                                   [Done]
convert option underlying into futures object     [Done]
decrement tau                                     [Done]
update documentation                              [Done]
options on futures formula                        [Done]
options class - euro vs amer                      [Done]
options class - s is now underlying.price()       [Done]
simulation - program structure                    [Done]
feed_data - logic                                 [Done]
feed_data - basic implementation				  [Done]
rebalancing - logic                               [Done]



10/3

>>>>>> urgent II : short pos repr <<<<<<          [Done]
data import                                       [Done]
update documentation                              [Done]
tidy up futures class                             [Done]
test portfolio class                              [Done]



13/3
gamma & vega hedging                              [Done]
re-structure dictionaries to incl product         [Done]
Update documentation							  [Done]



14/3
Implement the following based on the matlab pricing code.

american option pricing                           [Done]
vol computation                                   [Done]
barrier option pricing: american                  [Done]



15/3
product specific information dic                  [Done]
update greeks based on lots etc                   [Done]
barrier option greeks : american                  [Done]


barrier option pricing: european                  [Done]
barrier option greeks : european                  [Done]




16/3
Debugging:
3_ options object                                 [Done]
4_ futures object                                 [Done]





17/3:
testing:
1_ all computational methods.                                     
	> vanilla options                             [Done]
	> vanilla greeks                              [Done]
2_ portfolio object                               [Done]



20/3
PRIORITY: Finish testing calc.py
	> American barrier                            [Done]
	> implied volatility                          [Done]
	> Call put spread pricing                     [Done]
	> Call put spread greeks                      [Done]
	> American barrier greeks                     [Done]
delta hedging logic                               [Done]



21/3:
familiarize with data                             [Done]
prep_data steps                                   
	> compute ttm from label for op               [Done]
	> reading in data                             [Done]
	> alter prep_portfolio                        [Done]
	> clean price data                            [Done]
flag indicating target or hedge in readin         [Done]




23/3:
tidy up expiry month for each class per month     [Done]
knockout for european barriers                    [Done]
European barrier                                  [Done]
European barrier greeks                           [Done]
Figure out why vols and 0 and what to do abt that [Done]
fix the nonetype error                            [Done]
documentation for:
	> option                                      [Done]
	> portfolio                                   [Done]
	> prep_data                                   [Done]
	> calc                                        [Done]
write-up                                          [Done]





24/3:

hedge vs otc dichotomy         
	> target : {'product': {'month': 
	    set(options), set(futures, greeks)}}      [Done]
	> hedges: {'product': {'month': 
	    set(options, set(futures), greeks)}}      [Done]
	> adapt compute_net_greeks to take in two 
	random dictionaries instead of hardcoded 
	to self.long and self.short                   [Done]
	> replaced long with OTC                      [Done]
	> replaced short with hedge                   [Done]
	> reflected changes in portfolio_specs        [Done]
	> updated prep_portfolio                      [Done]
	> updated tests                               [Done]


27/3:

Basic stuff:
> getting the right data.                         [Done]

Data processing steps:
> allocating C1 C2 C3 based on front month        [Done]
> add in ordering/month flags in options          [Done]




28/3:

> Not all options in expiry list; N7.Z7 for ex.   [Done]
> Debug all prep functions                        [Done]
> follow up with kishor about expiries            [Done]
> price returns transformations                   [Done]
	> log(curr - prev) taking rollover into acct
> price transformation                            [Done]


29/3
> vol surface difference transformation           [Done]
	> atm_curr - atm_prev 	                      [Done]
	> helper to isolate option ordering mth       [Done]
> selection rollover logic                        [Done]
> selecting vol from vol surface based on strike  [Done]



30/3 [Morning]
> finish indexing vols ci option indexng          [Done]
> testing find_cdist                              [Done]



30/3 [At work]. Priority: finish the CI stuff/testing.
> rollover in price                               [Done]
	> testing                                     [Done]
> fix find_dist and assign ci to take into account   
multiple years.                                   [Done]
	> testing                                     [Done]


31/3
> New task: delta interpolation dataframe.        [Done]
2) testing from xtrader                           [Done]
3) testing for european barrier greeks            []






1/4 (Saturday):

Outstanding: vol/price dumps. gaining access to these mandatory.


1) think about how mapping CI and symbols is gonna  [Half]
happen in options/futures objects, implement 
accordingly.
	2_ use same logic from ciprice to construct     [Done]
	ci vols from vol by delta	            

2) moneyness error.                                 [Done]
	> was a problem with self.active/self.knockedout
	 interaction in options class during 
	 self.moneyness and self.check_active calls.


2/4 (Sunday):

Conceptualize ordering in options/futures.          [Done]
	> testing

	Notes:
		
		1_ ordering in futures: taken care of during read-in, 
		based on current month and the number of contract months between target and current month.
		
		2_ ordering in options: for each given C-i price series, 
		bucket overlying options based on time to expiry (i.e. tau < x = bucket 1, x <= tau <= y = bucket 2, etc.)
		
		> implication: logic to check for which price/vol series 
		to read in should be done in simulation.py, not in any of the classes.
		
		> todo: isolate c_i from readin, which is already being done by read_data. 
		once access to vol/price dump is given, implement get_all_underlying in portfolio to specify the data that is required.
		   - done.
		> during simulation:
			1) for each product, keep track of rollover dates.                  [Done]
			2) if value_date == rollover date, update orderings                 [Done]
			(i.e. decrement ordering for options and futures, discard those        
			options that correspond to ordering 0)

			> related TODOs: 
			1) generate and store rollover dates in sim.py                      [Done]
			2) new active condition (i.e. if cond == 0, inactive, regardless).  [Done]
			3) question: barrier option expiries?                             

 
> isolate distances using get_cdist, pass that into [Done]
underlying.ordering.                                
> updating future and options object with ordering  [Done]




Simulation logic notes:

1) Timestep --> immediately handle expiries/rollovers
2) Feed in price and vol data:
	> price --> knockouts/knockins that changes activity status.
	> price --> change in value of futures and options. 
	> vol --> change in value of option
3) Handle options:
	> daily vs bullet payoff? might not actually be necessary.
4) delta/vega/gamma hedging. 
5) store changes from above 4 steps, figure out how the hell brokerage/lots/multipliers etc factor into PnL calculation.
6) store PnL against run_number --> plot a histogram, and summary statistics. 





3/4 (Monday) :

# Notes:
	> use the regular ci_vols_final dataset to actually update the portfolio. 
	vol_by_delta is purely for database purposes, can later be adapted to portfolio if so desired.
	> currently using Piecewise continuous hermite interpolation (pchip) 
	to interpolate deltas. ensures function and derivative match at those points.

Debugging ci_price                                  [Done]
debugging ci_vol                                    [Done]
resolving imports                                   [Done]
updating documentation in prep_data                 [Done]

### Simulation TODOs: ####
figure out how to check for rollover.               [Done] -> in feed_data
main simulation loop                                [Done] -> notes written.



4/4 (Tuesday):

feed_data                                           
	> timestep --> expiries.                        [Done]
	> rollovers                                     [Done]
	> price --> knockouts/knockins that              
				changes activity status.            [Done]
	
	> price --> change in value of futures          [Done]
	                          and options.          [Done]
	> vol --> change in value of option             [Done]
>> Check: if greeks updated when new price/vols 
	passed in                                       [Done]
>> check if expiry zeros out greeks of options      [Done]
>> check if decrementing ordering has the 
   desired effect                                   [Done] 
>> understand readin process and how tau is 
   computed, in relation to simulation logic  
   as a whole.                                      [Done]
>> vega/gamma hedging                               [Done]




5/4 (Wednesday):
> prep notes, send out tomorrow morning.            [Done]

# Notes:
> discussion with Evert on structure of the code.   [Done]
>> testing: feed_data.                              [Done]
	> tested all constituent components           
		- rollovers                                 [Done]
		- expiries                                  [Done]
		- change in val upon price/vol              [Done]
	> test compute_ordering                         [Done]
> Delta Hedging.                                    [Done]

FFDSDdfsd
dd



6/4 (Thursday):

calc-related work:
> rework calc testing. use exclusively xtrader.     [Done]
> debug euro barrier pricing.                       [Done]
> euro barrier greeks                               [Done]
	> delta [Done]
	> gamma [Done]
	> theta [Done]
	> vega  [Done]
> american barrier greeks                           [all but gamma]
	> delta [Done]
	> theta [Done]
	> vega  [Done]
> rework compute value & testing                    [Done]


7/4:
>> testing: gamma hedging                           [Done]
>> testing: vega hedging                            [Done]
>> testing: joint vega/gamma                        [Done]
	- issue is with shorting and greeks             [Done]
>> testing: delta hedging                           [Done]


PnL:
> brokerage in hedging steps                        [Done]
> lots for delta hedging.                           [Done]

Pnl calculations.                                    
> fully understand what is expected.                [Done]
> pnl mult                                          [Done]




10/4
> exercise                                          [Done]

> make sure shorted values option payoff at expiry 
is being handled correctly.                         [Done]

> bullet vs daily payoff                            [Done]
	>> convert daily into 1 dtm payoff options      [Done]
	>> testing daily to bullet conversion.          [Done]

> isolate zero div errors                           [Done]





12/4
> timestep issues, check with pnp             [Done]




13/4
> debug net greeks and updating               [Done]
> implement futures = 1 delta.                [Done]

> pnl into daily and cumulative               [Done]
> check pnp options                           [Done]


14/4
> E barriers                                  [DONE]
> FINISHED THE SIMULATION                     [DONE]




17/4
> Debug gamma calculations                    [Done]
> A barriers                                  [Done]
> feeding in now Done through CSV             [Done]
> data viz                                    []
	> daily pnl                               [Done]
	> cumulative pnl                          [Done]
> run the corn portfolio                      [Done]



18/4
> standardize date ranges                                    [Done]
> figure out the expiry issue                                [Done]
> debug handle options                                       [Done]
> get solid on the time delta stuff                          [Done]




19/4
> adapt hedges as input.                                     [Done]
	- from csv to dictionary          	                     [Done]

> fix test_simulation                                        [Done]

> make sure hedging works for k7 corn                        [Done]
	> changed location of break function                     [Done]

> theta hedging with atm straddles                           [Done]

- hedge only according to frequency                     
		> counters                                           [Done]
		> day number                                         [Done]

> vol by delta implementation in feed data                   [Done]
	- understand reason why this is required.
	> testing required



20/4
> computational block:                                       [Done]
	> price * returns                                        [Done] 
	> vol + change vol                                       [Done]
	> feed these two into portfolio as per usual.            [Done]


change the data selection process.
	> need to save current local sim date, use these values  []
	  to construct the portfolio.
	> generate ordering based on earliest date in dataset.   []
	> decrement tau as per usual.                            []
		- test to make sure that simulation only lasts as 
		long as ttm in days.                                 []
	> 


apply on BO portfolio


Aux Concerns:
> adapt feed_data to check for high/low/close for ki/ko      [WIP]
> clean bo data, run the bo portfolio                        [WIP]


TODO once dumps are ready:> finalize database updating stuff. 
> batch script to run the file every day. 


Historical Simulator TODOS:



3/5:

>				  ## Data-Related ## 

	> [Done] read_data: sim_start as start_date; figure out what is required, 
	  and what is being piped downstream into clean_data and related functions.

	> [Done] functions related to start_date: 
	    1) assign_ci
	    2) get_expiry and expiry-related

	> [Done] prep_portfolio
		- debug get minimum dates
		- update datasets 

	> [Done] generatin hedges                                                                        

	


				## Simulation-related ##

	> [Done] global_vars refactoring of everything. [HIGH PRIORITY]
	> Changes to simulation logic; include placeholders for measure input.  
		- Notes:
			1) Following variables are NOT REQUIRED: [daynum, run_size, hedge_today, op_to_strike]
			2) Differences in simulation loop:
				- in PVM, init_val is reset whenever simulation block ends. in HSM, init_val is never reset, since 
				we don't have simulation blocks, nor portfolio restores. 
				- Simulation order in PVM:
					feed --> (handle exercise) --> pnl calc --> rebalance.
				- Simulation order in HSM:
					feed --> handle exercise --> pnl --> rebalance. Should probably move handle exercise into feed. 
				- decrement ordering has validity in HSM but not in PVM. 
				- PVM currently has brokerage/slippage included in PnL calculations and repr. HSM does not. 

	> [Done] Brokerage/slippage and associated PnL representation changes. 
	> [Done] changes to hedging implementation 
	> [Done] changes to feed_data implementation
	> [Done] handling exercises/expiries/decrement ordering distinction btwn var and historical
		- handle_exercise: cash and future settlement




4/5:

>				## Others: ##
	> [Done] fix imports and implicit/default function calls in other files. 
	> fix tests. 

				## Object-Related ##
	> [Done] Changes made to Portfolio/Option/Futures class
		- [Done] Futures:
			+ cash vs future settlement
		- [Done] Options:
			+ cash vs future settlement
			+ [Done] str representation 
		- [Done] Portfolio
	
				## Calculation-related ##
	> [Done] Updates made to calc.py
		+ update read_data imports and function calls.
		+ make sure european barriers are handled correctly
		+ calculations are right, just need to make sure that variables are passed correctly. 

				## Testing-related ##
	> [Done] changes to nosetests: can more or less port over tests from PortfolioVARModule wholesale.


				## Implementation TODOs: ##
	> delta roll hedging [Done]


	> debugging the inevitable. 


> after all [DONE]
	- run with PnP and make sure it works. 




5/5: 
> debug delta hedge roll
	- for some reason, offending option is not getting removed.                   [Done]
	- not finding the relevant vol in the dataframe randomly.                     [Done]
	- figure out why delta of new option isn't exactly 25 delta.                  [Done]
		- due to the strike being rounded up/down to the nearest 10.  

> refactor get_min_start_date so that it depends on portfolio rather than on the dataframe. [Done]

> refactor simulation loop so that it buys/sells according to signal provided.

	- keep loop to be in terms of voldata dates
		- change dates in signals df to account for SOD/EOD difference; signals are given a the start
		of the day <=> eod of the previous day. 

		- make changes to simulation functions to accomodate empty portfolio
			> added in .empty() method to portfolio that checks if self.OTC and self.hedge dictionaries are
			empty. Preferred checking dictionaries rather than just OTC dict to distinguish between portfolio state
			when all OTCs have expired vs an initially empty portfolio with no signal yet. 

			> changed get_min_start_date to reflect min between signals and voldata, with a bunch of extra conditions
			so that it can be applied unchanged to both prep_portfolio and prep_data with the same results.

	- [Done] need to figure out how to specify a particular month's contract even if initial portfolio is empty.
		+ for now, add in vol_id flag into the signals dataframe, default to C  N7.N7 

	- include handling logic for rollover upon crossing expiry buffer.





8/5
	
	- [DONE] sort out the inconsistency in dates between signals and voldata
		> need to somehow cross reference dates in voldata and signals such that signals_date - numdays = vdf.date
		> prelimiary steps in diagnostic. 
		+ solution was to roll all dates reported on sunday to the previous friday. introduced match_to_signals function in prep_data.py 

	- added in match_to_signals function to implement aforementioned change
	- debugged problems in prep_data when dealing with product names that had more than 1 letter --> checks to standardize the two space convention in vol_id and underlying_id. 

	- function to parse signal input:
		- date | 25d call vol | 25d put vol | signal
		- for each day:
			1) 

	- TODO: figure out why timestep isn't working
		> solution: created the option objects individually. my guess is that objects were being overwritten when added as part of one large list, but honestly not sure. 


9/5
	- solve timestep issue (done)
	- fix rounding strike per commodity (done)
	- change order of simulation loop to rebalance after signal (done)
	- ensure pnl computation only considers comparable portfolios (done)
	- logging to csv (done)

	- finish interfacing (done)



10/5
	- notes:
		1) currently apply_signal not taking any brokerage/slippage/cost of options into count since all of this happens after prev PnL computation and before next feed. 
	Further todos:
	
	> figure out how to intelligently sell the equivalent of 25 skew pos using current-day values. 
		- more specifically: vega amount specified in signal. figure out how to sell off that much vega from current position using various methods such as:
			1) first-in-last-out
			2) furthest from 25delta 

	> Notes for Ujjwal:
		- generate signal using historical data. 
		- multipliers on the buy/sell signal on the basis of skew percentile compared to history.
		- rolling bands; perhaps some sort of rolling weighted average based on historical analysis. 



11/5:

	> [Done] double check if changing lots changes greeks [high priority]
		- if not, add in new method decrement_lots that updates greeks after decrementing lots 
	> [Done] finish implementing close_off_delta 
	> [Done] testing
	> [Done] liquidation logic


12/5:
	> [Done] filo implementation: change OTC_options, hedge_options, OTC_futures and hedge_futures into deques, which have the same functionality as lists. 
	> update liqudation such that we always have vega_req * num_skews on each leg, regardless.
		- [Done] if insufficient, let it go to an empty portfolio




15/5: 
	> run test with two different products in portfolio. 
	> update coding documentation
	> flow chart for Historic/PortfolioVARModule
	> [Done] make sure generate_skew_op works with:
		- negative vega 
		- call and put flags 

	> [Done] implement liquidation logic. 
	> [Done] debug 



18/5:

High priority:
	> finalize data processing pipeline and storage of files [Done]
	> determining start date/end date given:	             [Done]
		1. user inputs                                       [Done]
		2. signals                                           [Done]
	> fully interface prep_data with database so that relevant info is pulled [Done]
	> sort out datetime issues                               [Done]


	


19/5:
	> [Done] run the soybean atm straddle  - data constraints = only 2015 and 2016 can be run. 
	> [Done] fetch_data and prep_data should accept (pdt opmth ftmth) as inputs for drawing data --> would allow for quick creation of positions 
	> [Done] data cleaning from alternate table would be useful; would allow that table to be used and integrated rather than depending on the opera one. 
		- at the very least nail down the transformation logic, give to Ujjwal  [Done]

	



20/5:
	> [Done] defining hedges in create_portfolio 
	> [Straddle - Done] 
		easy creation of certain positions (skew, straddle, etc) given relevant information. 
		- either update prep_portfolio, or make it easy to specify somehow. 	
	> [Done] hedge rolling logic
	> [Done] change sql queries to draw all data <= current contract month value. 
		- done for pdt/opmth/ftmth draws, not for others. 




	
22/5:
> take stock of all work that has been done so far - cut out redundancies and refactor as necessary. 
	- crucial :
		1. prep_portfolio vs create portfolio. (latter honestly seems superior/more flexible for composite structures (e.g. straddles, etc))

		Notes: 
			CSV --> List of dictionaries --> create portfolio
			split into OTC and hedges
				> define each option individually in its own row
				> create all OTCs, then generate hedges so that hedges can use OTC values as inputs. 
				> some kind of linkage between a hedge and the thing it is hedging? 
				> specify a greek value OR lots, depending. 


		2. util functions; where do they fit into everything?
			> left to develop: 
				- butterflies
				- spreads 




23/5:
	> new read method csv --> dict --> portfolio. Try to keep current work intact 
	> portfolio-wide hedging as a flag --> how does this work? numerical equivalent? 
	> multiple signals per day [Done]


25/5:
	- creating separate underlying for each option created. CANNOT USE SAME UNDERLYING FOR DIFFERENT OPTIONS. -> Done. 
	
	- finish up util implementations -> butterfly left to do; finalize kwarg inputs for other composites.
		> Straddles: Done. 
		> Strangles: Done. 
		> skews/fences: Done
		> butterfly: Done 
		> spreads: Done


26/5
	
> multiple signals per day [Done]



23/6:

> [Done] implement barrier volatility updates for european barriers
> [Done] include bvol in class constructor 
> [Done] change up compute_value, compute_greeks etc to include barrier volatilities.
> [Done] include bvol in construction methods for option combos in util.py
> [Done] include bvol calculation in prep_data
> [Done] include bvol in handle_dailies
> [Done] include bvol in get_properties method of options.
> [Done] ensure it all still works; run some samples

> read through and understand skew code, what it does and what is required. 

> update documentation. 



>>>>>>> TODOS AS OF 7/12 <<<<<<<< 

1) [Done] Multiple signals per day:
	> currently relies on net_vega_pos of the entire portfolio. need to re-implement this in order to make sure different signals are applied appropriately.

> create skew apparently has some problems to be debugged. 


2) [Done] Go through skew signal application with a fine-toothed comb. PnL profiles look very similar to previous runs, consistently making money on gamma and losing on vega. Why?
	> long theta long vega long gamma, overall. gamma PnL is therefore consistently positive (since gamma is timestep + price change). 

4) [Done] run it through the debugger and make sure it is outputting correct results. 




>>>>>>>>>> TODOS as of 7/20 <<<<<<<<<<<<


1) Expand historic simulator to accept currencies. 
	> might need alternate methods in feed_data to address differences in data.
	> valuation of these options (e.g. delta gamma theta vega scaling) would not be accurate --> need to redo.
		> might need different hedging strategies/implementation? rebalance might not work explicitly. 


2) Improve on hedging code to do the following:
	> apply bound, static to each greek independently (i.e. don't assume that delta is zero and gamma/theta/vega are bounded)
	
	> accept multiples of current TTM
		- need some kinda way to check wha
	
	> accept fixed ttm for hedging that decrements with each passing day. 

	> [Done] ensure that rollovers work for  multiple products. 
	> [Done] edit grab data to search for and accept data if it is already saved rather than querying database. 
	> [Done] updated all dictionaries to change LCC to QC as per database.
	> [Done] implemented flexible rollovers for multiple vol_ids 
	> [Done] updated scripts/util.create_vanilla and create_straddle to accept atm as a strike value. 
	> [Done] debugged construction errors with create_vanilla and create_underlying. 

delta,roll,25,1,"(10,40)"
vega,bound,"(-5000, 5000)",1,,0.164383562
theta,bound,"(-5000, 5000)",1,,0.5
gamma,bound,"(-5000, 5000)",1,,0.5






>>>>>>>>>>>>> TODOS as of 7/24: <<<<<<<<<<<<<

1) [Done] Finish implementation of the hedge object.
	- _calibrate:
		> applying ttm multiplier/specifying ttm based on value passed in.
		> generating parameter dictionary for each case:
			+ exp: (product, ttm) -> vol_id 
			+ uid: vol_id -> vol_id (self referential mapping)

	- hedge implementation via hedge object; use existing methods or re-implement? 

2) [Done] Handle missing data properly; i.e. timestep needs to be done, but pnls recorded should be 0.
	> removed 'broken' boolean and break statements when data is missing. 

3) [Done] reconcile rolling over with target_product with ttm_tolerance for each option (buy and hold example)
> [Done] updated the hedges_satisfied function to deal with more complex hedges 



>>>>>>>>>>>>>> TODOS as of 7/25: <<<<<<<<<<<<<

+++ Hedging-related concerns: +++

> [Done] update utils and see if greek-based lot selection has sign errors like create_straddle did. 
> [Done] e-write hedge delta to dispense with ordering input and to rely on utils.create_underlying
> [Done[ figure out whether or not to keep delta hedging as it is, or to move it into the hedge object
	_ would require capability to handle multiple representations. 
	- adding/removing futures/options? 
	- implementation decision: keep delta hedging and delta rolling out of the hedge object because they do not depend on the representation of the greeks. 
> [Done] expand hedging to allow for the construction of different objects to hedge based on input flags. ]
	- taken care of with self.params dictionary, which will specify the actual structure to be used to hedge. 
	- can currently construct any and all pre-defined structures with constructors found in scripts.utils. 

>>>>>>>>>>>>> TODOS as of 7/26 <<<<<<<<<<<<<

> [Done] re-write the hedging functions (most notably rebalance, gen_hedge_inputs and hedge) based on what convention is decided upon. 
> [Done] Debug hedging implementation, include delta hedging in hedge object. 
> [Done] implement slippage/brokerage 
> [Done] Assign more meaning to the 'static' and 'zero' keywords passed into the dict, or remove if unnecessary. 
	> zero and bound hedging are both taken care of now. fuck yeah. 
> [Done] Debug option hedging
	- runs properly for days with missing data 
	- non-delta greeks can be zero-hedged with no issues. 


> recall example where pf was not rolled --> ensure that close out works, checks to see if OTC options are empty. 



>>>>>>>>>>>>>>> TODOS as of 7/28 <<<<<<<<<<<<<<<<<

> debug Hedge object under the following scenarios:

1) [Done] missing data. 
2) [Done] done_hedging always returns false for some reason. 
3) Proper handling of expiries. 
	> issues:
		- when the whole portfolio is near expiry, some weird things start to happen:
			- [Done] options of identical TTM are not evaluated all at the same time for expiration/exercise.
				+ resolved by replacing op.tau <= tol with np.isclose(op.tau, tol)
			- gamma PnL goes batshit crazy. PnL
				+ resolved (partially) by adding in intermediate_val == 0 check when exercised = True. 
			- HUGE delta accumulations close to expiry. 





>>>>>>>>>>> TODOS as of 08/01 <<<<<<<<<<<<<


> [Done] alter pull_alt_data to accept write_dumps, start_date and end_date as parameters 
> [Done] test new pull_alt_data and make sure it all works. 
> [Done] Move signal generation files from skew_lib to HistoricSimulator.scripts 
> [Done] Sketch out framework of applying signals to straddles. 




>>>>>>>>>>> TODOS as of 08/02 <<<<<<<<<<<<<

> [TODO] Figure out import statements for runs when moving to runs folder. 
> [In Progress] closing out portfolio criterion when all OTC options expire (simulation.py, lines 744-762)
> Figure out if match_to_signals etc are actually required.
> Standardize date conventions for signals. 
> [In Progress] Generalize signal application to more than just skews:	
	0) [Done] Format of the signals file
	1) [Done] General apply_signal loop. 
	2) [Done] add_pos 
	3) liquidate_pos
		> actual liquidation loop. 
		> strat = dist and metric of choice. 
	4) [Done] close_position
	5) [Done] Cumulative signal value etc; make sure it makes sense. 


>>>>>>>>>>> TODOS as of 08/03 <<<<<<<<<<<<<

> [Done] liquidate_pos
	> actual liquidation loop. 
	> strat = dist and metric of choice.  
> [Done] New concern: maintain flag. 
> [Done] Test with fake signals, just to see if it all gets applied properly. 




>>>>>>>>>>>> TODOS as of 08/04 <<<<<<<<<<<<<

> [Done] finish up implementations in util regarding greeks -> lots 
> [Done] specifying type of option to use in hedging in hedging.csv, and params.

> roll portfolio, roll product and ttm tol in hedging file. 
	- [Done] updating generate_hedges to take into account the new xlsx format. 
	- [Done] update hedge object to read buckets from row.subcond if desc = exp. 



>>>>>>>>>>>>>> TODOS as of 08/05 <<<<<<<<<<<<<<<

> [Done] Port over changes from HSM to PVM 
> [Done] done_hedging returning None; why?
	- issue was a stray return statement when delta is 0 and we could skip hedging. 
> [Done] Debug and make sure everything recently implemented is working with no glitches/problems:
	1) new hedging object - process_hedges/calibrate/add_hedges. 
	2) grab_data with filtering. 
	3) implementation regarding greeks and lots in utils. 
> [Done] hedging after timestep. 
	> rolling over is still done before timestep, but rebalancing now happens after. 
> [Done] callstrike vs putstrike paradigm for options


>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<


>>>>>>>>>>>>>> TODOS as of 08/08 <<<<<<<<<<<<<<<
> [Done] calculating the conditions for op-delta roll. 

> [Done] instead of zeroing delta, requiring delta at a fixed value (i.e. maintain 5)

> [Done] ISSUE: specifying a value for hedging gamma/theta/vega forces that value to be met, irrespective of sign of the desired position. 
	- e.g. : initial position is a short CC straddle with -9 gamma, value in hedging requires 50. 
			--> adds on 59 gamma with of long straddles to 'hedge', resulting in overall 
				long CC position when we desired a short position. 
	- solution was one liner: if greekval < 0, target = -target

	- Easy enough to do with UID representation, but what about exp representation?

>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<


##################################################### Open Issues #######################################################

---------------------
| As of: 08/07/2017 |
---------------------

BUG: calling update_sec_by_month in uid_hedges_satisfied zeroed out initially bound self.greek_repr. Why?
	- fixed by calling update_sec_by_month and then rebinding self.greek_repr to the new net_greeks produced. 


> [Done] ISSUE: generalizing atm roll in roll_over. 
> [Done] Move bounds to bounds based on difference (doesn't really matter)
> [Done] ISSUE: checking when all options have expired/exercised and we need to close out the rest of the portfolio.

> Family-based construction and hedging parameters.
	> [Done] combine_portfolio
	> [Done] specifying hedging logic for each family (both read in and otherwise).
	> [Done] handling add/removes from families
		- additional recursive call in remove_security when dealing with families. 
		- overall containing portfolio (i.e. super-portfolio) contains all the futures.
	> [Done] Portfolio constructor now takes in a dictionary of hedges and a name; relevant hedges are assigned by searching for portfolio's name in hedge dictionary. 
	> fully specify rolling logic
		- [Done] delta rolls
	> [Done] fully specify gamma/theta/vega hedging logic.
	> [Done] fully specify relationship between hedge objects and portfolios they are attached to. 
	> [Done] Make sure this all works for simple portfolios. 
	> [Done] Trial run. 
	> [Done] go through and make sure refresh/recomputations are called correctly and aren't messes. 
	


--------------------
| As of 08/11/2017 |
--------------------

> [Done] Fuse together families and composites. 
> [Done] callspread/putspread roll example --> each leg can be placed into a different 'family' and it would work just fine with no hitches. 
> [Done] check_roll_status and roll_hedging for simple portfolios. 
> [Done] updated check_roll_status and hedge_delta_roll to be applied to all options rather than just OTC.
> [Done] Contract rolls:
	- make compatible with families paradigm.
	- handle option composites.
> [Done] ISSUE: net vega position problem --> maintaining a primary position of sorts. 
	- example: in the case of an event in CC and no corresponding event in QC, we want to still maintain net 0 vega across both positions; i.e. one will have
		to be hedged basis the other
	- solution: some kind of flag passed into kwargs specifying what the default position is?
	- is there a more general formulation for a primary position?
	>>> Note: this would fall under the paradigm of portfolio-wide hedging rules. <<<








--------------------
| As of 08/14/2017 |
--------------------

> [Done] finalize (more or less) the layout of the gui. 


>>> Priority <<< 
1) [Done] see if code can be refactored.
2) [Done] understand how the code snippets talk to each other.

4) [Done] data pulling from frames.
	- get data done today.
5) [Done] pass date into PortfolioFrame object to allow for creation. 
> [Done] adapt grab_data to default to SQL query rather than dumps so that directory is not at all relevant. 
> [Done] figure out how to clear grid of labels when deselecting. 
> [Done] Figure out import issue. 


> [Done] Tickbox for writing text log. 
> [Done] Writing log to csv. 
> [Done] Plotting functionality in plotly. 


> [Done] Visualize vol curve button/functionality. 
> [Done] button option to keep vols/price constant.
	> [Done] add in to simulation as well.  

> [Done] Sumit's visualization requirements 


--------------------
| As of 08/21/2017 |
--------------------

> [Done] Reconcile timestep before hedge with timestep after pnl calc
> [Done] dropdowns for hedging requirements.
> [Done] Updating test suite. (Issue 1) <<<<<<<<<<< Priority this week. 
	- [Done] Portfolio
	- [Done] Option
	- [Done] Simulation
	- [Done] Util
	- [Done] error handling in ui 
	- [Done] Hedge 
	- [Done] error handling in simulation script. 
	- [RMS] fixing ttm_mod cases with exp repr 
> [Done] Solve issue where all options and all families could be removed, resulting in an empty screen. 
> [Done] Solved plotting issue for multi-commodity future prices.
> [Done] Check to make sure skews and strangles work properly.
	- strangles with > 1 strike or 1 delta. 
	- skews with > 1 strike or 1 delta. 
> [Done] raise errors appropriately. 
> [Done] General Dev:
	- [Done] drawdown requirement
> [Done] drawdown in plots 
> [Done] filtering out volids from full dataset for vis purposes. 


-------------------
|As of 08/28/2017 |
-------------------

> [DONE] Compile current simuator into an executable file, distribute. 
	- failed attempts:
		- py2exe - Does not support python 3.6
		- pyinstaller - no support for 3.6
		- nuitka - doesn't actually produce an executable for whatever reason.
		- cx_freeze 
		   -> [Done] freezing data-files
		   -> [Done] resolving __file__ errors. 
		   -> [Done] successful build and setup.py init. 
		   -> scipy-related import errors. 
		   		- [fixed] added explicit path to scipy distribution.
		   		- [fixed] added setuptools to path to counter pkg_resources error.
		   		-  new issue: appdirs module not found.

		- FIGURED IT OUT WITH PYNSIST.
			- issue was with different bit compatibility between tkinter-required dll files and the python version, plus
			the additional dependencies for pandas and plotly that had to be
			manually specified. 
		
			> COMPILATION TO EXE FILE IS COMPLETE.
				- required that: 
				1) DLLs required for tkinter and python version be of the same bitness (i.e. either x86 or x64)
				2) sqlalchemy requires psycopg2 
				3) pandas requires six, setuptools, pytz and dateutils
				4) plotly requires requests, pprint, pkg_resources and decorator.
				5) matplotlib should be built from wheels, as should psycopg2 to avoid DLL load errors. 
				6) expdate query should be straight from database rather than from a file. 


> Debugging:
	- [Done] strangle issue.
	- [Done] edit expiry to pull from database. 



-------------------
|As of 09/01/2017 |
-------------------

- [Done] composites in simulator that are NOT part of the same family. --> flag to mark these two options as composites. 
	Sub-components:
	0) [Done] include composite_name checkbox. 
	1) [Done] ensure naming works properly in UI. 
	2) [Done] update FFC.process() method:
		- pulls options
		- assigns composites
		- constructs pflist. 
		- returns pflist 



--------------------
| As of 09/05/2017 |
--------------------

- [Done] replace writepath str with directory navigation. 
- [Done] not specifying writepath throws error. 
- [Done] PnP format to prep_portfolio format; currently, everything would be under a single family. 
- [Done] Hedge specification for spec portfolios. 
- [Done] File navigation in the UI 
- [Done] Added breakeven method to portfolio.py 



--------------------
| As of 09/06/2017 |
--------------------

#### Handling errors in other Volgraphix components. 
- [Done] skewhistory
- [Done] skewttm
- [Done] skewseason
- [Done] heatmap 
- [Done] comps 


#### As of 09/07/2017
##### Data Processing steps. 
- [Done] basic cleaning steps. 
- [Done] adding settlement data to intraday 
- [Done] timestep reconciliation between uids
- [Done] switching variables: i.e. settle_value -> price, settle_vol -> vol 
- [Done] reconcile intraday prices with clean_data with price flag. 
- [Done] Refactor to remove dependence on vol by delta. 
- [Done] Re-write construction methods to default to settlement data
- [Done] add time column to vol data 
- [Done] conceptualize intraday loop -> settlement flag trigger for vol update.


#### As of 09/11/2017
- [Done] Roll product/TTM TOL should be part of individual families. 
- [Done] error log in volgraphix.
- [Done] debug straddle/strangle issue in the UI. 
- [Done] added time column to price data; settlement to settlement should work with absolutely no issues now. 
- [Done] catching errors with option construction. 

#### As of 09/12/2017
- [Dev] Rollovers per family.
- [Done] speeding up data processing; cleaning intraday prices takes a long, long time.
- [Done] marking beginning of settlement period. 
	> issue: settlement vs last data entry of intraday data
		- solution: hedge all remaning deltas using first price at beginning of settlement 
		period. wait till settlement, and if settlement price results in residual delta, 
		buy/sell to close it off. 

#### As of 09/28/2017
- [Done] Pulling intraday data from the database/cleaning. 
- [Done] reconciling intraday data with settlements. 


#### As of 10/5/2017:
- [Done] parallelize to run multiple simulations at once. 
- [Done] proper implementation of hedging per breakeven and per price move.


#### As of 10/10/2017:
- [Done] Go through profile results to isolate/refactor bottlenecks. 
- [Done] double check the PnL computation for HBPB and HBPS modes. 



##### As of 11/2/2017:
1) [Done] fix granularize function. 
2) [Done] standardize inputs for OHLC and intraday types 
3) ensure that rolling works properly with book vols
	> [Done] delta rolling
	> [Done] contract rolling.
		- [Done] fix the dataframe issue with rollovers not finding price data. 
		- [Done] remark all rolled products to settlement vols on that day, pending flag. 

4) [Done] re-run the hedging simulator for LH. 




###### As of 11/13/2017:
1) over night market time step reconciliation:
	1) change settlements to be inserted one minute after close. 
	2) [Done] multiple products = standardize the timezone. 
	3) [Done] overnight market data pulling caveat. pull one business day earlier. 

2) Pending bug fixes:
	2_ testing.



##### As of 11/29/2017:

1) TrailingStop implementation:
	
	> Finish implementing/testing trailingstop object. 
		- issue: active updated whenever new prices are passed in --> gap moves beyond the stop limit don't trigger stop hits because active is set to False. 
			solution_: implemented a lock that is activated when trailing stop monitoring is on, and deactivated only when the stop is hit. 

	> How the hell are TrailingStop and Hedge going to communicate?
		- 3 cases to consider:
			> trigger bounds > breakeven multiple.
				- while within range, hedge as per normal. when outside, trailing stop logic kicks in. run deltas until stop loss is hit,
				upon which we zero out deltas completely. 
			> trigger bounds < breakeven multiple:
				- going to run deltas anyway. do nothing. this is equivalent to just hedging according to breakeven multiple. 
			> trigger bound == breakeven multiple.
				- want to preferentially follow the trailing stop until the stops are hit.
				= solution: 1) stop loss monitoring should be active as soon as breakeven is hit --> run_deltas will return true. 
						    2) in the case where ratio is specified, want to ret

		DONE
		- full flow between trailingstop and hedge object:
			hedge_delta(intraday=True) -> passes price dictionary into trailingstop object --> checks monitoring status.
			--> run_delta returns TRUE if trailing stop is being monitored, FALSE otherwise --> should also return a FRACTION that will determine how much of the delta to hedge. 

		TODO 
	> Incorporating TrailingStop into is_relevant_price_move. 

		DONE
	> testing HedgeParser object and evaluating usefulness. 




------------- Notes -------------

##### Scenario-based implementations 
- [Done] Three different scenarios:
	+ Case A: Flat vols + step-update.
		* EOD update step.
	+ Case B: Settlement to settlement
		* Sanity check and ensure that results work the same when just dealing with settlements as opposed to settlements + intraday.
	+ Case C: hedging basis book vols and pnl basis settlement. 
		* update vols function that takes in a dataframe, updates the vols of all relevant options.

- common implementation requirements across scenarios:
	* [Done] Market Minuses update/reset
	* [Done] book vol variable/dataframe
	* [Done] refactor PnL computations. 
	* [Done] book vols and contract rolls. 
	* [Done] book vols and delta rolls
	* [Done] book vols and hedging with options. 
		- if using book vols, initialize/add with book vols, but compute cost using settle vols. 
		- [Done] ISSUE: TTM is wrong when hedging basis book vols
	* double check the PnL differential between HBPB and HBPS
	* testing. 

		 

##### Hedging Updates:
- move delta roll-hedging into the hedge object. 
- Re-implementing hedging logic:
	+ {breakeven multiple} -> {percentage} -> {buyback logic}


##### UI Implementation:
- Volgraphix implementation for each of the scenarios. 
	+ checkboxes for each scenario:	
		* Book PnL and Book Hedging
		* Settle PnL and Settle Hedging
		* Settle PnL and Book Hedging
	> pipe these booleans as flags into the simulation to determine behavior. 

- volgraphix implementation for intraday hedging conditions. 

--------------------------------------