Monday, November 16, 2015

Comment on Overnight SPY Anomaly

This post is in response to Michael Harris' Price Action Lab post, where he uses some simple R code to evaluate the asymmetry of returns from the day's close to the following day's open.  I'd like to respond to his 3 notes, which I've included below.
  1. The R backtest assumes fractional shares. This means that equity is fully invested at each new position. This is important because it affects drawdown calculations.
  2. When calculating the Sharpe ratio, the “geometric = FALSE” option must be used otherwise the result may not be correct. It took some time to figure that out.
  3. The profit factor result in R does not reconcile with results from other platforms or even from excel. PF in R is shown as 1.23 but the correct value is 1.17. Actually, the profit factor is calculated on a per share basis in R, although returns are geometric.
I completely agree with the first point.  I'm not sure Mike considers the output of  SharpeRatio.annualized with geometric=TRUE to be suspect (he doesn't elaborate).  The overnightRets are calculated as arithmetic returns, so it's proper to aggregate them using geometric chaining (i.e. multiplication).

I also agree with the third point, because the R code used to calculate profit factor is wrong.  My main impetus to write this post was to provide a corrected profit factor calculation.  The calculation (with slightly modified syntax) in Mike's post is:

getSymbols('SPY', from = '1900-01-01')
SPY <- adjustOHLC(SPY, use.Adjusted=TRUE)
overnightRets <- na.omit(Op(SPY)/lag(Cl(SPY)) - 1)
posRet <- overnightRets > 0
profitFactor <- -sum(overnightRets[posRet])/sum(overnightRets[!posRet])

Note that profit factor in the code above is calculated by summing positive and negative returns, when it should be calculated using positive and negative P&L.  In order to do that, we need to calculate the equity curve and then take its first difference to get P&L.  The corrected calculation is below, and it provides the correct result Mike expected.

grossEquity <- cumprod(overnightRets+1)
grossPnL <- diff(grossEquity)
grossProfit <- sum(grossPnL[grossPnL > 0])
grossLoss <- sum(grossPnL[grossPnL < 0])
profitFactor <- grossProfit / abs(grossLoss)

I'd also like to respond to Mike's comment:
Since in the past I have identified serious flaws in commercially available backtesting platforms, I would not be surprised if some of the R libraries have some flaws.
I'm certain all of the backtesting R packages have flaws/bugs.  All software has bugs because all software is written by fallible humans.  One nice thing about (most) R packages is that they're open source, which means anyone/everyone can check the code for bugs, and fix any bugs that are found.  With closed-source software, commercial or not, you depend on the vendor to deliver a patched version at their discretion and in their timing.

Now, I'm not making an argument that open source software is inherently better. I simply wanted to point out this one difference.  As much as I love open source software, there are times where commercial vendor-supported software presents a more appealing set of tradeoffs than using open source software.  Each situation is different.

Friday, July 24, 2015

New quantmod and TTR on CRAN

I just sent quantmod_0.4-5 to CRAN, and TTR_0.23-0 has been there for a couple weeks. I'd like to thank Ivan Popivanov for many useful reports and patches to TTR. He provided patches to add HMA (Hull MA), ALMA, and ultimateOscillator functions.

James Toll provided a patch to the volatility function that uses a zero mean (instead of the sample mean) in close-to-close volatility. The other big change is that moving average functions no longer return objects with column names based on the input object column names. There are many other bug fixes (see the CHANGES file in the package).

The biggest changes in quantmod were to fix getSymbols.MySQL to use the correct dbConnect call based on changes made in RMySQL_0.10 and to fix getSymbols.FRED to use https:// instead of http:// when downloading FRED data. getSymbols.csv also got some much-needed love.

I'd also like to mention that development has moved to GitHub for both TTR and quantmod.

Monday, April 20, 2015

plot.xts RFC

We have been working on a new charting engine for xts::plot.xts for the past couple years. It started with Michael Weylandt's work during the 2012 Google Summer of Code, and Ross Bennett took up the torch during the 2014 GSoC.

This new engine improves the functionality, modularity, and flexibility of plot.xts by building off the framework Jeff Ryan began with quantmod::chart_Series.  The modular framework allows users to plot an xts object and incrementally build custom charts by adding panels of new data (including transformations of the original xts object).

The main objective was to provide functionality similar to chartSeries and addTA for xts objects. The current code includes support for:
  • Basic time series plots with sensible defaults
  • Plotting xts objects by column "automagically" as separate panels
  • Small multiples with multiple pages
  • "Layout-safe" so multiple specifications/panels can be charted in a single device
  • Easily add data to an existing plot or add panels similar to quantmod::add*
  • Event lines
The xts team would greatly appreciate any comments, feedback, and bug reports before the upcoming CRAN release at the end of April.

The new version of plot.xts is in the main xts development code base, which is available on GitHub in the develop branch.  GitHub is also the place to submit bug reports and feature requests.

Note that the new plot.xts includes breaking changes to the original (and rather limited) plot.xts.  However, we believe the new functionality more than compensates for the potential one-time inconvenience.

Tuesday, March 31, 2015

Registration Open for R/Finance 2015!

You can find registration information and agenda details (as they become available) on the conference website.  Or you can go directly to the registration page.  Note that there's an early-bird registration deadline of May 15.

The conference will take place on May 29 and 30, at UIC in Chicago.  Building on the success of the previous conferences in 2009-2014, we expect more than 250 attendees from around the world. R users from industry, academia, and government will joining 30+ presenters covering all areas of finance with R.

We are very excited about the four keynote presentations given by Emanuel Derman, Louis Marascio, Alexander McNeil, and Rishi Narang.  The main agenda (currently) includes 18 full presentations and 19 shorter "lightning talks".  As in previous years, several (optional) pre-conference seminars are offered on Friday morning.

There is also an (optional) conference dinner that will once-again be held at The Terrace at Trump Hotel. Overlooking the Chicago river and skyline, it is a perfect venue to continue conversations while dining and drinking.

We would to thank our 2015 sponsors for the continued support enabling us to host such an exciting conference:

International Center for Futures and Derivatives at UIC

Revolution Analytics
MS-Computational Finance at University of Washington

Ketchum Trading

On behalf of the committee and sponsors, we look forward to seeing you in Chicago!

For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson, Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich

Tuesday, March 10, 2015

Import Japanese equity data into R with quantmod 0.4-4

I pushed quantmod 0.4-4 to CRAN this weekend.  It adds a getSymbols.yahooj function to pull stock data from Yahoo Finance Japan, and fixes issues in and getSymbols.oanda.

Changes to the Yahoo Finance and Oanda websites broke the and getSymbols.oanda functions, respectively.  I didn’t use much, so I’m not certain I restored all the prior functionality.  Let me know if there’s something I missed. I’d be glad to add a test case for that, or to add a test you’ve written.

The getSymbols.yahooj function is a major enhancement provided by Wouter Thielen.  It allows quantmod users to pull stock data from Yahoo Finance Japan. 

Japanese ticker symbols usually start with a number and it is cumbersome to use variable names that start with a number in the R environment, so the string "YJ" will be prepended to each of the Symbols.  I recommend using setSymbolLookup to prepend the ticker symbols with “YJ” yourself, so you can just use the main getSymbols function.

For example, if you want to pull Sony data, you would run:

The full list of supported data sources for quantmod is now: Yahoo Finance-US, Yahoo Finance-Japan, Google Finance, csv, RData (including rds and rda), FRED, SQLite, MySQL, and Oanda. 

Contributions to add support for additional data sources are welcomed.  The existing getSymbols functions are good templates to start from.

Tuesday, March 3, 2015

Google Summer of Code 2015

The R Project has once again been selected as a mentoring organization for this year's Google Summer of Code (GSoC).  If you're not familiar with GSoC, it's a global program that offers students a stipend to write code for open source projects, under the direction of a mentor.  Mentors get code written for their project, but no money.  Students get something like a paid summer internship, with open-source contributions they can reference on their resume.

If you’re interested in participating as a student or a mentor, there's an overview of the GSoC program on The R Project GSoC 2015 Wiki.  The wiki also includes a timeline and links to prior year's projects.

Several mentors from various backgrounds have already proposed projects for students to work on this summer.  Mentors have until March 9th to submit projects they would be willing to support, and student applications begin on March 16th.

Monday, December 15, 2014

Updated quantmod on CRAN

An updated version of quantmod has just been released on CRAN.  This is my first submission as the new maintainer.  The major change was removing the dependency on the now-archived Defaults package.  End-users shouldn't notice a difference, since I basically copied the necessary functionality from Defaults and added it to quantmod.

There are also several bug fixes. A few worth noting are: