Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How to launch an algo trading side project?
58 points by meifun on Feb 14, 2018 | hide | past | favorite | 88 comments
I've been writing my own implementations and using 20 year historical stock data. Have been test trading to see how my logic is working.

However, I have an issue. When I think I am ready to use real money how do I do this? Do I need to get ETrade or Robinhood to actually start automated trading?




Interactive Brokers (IB) is your best bet. However, I’d like to reiterate as others have stated that you need to be prepared to lose it all (and moreover that you likely need much more capital than you realize to stay solvent on down swings).

Don’t take this the wrong way, but it seems like you haven’t done a lot of research into this: Interactive Brokers is one of the first things that comes up when you search on Google (etrade, Robin Hood or any other retail trading just isn’t credible).

Finally, is your 20 years of historical data for just open/close? Is that your execution plan? (Otherwise you can get really tripped up as you move to intraday trading)


I know about IB but initially I thought that I couldn't use their API unless I also ran on their platform. Meaning I can't use my own VPS, etc.

I have been using 1 minute bar data.


I don't know about setting up VPS, but the standard way to use IB is to run their main client (either the GUI-based "TWS" or the GUI-less "IB Gateway"), then connect to the main client. Your binary is totally separate from the main client, which you do not modify.

Have you looked at Lightspeed Trading?


You can use their API from your own system.

https://www.interactivebrokers.com/en/index.php?f=5041

I strongly urge you to paper-trade the live market before risking real money. https://www.interactivebrokers.com/en/software/am/am/managea...


When using IB, you can trade on your own machine/vps/whatever. I suggest you do this, and set it up such that you trade on an IB paper account, make sure everything is running correctly and in line with your backtests, and then start trading with some real money. Good luck!


Great advice. Thank you.


To other folks in this thread - your warnings are justified but please stop conflating “algo” and “HFT”.


1. Only use money that you can 100% afford to lose.

2. Once (if..) you've made some profit, take out your initial investment and now you're gambling with free money.


"Once (if..) you've made some profit, take out your initial investment and now you're gambling with free money."

I never understood this sentiment, and I'm surprised to see it so often on Hacker News. It's irrelevant whether the money came from profits or initial investment - money is money? There is no need to distinguish the both, it's purely psychological.


It's a classic example of loss aversion[0]. In my mind the real loss risk of this venture is the opportunity cost spent developing the algorithm. Many people that fail to build trading algorithms would have been very successful if they invested their time/expertise doing something else. The person stands to lose not only his cash investment, but all of the cash he could have received if they focused their efforts on a less risky venture[1].

[0] https://en.wikipedia.org/wiki/Loss_aversion

[1] I'm all for risk, but Vegas has better odds.


It makes no sense. That's why I suggest you look for advice from seasoned guys. The average guy is very emotional to trade regardless of their technical and financial background.

There is zero reasons to lose money or carry on running a losing strategy. If you are trading professionally and quantitatively, you are runnings lots of positions. Unless you have a technical issue, you should not worry about losses on one or a few positions since you only care about what your average is doing.

That assuming your models work but you should not start trading with real money before you are certain of that.


Exactly. Thanks for mentioning this. It is important to get the initial investment back into my bank.


1. Read Flashboys

2. a) Realize how CRAZILY far many HFT firms have gone to inch out literally every single microsecond and achieve economies of scale with billions in assets

2. b) individual user-scale tools like ETrade won't get your trades in fast enough/any cost per transaction fees will erase any gains

3. Invest said capital in something like Betterment

If your model has been truly beating the market for a sustainable amount of time, I apologize for my comment and go all out :)


HFT is only one type of algo trading.

You can make money with simple trend following algos that execute perhaps only a few times a week.

To the OP, if you aren't sure about market execution, it makes me think you haven't modeled for slippage. Make sure you do, as that nudges many profitable algos into the red.


And maybe trading costs.


HFT is a subset of algorithmic trading, but algorithmic trading isn't all HFT.

For an example, a value based investment strategy can be implemented as an algorithm.


I've been ahead for 2 months in just testing my models.

Edit: granted not FY ahead.


Cool! Pretty cool to see some alpha generated by a single IC. I still recommend Flash Boys as a book though about HFT, very interesting and a good read


I’ve long thought that “Dark Pools” was a better book about HFT.


I ordered this book a while back and never started reading it I just got caught up in writing code for to long. Thanks for mentioning it.


May I ask what's IC?


I ordered it on Amazon :-) I'll read it this weekend.


I have only tried one algo trading platform, Interactive Brokers. API documentation: http://interactivebrokers.github.io/tws-api/index.html#gsc.t...

Their live market data is gated at 250 ms, so definitely not for HFT. Market data subscriptions are only $5 / month (US Equities, and others I don't remember), and commissions are low.

You can program in C#, Java, VB, C++, or Python. I use their C++ platform. It is not particularly pretty, but it seems well-tested and works. Perhaps most importantly, they have an awesome support phone number. They are responsive and very helpful.


The easiest way to make a small fortune in trading is to start with a large one. :-)

HFT is like picking up pennies in front of a steamroller.


> HFT is like picking up pennies in front of a steamroller.

here's to hoping I can stay one step ahead, lol. It's risky.


Check out https://www.quantopian.com/


+1 Read the zipline source, I learnt a lot in how to structure trading code / solving common problems.


I strongly second this. I've had some other projects get in the way, but I signed up for quantopian and have been very impressed with the things I have gleaned from reading others algos and postings. If your algo is really that good you get a cut.

That said, I do think ultimately you'd want to find something with a fast api for yourself lately.

Oh, and for entertainment purposes read /r/wallstreetbets


How often do you trade and how important is fast executions? I've noticed many programmers try to shoehorn in execution algorithms into a strategy that doesn't need it. If you can run an analytics program that just sends you a trade alert from which you manually execute, that will save you a lot in systems risk.

Also, the fact that you're asking this question tells me this is very very unlikely to be a good idea. Be careful.


My thought is to trade as infrequently as possible, sort of batching trades to do all at one. Nothing is automated for me yet. I am just manually reviewing my output.


Check out this podcast. Dis guy had a algo trading as a sideproject (now fulltime): https://www.youtube.com/watch?v=Co_GYku903k


Looks like this guy is looking to create a Quantopian clone, so in the end it feels like the whole interview was a bit of advert for the signal service to come...

In fact all podcasts of this guy are like than, a lot of charlatans and scammers are appearing on the podcast all the time, probably 99% of the guests...


I listened to ~10 podcasts and most people are very humble and talking about a lot of mistakes they made and I learned a lot. I cannot share that 99% of the guets should be charlatans or scammers. Did you listened to most of the episodes?


Yep at least the first 50, maybe more, beside the occasional guest, every one is selling a chat room, indicator or similar scam, just listen to this particular podcast where the host comments on the expected 60% yearly return and that it should be easy to do...


It's crazy in this business how everyone tries to sell something but I think there is still value there to extract, don't you think? It's a honest question because I'm new in this space and I always enjoy learning from others, especially from their mistakes so I can try to avoid them.

I would never copy anything they say or buy anything they offer.


The problem is that they will “teach” you the wrong things, because they do not have trading experience... all is smoke and mirrors with made up stories and nothing to learn... I was listening in the beginning as entertainment and because there were one or two real guests, but in the life is too short to waste your time on those scammers...


Awesome, I'll watch it later today.


Mind sharing where you get your market data? Are you using a paper account that you can simulate the trades? what markets are you participated in?

Apologies for the burst of questions. Am interested in testing out some ideas too


Historically I have scraped data, but now: http://eoddata.com/default.aspx


Be careful. EODdata has a lot of data issues. Seriously...a lot. I would recommend checking out Alpha Vantage. It's free and I didn't see any of the issues with their data that I saw with EODdata.


WOW! OMG. Thanks for mentioning your experience. Can you share more about this? My e-mail is in my profile if you want to do it privately.

Thanks for mentioning Alpha Vantage. I will check them out.


No need for private communications. Probably better to have here anyway for anyone else who is curious.

I was thinking of using their data to help test and run a system I am working on. I was excited to see that their stuff is free for what I need, as long as I wait until after a specified time to grab it (9 PM EST). I started looking at the data and noticed that it doesn't track corporate actions well at all. This becomes very problematic when you need to track your day over day returns for a security. There were also unexplained gaps and spikes in historical data. For index data, the lists seemed inconsistent in that things would seemingly drop out and then back in from day to the next leaving you with having to manually acquire the value of certain indices manually or having to process a historical file for that index to fill the gap. Historical files are not adjusted properly for actions like splits and ticker changes. Basically it boils down to there being a very, very limited use case for a provider like EODdata. Also, it should be noted that while I was never a paying subscriber, their subscribers have said there is virtually no customer service. I thought that might be the case looking at the website. It seems like EODdata was basically put on autopilot a long time ago and just left to run with no one at the helm. From a security standpoint, I recall them not using HTTPS (that should give you an idea of how long they haven't updated their site or security) and I clearly remember them storing passwords as plain text because I did a "forgot password" and it emailed it to me. In fact, I just did it again now to make sure I wasn't crazy, and sure enough, they haven't fixed anything. Still no HTTPS (even when manually typing in the HTTPS:// prefix) and it just emailed me my password (which was set to a bullshit password that I would never remember after discovering this gaping, unforgivable security travesty). I also remember having issues with some of the downloads not working correctly.

All in all, I'm damn near certain someone has this site on full autopilot and never fixes, updates, or even maintains anything. The plain text password nonsense should be enough right there to disqualify its usage, even if you are only using the free service(s) and never provide your payment details, and even if you use a unique password. If you were actually thinking of providing payment details, it's a clear and total no-go.

There are additional reviews from other people at these links: https://quant.stackexchange.com/questions/3284/is-eoddata-a-... http://www.trade2win.com/reviews/data_feeds/82-eod-data#revi...

They mention a lot of what I have talked about and more.


Thank you. I appreciate the word of caution. I'll look at the place you mention too.


If you are not planning to do HFT then let your system create signals and send them to your e-mail. Then buy/sell manually.


I went with Interactive Brokers and their python API.


I am working on the same thing. I'm still in the model building phase and I still need to complete my backtesting framework. You seem to be ahead but I've already looked into the trading API process.

Go with whatever broker offers a trading API obviously, and has a fee structure that matches your trading strategy. If you can afford to lose a bit on the trade through commissions and your strategy is swing trading then you can pick pretty much any broker. If you need specific fill times and commission percentages before your strategy pays off then you are stuck with Interactive Brokers ($10k minimum[1], low comms, fast exec) or Robinhood (low minimum, free comms, slow exec). And technically Robinhood doesn't have a public API[2] so you would have to do a little reversing of their app's messaging. And then there's terms of service... yeah. Probably not a big deal for a small fry like yourself.

Next would be testing. You've got isolated tests with historic data but try get a broker that has a paper money version. TDAmeritrade's paper & api is good (but they are expensive). To be clear you would be using your algo to trade in a paper money account. That lets you get more realistic trade tests and if you blow something up you are still ok!

Keep in mind Robinhood is free because they are paid to route your orders to hedge funds and they are re-ordered in the book. You will get ok price execution through them but never the best. One thing you should include in your backtests/simulations is order execution delay and fill price variance. If you place market orders they won't always fill at the quote price. If you place limit orders they may take a while to fill (or not at all) unless the price is within the other side's order prices.

Last is the pattern day trading rule. Make sure you have a strategy that won't make more than 3 day trades in a 5 day window, otherwise your account might get suspended for a few months. It's a pile of horses but you can get around that by simply having more money in your account. > $25k will buy you unlimited day trading privileges.

Just curious what your strategy is based on if you don't mind giving a few details.

Good luck!

[1]https://www.interactivebrokers.com/en/index.php?f=4969 [2]https://support.robinhood.com/hc/en-us/articles/210216823-Ro...


One suggestion to avoid the PDT rule (and in general be more efficient with your capital) is to look at futures. For instance rather than trading SPY trade ES. ~8k per contract minimum to control ~140k (50*Index Points) of the SP500 and no PDT.


You say more efficient, I say leveraged. Potato Potato. :) Yea futures are much more interesting in my opinion and no PDT is way better!


Wow, thanks for this advice. I think I have a few questions. Let me read more carefully to be sure later.


What if I want to do fundamental analysis? Anyone know of any cheap or free data sources that have parseable financial statements?


If I may, what is the timeframe for your algo trades?

If it's short one like say in minutes, then you might want to test it with small amount of real money before going all in. The reason is orders fills are imperfect. So, cost price you had during backtests vs realtime might differ a lot.


This is the hard part. How much money am I willing to part with :-) Some of these places I think need $25k in an account.


Actually, that threshold is an SEC one for being a “frequent trader”. So anyone who doesn’t is just not telling you that you’ll need at least 25k (and IIRC, at all times) to make the volume of trades you want.


A friend of mine did this and had the algorithm output the trades and then he made them manually. If it trades too quickly for that to work, trading costs will likely be an issue for you anyway.

Remember hindsight bias, and as others have said, don't invest more than you can lose.


This is how I did it. My problem was that I listened to my algos for entry points but let my emotions take over for exits, so I often left money on the table.


What you did is a natural instinct/emotion! I can see this happening to me also.


Depends...

What instruments are you trading?

What frequency?

What size?

Why does it need to be automated?


You said you have been test trading. What does that mean? Where are you testing? Which platform are you using?


Since this topic is trending, I would like to grab this opportunity to ask if anyone's knows any material on how FPGAs are used in HFT/ Quantitative trading? Any books/ web sites / blogs?

Any pointers would be highly appreciated?


for development in Python you can use https://www.backtrader.com with simulations and live trading


TradeStation. I wouldn't bank on any HFT, unless you're doing FPGAs, and you've got the 'good' port on the switch.


I have always wondered if FPGA's are really needed. Can you share more info about this?


Are you doing HFT trading? If so, then the FPGA could help. Are you just doing plain ol' algo trading then you wouldn't really need it.


plain ol'. Currently I dont even have this running on a VPS yet. I am just using a Mac Pro in my house, lol. But I am not "live" yet.


May I ask what do you mean by good port on the switch?


I think he/she is referring to co-locating servers at exchanges or using companies that are already hosting their servers there.


Do you have any experience with InteractiveBrokers?


Robinhood have an unplublished api. There are hard limits on the number of trades you can do. But i would use that


Can you share more about this? IIRC web isn't ready yet so mobile only.


https://github.com/Jamonek/Robinhood I used that a few months ago, you might need to sniff the traffic if there has been changes. it wasnt too difficult


I am glad you ran with my idea and found an edge pretty quick!

What is your average yearly return on the 20 years back testing?


Your advice was absolutely amazing and cemented my desire to continue! Now I need to figure out how to actually execute. If you have any advice! I keep staring at a list of possible transactions and manually checking how my trade might have turned out and keeping a gain/loss figure.

I was hoping you would chime in on this thread. I was not so comfortable with the work I was doing and felt like my direction was wrong and I was questioning why I was even going to try this. Your advice helped me decide to continue.


Hey Jason, I am glad I could help!

Before you start with the execution, make sure you did use the 20 years of data, go and back test all the years, they encompass few market cycles, so you will know how your strategy will perform in bull and bear markets.

For the execution part, when I started, I had an ssh session open to my server with the output of my bot running on the screen and had it configured to chime with something like” EXECUTE buy 2000 HYZ at $65 stop 56. I did it manually for a few weeks until I made sure everything worked properly and then did the order execution with the API. Interactive Brothers are a good choice with a good API.

I would stay away from Quantopian and other platforms that are holding your code, because this is the prefect target for a hacker and the perfect crime – nothing to steal or sell, they will just copy your winning strategy…


The Interactive Brokers (IB) API would probably be your best option for trading on the side.

Here are a few words of advice regarding IB:

- Thoroughly read and understand the IB documentation. - Be sure that you never get close to any of IB’s risk limits that can result in the automatic liquidation of some, or all, of your positions. - If you don’t want to put up enough cash in your IB account for day trading, be sure you don’t exceed the trading limits that will get you classified as a day trader and thus prevent you from further trading until you add more cash. - Be aware that the IB market data can have significant inaccuracies and/or delays.

Here are a few words of advice on automated trading:

- Thoroughly understand the exchanges that handle the instruments you want to trade. Each exchange has online documentation explaining details of traded instruments, matching algorithms, pricing, and market data. Even though you won’t be accessing the exchanges directly, these details have an effect. - Thoroughly understand IB’s order routing and how they construct their market data (if you use it). - Just because your historical data says an instrument closed at a particular price, that does not mean you could have actually bought or sold at that price with IB or even with direct market access. The same is true of tick data constructed at 1 minute, 1 second, or even millisecond resolution. Trading simulations can be terribly inaccurate if you don’t have a reasonable approximation of the exchange order books, the exchange latencies, IB’s latencies, and your own trading system latency. To counter this, you need to be extremely conservative in your assumptions about getting an order filled at a particular price. - Even with accurate market data and exact latency measurements, predicting future profits by backtesting over historical data is perilous. I advise having a strong background in the statistical/machine learning methods for constructing and analyzing appropriate training, testing, and validation sets. - Don’t make a trade just because some analysis suggests the market will go in that direction. Doing this will probably get you hurt and it harms markets by pushing them to overshoot or undershoot the correct prices. The trend is your friend until he turns around and stabs you in the back. Make a trade because that’s where the market should go and probably will go eventually.

My best advice for trading profitably is to do something to improve a market. Focus on areas that can benefit from your strongest personal advantages (mathematical, technical, domain knowledge, etc.). Gain an understanding of the exchanges and instruments that allows you to recognize ways to make prices more accurate. This improvement could be in an absolute sense for individual prices or it could be in a relative sense in terms of how instruments should be priced with respect to one another. It could also be in terms of adding liquidity (offering to buy or sell) where needed and justified. Improving markets improves our economy by helping it to better prioritize resources and limit risks. That is a valuable service and the market will pay you for it.


Consider cryptocurrency over stocks, as it comes with exchange API's and low/no fees.


I agree. There seem to be several good leading indicators for cryptocurrency prices, especially for new altcoins. I'm building https://coinparse.com/ exactly for this. I wrote a post on this recently: https://coinparse.com/articles/omg-anatomy-of-price/ as an introduction.


Crypto doesn't have 20 years of historical price data though.


True, but there are many other market conditions in play there to be exploited, and a year or more of data is still useful.


Isn’t history-based trading a myth? I mean, technical analysis of markets in general.


It's not about using the 20 years of data to fit your model, it's about using the 20 years of data to test your strategy on (a process called backtesting). Once you know how it performs historically, you can develop an intuition for where it breaks down, and try to alleviate the problem (keeping in mind overfitting and all those types of issues).


From my understanding you basically only have technical data when you're trading crypto. Or you can do HFT I suppose and basically supply a volume service to these exchanged.


have you looked into quantopian?


Thanks. I just looked at them. I like their idea of letting you test, but all of my code is in c++. I have 6 apps that need to continually run to fetch new data, import it, create models, make predictions and execute them. I'm not ready to share code, but I could always share how my architecture is works.


Then probably something like IB is your best bet. Do you have margin requirements?

Apologies if you already considered this, but make sure you consider execution costs. Depending on your type of strategy and selection of stocks, it may be a major determinant of the success or failure of the strategy.


http://quantconnect.com is a better alternative for you, take a look.


Thanks. Why do you feel they are a better alternative?


Considering you can't use Quantopian, it can't be much worse.

TBH I've never used them, I just remember looking at a platform that was much more tech agnostic, I'm not sure if this was what I looked at, but looked potentially useful.


As others have said checkout cryptocurrency, im founder of https://bitbank.nz providing ai forecasts, charting and bulk data for backtesting bots. Checkout our blog at http://blog.bitbank.nz for info on how to easily develop profitable trading algorithms from some of our open cryptocurrency data. Also checkout our GitHub




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: