Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Open problems in finance?
55 points by edithv on Feb 6, 2016 | hide | past | favorite | 67 comments
Hello everyone,

I've seen people working in finance or interested in finance on this board, so I thought I may ask.

According to you, what are some open problems in finance, old or new? I am especially interested in quantitative finance.

I would ultimately like to come up with a list of them, a little bit like we do in mathematics, physics and computer science.

Thanks and have a great day!

Not sure if this would come under your defintion of an "open problem", but in the financial risk space there are many time consuming and costly challenges implementing economic capital models.

These models are used to determine the probability of an account/customer/contagion risk group defaulting (aka PD) and the loss incurred, given a default (LGD).

The current models require a lot of historical data to train them, and need to incorporate economic cycle factors.

There are some vendors in this space, but their products are very dated, expensive and leave a lot to be desired. Also the products really provide a platform for an actuarial team to build upon, rather than having an out of the box solution (which is probably desirable in the banking space)

While your interest is in fintech, do take a look at the forces shaping the broader global macro economic environment because you can literally see some of these "open questions" playing out in real-time!

List of Unsolved Problems in Econ:


What happens when central bankers run out of ammo and realize monetary policy may not be an ideal path to fiscal stimulus? Or having the bright idea of devaluing your currency to depreciate your outstanding liabilities results in a mutually assured destruction scenario when every sovereign on earth implements the same idea simultaneously? And when "too-big-to-fail" entities amass derivative exposure that is 100X global GDP does it become time to panic yet?

But still its innovations like ZeroCoin and SmartContracts that give one hope that out of the rubble, creative solutions will arise. As long as they are grounded in economic reality and not fantasy, humanity will abide...

Thanks for posting that list. Interesting problems.

One thing that I think is missing from that list is how to factor in technology into economic models or measurements like GDP.

Right now this dark horse which in my view have the biggest single impact on our world is mostly left out as an externality which if you think about it is not just absurd but straight out irresponsible.

Politicians listen to economists and their models and as long as it's kept out of those models they can't react politically to it. And thus technology will keep moving faster than legislation.

1. A question that has interested me for some time: what is the best quantitative measure of "risk"? Intuitively we understand risk to be related to the distribution of investment returns. More information means that we are more confident about our prediction, hence our risk is lower.

But pinning risk down "mathematically" is kind of tricky. There are several proxies for risk under modern portfolio theory, including variance, standard deviation, shortfall, half-normal variance. Why are there so many different measures? If analytical tractability was not a concern, what should risk models look like?

2. Multi-horizon portfolio optimization. Portfolios are typically optimized one time step (horizon) into the future. However, if one wishes to optimally allocate a portfolio with extremely long-term investment periods (think Berkshire), one needs to subdivide that period into multiple horizons and optimize over these together. This is computationally expensive - is there a way to do this efficiently?

#1 I don't think there is a perfect measure. In practice portfolio managers use multiple views of risk, including multi-factor risk models, stress testing, value at risk, etc. Any tool money can buy they will use.

#2 has already been solved: One formulation is here: https://stanford.edu/~boyd/papers/pdf/dyn_port_opt.pdf There are optimizers out there already on the market with these capabilities.

#1 - Whether or not there is a formal risk measure that is less "ad-hoc/in-practice" than the others is still up in the air, no?

1: Risk, just like reward, is an umbrella term. It doesn't specify one particular thing. Take reward for work, some of it is monetary, some in terms of learning experiences, some in terms of friends from work, some reward is in working close to home or at home etc. Risk is similar. The variance you mentions measures volatility, but value-at-risk does not, it's also a risk but it measures the most you could lose at some high degree of probability. For different perspectives and measurements you have different models.

2: My biggest concern here is that the data isn't really reliable in extremely long-term investments, MPT mostly runs on historical data, if you're wanting to pick stocks to invest in 10 years from now on today's data, it's like investing in today's stocks on data from 1996-2006, which is pretty silly. Such very long-term models are just not very feasible. The weather is a great example, I think. We're pretty good at calculating patterns an hour or a day into the future, two weeks into the future is extremely hard. There's just too many small things that can explode into significant changes.

You mentioned Berkshire as an example, they're pretty much the opposite of modern portfolio theory. They look first and foremost at fundamental business analysis, i.e., is it a good business, is there a solid management etc... MPT disregards virtually all of that.

Every model has assumptions, MPT relies pretty heavily on the notion that 'all knowledge is priced in', so that analysing businesses and building portfolios like Berkshire is not going to get you anywhere. Rather, you're left with looking at historical data and building a portfolio like that, assuming that this data includes all information in the market, because it's price data and everything is priced in, so it gives the most accurate reflection of the market. Lots of counterexamples show this isn't the case, it's still a nice theory but it does better in the short (or medium) than the long term imo, and not because it's computationally expensive.

I see - thanks for the clarification! Looks like they were only "open problems" in my mind :)

The measure for risk is probability. Variance, sd, etc. are just attempts to simplify complete probability distribution.

This isn't a quantitive finance problem but more of a consumer finance problem: I really think it is about time there is an API for consumers to access their data at financial institutions. I could see that opening up all kinds of innovation. (Selfishly it would allow the startup I'm working on to multiply the value we offer to users.)

Obviously a standard would be nice but in practice I'm sure each institution would come up with their own or embrace and extend the standard. e.g. FIX with trading.

I looked into this problem extensively a few years ago. Banks are actually moving backwards. The ones that previously exposed an API are killing them off and moving to a proprietary API that talks only to their app. Every retail bank is basically creating their own version of Mint, where you add your accounts from other institutions to their app.

Ready only access would be key too.

No exactly an "open problem" (as in science), but an underlying problem is that there is no "google for finance". All financial data is scattered over different places, some of it not openly accessible, in crude formats, etc.

Recently mentioned on HN: https://www.tiingo.com/welcome

And of course https://www.quandl.com/

Thanks, I didn't know those yet.

IMHO financial data is scattered for two primary reasons:

1) "Finance" is extremely fluid because of the speed with with transactions are conducted...(e.g., the stock market)...during the course of a day's trading market reports are given minute by minute, but the most useful "summary" of "performance" is often available only at the end of the day when the market closes...

2) Entities derive a capitalistic advantage by protecting their own data, until, and unless, they're given sufficient incentive to share it...

To me this implies that a system sufficiently complicated enough to provide "meaningful" data captures might have to be nearly as complicated as the field of finance itself...the best we seem to be able to do just now is provide snapshots--AKA, the "leading indicators", etc...

3) Data is protected/hoarded/scattered because for regulatory reasons. Some countries (LU and CH come to mind) don't allow "their" financial data to be stored abroad, for instance.

4) There are many (MANY!) financial standards and formats. But at the end of the day, a payment is a payment. There are only so many different kinds of financial instruments and transactions.

There's a lot of noise, but the signal is still there. And solutions exist to make sense of it all.

Good points...

The "signal" is there, I agree...I used the word "snapshot" and I'm guessing that we mean something similar...

Global finance is a moving target...gleaned information is useful one minute, sometimes meaningless an hour later...

Agree 100%. Good data is very expensive and even then many times needs additional cleaning. Public data sources are for the most part unusable.

what do you mean when you say that public data sources are mostly unusable? what public data sources are there and what renders them unusable?

There are a couple of "googles for financial transactions" out there: http://www.intix.eu

Remittance is still one the biggest unsolved problem according to me.

I could consider cryptocurrency as the ultimate solution but online if everywhere its being used in equal importance and liquidity. For more traditional money, banks are squeezing fees & spreads out of regular people. Companies like TransferWise are trying to resolve some of these problems but still they rely on the existing banking system to move the funds around.

So there you go - open problem!

It may be just my ignorance. But I always thought we need a better measure of the health of an economy. (All?) the proxies like GDP are either simplistic or flat out deceptive. GDP for example seems to suggest that throwing a bunch of money around without producing anything improves the health of economy. I realize this is not really a quant finance related question.

A metric that I like is mean(log(after-tax income)). It is a reasonable measure of the average "utility" that income provides to a population. If money circulates among corporations but does not change personal income, then the metric remains constant. If everyone's income increases 1%, then the metric increases 1%. If inequality increases, then the metric decreases since average utility also decreases.

If money circulates among corporations but does not change personal income, then the metric remains constant.

This is true of GDP as well, since GDP only includes final goods. I.e. if a farmer produces wheat, and the wheat is then used to produce bread, the farmer -> baker transaction is excluded from GDP. Only the baker -> consumer transaction counts towards GDP.

If you want to try and measure utility, your metric should be based on consumption - e.g. mean(log(after tax consumption)). Income isn't utility, it's only potential utility.

If we used income, your metric would unfairly penalize volatile income. I.e., a person with a stable income of $50k/year would be considered 70% better than a person who earns $100k then $0k (saving $50k in the first year and spending it in the second while having an identical lifestyle.

Good point on the final goods distinction, thanks.

The distinction between consumption and income is not significant for a large fraction of the population, and if you are going to track just one number, it really shouldn't be consumption. If you do that, you have no hope of tracking changes in social structures or inequality or wealth accumulation. You would have to track wealth or income too.

Addressing your concern over volatile income is not that important and could compromise the usefulness of the metric. If you propose using GDP or any other linear function instead (rather a logarithm or power-law) then you cannot approximate utility and your number will be insensitive to changes in inequality. If incomes go up for the bottom half by 10%, then GDP would not change by much, though many would be better off. Income taxes are already calculated annually to smooth out these income fluctuations for seasonal workers.

If you do that, you have no hope of tracking changes in social structures or inequality or wealth accumulation. You would have to track wealth or income too.

But you wanted to measure 'average "utility"', not "social structures". Utility comes from consumption, not income. You are right that tracking consumption would fail to be a proxy for income inequality (as you now seem to want it to be) because consumption inequality is vastly lower than income inequality.

At this point, your metric sounds less like an improvement on GDP, and more like just some other random thing you want to track.

The idea of using log-income as a proxy for utility is not new. It is a standard measure in the economics of taxation. Yes it is different than measuring consumption. It is an improvement over GDP in the sense that it measures how the economy is serving the population. Some consider this to be a laudable goal.

The economy is serving the population when it enables consumption. Income is shifting numbers around in banks, consumption is delicious mac&cheese.

That said, you criticized log(consumption) above because it doesn't measure "social structures" and "inequality". Those things are pretty explicitly not how the economy is serving the population.

If you include wealth transfers such as inheritance in consumption, and you include mortgage spending, then log-consumption can work fine too. The only problem is that it is more difficult to measure because you would need to track either either consumption per-person or both income and savings per-person.

Income is easily measured for individuals, and aggregate consumption is easily measured by merchants. This is why we have a progressive income tax and not progressive consumption taxes. It could be done, it is just more difficult.

You can repeat as often as you wish that "inequality" is not relevant, but you are wrong. Utility is increased more if $1 is earned/spent by a poor person than a rich person, ipso facto inequality is bad for utility, all else being equal. My reference to the economy serving the population, is just a reference to utility, as opposed to any other linear measure such as GDP.

There are many economic metrics apart from the GDP, and they are freely available. Things like national productivity, currency inflation, average amount of savings, etc.

The most telling one, for me personally, is the human condition of the bottom 20% of the residents.

Yes. Measuring inequality or happiness etc would seem better but are they quantitative and objective? I don't know how this human condition metric is quantified. I will look it up though. Moreover this has the same problem as GDP. Supposing every one is happy and there is very little inequality in the society. But it does not tell me anything about the actual economic progress. Unless I define progress as reducing inequality, in which case this argument is moot.

Any sources of that metric that aren't plagued by political bias, be it progressives or neocons?

Good question. I would like to know what's been debunked in finance.

Personally, I cringe when I hear talking heads correlating then Fibonacci number with the stock market.

And I've come to realize, if Jim Crammer recommends a stock, don't buy it. I would like to know which professional out there has the best track record in stock speculation. It seems like something that we could accurately study, and publish the data?

I don't have the money to speculate in the stock market. It just seems like it's as bad as gambling at this point in history. It interests me though. If I had money it would go to realestate speculation. I'm so wrong on my stock picks, I've though about contrarian investing. Supposedly, there are such people?

You're not "so wrong on your stock picks", you're wrong almost exactly 50% of the time (well actually you should be wrong about 40% of the time, but those 40% of bad picks will lose you about the same amount of money as the 60% of good picks earn you).

Just like everybody else, even most professional fund managers, although they wouldn't like to admit it.

Quantitative risk measures for longer term fundamental investing. Beta, cointegration, correlation, VAR all break down when there is a discrete event that completely changes the fundamental outlook.

The most popular stack with IDE lacks most modern programming tools. I am of course talking about Excel. Where are my unit tests, version control, and so on?

A good start would be a readable and round-trippable format that Excel documents can be saved as

In my university finance classes we were forced to use calculators that were invented in 1976[1] to run calculations on formulas we had to memorize that could easily have been automated in Excel. Since then I've been prejudiced that "the invisible hand" ;) was deliberately keeping electric technology out of finance in order to maintain a legacy business model. Like that of the MPAA towards VCRs and DVDs etc.


> A good start would be a readable and round-trippable format that Excel documents can be saved as

What happened to CSV?

Edit: Maybe you want to be able to save the equations etc?

Risk neutral model selection. In general, if your model has too few parameters, it doesn't capture some features, and if it has too many, it overfits. In statistics, the balance between fit and model parsimony is achieved by information criteria, like Akaike. There is nothing similar for risk neutral models. How do you choose between a one factor Vasicek model and a 10 factor Cheyette with stochastic-local volatility and time-dependent mean-reversions? Currently it's much closer to an art than to a science.

True digital cash: secure, anonymous, portable, offline etc.

+ universally and instantly spendable :)

That's not a problem, that's a product

Offline and secure it's the possibly impossible part, but it would be nice!!

That doesn't really have all that much to do with what is considered finance. It's maybe a bit like saying an open issue in computer science is having a great keyboard.

The measurement problem, if by quantitative you mean "relating to, measuring, or measured by the quantity of something".

There is no consistent mathematical standard of measurement by which money is created. Instead, the monetary value of some good or service used as capital for a loan is largely up to the arbitrary appraisal of the bank.

I know not all countries have this problem. But i live in Switzerland and not a single bank offers me some kind of API. All i was asking for is a readonly interface to allow me personal tracking. (We dont do credit card payments for everyday stuff here, we use money)

How to prevent bubbles from forming. How to provide a viable way to invest in ones retirement that provides some surety.

How to calculate rate variances and volume variances. There is currently a standard used, but it is misleading.

How to make money. Unsolved, or at least those who've solved aren't sharing. (100% serious answer.)

Since you insisted it was a serious answer, I'll bite. Making money relatively straightforward. Money is a type of medium of exchange, it possesses certain qualities that other media do not.

You make money first by creating the tangible / intangible artifacts of that exchange, like currency, systems of accounting, and such. You then get other people to use the system to exchange goods and services.

This typically requires a government, but other entities have made money in the past, and even today. Bitcoin satisfies all the criteria of money.

The practice of fractional-reserve banking is another example of money creation. Most banks will lend out more money than they have access to in deposits, as a relatively safe form of leverage.

This has the effect of slightly expanding how much money is available to the economy, aggregated over all the banks and loans being made by those banks, the expansion is significant.

Things like video game currencies also satisfy all the criteria of money, though the things you can buy with them are limited, sometimes you can trade them for other currencies.

So there's lots of ways to make money.

100% serious response: What do you mean when you say "how to make money"? One can earn wages through employment and invest that money in the stock market (specifically, something relatively safe like an index fund will give you a predictable return). Do you mean how to maximize earnings with minimum effort? How to make over $X per time period?

Have lots of it to begin with. Pay the law makers to rig the game in your favor.

There's a potential unsolved problem if you think Bernie Sanders would eventually get his wish of taxing stock trades.

In fact, I suspect there's a Superpac somewhere that would fund research showing the net effect it might have. Or some organization that would want to know what that does to algorithmic trading.

I'm guessing though, the likelihood of a bill like that making it all the way is pretty low.

Canada tried such a tax, and there is already research showing the net effect it has.



The net result is that institutional investors (Goldman) won, retail investors (grandma) lost and spreads (the cost of trading) increased 9%.

I haven't been paying close attention, but I thought Bernie was contrasting himself against Hillary who is supposedly in the pockets of banks. That image doesn't really fit with such a policy.

Here's what he has on his official site regarding it:

"Has proposed a financial transaction tax which will reduce risky and unproductive high-speed trading and other forms of Wall Street speculation; proceeds would be used to provide debt-free public college education."[1]


I'm trying to avoid getting into a debate about whether it's a good idea. If you think, though, it has some chance of happening...it does specifically become an "Open Problem in Quantitative Finance", which is what the OP is asking for.

Yeah, a financial transaction tax is exactly what Canada did. It did reduce "risky" [1] high speed trading, and in the process hurt retail while helping (a little) big traders. This is pretty much exactly what microstructure theory predicts. Figuring out the effects is a solved problem.

The "open problem" I guess will be improving the existing tax avoidance software for traders?

[1] High speed trading is extremely non-risky. Look at graphs of the most extreme HFT incidents: https://www.chrisstucchio.com/blog/2012/flash_crash_flash_in...

>helped big traders a little >reduced HFT

don't big traders use HFT as a market advantage over retail?

Not at all. The HFT debate has two real players: Large banks and large active investors (on the one hand) and HFT firms on the other hand. HFT firms profit by driving margins down at the expense of established players, and by reacting quickly to very large orders. Retail investors, however, benefit from the lower margins, and weren't placing very large orders. They're fairly clearly winners. (So are large passive investors. Vanguard is on record as being very pro-HFT, because it drives their costs down.)

One description of the HFT (from the always thought provoking Matt Levine): "It's an incremental efficiency improvement, with some opportunity for gamesmanship, that overall allocates some money out of the pockets of banks and hedge-fund managers and into the pockets of exchanges and HFT technologists."

That's basically correct. And as a bonus, the small retail investor buying twenty shares of Apple now gets his shares very slightly cheaper, which is kind of nice I suppose. (Unless you work for a large Wall Street firm who's revenue depends on retail investors paying high margins. But in that case my sympathy is quite limited.)

See also this article I wrote explaining this: https://www.chrisstucchio.com/blog/2014/fervent_defense_of_f...

Concretely speaking, HFTs price discriminate when providing liquidity. Grandma gets a good price to sell her 2 lots of GOOG, but Bill Ackman and George Soros need to pay more to sell 20000 lots.

One of the problems with the Canadian tax law, was that Canada has to remain competitive to the same-language speaking country south of its borders that has a bigger and more advanced stock exchange. It is explained in the last section of your second link.

The same problems will not be necessarily manifested if the law is applied in the US, especially if Canada has similar laws. However, the points of the first link seem to hold.

That only applies to cross listed symbols (e.g. Blackberry), currencies, etc.

The fact that one can shift trading in cross listed symbols to other venues doesn't mean that supply&demand also doesn't reduce market making (thereby increasing costs). In fact, Bernie Sanders explicitly hopes the tax will do exactly that.

The "breakthrough models" are not stable and fall apart as soon as the world financial landscape drastically changes. It took me 17 years to build a working model (black box) and my secret ingredient was an "organic" sub formula that allow the core structure of the model to adjust with the changing financial landscape.

Changing legacy business processes to align with the post-recession regulatory landscape.

Predicting the markets


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact