I've been regularly monitoring the machine learning job postings on upwork.com and absolutely love some of the requests coming through, especially those focusing on stock picking or FX (forex / currency exchange) prediction. And they want it cheap. Which begs the question that if you had the ability to create a learning algorithm that could make stock predictions and you could make money on it, would you really sell it to someone for $100?
Here's a typical one:
Forex Trading Algorithm Fixed-Price - Expert ($$$) - Est. Budget: $250 - Posted 5 days ago
Develop a Forex Trading Algorithm/robor, particularly to trade on the currency markets with a high wining streak.
- Project Type: One-time project
- Other Skills: Algorithms C++ Machine Learning Node.js Python
To some machine learning has become the new snake oil and with a little effort there is untapped knowledge just waiting to be exploited. I see a few problems with these types of requests for making money:
1. In finance courses and prospectus for mutual funds the common disclaimer is "past performance is no guarantee of future returns". That is, just because something did well in the past doesn't mean it will do well in the future.
2. More importantly, if one person has figured something out, then chances are everyone else has too. Looking at hedge funds, definitely one of the groups on the cutting edge of stock market prediction with the most invested in new algorithms and processes (smart, automated high frequency trading), few do better than the market average, and it is a rarity to beat the market consistently over the long term.
This exposes an interesting tension. Machine learning can learn data, and with enough data, make predictions that match human ability, or even exceed it, in a fraction of the time it would take a human to do it. The advances we've seen with translation and autonomous driving bears this out, and we can only expect it to get better over time.
So why can't machine learning become better at predicting the stock market? There is a lot of historical data that can be used for training. Part of the problem is that predicting high returns is difficult as more smart algorithms look at the same historical data and will make the same predictions, more or less. It is harder to make a high return if everyone is making the same investment.
Perhaps too there is an issue with the number of features within the data (i.e. the different categories of data points that can be analyzed). A similar problem exists with current climate models. They have been notoriously inaccurate at predicting temperature changes in relation to CO2 levels due to the complexity of the model and the amount of CPU power needed to calculate all the variables. In addition, climate models are based upon theory and a lot of the science and correlation is still being discovered and has not been incorporated into the model. For stock market data causation is fairly clear, based upon past performance. In both cases the climate models and stock market models work well with the data given and accurately predict past results. Clearly the climate models fail in predicting future results, which we know because the results are publically available. Less clear is how well machine learning algos are doing at stock market predictions as these are closely held trade secrets.
If past data can correctly predict translated data or autonomous driving, why can it not predict the stock market? Data for translations and autonomous driving is fairly fixed, with enough observation and adjustment only the tails of the data are difficult to predict but the majority of testing data is solvable. The stock market is dealing with changing live data due to other algorithms affecting the markets but also due to human behavior, that may or may not be predicatable depending on what assumptions are made on the agent involved in the trading. It is much harder to make a prediction if the base assumptions are constantly changing.
I'd be willing to guess that success in stock market returns via machine learning will emerge from an area unrelated to financial data, perhaps sociological data related to human behavior.