Hybrid cloud typically is a coupling of on-premise and off-premise cloud environment. However, hybrid data architecture not only entails hybrid cloud, data-warehouse (DWH) design too has the potential of being hybrid to get the optimal benefits. With the growing business complexities and volumes of data clubbed with data securities & compliance issues prompting many to consider hybrid data architecture which is beyond just on-premise DWH. This involve migration of some of the on-premise workload to WAN environment in public cloud. The objectives are to have the best of both in terms of greater control & high level security combined with…

Trend-following which also termed as momentum trading strategies which buys the past winners (stocks with the highest returns in the recent past) and sells the losers (stocks with the lowest past returns). To better understand the nature of trend-following trading strategies and discover the corresponding optimality conditions, let us consider the cases when the market trends i.e. the general direction the market is taking during a specified period of time. In mathematics trend is the degree of serial correlation in the data while a trend is a series of higher highs or lower lows in trading. If we take a…

Relative Strength Index (RSI) introduced by J. Welles Wilder Jr. in his classic book “New Concepts in Technical Trading”. He used RSI as a momentum oscillator to identify reversals in security prices. Momentum strategy basically bets on the continuation of an existing market trend. While it is possible to time reversals using RSI, reversal strategies do not utilize the inherent strengths of the indicator.

Let us understand momentum first. When using Technical analysis for prediction of stock values, few assumptions are made:

- It is assumed that market moves in trends.
- History repeats itself i.e. under similar kinds of inputs the…

Modeling real-world problem is rather an art than science and this has been agreed by many scholars. Nevertheless, despite of we need to incorporate scientific elements and rules but there is still some consensus that certain models are elegant and beautiful while others lacking aesthetic appeal. Here, we are going to discuss about optimization techniques using mathematical formulation & model. Before diving into it, let us understand the problem.

Optimization comes in many forms and state; whether it’s a ML model optimization (hyperparameter tuning) or feature selection or selection best asset distribution or it could be cross validation / walk-forward…

Predictions over time tend to become error prone in the case of time-series modeling; thus a realistic approach would be to re-train the model with actual data as and when available for future predictions and validate simultaneously. Unless we have a robust optimization process, our modeling exercise will remain in hypothesis state. Though walk-forward framework could be time expensive in the case of big data, but for statistical model, time is not a constraint and walk-forward validation is the most preferred solution to get most accurate results. This involves moving along the series one step at a time. …

Mean reversion actually is originated from regression toward the mean. We always tend to apply our intuition for devising a trading strategy and that is perfectly all right. Considering if markets have been on roller-coaster ride, moving up/down all the time, the first thing our intuition will guide us, that probably it will continue to move in the same direction. Fundamentally that leads to trend following strategies. On the other hand, there is mean reversion which comes from the argument that, poor performing stocks will perform well in the subsequent periods and vice versa. …

Class distribution in general comprises of either skewed or imbalance classes and somewhat balanced classes. Either of these have their own set of challenges and metrics to evaluate classifier’s performance. The effective classification with skewed which is imbalanced data here, is an important area of research. High class imbalance is naturally inherent in many real-world data set e.g. ** medical diagnosis, fraud detection (credit card, phone calls, insurance), network intrusion detection, pollution detection, fault monitoring, biomedical, bioinformatics and remote sensing (land mine, under water mine)** to name a few, suffer from these phenomena. Likewise balanced class could be prevalent in stock…

Adaptive Boosting (AdaBoost) has popular use as an Ensemble Learning Method in Supervised Machine Learning and was formulated by Yoav Freund and Robert Schapire, 2003. It is a meta-estimator and begins by fitting a classifier on the original dataset and then fits additional copies of the classifier on the same dataset but where the weights of incorrectly classified instances are adjusted such that subsequent classifiers focus more on difficult **cases**. We have seen **earlier** that, creating an effective classification model is a daunting task in the presence of imbalance classes in data set.

We will experiment with Boosting mechanisms to…

Accuracy and F1 score computed on confusion matrices have been among the most popular adopted metrics in binary classification tasks and a lot of businesses are still relying on these when dealing with imbalance data set. Imbalanced distribution of data is a big challenge for standard learning algorithms and statistical measures can dangerously show overoptimistic inflated results. Imbalanced class is persistent in many real world problems, especially when connected with anomaly detection, such as in financial fraud, email fraud detection, medical diagnosis or computer intrusion detections.

Here, we will talk about **Matthews Correlation Coefficient (MCC) and Cohen’s Kappa**; these two…

Exponential Moving Average is a kind of weighted moving average which gives more weightage to recent price data compared to simple moving averages. To simplify, exponential moving average reacts faster to the recent changes in the underlying data. Here, we will use EMAs crossover to time buy/entry and sell/exit points in stock trade. Testing is extremely critical for whatever trading strategy we take; more hypotheses we create during testing stage, better the evaluation and outcome result.

Let us observe the data points we have; we have intraday minute frequency data of S&P E-mini as shown below.

Original data was in…