Big data and data analysis

1 Big data

Syllabus

This chapter serves the whole of BA1 C2: (a) the principal business applications of big data and analytics, (b) the relationship between data variables — cross-sectional and time-series analysis, the correlation coefficient, the coefficient of determination, and BOTH Spearman’s rank and Pearson’s coefficients, (c) trends and patterns, seasonal factors and predicted values under both the additive and the multiplicative models, (d) preparing a trend equation by graph or by regression, and (e) the limitations of forecasting models.

BIG DATA means an extremely large collection of data that can be analysed by computer to reveal patterns, trends and associations — especially relating to human behaviour and interactions.

Two separate processes are implied, and it is worth keeping them apart:

the COLLECTION of the data and its storage; and
the ANALYSIS of the data, to turn it into useful information.

The first is an engineering problem. The second is what this chapter, and the syllabus, are actually about.

There are two recordings for this chapter. The first covers sections 1, 2, 3, 5 and 6; the second covers Spearman’s rank correlation coefficient in section 7 and then works its own time-series example. Two things to know.

Neither recording reaches section 4 (cross-sectional and time-series data) or section 9 (the limitations of forecasting models). Both are examinable and both are written here.
The second recording works a DIFFERENT time-series example from section 8 — different data, and it does the additive model only. Work section 8 from these notes; use the recording for the method.

YouTube video

2 The characteristics of big data

In February 2001 Doug Laney, then an analyst at the META Group, described big data as having three characteristics, known as the 3Vs. (META Group was acquired by Gartner in 2005, which is why the note is often attributed to Gartner.) A fourth V is commonly added.

Characteristic	What it means
VOLUME	A very large amount of data — more than can be handled by a single computer, spreadsheet or conventional database.
VARIETY	Disparate, non-uniform data of different sizes, shapes and sources, arriving irregularly, some internal and some external, some structured but much of it unstructured.
VELOCITY	Data arrives continually and often has to be processed very quickly for the result to be of any use.
VERACITY (the fourth V, added later)	Is the data true? Can its accuracy be relied on? Volume is no substitute for quality.

2.1 Volume

The volume of big data held by companies such as Walmart, Apple and eBay is measured in multiple PETABYTES. A petabyte is 10¹⁵ bytes of information. A typical personal computer disc holds around 10⁹ bytes — a gigabyte — so these depositories hold at least what a million PCs could hold, and perhaps ten or twenty million.

Those numbers mean little even converted into PCs. It is more instructive to list what large organisations actually store:

RETAILERS. Through loyalty cards swiped at checkouts: every purchase you make, when, where, how you paid, which coupons you used. Through their websites: every product you have looked at, every page you have visited, everything you have bought.
SOCIAL MEDIA. Friends and contacts, postings, your location when you posted, photographs that can be scanned for identification, and anything else you choose to reveal.
MOBILE PHONE COMPANIES. Numbers called, texts sent (which can be scanned automatically for key words), every location the phone has been while switched on, browsing habits, voicemails.
INTERNET AND BROWSER PROVIDERS. Every site and page visited, downloads, emails, and the search terms entered.
BANKING SYSTEMS. Every receipt and payment, card transaction amounts, dates, retailers and locations, and the ATMs used.

2.2 Variety, and structured against unstructured data

The variety is visible in the list above: browsing activity, financial transactions, interests, buying habits, responses to advertising, geographical information, social and business contacts, text, numbers, photographs, voicemails, and technical data such as jet-engine vibration and temperature readings.

	Structured data	Unstructured data
What it is	Data stored in defined fields — numeric, text, date — often of defined length, within a defined record in a file of similar records	Data with no pre-defined data model. It comes in all shapes and sizes
How it is designed	A DATA MODEL defines the types and formats to be recorded and how they will be stored, processed and accessed — and in doing so LIMITS what can be collected and what can be done with it	There is no model to design, which is why it is so easy to collect and so hard to use
Example	A bank statement: date, amount, receipt or payment, a short description	Word processor documents, spreadsheets, presentations, audio, video, social media interactions, map data
Getting at it	Straightforward, using well-established database query languages	Difficult: it has to be stored in a way that will let it be searched and analysed at all

A frequently quoted estimate is that about 80% of business data is unstructured.

2.3 Velocity

Information has to arrive quickly enough to be of use in a decision. There is little point in a supermarket obtaining price-comparison information and texting a customer an offer after they have left the store. If facial recognition is to be used to welcome hotel guests by name, it has to be more or less instant.

Volume and variety conspire against velocity: methods have to be found to process huge quantities of non-uniform, awkward data in real time. That tension is the whole of the engineering problem.

3 Big data analytics and what it is used for

The processing of big data is generally called BIG DATA ANALYTICS.

3.1 The techniques

DATA MINING: analysing data to identify patterns and relationships — ASSOCIATIONS, where several events occur together; SEQUENCES, where one event leads to another; and CORRELATIONS, where two measures move together.
PREDICTIVE ANALYTICS: a form of data mining aimed at predicting future events — for example, which customers can be persuaded to upgrade a flight.
TEXT ANALYTICS: scanning text such as emails and documents to extract useful information, which may be no more than key words indicating an interest in a product or a place.
VOICE ANALYTICS: the same applied to audio.
STATISTICAL ANALYTICS: identifying trends, correlations and changes in behaviour — which is what the rest of this chapter teaches.

The syllabus then names four things this is used FOR, and the next four subsections take them in turn.

3.2 Identifying customer value

Not all customers are equally valuable, and revenue alone does not say which are. Analytics combines what a customer spends with what they COST to serve — discounts taken, returns made, deliveries, calls to the helpdesk, credit taken — and with how long they are likely to stay, to give a customer LIFETIME VALUE.

A retailer might find that a segment producing 15% of revenue produces 40% of returns and half the helpdesk contacts, and is therefore worth less than a smaller-spending segment that costs almost nothing to serve. The decisions that follow are practical ones: where to spend the marketing budget, which customers to keep on a discount and which to let go, and what a new customer is worth paying to acquire.

3.3 Understanding customer behaviour

This is about what customers DO rather than what they are worth. Basket analysis finds associations — which products are bought together, so that they can be placed, priced or promoted together. Sequence analysis finds what leads to what: which browsing path ends in a purchase, and which ends in an abandoned basket. Churn analysis finds the behaviours that precede a customer leaving, early enough to do something about it.

The volume matters here. A pattern that shows up in a thousand transactions may be noise; measured across fifty million, the estimate is far more precise. But size is not the same as validity. A huge data set can still be unrepresentative, can be biased in what it records, and will throw up relationships that arise purely by chance — so a pattern still has to be checked, and explained, before it is treated as a fact about the business.

3.4 Understanding cost behaviour

Management accounting has always split costs into fixed and variable elements, usually from a handful of monthly totals. Analytics does it from transaction-level data: every job, machine, delivery and shift, rather than twelve monthly figures.

The practical gains are better cost drivers — because the driver can be tested against the data rather than assumed — a properly identified stepped fixed cost, and cost information at the level of the individual product, customer or route rather than the department. The regression technique in section 5 of this chapter is the simplest version of exactly this analysis.

3.5 Assisting logistics decisions

Logistics is where big data has had its most visible effect, because the data is generated continuously and the decisions are made continuously. Vehicle telematics and traffic data give routing and scheduling; point-of-sale and weather data give demand forecasts store by store, and therefore stock levels; sensor data on machinery gives predictive maintenance, so that a vehicle or a machine is serviced before it fails rather than after; and tracking across the supply chain gives early warning of a delay while there is still time to react.

The common thread across all four applications is the same one: the data was always being generated, and what has changed is the ability to store it and analyse it fast enough for the answer still to be useful.

The findings can lead to better marketing, better customer service and relationship management, increased customer loyalty, greater competitive strength, higher operational efficiency, and the discovery of new sources of revenue.

4 Cross-sectional and time-series data

	Time-series (longitudinal) study	Cross-sectional study
What is measured	The SAME variable at different points in TIME	Data from a number of members of a POPULATION at a single point in time
Example	Monthly sales for four years; profit over a number of years	The gross profit percentage achieved by twelve branches of a company in one year
What it can show	How something has changed, and what it might do next	How members differ from one another, and what is associated with the difference
Technique used	Time series analysis (section 8)	Regression and correlation (sections 5 to 7)

Other studies measure TWO variables and ask how one moves with the other — cost and production volume, or advertising expenditure and sales. For simple patterns, linear regression and the coefficient of correlation describe the relationship. For more complex patterns, and particularly for data recorded over time, time series analysis is used instead.

5 Linear regression

Linear regression is a method of fitting the best straight line through a set of points — the arithmetic version of the trend line drawn by eye through the scatter diagram in chapter 7 section 3.3.

In business the line would typically connect cost and volume, selling price and sales volume, or hours worked and units produced.

5.1 The regression equation

Linear regression produces constants that fit a line of the form:

y = a + bx

Symbol	What it is	In a cost example
y	the DEPENDENT variable — the one being predicted	total cost
x	the INDEPENDENT variable — the one doing the explaining	units made
a	the INTERCEPT: the value of y when x is zero	the fixed cost, incurred even if nothing is made
b	the SLOPE: the change in y for each one-unit change in x	the variable cost per unit
n	the number of pairs of observations	the number of months of data

Textbooks write the equation in two ways: y = ax + b, with a as the SLOPE, and y = a + bx, with a as the INTERCEPT. Both conventions exist, but they cannot both be used in the same working. THIS CHAPTER, AND THE ANSWERS, USE y = a + bx WITH a AS THE INTERCEPT. If a question gives you the other convention, read the definitions it supplies rather than assuming.

Cost depends on output, not the other way round, so cost is always y and output is always x. Getting them the wrong way round produces a line that predicts output from cost, which is not what anybody wants.

5.2 The formulae

The values of a and b are calculated from five totals: n, Σx, Σy, Σxy and Σx². (The symbol Σ means “the sum of”.) Both formulae are given to you in the examination.

	Formula
Slope	b = (nΣxy − Σx Σy) ÷ (nΣx² − (Σx)²)
Intercept	a = (Σy − bΣx) ÷ n, which is the same as a = ȳ − bx̄

Work out b FIRST. The formula for a uses b, so there is no way round it, and the commonest way to lose the marks is to start with a.

5.3 A worked example

The following table shows the number of units produced each month and the total cost incurred.

	Units	$000
January	100	40
February	400	65
March	200	45
April	700	80
May	600	70
June	500	70
July	300	50

Required

Calculate the regression line y = a + bx, taking x as output in HUNDREDS of units and y as total cost in $000, and use it to forecast the cost of a month in which 475 units are made.

Scale first: x is output in hundreds of units, y is cost in $000.

x	y	xy	x²
1	40	40	1
4	65	260	16
2	45	90	4
7	80	560	49
6	70	420	36
5	70	350	25
3	50	150	9
Σ 28	Σ 420	Σ 1,870	Σ 140

n = 7.

b = (7 × 1,870 − 28 × 420) ÷ (7 × 140 − 28²) = (13,090 − 11,760) ÷ (980 − 784) = 1,330 ÷ 196 = 6.7857.

a = (420 − 6.7857 × 28) ÷ 7 = (420 − 190.00) ÷ 7 = 230.00 ÷ 7 = 32.857.

So y = 32.857 + 6.7857x, where y is in $000 and x is in hundreds of units.

UNSCALING. $32,857 is the fixed cost per month, because that is what is incurred when x = 0 — that is, when nothing is produced. Each HUNDRED units adds $6,785.70 of cost, so the variable cost is $67.86 per unit.

THE FORECAST. 475 units is x = 4.75. y = 32.857 + (6.7857 × 4.75) = 32.857 + 32.232 = 65.089, that is about $65,090.

6 Correlation and the coefficient of determination

Regression will fit the best straight line it can through ANY set of points. If you numbered the days of the year 1 to 365, noted the day each person was born and the amount of money in their bank account, regression would produce a line. There would obviously be no relationship. Something else has to tell you whether the line means anything, and that is CORRELATION.

6.1 Pearson’s correlation coefficient

PEARSON’S correlation coefficient, r, measures how linear the relationship between two variables is. It is used when the ACTUAL VALUES of both variables are known — output and cost, advertising and sales.

r = (nΣxy − Σx Σy) ÷ √[(nΣx² − (Σx)²) × (nΣy² − (Σy)²)]

Value of r	What it means
r = +1	Perfect POSITIVE correlation. Every point lies on the line, and as one variable increases so does the other.
r = −1	Perfect NEGATIVE correlation. Every point lies on the line, and as one variable increases the other decreases.
r = 0	No correlation at all.
Between	The further from +1 or −1, the weaker the linear relationship.

6.2 The coefficient of determination

The COEFFICIENT OF DETERMINATION is r², and it is the more useful of the two, because it has a direct meaning: it is the proportion of the variation in one variable that is EXPLAINED by variation in the other.

If r = 0.7, then r² = 0.49, so about 49% of the change in one variable is explained by the change in the other — and, just as importantly, 51% is not.

Required

Using the data in Example 1, calculate Pearson’s correlation coefficient and the coefficient of determination, and interpret them.

Two further totals are needed: Σy² = 40² + 65² + 45² + 80² + 70² + 70² + 50² = 1,600 + 4,225 + 2,025 + 6,400 + 4,900 + 4,900 + 2,500 = 26,550.

Numerator = nΣxy − ΣxΣy = (7 × 1,870) − (28 × 420) = 13,090 − 11,760 = 1,330.

nΣx² − (Σx)² = (7 × 140) − 784 = 980 − 784 = 196.

nΣy² − (Σy)² = (7 × 26,550) − 420² = 185,850 − 176,400 = 9,450.

r = 1,330 ÷ √(196 × 9,450) = 1,330 ÷ √1,852,200 = 1,330 ÷ 1,361.0 = 0.977, say 0.98.

r² = 0.98² = 0.96.

INTERPRETATION. The correlation is strongly positive: cost rises with output, and the points lie very close to a straight line. About 96% of the variation in cost is explained by the variation in output, which is what you would hope for in a cost relationship — the remaining 4% is everything else, including measurement error and one-off items.

CORRELATION IS NOT CAUSATION. A high coefficient shows that two variables move together. It does not show that one causes the other, and it never shows which way round the causation runs. Both may be moving under the influence of a third variable — a general improvement in the economy, or inflation — and with enough series to choose from, some strong correlations arise purely by chance.

7 Spearman’s rank correlation coefficient

SPEARMAN’S RANK correlation coefficient measures the same idea — how well two sets of observations line up — but it is used when the data is RANKED rather than measured. It is the right tool when there are no actual values to work with, only positions in an order.

	Pearson’s correlation coefficient	Spearman’s rank correlation coefficient
Use it when	The actual values of both variables are known	The observations are RANKED — placed in order
Typical data	Output and cost; advertising and sales	Positions in two examinations; two judges’ orderings; two guests’ ratings
What you need	n, Σx, Σy, Σxy, Σx², Σy²	n and Σd², where d is the difference between the two ranks for each item
Range and reading	−1 to +1, read the same way for both

Do not call the coefficient in section 6 “Spearman’s”. A coefficient computed from the ACTUAL VALUES of both variables is PEARSON’S. Spearman’s is computed from RANKS. They are two different measures for two different kinds of data, and the table above is the distinction.

YouTube video

The formula is:

r = 1 − (6 Σd²) ÷ (n(n² − 1))

where n is the number of pairs of data and d is the difference between the two rankings for each item. It is interpreted exactly as Pearson’s r is: +1 is a perfect match, −1 a perfect reversal, and 0 no relationship.

The positions of seven students in their examinations in accountancy and economics were as follows.

Student	Accountancy position	Economics position
A	1	3
B	4	7
C	2	1
D	5	6
E	3	2
F	7	4
G	6	5

Required

Judge whether the students’ positions in accountancy correlate with their positions in economics.

Student	Accountancy	Economics	d	d²
A	1	3	−2	4
B	4	7	−3	9
C	2	1	+1	1
D	5	6	−1	1
E	3	2	+1	1
F	7	4	+3	9
G	6	5	+1	1
			Σd = 0	Σd² = 26

Σd should always come to zero, which is a useful check that no difference has been taken the wrong way round.

n = 7, so n(n² − 1) = 7 × (49 − 1) = 7 × 48 = 336.

r = 1 − (6 × 26) ÷ 336 = 1 − 156 ÷ 336 = 1 − 0.4643 = 0.536.

INTERPRETATION. The correlation is positive — as performance in one paper rises, so does performance in the other — but at 0.536 it is not particularly strong. Squaring it, only about 29% of the variation in one ranking is associated with the other.

8 Time series analysis

Linear regression assumes the relationship between two variables is a straight line. Time series analysis is more adaptable, and recognises that a series of observations recorded over time can contain four separate effects.

8.1 The four components

Component	What it is	Examinable?
TREND	The underlying smooth increase or decrease as time passes	Yes — sections 8.2 and 8.4
SEASONAL VARIATION	Cycles that repeat in LESS than a year: the four quarters, or the days of the week	Yes — section 8.3
CYCLICAL VARIATION	Cycles that repeat in MORE than a year — typically the long-term trade cycle	Known about, not calculated
RANDOM EFFECTS	Non-repeating, unpredictable variation	Known about, not calculated

Time series analysis separates out the first two. C2(c) also asks for GRAPHICAL analysis, and plotting the series is always the first step: it shows at a glance whether there is a trend, whether there is a seasonal pattern, and how regular that pattern is. Plotted, the seventeen quarters of Example 4 below look like this — a rising trend, a dip in the third quarter of most years and a high second quarter.

The vertical axis measures sales ABOVE $950,000, so that the seasonal pattern is visible; on a full scale from zero the whole series would be a nearly flat line across the top of the chart. The dashed trend line is the one calculated in Answer 4, and the whole of section 8.2 is about how to get from the jagged line to the smooth one.

8.2 Finding the trend: moving averages

The trend is found by MOVING AVERAGES, a smoothing technique that averages the seasonal pattern away. With quarterly data the average is taken over four quarters, so that every quarter of the year is represented exactly once and the highs and lows cancel out.

A four-quarter average falls BETWEEN two quarters rather than on one, so it has to be CENTRED: each pair of consecutive moving averages is averaged again, and the result is the trend value for the quarter that sits between them. This is why the first two and the last two quarters of a series never have a trend value. (Centring is only necessary where the cycle has an EVEN number of periods; with a five-day week the five-part average already falls on a day.)

A company’s quarterly sales, in $000, from the first quarter of 2012 to the first quarter of 2016 were as follows.

Year	Quarter	Time	Sales $000
2012	1	1	989.0
	2	2	990.0
	3	3	994.0
	4	4	1,015.0
2013	1	5	1,030.0
	2	6	1,042.5
	3	7	1,036.0
	4	8	1,056.5
2014	1	9	1,071.0
	2	10	1,083.5
	3	11	1,079.5
	4	12	1,099.5
2015	1	13	1,115.5
	2	14	1,127.5
	3	15	1,123.5
	4	16	1,135.0
2016	1	17	1,140.0

Required

Find the trend using a four-quarter moving average; calculate the average seasonal variation for each quarter using BOTH the additive and the multiplicative models; and use the results to forecast sales for the third quarter of 2016.

Step 1 — the four-quarter moving average, the centred trend, and the two measures of seasonal variation. Each moving average is shown against the earlier of the two quarters it falls between.

Year	Qtr	Time	Sales $000	4-quarter moving average	Trend (centred)
2012	1	1	989.0
	2	2	990.0	997.000
	3	3	994.0	1,007.250	1,002.125
	4	4	1,015.0	1,020.375	1,013.813
2013	1	5	1,030.0	1,030.875	1,025.625
	2	6	1,042.5	1,041.250	1,036.063
	3	7	1,036.0	1,051.500	1,046.375
	4	8	1,056.5	1,061.750	1,056.625
2014	1	9	1,071.0	1,072.625	1,067.188
	2	10	1,083.5	1,083.375	1,078.000
	3	11	1,079.5	1,094.500	1,088.938
	4	12	1,099.5	1,105.500	1,100.000
2015	1	13	1,115.5	1,116.500	1,111.000
	2	14	1,127.5	1,125.375	1,120.938
	3	15	1,123.5	1,131.500	1,128.438
	4	16	1,135.0
2016	1	17	1,140.0

How the columns are built. The first moving average is (989.0 + 990.0 + 994.0 + 1,015.0) ÷ 4 = 997.000, and it falls between quarters 2 and 3 of 2012. Moving down one quarter, the second is (990.0 + 994.0 + 1,015.0 + 1,030.0) ÷ 4 = 1,007.250, between quarters 3 and 4. Centring the two gives (997.000 + 1,007.250) ÷ 2 = 1,002.125, which IS the trend for the third quarter of 2012.

The additive seasonal variation is then the actual less the trend: 994.0 − 1,002.125 = −8.125. The multiplicative variation is the actual divided by the trend: 994.0 ÷ 1,002.125 = 0.9919, so that quarter ran at about 99.2% of its trend. Seventeen quarters give thirteen trend values: the first two and the last two quarters have none.

Year	Qtr	Sales $000	Trend (centred)	Additive: sales − trend	Multiplicative: sales ÷ trend
2012	3	994.0	1,002.125	−8.125	0.9919
	4	1,015.0	1,013.813	+1.187	1.0012
2013	1	1,030.0	1,025.625	+4.375	1.0043
	2	1,042.5	1,036.063	+6.437	1.0062
	3	1,036.0	1,046.375	−10.375	0.9901
	4	1,056.5	1,056.625	−0.125	0.9999
2014	1	1,071.0	1,067.188	+3.812	1.0036
	2	1,083.5	1,078.000	+5.500	1.0051
	3	1,079.5	1,088.938	−9.438	0.9913
	4	1,099.5	1,100.000	−0.500	0.9995
2015	1	1,115.5	1,111.000	+4.500	1.0041
	2	1,127.5	1,120.938	+6.562	1.0059
	3	1,123.5	1,128.438	−4.938	0.9956

Step 2 — average the variations for each quarter, and adjust. The third quarter has four readings and the others three.

Additive	Q1	Q2	Q3	Q4	Total
2012			−8.125	+1.187
2013	+4.375	+6.437	−10.375	−0.125
2014	+3.812	+5.500	−9.438	−0.500
2015	+4.500	+6.562	−4.938
Average	+4.229	+6.166	−8.219	+0.187	+2.363
Adjustment	−0.591	−0.591	−0.591	−0.591	−2.364
Adjusted	+3.638	+5.575	−8.810	−0.404	−0.001

Multiplicative	Q1	Q2	Q3	Q4	Total
2012			0.9919	1.0012
2013	1.0043	1.0062	0.9901	0.9999
2014	1.0036	1.0051	0.9913	0.9995
2015	1.0041	1.0059	0.9956
Average	1.0040	1.0057	0.9922	1.0002	4.0021
Adjustment	−0.0005	−0.0005	−0.0005	−0.0005	−0.0020
Adjusted	1.0035	1.0052	0.9917	0.9997	4.0001

Why the adjustment is made. Under the ADDITIVE model the seasonal variations are amounts either side of the trend, so over a full year they must sum to ZERO. Here the averages sum to +2.363, so a quarter of that, 2.363 ÷ 4 = 0.591, is deducted from each quarter. The adjusted figures total −0.001, which is rounding. The excess is large here mainly because the 2015 third-quarter dip (−4.938) is much shallower than in earlier years — a sign that the seasonal pattern itself may be changing, which is one of the limitations in section 9.

Under the MULTIPLICATIVE model the variations are proportions of the trend, so for quarterly data the four factors must sum to 4 (that is, 400%; for monthly data, 12). Here they sum to 4.0021, so 0.0021 ÷ 4 = 0.0005 is deducted from each; the adjusted factors total 4.0001, again rounding.

Step 3 — forecast the third quarter of 2016.

First project the TREND. It rose from 1,002.125 in the third quarter of 2012 to 1,128.438 in the third quarter of 2015 — an increase of 126.313 over 12 quarters, so 126.313 ÷ 12 = 10.526 per quarter. The third quarter of 2016 is four quarters after the third quarter of 2015, so the projected trend is 1,128.438 + (4 × 10.526) = 1,128.438 + 42.104 = 1,170.542.

Then apply the third-quarter seasonal adjustment.

Model	Working	Forecast ($000)
Additive	1,170.542 − 8.810	1,161.732
Multiplicative	1,170.542 × 0.9917	1,160.827

So sales for the third quarter of 2016 are forecast at about $1,161,000 on either model. The two answers differ by under $1,000 here because the trend has not moved far from the level at which the seasonal variations were measured; the further ahead the forecast, the wider that gap becomes — which is the reason for preferring the multiplicative model where the trend is rising strongly.

8.3 The two models

	Additive model	Multiplicative model
The variation is	an AMOUNT added to or deducted from the trend	a PERCENTAGE of the trend
Forecast =	trend + seasonal variation	trend × seasonal index
The variations must sum to	ZERO over a full cycle	400% for quarterly data, 1,200% for monthly
Use it when	the size of the seasonal swing is roughly constant however large the trend gets	the seasonal swing grows in proportion to the trend — which is usually the more realistic assumption when the trend is rising strongly

The reasoning behind the second row of that table is worth stating. If a quarter runs 3% above the trend when sales are 90,000, it is more natural to expect it to run 3% above the trend when sales are 180,000 than to expect it to run the same 2,700 units above.

8.4 Forecasting from the model

Neither model is any use until it produces a number. Forecasting from a time series is always the same two steps:

PROJECT THE TREND to the period required. The trend has been smoothed, so it can be extended by eye from a graph, by regression on the trend values, or — most simply — by taking the average movement per period, as Answer 4 does.
ADJUST IT BY THE SEASONAL VARIATION for that period: ADD the adjusted seasonal variation under the additive model, or MULTIPLY by the adjusted seasonal index under the multiplicative model.

Count the periods carefully. Going from the third quarter of 2015 to the third quarter of 2016 is FOUR quarters: Q4, Q1, Q2, then Q3. Miscounting the increments is the commonest way marks are lost in this calculation, and it is worth writing the intervening periods out.

9 The limitations of forecasting models

The syllabus asks you to identify the limitations of forecasting models, and it is a question that is regularly asked. Both of the techniques in this chapter share most of them.

CORRELATION IS NOT CAUSATION. Two series may move together without one driving the other. Both may be responding to a third factor, and with enough series to choose from some strong correlations arise purely by chance.
TOO FEW OBSERVATIONS. The fewer the pairs, the less reliable anything built on them. Two points are always perfectly correlated and prove nothing at all.
EXTRAPOLATION BEYOND THE OBSERVED RANGE. A relationship estimated between 100 and 700 units a month says nothing about 5,000 units a month. Costs may step, capacity may run out, discounts may change. There is no evidence in the data about a region the data does not cover.
FORECASTING TOO FAR AHEAD compounds the error. In a time series it also assumes the seasonal pattern itself is stable, which is a stronger assumption than it looks.
THE PAST IS NOT NECESSARILY A GUIDE TO THE FUTURE. The data was generated under conditions that may already have changed — a different product mix, a different supplier, a new competitor, a change in the law.
THE DATA MAY BE POOR. It may contain one-off items, cover a period of unusual trading, or have been recorded inconsistently. Anomalies should be identified and removed BEFORE any technique is applied, not explained away afterwards.
INFLATION DISTORTS ANY MONEY SERIES. Where prices have moved materially, the figures should be adjusted to a common price level before a relationship is estimated, or the model will read inflation as a real trend.
A LOW r² MEANS A POOR PREDICTOR. If little of the variation in one variable is explained by the other, any prediction from the line is weak however neatly the line is drawn.

None of this makes forecasting worthless. A forecast built on historic evidence is far better than a guess. But it is an INPUT to a decision and not the decision itself, and it should always be used alongside management’s own knowledge of the business.

10 Tests

The chapter’s own five questions follow. Work each one before you open the answer.

What are the 4Vs of big data?

VOLUME, VARIETY, VELOCITY and VERACITY.

The first three are Doug Laney’s original 3Vs, published at the META Group in 2001. Veracity — is the data actually true? — was added later, and it is the one that matters most in practice: volume is no substitute for quality.

A sales director has carried out a linear regression exercise to examine the connection between sales revenue and advertising. The coefficient of correlation is 0.96.

Does this show that more advertising causes sales to increase?

No.

Even a very high coefficient of correlation does not prove cause and effect. The result is CONSISTENT with a causal relationship but does not establish one. The increase in advertising might have coincided with an improvement in the economy, and sales might have risen because of that. It is also possible that the causation runs the other way: firms often set the advertising budget as a percentage of sales, in which case higher sales cause higher advertising.

A comparison of the ratings given by two guests who each visited four hotels shows the following rankings.

	Guest 1 ranking	Guest 2 ranking
Hotel 1	2	3
Hotel 2	4	4
Hotel 3	3	1
Hotel 4	1	2

What is Spearman’s rank correlation coefficient?

	Guest 1	Guest 2	d	d²
Hotel 1	2	3	−1	1
Hotel 2	4	4	0	0
Hotel 3	3	1	+2	4
Hotel 4	1	2	−1	1
			Σd = 0	Σd² = 6

n = 4, so n(n² − 1) = 4 × (16 − 1) = 4 × 15 = 60.

r = 1 − (6 × 6) ÷ 60 = 1 − 36 ÷ 60 = 1 − 0.6 = 0.4.

A weak positive correlation: the two guests broadly agree, but not closely.

The seasonal variations for the sales of a product are:

Season 1	−$20,000
Season 2	+$4,000
Season 3	+$25,000
Season 4	−$9,000

The trend has been calculated as $2,000 per season, and the trend figure for season 3 of 20X7 is $73,000.

What is the forecast figure, including the seasonal adjustment, for season 4 of 20X8?

Count the seasons first. From season 3 of 20X7 to season 4 of 20X8 is FIVE increments: 20X7 season 4, then 20X8 seasons 1, 2, 3 and 4.

Projected trend = $73,000 + (5 × $2,000) = $83,000.

The seasonal variation for season 4 is −$9,000, so the seasonally adjusted forecast = $83,000 − $9,000 = $74,000.

Which of the following elements of a time series are analysed using the moving averages technique?

A Cyclical variations

B Seasonal variations

C The trend

D Random variations

B and C.

Moving averages smooth the seasonal pattern away in order to isolate THE TREND, and the SEASONAL VARIATIONS are then obtained by comparing each actual figure back against that trend. Cyclical variations repeat over more than a year and are not extracted by a four-quarter average, and random variations are by definition not predictable.

11 Test your knowledge

Two quick checks before you move on: work through the flashcards to fix this chapter’s key terms and definitions, then sit the objective questions for exam-style practice. Both mark themselves and explain the answers as you go.

Answer the questions one at a time. Your progress is saved so you can leave and come back.

Open chapter practice