Cross-Correlation vs Convolution — understanding technical nuances.
There are numerous articles online explaining the difference between Cros-Correlation and Convolution operation. But is there really a difference between the two in the context of Deep Learning - specifically Convolutional Neural Networks? Yes, there is. Hence the two different terms… or is there?🤔🙄
This article limits the use of math so that it’s easier to follow.
What’s the formal difference between Cross-Correlation vs Convolution?
Let’s recap what we mean by Cross-Correlation and Convolution in the case of signal processing.
Let’s take three functions f(x), g(x), and h(x).
For simplicity, let’s take f(x) and g(x) to be step functions but they can be different, and h( x ) is a time-reversed (flipped) version of g(x).
Now,
Cross-Correlation of f( x ) and g( x ) is found by fixing any one of the two functions, let’s say we fix f( x ), and slide the other, g( x ), over f( x ) while multiplying the functions and adding at each shift.
Cross-Correlation of two functions gives a measure of similarity between the functions.
On the other hand,
Convolution of f( x ) and g( x ) is found by fixing any one of the two functions, let’s say we fix f( x ), and flipping the other function, g( x ), about the y-axis and then sliding it over f( x ) while multiplying the functions and adding at each shift.
Convolution of two functions gives the output fuction when one function is transformed by the other.
In conclusion we see, Convolution of f( x ) and g( x ) = Cross-Correlation of f( x ) and g( - x ),
hence,
f( x ) * g( x ) = Cross-Correlation of f( x ) and h( x ),
where h( x ) = g( - x )
NOTE :- “ * “ is the symbol for Convolution.
Q - What’s the difference between Cross-Correlation vs Convolution in the context of Deep Learning?
Ans - None.
Allow me to explain…
In the case of Deep Learning (Convolutional Neural Networks to be precise), weights of the kernel(a small matrix) are learnt during training using backpropagation and are not defined/set explicitly.
If we expect a particular result after a convolution operation using a particular kernel, we can get the same result if we do cross-correlation with the same kernel flipped horizontally and vertically (i.e in both X & Y directions).
For example, let’s say, the model should learn kernel “A” performing actual convolution with the image and through backpropagation.
Instead of kernel “A”, if the model learns a flipped version of kernel “A”, say kernel “B”, we can directly perform the cross-correlation i.e not wasting any computation power to flip the kernel!
That means Cross-Correlation is equivalent to Convolution in case of CNNs, provided the kernels learnt are mirror images of each case in both the directions.
This also demystifies the reason why, one of the most popular Deep Learning frameworks, Tensorflow does not perform a proper/actual convolution — because it doesn’t need to.
So for future reference, is there any difference between Cros-Correlation and Convolution? Let’s just say “Yes, but No.”
