AI Like I'm Five

Search AI Like I'm Five

By Richie Thomas in Vision6 min read

Paper #5: Dilated Convolutions, a Filter with Gaps

A dilated convolution spaces out the positions a filter reads, so the same few weights cover a wider area. Stack a few with growing gaps, and the area each output sees grows exponentially, while the map never shrinks.

dilated convolutionsconvolutionsreceptive field

Share

Where we are: the last post showed that classification networks shrink their maps so later layers can see wide areas, and that segmentation needs the wide view without the shrinking. This post covers the paper’s tool for that: the dilated convolution.

A filter on a single row

Start with one row of 9 numbers and a filter with 3 weights, all set to 1, so the filter just adds up the values it lands on. The row is 2, 7, 1, 8, 2, 8, 1, 8, 2, and the filter is centered on the middle value, 2.

A regular convolution centered on the middle value reads it and its two neighbors:

A row of 9 numbers: 2, 7, 1, 8, 2, 8, 1, 8, 2. The three middle values, 8, 2, and 8, are highlighted, with arrows down to a filter of 3 boxes, each labeled times 1. Below, the sum 8 times 1 plus 2 times 1 plus 8 times 1 equals 18, labeled the output for the middle position. A caption says a regular convolution reads the middle value and its two neighbors, multiplies each by its weight, and adds them up.

A dilated convolution reads values that are a fixed distance apart instead. That distance is the dilation.

  • Dilation 1 (a regular convolution) reads the 3 values next to each other: 8 + 2 + 8 = 18. It spans 3 positions.
  • Dilation 2 reads every 2nd value: 1 + 2 + 1 = 4. It spans 5 positions.
  • Dilation 4 reads every 4th value: 2 + 2 + 2 = 6. It spans 9 positions, the whole row.
The row 2, 7, 1, 8, 2, 8, 1, 8, 2 drawn three times. At dilation 1, the three middle values 8, 2, 8 are highlighted, spanning 3 positions, summing to 18. At dilation 2, the values 1, 2, 1 at every 2nd position are highlighted, spanning 5 positions, summing to 4. At dilation 4, the values 2, 2, 2 at the ends and middle are highlighted, spanning all 9 positions, summing to 6. A caption says the filter weights are all 1 and only the spacing changes.

The filter has the same 3 weights every time. Only the spacing between the positions it reads changes. No larger filter is built; the convolution just reads its inputs farther apart.

Why the map does not shrink

The filter then slides one position at a time and produces an output at every position, the same as a regular convolution. Near the edges, where some taps would fall off the row, the row is padded. The output row has as many values as the input row.

That is the difference from pooling and striding. A 2 × 2 pooling step turns a 56 × 56 map into a 28 × 28 map; a dilated convolution with dilation 2 turns a 56 × 56 map into another 56 × 56 map, while still reading from a wider area.

In two dimensions

On an image or a feature map, a 3 × 3 filter has 9 weights. At dilation 1 its taps sit next to each other and cover a 3 × 3 area. At dilation 2 they spread out to cover 5 × 5, with a gap between each pair. At dilation 4 they cover 9 × 9. It is still 9 weights each time.

Three 9 by 9 grids. In each, 9 blue dots mark the positions a 3 by 3 filter reads. At dilation 1 the dots are adjacent and cover a 3 by 3 area. At dilation 2 they are spaced 2 apart and cover 5 by 5. At dilation 4 they are spaced 4 apart and cover the whole 9 by 9 grid. Each is labeled 9 weights.

Stacking them

With one layer, the wider view has gaps: dilation 2 skips every other value. Stacking layers fills in the gaps and widens the view much faster. Each layer reads from the layer before it, and each position in that earlier layer already summarizes an area of the image. The area a single output depends on is called its receptive field.

Two panels using the row 2, 7, 1, 8, 2, 8, 1, 8, 2, with filter weights all 1. Left, 1 layer with dilation 2: the output reads 1, 2, and 1, giving 4; the two 8s between them are marked orange as skipped. Right, 2 layers, dilation 1 then 2: layer 1 sums each value with its two neighbors, and layer 2 reads three of those sums spaced 2 apart, 16, 18, and 17, giving 51. Those come from the 7 input values 7, 1, 8, 2, 8, 1, 8, so the two skipped 8s now reach the output. Labeled receptive field: 7 positions, no gaps.

The paper’s Figure 1 stacks three 3 × 3 layers with dilations 1, 2, and 4:

  • Layer 1, dilation 1: each output depends on a 3 × 3 area.
  • Layer 2, dilation 2: each output reads 3 × 3 positions from layer 1, spaced 2 apart, so they span 5 × 5 positions, and each of those already covers 3 × 3. Together: 7 × 7.
  • Layer 3, dilation 4: by the same reasoning, 15 × 15.

The figure below shows one filter in each layer. Dark blue squares are the 9 positions the filter reads; the yellow area is how much of the input its output depends on. Lines show which values are added up to make the next layer’s value.

Three 17 by 17 grids arranged as a descending staircase, one per layer, each showing one filter. Layer 1, dilation 1: 9 adjacent dark blue squares, the input values one filter reads, labeled covers 3 by 3, 9 input values to 1 layer-1 value. Layer 2, dilation 2: 9 dark squares spaced 2 apart, layer-1 values, on a yellow 7 by 7 area of the input, labeled covers 7 by 7. Layer 3, dilation 4: 9 dark squares spaced 4 apart on a yellow 15 by 15 area, labeled covers 15 by 15. Lines run from each panel's 9 dark squares down to the middle dark square of the next panel, and from layer 3's squares to a final layer-3 output square. A legend says dark blue is the 9 positions one filter reads and yellow is the area of the input its output depends on.

Doubling the dilation at each layer roughly doubles the receptive field each time. In the paper’s words, “the receptive field grows exponentially while the number of parameters grows linearly”: every layer adds the same 9 weights per filter, but the area it covers keeps doubling. Three regular 3 × 3 layers, with the same number of weights, would cover only 7 × 7, against 15 × 15 for the dilated stack.

The context module in a later post uses dilations 1, 1, 2, 4, 8, 16, which gives receptive fields of 3 × 3, 5 × 5, 9 × 9, 17 × 17, 33 × 33, and 65 × 65 after each layer. Those are the numbers in the 1st post.

An old idea under a new name

The operation itself was not new. It appears in the “algorithme à trous,” French for “algorithm with holes,” a method for analyzing signals with wavelets from 1987. DeepLab, a segmentation network from 2015, also used it. The authors chose the name “dilated convolution” and built a network that, in their words, “systematically uses dilated convolutions for multi-scale context aggregation,” meaning combining information from both small and large areas around each position.

What’s next

The next post covers how the authors used this: first to stop VGG-16 from shrinking its last maps, and then to build a small “context module” that stacks dilated layers to see wide areas at full resolution.

Keep reading