go range variables

Today I was having a chat with a friend about a thing that is often surprising or puzzling to new go programmers.

wg := sync.WaitGroup{}
s := []string{"hello", "world", "!"}
for _, v := range s {
    wg.Add(1)
    go func() {
        fmt.Println(v)
        wg.Done()
    }()
}
wg.Wait()

The surprising output of this block of code is

!
!
!

A brand new gopher might have expected

hello
world
!

More experienced gophers might have expected these 3 lines, but not necessarily in that order.

We can break down what’s going in this sample code into 3 items:

1. Variable capture/closure semantics
2. Variable re-use in range loops
3. Out of order execution (aka concurrency)

Variable capture/closure semantics

Let’s start with item #1. This may not look related to the sample code above, but bear with me.

v := "hello"
f := func(s string) {
    v = s
}
fmt.Println(v)
f("world")
fmt.Println(v)
v = "!"
fmt.Println(v)

The output of this little block is

hello
world
!

The first print prints the current value of v. Next we use the function f to change the value of v to “world”. Print again. Next is just a plain assignment to v and the final print.

When written this way, it’s a bit more clear that the function f setting the value of v is affecting the variable v that was declared outside the function. This is a closure. The function f has captured the variable v and can change its value. The fact that the setting of the variable by this function is executed later is of no matter. The variable has been captured: it’s still the same variable.

Variable re-use in range loops

In go the magical range keyword exists to work with for loops. (As far as I know, it cannot be used anywhere else.)

Consider this non-idiomatic go:

s := []string{"hello", "world", "!"}
var v string
for _, v = range s {
    fmt.Println(v)
}

While this does work, I’ve never seen any code with a for ... range loop that used = and not :=.

To be explicit, the difference between = and := is that = assigns a value to an existing variable, while := declares a variable and also assigns a value to it.

So, when a new gopher sees this:

s := []string{"hello", "world", "!"}
for _, v := range s {
    fmt.Println(v)
}

the expectation is that v is a newly declared variable for each iteration of the loop. Many would naturally expect that the v variable that holds “hello” is allocated, assigned “hello”, then released, and for the second iteration a new v is allocated, assigned “world”, and released.

The very common assumption is that it’s equivalent to something like this:

for i := range s {
    v := s[i] // a new v is declared each iteration
}

But the reality is that for ... := range ... loops (even with :=)reuse their variables, and do not de/reallocate on each iteration.

That is, even though it uses :=they actually behave like this:

var v string
for _, v = range s {
    ...
}

The iteration variables may be declared by the “range” clause using a form of short variable declaration (:=). In this case their types are set to the types of the respective iteration values and their scope is the block of the "for" statement; they are re-used in each iteration. (https://go.dev/ref/spec#For_statements)

Yes, this is confusing. And, yes, it is an example of go back-pedaling on it’s claims of simplicity and orthogonality. I cannot speak for the designers/creators of go as to why it’s like this, but that’s the way it is.

Out of order execution (aka concurrency)

Finally we come to our last item, and putting it all together, we’ll see why we get the unexpected output of

!
!
!

Here’s the code again:

wg := sync.WaitGroup{}
s := []string{"hello", "world", "!"}
for _, v := range s {
    wg.Add(1)
    go func() {
        fmt.Println(v)
        wg.Done()
    }()
}
wg.Wait()

Looking at this code, we can see that both our previous items are at play here.

1. The variable v is being captured in a function within the loop.
2. Each iteration of the loop is re-using the same variable v.

Layering on top of these two facts, we also have the fact that (officially) we do not know when, or in what order the 3 goroutines that we created will execute. (What experience tells us is that the 3 goroutines will actually execute after the completion of the for loop.)

So, what actually happens here is

1. Iterate thru the loop 3 times.
2. Each time thru the loop, we create a new func that captures the variable v.
3. Each time thru the loop, the value of v is updated.
4. At the end of the loop, 3 independent funcs exist, each with a captured reference to the variable v.
5. At the end of the loop, the value of v is “!”.
6. After the end of the loop, the variable v is out of scope in our mainline code, but still exists, because it was captured by the goroutines.
7. The 3 goroutines actually execute (in some order).
8. Each goroutine prints the current value of v, which is “!”.

I hope that this helps clarify this common point of confusion (or maybe a couple point of confusion) about go and it’s handling of variables in range loops.

P.S.

By the way, here’s how to “fix” the example code:

wg := sync.WaitGroup{}
s := []string{"hello", "world", "!"}
for _, v := range s {
    wg.Add(1)
    go func(v string) { // accept the value as a parameter
        fmt.Println(v)
        wg.Done()
    }(v) // pass the value as an argument
}
wg.Wait()