write unix @->@: write: broken pipe

[tmirmota]

Hey Chris, awesome package as it let me successfully connect to my postgres db on Cloud SQL with GAE standard.

It's been running for a few days but just produced this error:

at panic (go/src/runtime/panic.go:489)
at main.checkErr (cloudsql.go:40)
at main.censusTracts (cloudsql.go:244)
at net/http.HandlerFunc.ServeHTTP (server.go:1942)
at net/http.(*ServeMux).ServeHTTP (server.go:2238)
at appengine_internal.executeRequestSafely (api_prod.go:296)
at appengine_internal.(*server).HandleRequest (api_prod.go:232)
at reflect.Value.call (value.go:434)

at main.checkErr (cloudsql.go:40)

func checkErr(err error) {
	if err != nil {
		panic(err)
	}
}

at main.censusTracts (cloudsql.go:244)

rows, err := db.Query(q, queries.Get("swlng"), queries.Get("swlat"), queries.Get("nelng"), queries.Get("nelat"))
checkErr(err)

Is the issue the connection or should I be retying the request?

[dmush]

I have the same. In my case it might be because I use the cheapest Cloud SQL instance type that is known for it's limited availability. Maybe the instance wasn't available for some time and the app didn't reconnect after that.

[andybons]

Got this as an intermittent issue (it fixed itself after about 40 minutes). @tmirmota what instance type were you running when you encountered this? I’m also running the cheapest instance type.

[tmirmota]

@andybons

Instance Type: PostgreSQL 9.6

I ended up just moving it back over to flex... Not sure why yours just started working...

[andybons]

OK this has nothing to do with instance type/class.

It’s using a local unix socket for it’s connection per the error message returned when it tries to send data over it. (For reference this is a *net.OpError).

Here is a stack trace for when it reaches pq’s error handler:

write unix @->@: write: broken pipe
at panic (go/src/runtime/panic.go:491)
at github.com/lib/pq.(*conn).errRecover (error.go:485)
at panic (go/src/runtime/panic.go:491)
at github.com/lib/pq.(*conn).errRecover (error.go:485)
at panic (go/src/runtime/panic.go:491)
at github.com/lib/pq.(*conn).send (conn.go:917)
at github.com/lib/pq.(*conn).simpleQuery (conn.go:662)
at github.com/lib/pq.(*conn).query (conn.go:855)
at github.com/lib/pq.(*conn).QueryContext (conn_go18.go:21)

• pq catches this and marks the connection as bad — https://github.com/lib/pq/blob/master/error.go#L495
• which should cause additional calls to Query to return driver.ErrBadConn — https://github.com/lib/pq/blob/master/conn.go#L844
• which should then cause database/sql to close the connection when it releases it back into the pool — https://golang.org/src/database/sql/sql.go?#L1170

However, it appears (it’s difficult to confirm this definitively) that the connection isn’t closed and continues to be re-used. When I spin up an instance, run a program that uses this driver and reports the number of open connections, then stop the database and reload the page so that it attempts to query it again, it doesn’t appear that the connection is being closed. The following output is from that program:

Open Connections: 1
Error: write unix @->@: write: broken pipe

Any additional debugging tips, @bradfitz?