rpc: guard grpc status stream send w/ mutex

gRPC has a fun quirk where if you call SendMsg on the same stream in
parallel from multiple goroutines, you can easily hit a deadlock in some
internal grpc writeQuota thing:
grpc/grpc-go#5393 (comment)

Was hitting some random deadlocks in dagger and saw from SIGQUIT stacks
that we were blocked on that writeQuota call. Wrapping the WriteStatus
implementation in a mutex fixed it.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Author: sipsma

rpc.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"io"
 	"net"
+	"sync"
 
 	grpc "google.golang.org/grpc"
 	"google.golang.org/grpc/credentials/insecure"
@@ -47,6 +48,7 @@ func DialRPC(ctx context.Context, target string) (Writer, error) {
 type RPCWriter struct {
 	Conn    *grpc.ClientConn
 	Updates ProgressService_WriteUpdatesClient
+	l       sync.Mutex
 }
 
 // NewRPCWriter returns a new RPCWriter.
@@ -59,6 +61,8 @@ func NewRPCWriter(conn *grpc.ClientConn, updates ProgressService_WriteUpdatesCli
 
 // WriteStatus implements Writer.
 func (w *RPCWriter) WriteStatus(status *StatusUpdate) error {
+	w.l.Lock()
+	defer w.l.Unlock()
 	return w.Updates.Send(status)
 }