API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jul 19th, 2024
43
0
21 days
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 94.14 KB | None | 0 0
raw download clone embed print report
  1. 2024-07-19T18:37:02+03:00 INFO Stop/Shutdown request received.
  2. 2024-07-19T18:37:02+03:00 INFO lazyfilewalker.trash-cleanup-filewalker subprocess exited with status {"satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "status": 1, "error": "exit status 1"}
  3. 2024-07-19T18:37:02+03:00 ERROR pieces:trash emptying trash failed {"error": "pieces error: lazyfilewalker: exit status 1", "errorVerbose": "pieces error: lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:422\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
  4. 2024-07-19T18:37:17+03:00 WARN services service takes long to shutdown {"name": "retain"}
  5. 2024-07-19T18:37:17+03:00 WARN servers service takes long to shutdown {"name": "server"}
  6. 2024-07-19T18:37:17+03:00 INFO servers slow shutdown {"stack": "goroutine 1077 [running]:\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x78\nsync.(*Once).doSlow(0xc004910270?, 0xc004c2ca40?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0007d9ee8?)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x3c\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x1ad3230?, 0xc000918990?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2b2\nruntime/pprof.Do({0x1ad3498?, 0xc000878140?}, {{0xc000860140?, 0xc0007d9fd0?, 0xf65f65?}}, 0xc00087e200)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 35\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x4cb\n\ngoroutine 1 [syscall, locked to thread]:\nsyscall.SyscallN(0x7ff93c1a7e20?, {0xc0000b9c40?, 0x3?, 0x1751aa7?})\n\t/usr/local/go/src/runtime/syscall_windows.go:544 +0x107\nsyscall.Syscall(0x14c3220?, 0x60?, 0xc?, 0xc0000b9ca0?, 0xda5279?)\n\t/usr/local/go/src/runtime/syscall_windows.go:482 +0x35\ngolang.org/x/sys/windows.StartServiceCtrlDispatcher(0x1751aa7?)\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/zsyscall_windows.go:1353 +0x4f\ngolang.org/x/sys/windows/svc.Run({0x1751aa7?, 0xb?}, {0x1ac5da0?, 0xc000008960?})\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:295 +0x11e\nmain.startAsService(0xc0000f2c08)\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:62 +0x347\nmain.main()\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:26 +0x225\n\ngoroutine 17 [select, locked to thread]:\ngolang.org/x/sys/windows/svc.serviceMain(0x1, 0x25b8788c638)\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:247 +0x3cb\n\ngoroutine 19 [semacquire]:\nsync.runtime_Semacquire(0xc0001100b8?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x85a857?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc0003b8000)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nmain.(*service).Execute.func2()\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:110 +0x17\nmain.(*service).Execute(0xc000008960, {0x0?, 0x0?, 0x0?}, 0xc000110000, 0xc000110060)\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:142 +0x553\ngolang.org/x/sys/windows/svc.serviceMain.func2()\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:235 +0x3f\ncreated by golang.org/x/sys/windows/svc.serviceMain in goroutine 17\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:234 +0x27e\n\ngoroutine 35 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc0008781e0?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0xc00087a100?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc00087e100)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/storagenode.(*Peer).Run(0xc00025eb08, {0x1ad3268, 0xc000a9a0a0})\n\t/go/src/storj.io/storj/storagenode/peer.go:967 +0x41b\nmain.cmdRun(0xc0000f3b08, 0xc000294488)\n\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:123 +0xdc5\nmain.newRunCmd.func1(0x15ba360?, {0xc0002505a0?, 0xc0000f3b08?, 0xc0002ea690?})\n\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33 +0x17\nstorj.io/common/process.cleanup.func1.4({0x1ad3498, 0xc000125400})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:393 +0x14a\nstorj.io/common/process.cleanup.func1(0xc0000f3b08, {0xc00010a580, 0x0, 0x2})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:411 +0x1c78\ngithub.com/spf13/cobra.(*Command).execute(0xc0000f3b08, {0xc00010a560, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaca\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc0000f2c08)\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039\nstorj.io/common/process.ExecWithCustomOptions(0xc0000f2c08, {0x1, 0x1, 0x1, 0x0, 0x19b4d98, 0x0})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:112 +0x1c9\nstorj.io/common/process.ExecWithCustomConfigAndLogger(...)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:77\nstorj.io/common/process.ExecWithCustomConfig(...)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:72\nstorj.io/common/process.Exec(0x0?)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:62 +0x5a\nmain.(*service).Execute.func1()\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107 +0x57\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 19\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 8 [select]:\nstorj.io/monkit-jaeger.(*ThriftCollector).Run(0xc00015c360, {0x1ad3268, 0xc00015a140})\n\t/go/pkg/mod/storj.io/[email protected]/thrift.go:174 +0x2dc\nstorj.io/common/process.cleanup.func1.2()\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:351 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 69 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc000008d80?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x166f060?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000254b40)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/common/debug.(*Server).Run(0xc000024fc8, {0x1ad31f8, 0x2381200})\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:203 +0x30f\nstorj.io/common/process.initDebug.func1()\n\t/go/pkg/mod/storj.io/[email protected]/process/debug.go:40 +0x112\ncreated by storj.io/common/process.initDebug in goroutine 35\n\t/go/pkg/mod/storj.io/[email protected]/process/debug.go:37 +0xde\n\ngoroutine 70 [syscall, 26040 minutes]:\nos/signal.signal_recv()\n\t/usr/local/go/src/runtime/sigqueue.go:152 +0x29\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x13\ncreated by os/signal.Notify.func1.1 in goroutine 35\n\t/usr/local/go/src/os/signal/signal.go:151 +0x1f\n\ngoroutine 71 [chan receive, 26040 minutes]:\nstorj.io/common/process.Ctx.func1()\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:139 +0x39\ncreated by storj.io/common/process.Ctx in goroutine 35\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:138 +0x2c7\n\ngoroutine 9 [chan receive, 26040 minutes]:\nstorj.io/common/debug.(*Server).Run.func3()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:182 +0x2c\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 10 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000262310, {0x1ad3268?, 0xc00015a190?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/common/debug.(*Server).Run.func4()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:186 +0x31\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 11 [select, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*listener).Accept(0xc000254b00)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/listener.go:37 +0x7e\nnet/http.(*Server).Serve(0xc000025008, {0x1acd750, 0xc000254b00})\n\t/usr/local/go/src/net/http/server.go:3255 +0x33e\nstorj.io/common/debug.(*Server).Run.func5()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:195 +0x50\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 36 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000262310, {0x1ad3268?, 0xc000090050?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 10\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 37 [IO wait, 26040 minutes]:\ninternal/poll.runtime_pollWait(0x25bcd045760, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0x870316?, 0x23827a0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001ae020, 0xc0002fdd20)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001ae008, 0x2a4, {0xc000220000?, 0xc0001f5ea0?, 0x167b9e0?}, 0xc00026e000?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001ae008, 0xc0002fdf00)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001ae008)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc0002660a0)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).Accept(0xc0002660a0)\n\t/usr/local/go/src/net/tcpsock.go:327 +0x30\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000262310)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 10\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 1039 [sleep]:\ntime.Sleep(0x8bb2c97000)\n\t/usr/local/go/src/runtime/time.go:195 +0x11a\ngithub.com/spacemonkeygo/monkit/v3.(*ticker).run(0x22f8040)\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/[email protected]/meter.go:203 +0x26\ncreated by github.com/spacemonkeygo/monkit/v3.(*ticker).register in goroutine 1038\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/[email protected]/meter.go:195 +0x89\n\ngoroutine 254 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0007176c0, {0x1ad3268, 0xc000526b90})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1095 [semacquire]:\nruntime.Stack({0xc005f22000, 0x100000, 0x100000}, 0x1)\n\t/usr/local/go/src/runtime/mprof.go:1429 +0x4f\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x78\nsync.(*Once).doSlow(0xc0013021a0?, 0xc000255b80?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0003e3ee8?)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x3c\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x1ad3230?, 0xc000a98660?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2b2\nruntime/pprof.Do({0x1ad3498?, 0xc0008781e0?}, {{0xc0008602a0?, 0xc000249930?, 0x1ad3268?}}, 0xc00087e9c0)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 35\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x4cb\n\ngoroutine 29 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716b60, {0x1ad3268, 0xc00015a460})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1064 [runnable]:\nstorj.io/storj/storagenode/retain.(*Service).next(0xc000346000)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:303 +0xf8\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:248 +0x165\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1065 [sync.Mutex.Lock, 7354 minutes]:\nsync.runtime_SemacquireMutex(0x175576c?, 0xe?, 0x7370089063810510?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0x0?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:273 +0x397\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 43317203 [sync.Mutex.Lock, 2067 minutes]:\nsync.runtime_SemacquireMutex(0xc004763750?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0xaf, 0x2c, 0x42, 0x0, 0x3e, 0xfc, 0x82, 0x6a, 0xb4, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0xaf, 0x2c, 0x42, 0x0, 0x3e, 0xfc, 0x82, 0x6a, 0xb4, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Retain(0xc000348000, {0x1ad3498, 0xc0002f8640}, 0xc000989a40)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:995 +0x478\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func5({0x16f76a0?, 0xc000348000?}, {0x1ad3498?, 0xc00202e500?}, {0x16d55e0?, 0xc000989a40?}, {0xc0008c1a00?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:328 +0xcb\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc005eb4388?, {0x1ad4e60, 0xc0008c1a00}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c19c0}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00391ed88}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc005656b40?, 0xc00391ed88, {0xc004aca3c0?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc000af6388?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002c26380?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 1066 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1067 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 33579483 [sync.Mutex.Lock, 6446 minutes]:\nsync.runtime_SemacquireMutex(0xc0039ff5c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc00051c360})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc0006f0c80?, 0x20?}, {0x1644a80?, 0xc0008c02c0}, {0xc0008c02c0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc0008c02c0}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c0280}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00321c908}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0039421e0?, 0xc00321c908, {0xc0006f0c80?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026ee008?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002548aa0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 242 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716d00, {0x1ad3268, 0xc0005269b0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 244 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716ea0, {0x1ad3268, 0xc000526a00})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 245 [select, 16 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000716ea0, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 246 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717040, {0x1ad3268, 0xc000526a50})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 248 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0007171e0, {0x1ad3268, 0xc000526aa0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 250 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717380, {0x1ad3268, 0xc000526af0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 252 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717520, {0x1ad3268, 0xc000526b40})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 46685679 [select, 27 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc0007176c0, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46685763\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 256 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717860, {0x1ad3268, 0xc000526be0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 258 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717a00, {0x1ad3268, 0xc000526c30})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 259 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717a00, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 291 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717ba0, {0x1ad3268, 0xc000526550})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 313 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717d40, {0x1ad3268, 0xc000526f00})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 330 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000249c70, {0x1ad3268, 0xc000527400})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1078 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc00058c960?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x166d020?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000990400)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/private/server.(*Server).Run(0xc0001aea08, {0x1ad3230, 0xc000984f00})\n\t/go/src/storj.io/storj/private/server/server.go:483 +0x123c\nstorj.io/storj/storagenode.New.func1({0x1ad3230, 0xc000984f00})\n\t/go/src/storj.io/storj/storagenode/peer.go:418 +0x1a5\nstorj.io/storj/private/lifecycle.(*Group).Run.func2.1({0x1ad3230?, 0xc000984f00?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:87 +0x2d\nruntime/pprof.Do({0x1ad3498?, 0xc000878140?}, {{0xc0009824a0?, 0x8616b6?, 0x0?}}, 0xc000449f28)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\nstorj.io/storj/private/lifecycle.(*Group).Run.func2()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:86 +0x287\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 387 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000249e10, {0x1ad3268, 0xc0006f80f0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1096 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc00029cbd0?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x164d040?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000346080)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/storagenode/retain.(*Service).Run(0xc000346000, {0x1ad3230, 0xc0009190e0})\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:282 +0x52f\nstorj.io/storj/private/lifecycle.(*Group).Run.func2.1({0x1ad3230?, 0xc0009190e0?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:87 +0x2d\nruntime/pprof.Do({0x1ad3498?, 0xc0008781e0?}, {{0xc0008c04a0?, 0x25bcd316748?, 0xc00017f500?}}, 0xc0005e3f28)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\nstorj.io/storj/private/lifecycle.(*Group).Run.func2()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:86 +0x287\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 434 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00013a680, {0x1ad3268, 0xc0006f8fa0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 441 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00013ad00, {0x1ad3268, 0xc0006f9180})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1062 [sync.Mutex.Lock]:\nsync.runtime_SemacquireMutex(0xc000907140?, 0x20?, 0xc000907140?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc0007bdf20?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Run.func1()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:223 +0x103\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1063 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1053 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000988af0, {0x1ad3268?, 0xc000598780?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/storj/private/server.(*Server).Run.func1()\n\t/go/src/storj.io/storj/private/server/server.go:389 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1055 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000988b60, {0x1ad3268?, 0xc000598780?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/storj/private/server.(*Server).Run.func2()\n\t/go/src/storj.io/storj/private/server/server.go:399 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1057 [semacquire]:\nsync.runtime_Semacquire(0x1ac5f60?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x1018002?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\nstorj.io/drpc/drpcctx.(*Tracker).Wait(...)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:43\nstorj.io/drpc/drpcserver.(*Server).Serve(0xc0002f8500, {0x1ad3268, 0xc0005987d0}, {0x1acd780, 0xc00058c810})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:133 +0x3f1\nstorj.io/storj/private/server.(*Server).Run.(*Server).Run.func4.func9()\n\t/go/src/storj.io/storj/private/server/server.go:422 +0x4f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1179 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000988af0, {0x1ad3268?, 0xc0005988c0?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1053\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 1180 [IO wait]:\ninternal/poll.runtime_pollWait(0x25bcd045570, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0xc000101408?, 0x0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001aef20, 0xc00082fcb0)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001aef08, 0x564, {0xc00396ea50?, 0x930f38?, 0x1c?}, 0xc000101008?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001aef08, 0xc00082fe90)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001aef08)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc000266740)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).AcceptTCP(0xc000266740)\n\t/usr/local/go/src/net/tcpsock.go:314 +0x30\nstorj.io/storj/private/server.(*tcpUserTimeoutListener).Accept(0xc0001320b0?)\n\t/go/src/storj.io/storj/private/server/listener.go:46 +0x1a\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000988af0)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1053\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 1204 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000988b60, {0x1ad3268?, 0xc0002a8500?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1055\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 1205 [IO wait, 26040 minutes]:\ninternal/poll.runtime_pollWait(0x25bcd045380, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0x870316?, 0x23827a0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001af420, 0xc000837cb0)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001af408, 0x6e0, {0xc0007f6000?, 0x0?, 0x0?}, 0x0?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001af408, 0xc000837e90)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001af408)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc0002667a0)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).AcceptTCP(0xc0002667a0)\n\t/usr/local/go/src/net/tcpsock.go:314 +0x30\nstorj.io/storj/private/server.(*tcpUserTimeoutListener).Accept(0x0?)\n\t/go/src/storj.io/storj/private/server/listener.go:46 +0x1a\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000988b60)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1055\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 46648660 [select, 7 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717520, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46648645\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 41062104 [sync.Mutex.Lock, 3427 minutes]:\nsync.runtime_SemacquireMutex(0xc0017a35c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc0028f04b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc00393ac40?, 0x20?}, {0x1644a80?, 0xc000a87ba0}, {0xc000a87ba0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc000a87ba0}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc000a87b60}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc003f62b48}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0041470e0?, 0xc003f62b48, {0xc00393ac40?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026efc08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc003664c30?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46391544 [sync.Mutex.Lock, 438 minutes]:\nsync.runtime_SemacquireMutex(0xc002247750?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x84, 0xa7, 0x4c, 0x2c, 0xd4, 0x3c, 0x5b, 0xa7, 0x65, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x84, 0xa7, 0x4c, 0x2c, 0xd4, 0x3c, 0x5b, 0xa7, 0x65, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Retain(0xc000348000, {0x1ad3498, 0xc004278280}, 0xc004164f50)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:995 +0x478\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func5({0x16f76a0?, 0xc000348000?}, {0x1ad3498?, 0xc00366bb80?}, {0x16d55e0?, 0xc004164f50?}, {0xc004288a40?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:328 +0xcb\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc005eb4388?, {0x1ad4e60, 0xc004288a40}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc004288a00}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00454fd48}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00411ef00?, 0xc00454fd48, {0xc0042f8720?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc002c59c08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc004c7e460?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46199905 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717380, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1092\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 28494251 [sync.Mutex.Lock, 9100 minutes]:\nsync.runtime_SemacquireMutex(0xc0000535c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc003980ec0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc000abc340?, 0x20?}, {0x1644a80?, 0xc0008c0de0}, {0xc0008c0de0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc0008c0de0}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c0da0}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc005071448}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0003481e0?, 0xc005071448, {0xc000abc340?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc003372008?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc003c3f8e0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46910192 [select, 27 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000249c70, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46910128\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 45604747 [sync.Mutex.Lock, 852 minutes]:\nsync.runtime_SemacquireMutex(0xc0025f95c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc0047b7490})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc005886020?, 0x20?}, {0x1644a80?, 0xc00377c380}, {0xc00377c380?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc00377c380}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc00377c340}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc003f63b08}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc004ef1c20?, 0xc003f63b08, {0xc005886020?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026eee08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002a5e4f0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 17564186 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000716d00, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1086\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 46803848 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc0007171e0, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1110\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 45728165 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717040, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1110\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 47101054 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717d40, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 47100956\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 47101022 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717ba0, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 47100947\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n"}
  7. 2024-07-19T18:37:17+03:00 INFO services slow shutdown {"stack": "goroutine 1095 [running]:\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x78\nsync.(*Once).doSlow(0xc0013021a0?, 0xc000255b80?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0003e3ee8?)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x3c\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x1ad3230?, 0xc000a98660?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2b2\nruntime/pprof.Do({0x1ad3498?, 0xc0008781e0?}, {{0xc0008602a0?, 0xc000249930?, 0x1ad3268?}}, 0xc00087e9c0)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 35\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x4cb\n\ngoroutine 1 [syscall, locked to thread]:\nsyscall.SyscallN(0x7ff93c1a7e20?, {0xc0000b9c40?, 0x3?, 0x1751aa7?})\n\t/usr/local/go/src/runtime/syscall_windows.go:544 +0x107\nsyscall.Syscall(0x14c3220?, 0x60?, 0xc?, 0xc0000b9ca0?, 0xda5279?)\n\t/usr/local/go/src/runtime/syscall_windows.go:482 +0x35\ngolang.org/x/sys/windows.StartServiceCtrlDispatcher(0x1751aa7?)\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/zsyscall_windows.go:1353 +0x4f\ngolang.org/x/sys/windows/svc.Run({0x1751aa7?, 0xb?}, {0x1ac5da0?, 0xc000008960?})\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:295 +0x11e\nmain.startAsService(0xc0000f2c08)\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:62 +0x347\nmain.main()\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:26 +0x225\n\ngoroutine 17 [select, locked to thread]:\ngolang.org/x/sys/windows/svc.serviceMain(0x1, 0x25b8788c638)\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:247 +0x3cb\n\ngoroutine 19 [semacquire]:\nsync.runtime_Semacquire(0xc0001100b8?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x85a857?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc0003b8000)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nmain.(*service).Execute.func2()\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:110 +0x17\nmain.(*service).Execute(0xc000008960, {0x0?, 0x0?, 0x0?}, 0xc000110000, 0xc000110060)\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:142 +0x553\ngolang.org/x/sys/windows/svc.serviceMain.func2()\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:235 +0x3f\ncreated by golang.org/x/sys/windows/svc.serviceMain in goroutine 17\n\t/go/pkg/mod/golang.org/x/[email protected]/windows/svc/service.go:234 +0x27e\n\ngoroutine 35 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc0008781e0?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0xc00087a100?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc00087e100)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/storagenode.(*Peer).Run(0xc00025eb08, {0x1ad3268, 0xc000a9a0a0})\n\t/go/src/storj.io/storj/storagenode/peer.go:967 +0x41b\nmain.cmdRun(0xc0000f3b08, 0xc000294488)\n\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:123 +0xdc5\nmain.newRunCmd.func1(0x15ba360?, {0xc0002505a0?, 0xc0000f3b08?, 0xc0002ea690?})\n\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33 +0x17\nstorj.io/common/process.cleanup.func1.4({0x1ad3498, 0xc000125400})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:393 +0x14a\nstorj.io/common/process.cleanup.func1(0xc0000f3b08, {0xc00010a580, 0x0, 0x2})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:411 +0x1c78\ngithub.com/spf13/cobra.(*Command).execute(0xc0000f3b08, {0xc00010a560, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaca\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc0000f2c08)\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039\nstorj.io/common/process.ExecWithCustomOptions(0xc0000f2c08, {0x1, 0x1, 0x1, 0x0, 0x19b4d98, 0x0})\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:112 +0x1c9\nstorj.io/common/process.ExecWithCustomConfigAndLogger(...)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:77\nstorj.io/common/process.ExecWithCustomConfig(...)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:72\nstorj.io/common/process.Exec(0x0?)\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:62 +0x5a\nmain.(*service).Execute.func1()\n\t/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107 +0x57\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 19\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 8 [select]:\nstorj.io/monkit-jaeger.(*ThriftCollector).Run(0xc00015c360, {0x1ad3268, 0xc00015a140})\n\t/go/pkg/mod/storj.io/[email protected]/thrift.go:174 +0x2dc\nstorj.io/common/process.cleanup.func1.2()\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:351 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 69 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc000008d80?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x166f060?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000254b40)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/common/debug.(*Server).Run(0xc000024fc8, {0x1ad31f8, 0x2381200})\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:203 +0x30f\nstorj.io/common/process.initDebug.func1()\n\t/go/pkg/mod/storj.io/[email protected]/process/debug.go:40 +0x112\ncreated by storj.io/common/process.initDebug in goroutine 35\n\t/go/pkg/mod/storj.io/[email protected]/process/debug.go:37 +0xde\n\ngoroutine 70 [syscall, 26040 minutes]:\nos/signal.signal_recv()\n\t/usr/local/go/src/runtime/sigqueue.go:152 +0x29\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x13\ncreated by os/signal.Notify.func1.1 in goroutine 35\n\t/usr/local/go/src/os/signal/signal.go:151 +0x1f\n\ngoroutine 71 [chan receive, 26040 minutes]:\nstorj.io/common/process.Ctx.func1()\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:139 +0x39\ncreated by storj.io/common/process.Ctx in goroutine 35\n\t/go/pkg/mod/storj.io/[email protected]/process/exec_conf.go:138 +0x2c7\n\ngoroutine 9 [chan receive, 26040 minutes]:\nstorj.io/common/debug.(*Server).Run.func3()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:182 +0x2c\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 10 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000262310, {0x1ad3268?, 0xc00015a190?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/common/debug.(*Server).Run.func4()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:186 +0x31\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 11 [select, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*listener).Accept(0xc000254b00)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/listener.go:37 +0x7e\nnet/http.(*Server).Serve(0xc000025008, {0x1acd750, 0xc000254b00})\n\t/usr/local/go/src/net/http/server.go:3255 +0x33e\nstorj.io/common/debug.(*Server).Run.func5()\n\t/go/pkg/mod/storj.io/[email protected]/debug/server.go:195 +0x50\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 69\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 36 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000262310, {0x1ad3268?, 0xc000090050?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 10\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 37 [IO wait, 26040 minutes]:\ninternal/poll.runtime_pollWait(0x25bcd045760, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0x870316?, 0x23827a0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001ae020, 0xc0002fdd20)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001ae008, 0x2a4, {0xc000220000?, 0xc0001f5ea0?, 0x167b9e0?}, 0xc00026e000?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001ae008, 0xc0002fdf00)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001ae008)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc0002660a0)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).Accept(0xc0002660a0)\n\t/usr/local/go/src/net/tcpsock.go:327 +0x30\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000262310)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 10\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 1039 [sleep]:\ntime.Sleep(0x8bb2c97000)\n\t/usr/local/go/src/runtime/time.go:195 +0x11a\ngithub.com/spacemonkeygo/monkit/v3.(*ticker).run(0x22f8040)\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/[email protected]/meter.go:203 +0x26\ncreated by github.com/spacemonkeygo/monkit/v3.(*ticker).register in goroutine 1038\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/[email protected]/meter.go:195 +0x89\n\ngoroutine 254 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0007176c0, {0x1ad3268, 0xc000526b90})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 29 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716b60, {0x1ad3268, 0xc00015a460})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1064 [runnable]:\nstorj.io/storj/storagenode/retain.(*RequestStore).Next(0xc000116c78?)\n\t/go/src/storj.io/storj/storagenode/retain/store.go:217 +0x6a\nstorj.io/storj/storagenode/retain.(*Service).next(0xc000346000)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:297 +0x8d\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:248 +0x165\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1065 [sync.Mutex.Lock, 7354 minutes]:\nsync.runtime_SemacquireMutex(0x175576c?, 0xe?, 0x7370089063810510?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0x0?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:273 +0x397\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 43317203 [sync.Mutex.Lock, 2067 minutes]:\nsync.runtime_SemacquireMutex(0xc004763750?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0xaf, 0x2c, 0x42, 0x0, 0x3e, 0xfc, 0x82, 0x6a, 0xb4, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0xaf, 0x2c, 0x42, 0x0, 0x3e, 0xfc, 0x82, 0x6a, 0xb4, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Retain(0xc000348000, {0x1ad3498, 0xc0002f8640}, 0xc000989a40)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:995 +0x478\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func5({0x16f76a0?, 0xc000348000?}, {0x1ad3498?, 0xc00202e500?}, {0x16d55e0?, 0xc000989a40?}, {0xc0008c1a00?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:328 +0xcb\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc005eb4388?, {0x1ad4e60, 0xc0008c1a00}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c19c0}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00391ed88}, {0xc004aca3c0, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc005656b40?, 0xc00391ed88, {0xc004aca3c0?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc000af6388?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002c26380?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 1066 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1067 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1077 [runnable]:\ngo.uber.org/zap/buffer.(*Buffer).AppendString(0xc003b3f060, {0xc0037f92e4?, 0x30?})\n\t/go/pkg/mod/go.uber.org/[email protected]/buffer/buffer.go:51 +0xdf\ngo.uber.org/zap/zapcore.safeAppendStringLike[...](0x19b5f58?, 0x19b5fe0, 0xc003b3f060, {0xc0037f4000, 0xb193})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/json_encoder.go:555 +0xb1\ngo.uber.org/zap/zapcore.(*jsonEncoder).safeAddString(...)\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/json_encoder.go:489\ngo.uber.org/zap/zapcore.(*jsonEncoder).AppendString(0xc004c2c7c0, {0xc0037f4000, 0xb193})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/json_encoder.go:295 +0x219\ngo.uber.org/zap/zapcore.(*jsonEncoder).AddString(0xc004c2c7c0, {0x174a8f5?, 0xc0007d9808?}, {0xc0037f4000, 0xb193})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/json_encoder.go:197 +0x3b\ngo.uber.org/zap/zapcore.Field.AddTo({{0x174a8f5, 0x5}, 0xf, 0x0, {0xc0037f4000, 0xb193}, {0x0, 0x0}}, {0x1ae0dc0, 0xc004c2c7c0})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/field.go:149 +0x415\ngo.uber.org/zap/zapcore.addFields(...)\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/field.go:210\ngo.uber.org/zap/zapcore.consoleEncoder.writeContext({0x1ac6040?}, 0xc00379f2e0, {0xc004c2ca80, 0x1, 0x1?})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/console_encoder.go:141 +0x166\ngo.uber.org/zap/zapcore.consoleEncoder.EncodeEntry({0xf8b12f?}, {0x0, {0xc19ebfeb75f09abc, 0x58d0433e4efd5, 0x22f8a80}, {0x174d257, 0x7}, {0x175465c, 0xd}, {0x0, ...}, ...}, ...)\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/console_encoder.go:119 +0x731\ngo.uber.org/zap/zapcore.(*ioCore).Write(0xc00011a5d0, {0x0, {0xc19ebfeb75f09abc, 0x58d0433e4efd5, 0x22f8a80}, {0x174d257, 0x7}, {0x175465c, 0xd}, {0x0, ...}, ...}, ...)\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/core.go:95 +0x7b\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc004910270, {0xc004c2ca80, 0x1, 0x1})\n\t/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:253 +0x11c\ngo.uber.org/zap.(*Logger).Info(0xc006280000?, {0x175465c?, 0xb193?}, {0xc004c2ca80, 0x1, 0x1})\n\t/go/pkg/mod/go.uber.org/[email protected]/logger.go:247 +0x4e\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:114 +0x15f\nsync.(*Once).doSlow(0xc004910270?, 0xc004c2ca40?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0007d9ee8?)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x3c\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x1ad3230?, 0xc000918990?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2b2\nruntime/pprof.Do({0x1ad3498?, 0xc000878140?}, {{0xc000860140?, 0xc0007d9fd0?, 0xf65f65?}}, 0xc00087e200)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 35\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x4cb\n\ngoroutine 33579483 [sync.Mutex.Lock, 6446 minutes]:\nsync.runtime_SemacquireMutex(0xc0039ff5c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc00051c360})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc0006f0c80?, 0x20?}, {0x1644a80?, 0xc0008c02c0}, {0xc0008c02c0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc0008c02c0}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c0280}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00321c908}, {0xc0006f0c80, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0039421e0?, 0xc00321c908, {0xc0006f0c80?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026ee008?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002548aa0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 242 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716d00, {0x1ad3268, 0xc0005269b0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 244 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000716ea0, {0x1ad3268, 0xc000526a00})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 245 [select, 16 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000716ea0, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 246 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717040, {0x1ad3268, 0xc000526a50})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 248 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0007171e0, {0x1ad3268, 0xc000526aa0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 250 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717380, {0x1ad3268, 0xc000526af0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 252 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717520, {0x1ad3268, 0xc000526b40})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 46685679 [select, 27 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc0007176c0, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46685763\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 256 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717860, {0x1ad3268, 0xc000526be0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 258 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717a00, {0x1ad3268, 0xc000526c30})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 259 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717a00, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 291 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717ba0, {0x1ad3268, 0xc000526550})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 313 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000717d40, {0x1ad3268, 0xc000526f00})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 330 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000249c70, {0x1ad3268, 0xc000527400})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1078 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc00058c960?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x166d020?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000990400)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/private/server.(*Server).Run(0xc0001aea08, {0x1ad3230, 0xc000984f00})\n\t/go/src/storj.io/storj/private/server/server.go:483 +0x123c\nstorj.io/storj/storagenode.New.func1({0x1ad3230, 0xc000984f00})\n\t/go/src/storj.io/storj/storagenode/peer.go:418 +0x1a5\nstorj.io/storj/private/lifecycle.(*Group).Run.func2.1({0x1ad3230?, 0xc000984f00?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:87 +0x2d\nruntime/pprof.Do({0x1ad3498?, 0xc000878140?}, {{0xc0009824a0?, 0x8616b6?, 0x0?}}, 0xc000449f28)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\nstorj.io/storj/private/lifecycle.(*Group).Run.func2()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:86 +0x287\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 387 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000249e10, {0x1ad3268, 0xc0006f80f0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1096 [semacquire, 26040 minutes]:\nsync.runtime_Semacquire(0xc00029cbd0?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x164d040?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc000346080)\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25\nstorj.io/storj/storagenode/retain.(*Service).Run(0xc000346000, {0x1ad3230, 0xc0009190e0})\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:282 +0x52f\nstorj.io/storj/private/lifecycle.(*Group).Run.func2.1({0x1ad3230?, 0xc0009190e0?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:87 +0x2d\nruntime/pprof.Do({0x1ad3498?, 0xc0008781e0?}, {{0xc0008c04a0?, 0x25bcd316748?, 0xc00017f500?}}, 0xc0005e3f28)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\nstorj.io/storj/private/lifecycle.(*Group).Run.func2()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:86 +0x287\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 35\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 434 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00013a680, {0x1ad3268, 0xc0006f8fa0})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 441 [select, 26040 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00013ad00, {0x1ad3268, 0xc0006f9180})\n\t/usr/local/go/src/database/sql/sql.go:1246 +0x87\ncreated by database/sql.OpenDB in goroutine 35\n\t/usr/local/go/src/database/sql/sql.go:824 +0x14c\n\ngoroutine 1062 [sync.Mutex.Lock]:\nsync.runtime_SemacquireMutex(0xc000907140?, 0x20?, 0xc000907140?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc0007bdf20?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Run.func1()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:223 +0x103\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1063 [sync.Mutex.Lock, 11579 minutes]:\nsync.runtime_SemacquireMutex(0xc00006ef08?, 0x8?, 0x8c57c7?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc000346040?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nsync.(*Cond).Wait(0x0?)\n\t/usr/local/go/src/sync/cond.go:71 +0x97\nstorj.io/storj/storagenode/retain.(*Service).Run.func2()\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:252 +0x3ed\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1096\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1053 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000988af0, {0x1ad3268?, 0xc000598780?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/storj/private/server.(*Server).Run.func1()\n\t/go/src/storj.io/storj/private/server/server.go:389 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1055 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).Run(0xc000988b60, {0x1ad3268?, 0xc000598780?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:90 +0x11a\nstorj.io/storj/private/server.(*Server).Run.func2()\n\t/go/src/storj.io/storj/private/server/server.go:399 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1057 [semacquire]:\nsync.runtime_Semacquire(0x1ac5f60?)\n\t/usr/local/go/src/runtime/sema.go:62 +0x25\nsync.(*WaitGroup).Wait(0x1018002?)\n\t/usr/local/go/src/sync/waitgroup.go:116 +0x48\nstorj.io/drpc/drpcctx.(*Tracker).Wait(...)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:43\nstorj.io/drpc/drpcserver.(*Server).Serve(0xc0002f8500, {0x1ad3268, 0xc0005987d0}, {0x1acd780, 0xc00058c810})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:133 +0x3f1\nstorj.io/storj/private/server.(*Server).Run.(*Server).Run.func4.func9()\n\t/go/src/storj.io/storj/private/server/server.go:422 +0x4f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1078\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96\n\ngoroutine 1179 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000988af0, {0x1ad3268?, 0xc0005988c0?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1053\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 1180 [IO wait]:\ninternal/poll.runtime_pollWait(0x25bcd045570, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0xc000101408?, 0x0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001aef20, 0xc00082fcb0)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001aef08, 0x564, {0xc00396ea50?, 0x930f38?, 0x1c?}, 0xc000101008?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001aef08, 0xc00082fe90)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001aef08)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc000266740)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).AcceptTCP(0xc000266740)\n\t/usr/local/go/src/net/tcpsock.go:314 +0x30\nstorj.io/storj/private/server.(*tcpUserTimeoutListener).Accept(0xc0001320b0?)\n\t/go/src/storj.io/storj/private/server/listener.go:46 +0x1a\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000988af0)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1053\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 1204 [chan receive, 26040 minutes]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorContext(0xc000988b60, {0x1ad3268?, 0xc0002a8500?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:106 +0x2d\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1055\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:87 +0xc5\n\ngoroutine 1205 [IO wait, 26040 minutes]:\ninternal/poll.runtime_pollWait(0x25bcd045380, 0x72)\n\t/usr/local/go/src/runtime/netpoll.go:345 +0x85\ninternal/poll.(*pollDesc).wait(0x870316?, 0x23827a0?, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27\ninternal/poll.execIO(0xc0001af420, 0xc000837cb0)\n\t/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe6\ninternal/poll.(*FD).acceptOne(0xc0001af408, 0x6e0, {0xc0007f6000?, 0x0?, 0x0?}, 0x0?)\n\t/usr/local/go/src/internal/poll/fd_windows.go:944 +0x67\ninternal/poll.(*FD).Accept(0xc0001af408, 0xc000837e90)\n\t/usr/local/go/src/internal/poll/fd_windows.go:978 +0x1bc\nnet.(*netFD).accept(0xc0001af408)\n\t/usr/local/go/src/net/fd_windows.go:178 +0x54\nnet.(*TCPListener).accept(0xc0002667a0)\n\t/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e\nnet.(*TCPListener).AcceptTCP(0xc0002667a0)\n\t/usr/local/go/src/net/tcpsock.go:314 +0x30\nstorj.io/storj/private/server.(*tcpUserTimeoutListener).Accept(0x0?)\n\t/go/src/storj.io/storj/private/server/listener.go:46 +0x1a\nstorj.io/drpc/drpcmigrate.(*ListenMux).monitorBase(0xc000988b60)\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:115 +0x44\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).Run in goroutine 1055\n\t/go/pkg/mod/storj.io/[email protected]/drpcmigrate/mux.go:88 +0x107\n\ngoroutine 46648660 [select, 7 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717520, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46648645\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 41062104 [sync.Mutex.Lock, 3427 minutes]:\nsync.runtime_SemacquireMutex(0xc0017a35c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc0028f04b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc00393ac40?, 0x20?}, {0x1644a80?, 0xc000a87ba0}, {0xc000a87ba0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc000a87ba0}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc000a87b60}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc003f62b48}, {0xc00393ac40, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0041470e0?, 0xc003f62b48, {0xc00393ac40?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026efc08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc003664c30?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46391544 [sync.Mutex.Lock, 438 minutes]:\nsync.runtime_SemacquireMutex(0xc002247750?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x84, 0xa7, 0x4c, 0x2c, 0xd4, 0x3c, 0x5b, 0xa7, 0x65, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x84, 0xa7, 0x4c, 0x2c, 0xd4, 0x3c, 0x5b, 0xa7, 0x65, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Retain(0xc000348000, {0x1ad3498, 0xc004278280}, 0xc004164f50)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:995 +0x478\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func5({0x16f76a0?, 0xc000348000?}, {0x1ad3498?, 0xc00366bb80?}, {0x16d55e0?, 0xc004164f50?}, {0xc004288a40?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:328 +0xcb\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc005eb4388?, {0x1ad4e60, 0xc004288a40}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc004288a00}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc00454fd48}, {0xc0042f8720, 0x1d})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00411ef00?, 0xc00454fd48, {0xc0042f8720?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc002c59c08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc004c7e460?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46199905 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717380, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1092\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 28494251 [sync.Mutex.Lock, 9100 minutes]:\nsync.runtime_SemacquireMutex(0xc0000535c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc003980ec0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc000abc340?, 0x20?}, {0x1644a80?, 0xc0008c0de0}, {0xc0008c0de0?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc0008c0de0}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc0008c0da0}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc005071448}, {0xc000abc340, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0003481e0?, 0xc005071448, {0xc000abc340?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc003372008?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc003c3f8e0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 46910192 [select, 27 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc000249c70, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 46910128\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 45604747 [sync.Mutex.Lock, 852 minutes]:\nsync.runtime_SemacquireMutex(0xc0025f95c8?, 0x38?, 0x14?)\n\t/usr/local/go/src/runtime/sema.go:77 +0x25\nsync.(*Mutex).lockSlow(0xc0002e6f58)\n\t/usr/local/go/src/sync/mutex.go:171 +0x15d\nsync.(*Mutex).Lock(0xc5656b?)\n\t/usr/local/go/src/sync/mutex.go:90 +0x32\nstorj.io/storj/storagenode/retain.(*Service).Queue(0xc000346000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/retain/retain.go:163 +0x52\nstorj.io/storj/storagenode/piecestore.(*Endpoint).processRetainReq(0xc000348000, {0x7b, 0x2d, 0xe9, 0xd7, 0x2c, 0x2e, 0x93, 0x5f, 0x19, ...}, ...)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1009 +0x165\nstorj.io/storj/storagenode/piecestore.(*Endpoint).RetainBig(0xc000348000, {0x1ad7df0, 0xc0047b7490})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:1045 +0x5d2\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func6({0x16f76a0?, 0xc000348000}, {0xc005886020?, 0x20?}, {0x1644a80?, 0xc00377c380}, {0xc00377c380?, 0x1ac0748?})\n\t/go/pkg/mod/storj.io/[email protected]/pb/piecestore2_drpc.pb.go:337 +0x134\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc001500a88?, {0x1ad4e60, 0xc00377c380}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/drpcmux/handle_rpc.go:33 +0x207\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000116858, {0x1ad51e0, 0xc00377c340}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/rpc/rpctracing/handler.go:61 +0x2e3\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00010c3a0, {0x1ad52e0, 0xc003f63b08}, {0xc005886020, 0x20})\n\t/go/pkg/mod/storj.io/[email protected]/experiment/import.go:42 +0x156\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc004ef1c20?, 0xc003f63b08, {0xc005886020?, 0x2381200?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:167 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc0002f8500, {0x1ad38f8, 0xc000985560}, {0x25bcd04d4f8?, 0xc0026eee08?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:109 +0x1e5\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1ad38f8?, 0xc000985560?})\n\t/go/pkg/mod/storj.io/[email protected]/drpcserver/server.go:157 +0x57\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc000985560, 0xc002a5e4f0?)\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:35 +0x25\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1057\n\t/go/pkg/mod/storj.io/[email protected]/drpcctx/tracker.go:30 +0x79\n\ngoroutine 17564186 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000716d00, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1086\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 46803848 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc0007171e0, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1110\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 45728165 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717040, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 1110\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 47101054 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717d40, 0x14aeec0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 47100956\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n\ngoroutine 47101022 [select]:\ndatabase/sql.(*DB).connectionCleaner(0xc000717ba0, 0x0?)\n\t/usr/local/go/src/database/sql/sql.go:1089 +0x9c\ncreated by database/sql.(*DB).startCleanerLocked in goroutine 47100947\n\t/usr/local/go/src/database/sql/sql.go:1076 +0x105\n"}
  8.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!