Everyone, welcome to the June 27th stand up.
Good to have you guys here. Let's get on to some points.
Okay, let me pull mine up here.
I had made a small proposal for a patch release that may include, let's see here, 5714 and
5708.
They're more so just like the annoying type of logs that we're printing from, I think
was like the duplicate...
seen duplicate blocks, I believe,
and also the node is syncing,
or the syncing logs.
So I was wondering if you guys are on the same page with this,
like, should we push out a...
a patch release with some of these PRs here,
and should it include anything else?
Does 1.9 include gnosis, Shapella?
- No, it doesn't.
Not yet.
- We could sneak that in.
- Sure.
That sounds like a good idea.
- Also, there is a bug in our metrics.
There is a one-line PR that came and merged.
We should add that in too.
- Got it.
Do you have the PR number by any chance?
- No.
- No, okay.
No problem, I'll look for it just in case
it doesn't capture here.
Any other ideas for things that should go in there?
I'm looking for, I guess, more like things
that are not large changes or anything,
but are annoying to users.
- I mean, we could include a fix for the beacon node
not shutting down in few cases,
and then we can just revisit it for 1.10 maybe
and do a proper fix.
Yep.
That sounds good.
That was the issue that SeaMonkey was reporting, right?
Yeah, I think it's the same thing I already noted down in the issue.
It happens in few cases.
And I think it's just safe to just exit the process explicitly right now.
I think this has the potential of have collateral damage.
So if it's not super urgent, best to do later?
Or what do you think?
Like, imagine by some chance there
is something that causes the process to exit too early,
and then we don't persist something,
and then the node goes into a bad state?
I don't know.
I think there is more risk on that PR than the others.
Yeah, but the thing is, so we would just
do it after all the close sequence is
done in the beacon node.
But yeah, I guess it could have some side effects.
But on the other end, I mean,
the beacon node was exiting way earlier
for several months, I guess.
I'm not sure how long that other issue was there, but yeah.
- Yeah, my feeling is it's pretty safe.
'Cause like you said, it's after all the things
are supposed to be closed.
- Yeah.
I guess it will mostly be annoying for people that don't use a process manager because they
would have to press control C twice.
Because usually, I mean, Docker and other like system view would anyway have a timeout
after 30 seconds and then just force close the process.
Okay.
In that case, we could definitely consider it and make a decision as we're getting closer
to that PR getting merged.
Any other ideas for inclusion for 1.9.1?
Otherwise, we'll just go with that for now and then discuss any additional stuff async.
Okay.
So we had a plan basically to after we released 1.9 to look into including node 20, then eventually
upgrading libv2p while dealing with some of the network stuff which with line line has
been dealing with with Ben on.
Do we have any sort of observations or concerns, I guess about node 20?
Cayman had deployed that on feat one and Tuyen had some notes here that I can read out as
well.
Okay, so just for the AI bot I guess I'll just read out what Tuyen wrote.
Set on feature one mainnet node, the first garbage collection pause time rate is greater
than equal to 50% higher than stable mainnet while the second garbage collector pause time
rate does not look correct scavenges greater than 10 million percent.
Whoa.
Okay.
But it's a bug that we fixed.
Okay.
In the metrics.
Got it.
Okay.
Seems feature one main net is able to process 50% more attestations 12 K versus eight K
per slot.
The metric till become head in feature one mainnet is consistently better.
It's 2.4 seconds versus 2.6 to 2.9.
Looks interesting only the garbage collection pause time rate number looks strange.
Is there anything else people want to add from what they observed on feature one?
So that metric, that 10 million, it should be removed six zeros on the end.
So it should be 10%.
I did do like a very quick comparison between between feature one and unstable on our
stable on mainnet and did see like some improvements, I believe, on some of the tracking that he
did for...
Let me just pull it up here so I can read it out.
The event loop lag, I believe, looked better on feature one.
There was less head drift.
I think some of the Prometheus grade duration
and API response times were better.
Did anybody notice anything like that?
I feel like there are definitely some performance.
- Yeah, I was looking more for like,
I was trying to find, I don't know,
errors that were happening that weren't happening on node 18.
And I didn't see anything.
So my feeling is that if there's no--
like if nothing's breaking and we're
wanting to move to node 20, then we should probably do it.
I guess it also is good that the metrics look happy
and there are no performance progressions.
Sounds good.
Okay, great work on that. What else do I have on here? I guess, yeah, you spend most of the week
on the Node 20 stuff. If you have any updates on the P2P or anything on the network thread, guys,
feel free to just give us a quick update on those.
>>My branch is ready to test.
I've fixed conflicts and merged in the, I guess,
the latest unstable.
Maybe I need to merge it again today.
But yeah, I guess I can snag feature one
and deploy the new libp2p there.
it. Right. Okay. Um, what else? Um. We in also had, um, an update to include. Um, we're
which was the implemented deterministic long-lived subnet.
But just for the recording here, he wrote,
The main change is to always connect
to exactly two subnets per node instead
of based on number of validators,
which reduced the subnet mesh peers a lot,
hence the I/O lag issue.
So of course, this is important for home stakers,
and then he would be able to enable
a new feature flag, deterministic long live
a test for the next release.
And then he's also debugging something with Nethermind
as well.
Any other points to add to that or to planning?
Looking forward to that fix, or I guess that change
that Tuyen’s doing to the long-lived subnets.
That should really help intermediate level stakers,
I guess, do less work.
- Yeah.
Cool.
So if there's no additional points for planning,
We can go into updates then.
Okay, today we'll start with Lion today.
Hello.
So I keep focusing on a bunch of spec things.
Basically, more of what we speak last time.
And thank you all for the feedback.
I shared with the researchers and I think we're drafting an EIP already.
So yeah, that's all for me.
Awesome. Thanks, Blaine.
Next, we have Gajinder.
Thanks, Phil.
So basically, I...
So there was an issue on DevNet 6 last time that I also talked about where Lodestar was not able to
sync blobs by range. And I thought it was the issue of Lighthouse because they were returning
32 blobs instead of 24 blobs that were required in a particular epoch. But similar issue was
observed with the clients as well. So I went debugging and figured out that, you know,
so in our max request, we were not passing. So basically, we were not having the factor of
multiplying the count by blocks per slot, because count is generally number of slots
that you are requesting. So there was a mismatch. And that was causing us noticing,
but this PR resolved the issue. Another thing, I mean this resolved the issue in the sense that
we started moving forward, but then we never actually sync to the head. And the reason
then I debugged again and figured out the reason for that is that, so in this entire range,
the chain wasn't finalizing because only a few nodes were up, mostly Lighthouse. And since the
the chain was not finalizing.
So let's say for more than 30,000 slots.
So we would basically stall after syncing about 15,000
or 11 to 15,000, basically, you know, we would stall.
And then I started looking into the logs,
all the request responses were going in and out.
That was not a problem.
So, but what I figured out was that,
so we would start, we would add a peer
and start syncing from it.
and over some period of time, it would disconnect.
And then we'd reconnect with that peer or some other peer,
and it would start syncing again from the last finalized,
which was like epoch 16.
And because whatever jobs we create,
we create with respect to finalize.
So whenever we connect with a new peer and start syncing,
I think we start syncing from finalize itself.
So we need to sort of figure out a solution for this
where the chain has not finalized for, let's say,
so many 30,000 slots.
And we have to figure out, okay,
whether the new peer which is joining in,
can we put it on the chain that we have already synced
instead of starting it from the last finalized
that we have in our local chain.
So I think we might need to tweak our sync process
for syncing for the scenarios where,
I mean, it's a net scenario, but other clients can do it.
So, and in test nets, this scenario can occur frequently
and it has been occurring.
So we basically should tune up our sync process
to sync to enable us go through this scenario as well.
There was no other error I could figure out,
just that, you know, it starts syncing whenever,
eventually, basically the peer disconnects
and we'll start syncing with new peers
from the same finalized epoch that we had in our local.
So this is, I will be,
that is something that I have in my head to work on.
But all of the things were sort of resolved for DevNet 6.
Now DevNet 6 is going to be relaunched as DevNet 7.
So no real work needs to be done over there,
unless something else comes up.
And for DevNet-- so the earlier DevNet 7 is now DevNet 8.
And basically, it will be launched in two weeks' time.
And two PRs for DevNet 8 are already in,
and one PR I'm working on.
And I also did PEEP and EEP presentation
for direct changes, basically, build and edit.
I think the recording will be out soon.
Thanks, Gijinder.
Okay.
We will move forward now with Nico.
Hey, so I was investigating the issue I mentioned last time about the doppelganger protection.
So I found that we definitely have a chance there that we have false positives.
I basically noted down the issue on GitHub.
Yeah, I also checked what other clients do, how they implemented it.
And I think our implementation is pretty similar to Lighthouse.
So basically the issues that we check for attestations in blocks, and those can happen
in the next epoch, which we then check
and that causes the false positives.
So I'm not sure how to properly resolve this
other than just increasing the wait epoch time
by one more epoch, which would be not that great
in terms of UX.
The other thing I wanted, or I also checked
is if we can maybe implement zero downtime,
doppelganger protection, which would be possible, I think,
if we just check the local attestations
that the client produced.
And if there's an attestation in the previous epoch,
we could skip doppelganger protection for that validator.
I think that should be secure,
but yeah, I still have to look into that
and how to implement that.
So there was one issue that the registration of signers
is sync at the moment and we would have to make it async,
which just causes a few downstream issues in the code.
So I haven't really further looked into that,
but I think in theory it should be secure to do.
I think it's even more secure than always waiting to epochs
because if you would start two validator clients
at the same time, one with a DB and one without one,
doppelganger protection would not work
because both would idle and then both would start
a testing at the same time.
And with that approach,
since you can only connect to one database
or the database has a lock mechanism,
this would not be possible because two validator clients
cannot connect to one DB.
So I guess it would even be an improvement
unless I missed something
in terms of the security around this.
Yeah, so besides that,
I looked at the latest speaking API spec,
created some issues for that.
Yeah, and just checked why the spec tests are failing.
And yeah, so this week I really want to focus
on the whole region topic.
And yeah, that will be mostly it, I think.
Thanks Nico for posting up those implementation things
that were missing there.
I have previously had people ask me
how to get started on Lodestar.
And we've always had trouble finding people who--
or issues that were easy enough for people
to get started or tackle.
So definitely if any of you guys spot anything where it's like,
hey, this would be a good intro for somebody
to pick up Lodestar and to start understanding it.
We definitely need more of those help wanted
or first time issues tagged.
So thanks for that.
All right, moving on, we have Cayman.
- Hey, yeah.
So last week, we put in a few PRs to get Node 20 ready
to put it over the weekend.
And now that we've got consensus on it,
I will put in another PR to bump all of our CI to node 20
and also to bump our Docker builds to node 20.
Other than that, I was doing some research,
just getting caught up on the discv5 spec
And some improvements that are happening there.
There's a discv5.2 that is in progress.
Right now we're on 5.1.
So looking at--
just getting prepared.
I don't think we need to implement anything right now
because the spec is still not fully formed.
Don't want to do it too soon and then have things change when
And there's no real benefit to us.
And other than that, I was looking into some--
I was looking at other libp2p issues,
working on--
still trying to convince these guys
to use TypeScript for some of their lower level libraries.
I know some of you guys are subscribed to that issue.
And there's a lot of back and forth.
But the maintainer just reviewed my PR
to convert to TypeScript.
So there's movement.
And he said he doesn't hate it.
But that his review does not mean that he checks off on it.
So it's still in an intermediate area.
Yeah.
So yeah, this week we'll deploy the libp2p branch today
so that we can start getting some metrics
and see how that goes.
But I'll be working on that.
And if I get to it, also trying to add
IPv6 support, which is almost all there.
Just haven't actually hooked it all together.
So that's it for me.
Thanks, Cayman.
Speaking of IPv6, I think Lighthouse
has some boot nodes in IPv6 out there right now.
- Wonderful, okay.
- Yes.
I will try to find the reference for you.
I remember reading that somewhere, but yeah.
- Great.
- And also speaking of boot nodes, John,
we've been wanting to actually deploy some boot nodes
for consensus and execution to help with the diversity.
I can talk to you more about that later as well on a 101.
- Sounds great.
- Cool.
- Are we compatible?
Because Barnabas and Pari were asking me about this.
- IPv6 compatible?
No.
Everything is not deployed or published yet.
All right, do you have anything to add, Dizar,
since you've been back?
No?
If not, we'll move on.
- Yeah, I am back since yesterday, starting on PR.
Hopefully we'll finalize that PR today for review.
It will add some test cases, end-to-end test cases
for rover package, and it will introduce a new package
called testutils, which will clean up a lot of code
which were scattered around testing
in different packages into one package.
So if you guys came across any useful thing
that you think can be helpful
or where you reuse during testing for other packages,
then you can include that in this new package testutils.
And apart from that, I had one on one with line
in the morning and we decided on the direction for the prover further.
And the direction is that after finalizing this end-to-end testing for the prover, we
will mark this public and that will make sure to release, like the first release of the
prover package.
And then I will use this first release into our light Client demo that we have, showcasing
the light client functionality. So that will help to minimize or reduce a lot of code inside the
light client demo and will help to troubleshoot and actually use this prover package in real
production case in the demo. And once that is done, then I will start working on MetaMask snaps
to create a snap as a proof of concept for the prover. That will be the discussion
initiation for MetaMask team. For production we are not sure if that is the right way,
like MetaMask snaps is the right way or not or it should be part of the MetaMask. But
once we have a snap then we can initiate this discussion with their team on it and then move
forward. That's something I will be doing this week. Tomorrow I will take off. If you
guys know, there's a Christmas-like event for Muslims tomorrow, Eid-ul-Adha, the slaughtering
Eid. So I will take off tomorrow. Thank you.
Thank you, Nizar. All right, and we have Matt.
I got the blst package finalized and fixed the bug.
So it's in the newest format that Gajinder was reviewing.
And it seems like it's working nice, got it deployed to Feature 2,
and metrics looked good for everything that it had touched.
and also turned on worker and the metrics improved even further, which is great.
So it's looking nice. We actually did get it deployed to mainnet as well today.
Mainnet feature 2, Tuyen was using.
And so we did get it deployed there so we can see how it's doing on mainnet.
And then also picked up the database duplication ticket
and dug into state and SSZ and whatnot.
And I was hoping to get that done for today,
but it's actually kind of a tricky little problem there.
But going smoothly.
And then I'll probably do another round of review
with Gajinder this week on the next part of blst,
and then finish up that PR and probably pull the next one,
possibly the one that you put in the channel just after,
because it was similar state stuff came in.
So I think maybe I'll...
I think it was mimicking Lighthouse, what they're doing.
I'll check that out, though.
Cool, thanks, Matt.
John, do you have anything you would like to bring up at all
for the team while we have you here?
Not at this time.
Okay.
Cool, guys.
I have nothing else other than ETH Waterloo was awesome.
I think they, I think 50% of the hackers there were actually new to the Web3 ecosystem, which was pretty cool.
So getting people excited and hoping to get more people to look at other parts of the ecosystem outside of just the DAP stuff.
But yeah, I think it'll be pretty cool, you know, to see these hackathons attract people
away from other parts of tech and into blockchain.
I did have a conversation with, I don't know if any of you guys know Sebastian from HopperNet.
He was kind of interested in some of the validator privacy stuff and introduced him a little
bit to like, the what Nym and Chainsafe were doing, and how they implemented that into
Lighthouse and sort of got them caught up on that. There might be some potential discussions
about having like having us get involved in terms of trying to implement this spec, perhaps
or testing it out and gathering some data. I believe I talked to Cayman a little bit about
this already. But, but yeah, validator privacy is something people are looking to tackle.
I guess not just in protocol, but also, also in the dapp layer, with like stealth addresses
and stuff like that. But, but I guess the concern for us would be more so the privacy
the protocol layer. Seb was the guy who did this experiment to sort of correlate validators
and IP addresses and stuff and was basically able to list out, you know, a bunch of validators
IPs correlate them and even potentially DDoS some of them. So he's actively looking for
a way to improve privacy at the protocol layer if anybody was interested. And he's also working
on them I think like a private RPC stuff. I think that's what Hopper net does.
Cool. Anything else to bring up for stand up this week.
Next Tuesday is July 4.
America Day.
Merca.
All right. Well, you guys have have fun on your Independence Day long weekend. I believe
Chain Safe, technically, like especially for the Canadians also will have the third off. I believe
it's the Monday. Yeah. July 3rd. The Canadians will celebrate Canada Day. So.
And then is Chain Safe days on the 30th?
Yeah, so technically, yeah,
chain safe days on.
Yeah, so there's quite a few.
Days to enjoy yourself as well.
But knowing us will probably
be around here and there so.
Yeah, I'm gonna be working, yeah, hi.
I'm planning on.
I'm planning on taking tomorrow off.
That way I.
Along with this are not for.
Not for the same reason.
Cool. All right.
Sounds good, guys. See you guys in the metaverse then.
Cheers.
One last thing. Protocol discussion on Thursday, I forgot. So we're going to talk about like
clients and see the prover. Yay, Nazar! See the prover. And yeah, very cool. Come on down,
everybody. Be cool. Yeah, let's do it. Cool. Thank you. Take care, guys. Have a good week.
Bye-bye. Bye-bye.
[BLANK_AUDIO]