Lioneye's Watch Episode #09 Transcript

Redditor /u/Shiverwarp transcribed Lioneye's Watch podcast episode 06 when Chris Wilson was the guest speaker, so that we all would have a written record (which could easily be searched and referenced) of GGG's opinion and insights.  As Shiverwarp put it, "The searching for quotes is, in part, what sparked this for me.  I've heard people say, "Didn't GGG say they were going to do so-and-so, in ___ podcast?".  And then we get a bunch of people just taking someones vague memory, as fact."

I've thought about transcribing other Lioneye's episodes, but I'm lazy and transcribing is time-consuming and exhausting.  This week we once again have a senior member of GGG's staff as a guest speaker -- If any week is a good one to transcribe, it's this week.

I tried to remove filler words, repetitions, that sort of thing.  False starts (changing sentences part-way through) are indicated with an endash --

My annotations are in [square brackets].  Timestamps [5:00] are a rough guide, with times based on the Youtube upload.


Meeting Minutes:
[0:45] What's the difference between the small team you started with and the large team you have now?
[7:48] What exactly is the "backend"?  Discussion of exploit protection in PoE.
[20:16] When did GGG decide that desync needed to be addressed?  Followed by performance improvements and DirectX 11
[34:53] Server performance.  Telemetry.  Potential new indicator to differentiate between local (harddrive) lag vs network lag?
[46:30] Should Hardcore players be allowed to set a lower automatic disconnection time?  (Bonus!  The dirty secret about Chris playing hardcore in Diablo II)
[54:00] Why can't you just add ten programmers to program ten times as fast?
[56:40] The Act 5 Beta.
[1:02:56] Updated microtransaction stash coming Soon(tm) (Quarter one next year maybe).
[1:06:50] Different programmers work on different things.
[1:09:00] Y U NO ADD PARTICLE SLIDER?

Lioneye's Watch Episode #09 Drinking Game:
Pick one word pair from this list.  Take a sip every time Johnathan says it:
Honestly, Right/Exactly, Actually/Really, Basically/Obviously, Stuff, Kind of / Sort of, Annoying/Unfortunate
Take a drink when Johnathan repeats a word three times in a row.
Take a drink when Johnathan forgets the name of an item or skill.
Take a drink when Johnathan swears. (Crap, Fuck, Shit, etc.)


------------------------------------------------------

Twitch stream on LiftingNerdBro's channel: https://www.twitch.tv/liftingnerdbro/v/95361704
Youtube mirror on Hegemony's channel: https://www.youtube.com/watch?v=Q_tBlZ-8xJo

------------------------------------------------------
YouTube Video Description
------------------------------------------------------

Published on 16 Oct 2016
In this podcast Lifting, ProjectPT and I discuss the issues in developing Grinding Gear Game's Path of Exile Expansion with technical director Jonathan Rogers.  Get some behind the scenes insight on what's to come to future exiles!

If you would like to submit a build of the week post a build guide on the official forums.  All we ask is that there is some video footage and you can send a private message to projectpt @ http://www.twitch.tv/projectpt to submit your entry. (We'd like to see some flameblast totem builds with a bunch of different ascendancies!)

You can also submit your hideout submissions to projectpt to be showcased on our podcast in the future!  Thanks in advance!

The Lioneye's Watch podcast is live every Sunday 21:00 (GMT+2) or 2:00PM CST on Lifting's Twitch: http://www.twitch.tv/liftingnerdbro

LiftingNerdBro's youtube can be found at this link: http://www.youtube.com/liftingnerdbro

ProjectPT's youtube and twitch are available at:
http://www.twitch.tv/projectpt
https://www.youtube.com/c/projectptsheep

Hegemony's Twitch: http://www.twitch.tv/HegemonyTV

Reference Material:
Dev Manifesto on Desync: https://www.pathofexile.com/forum/view-thread/710359
Dev Manifesto on Performance: https://www.pathofexile.com/forum/view-thread/1642228
Path of Exile's Mulithreading Video: https://www.youtube.com/watch?v=ajVc-uwdJ0Y
Build Guide For the Ice Trapper Background Video: https://www.youtube.com/watch?v=_j7SpKL_PEQ

All the Art was made by Maddhatterui.  This podcast would not be as high quality without him.  Thank you so much man!

Recommended Community Tools:
PoE Wiki: http://pathofexile.gamepedia.com/Path_of_Exile_Wiki
Trading Search Tool: http://poe.trade
Labyrinth Layouts: http://poelab.com
PoE Affix Info: http://poeaffix.net
Vorici Chrome Calculator: https://siveran.github.io/calc.html
Aura Reservation Calculator: https://poe.mikelat.com/

------------------------------------------------------
TRANSCRIPT
------------------------------------------------------

LiftingNerdBro: What's up, guys?  Welcome back to the Lioneye's Watch Episode #09, and as always I have my two co-hosts with me.  You guys want to say hello?

HegemonyTV: Hello!  Hi.

ProjectPT: Hello, everyone!

LiftingNerdBro: And as special guest, we're happy to have Johnathan Rogers, Lead Technical Director from GGG to join us for a talk about his role as a developer for the team and then a lot of technical questionos.  Thank you for joining us tonight Johnathan.  How are you doing?

Johnathan_GGG: Very good.  Thank you for having me.

LiftingNerdBro: We're happy to have you.  We're going to jump straight into this, and I want to start with a question here, so: Johnathan, I'm curious, can you walk us through the difference in designing Path of Exile from the small team that you started out with to the team you have now?

Johnathan_GGG: Honestly, the biggest difference is that when you're a small team, everything is just really fast.  Like, if you make a decision and then, "Oh let's do that," and then someone does it, and then it's done, you know.  It's just really easy to -- It honestly feels like you're making more progress than when you've got a big team.  Big teams can produce a lot more crap, but they're also extremely slow to do it.  There's a massive pipeline.   It is nice that I can go -- We'll design a league, and then at the end of the process there's going to be a ton of content in a modern league that we design.  But it's so hard to actually take the ship and turn it in a different direction when it's rolling.  So that's really the main difference, the agility you get when you're small.
Also, honestly, things are just more fun in a smaller team, just because you know everyone, you're friends with everyone.  When a team starts to get big, it's like, "There are people in this office I don't even really know."  There are probably some people I can't even remember the name of, which is terrible to admit.  At least Chris knows everyone's names, I know that.  He's a much better boss than I am.

LiftingNerdBro: What do you consider your team right now?  Small, large, medium?

Johnathan_GGG: You mean as a game company?  In terms of the game industry, it's only a medium team, but in terms of how I feel about it, it feels absolutely enormous.

HegemonyTV: Are you the biggest in New Zealand?

Johnathan_GGG: Yes we are, I believe.  Just.  There used to be a bigger studio, which was a GameLoft subsidiary I guess you'd call it.  So GameLoft had a studio in New Zealand.  But, unfortunately, the main office decided -- It was something to do with that hostile takeover they had.  So anyway, the New Zealand studio closed down, and that ended up being -- It was very unfortunate for all their staff, because they had like 160 people or something like that.  We actually got some really good hires out of that and an example of that is Vincent, who has been working on all the performance work we'll be doing, was someone who worked for GameLoft and we managed to snap him up, which was really good.  So that was pretty good, for us at least.  We have been expanding quite a lot recently.  China has meant that we had to expand.  The other thing is just the push for 3.0.0 is going to be really massive.  There's a huge amount of content there.  That's basically why the team is getting so big.

ProjectPT: With what you were discussing earlier, what was the transitional number, roughly, when it switched from the small team mentality to the large team mentality for GGG?

Johnathan_GGG: There's probably actually two points.  One of them was around 20.  That's the point where you go from "everyone knows everything" to "now not everyone knows everything."  What I mean by that is, generally speaking, a room with 20 people in it, everyone else knows what everyone else is working on.  You don't get any of that communication overhead you get here.  As you get larger, you get to the point where people will just work on the wrong thing because no one's told them that it's the wrong thing -- The founders just don't know about it.  It's not really their fault, you get lost to communication.  And I'd say that starts to happen around 20.
I think around 50 is when the tone really changes because now you're getting to past the point where you can really even know everyone really well.  When we're small, I know each of the people and what their strengths and weaknesses are.  I know if I ask X person to do this thing then he's going to be the right person to do that.  When you get above that, we have to rely on the structure more.  I can't just walk up to an artist and go, "Hey, do X," because he might have some totally different priority based on some other part of the company.  So I have to go use the -- Go talk to Erik and then find out which guy it needs to be and then blah blah blah, that kind of thing.  In terms of structural stuff it gets more -- You have to start to be more official and you have to do things through proper lines of communication.  It starts to turn into a company, right?  [Laughs]  Which I can't say I particularly enjoy, but it has to be done.


[5:00]


HegemonyTV: With that, how has your role changed, if at all, since you founded the company?  Do you find yourself delegating more tasks?  Are you less hands-on?  Or are you just taking stuff that is escalated to the highest role?

Johnathan_GGG: At the start, I wrote all the code because I was the only programmer.  And then we hired programmers and we had to start dividing stuff up.  I originally wrote a lot of the core system, like I originally wrote the renderer, and the skill system, and the stats, and the server backend.  You name it, basically.  The biggest system I didn't have any hand in at all really was the terrain generator.  That was handled pretty much entirely by Rhys.  I obviously worked with him a lot on design, but he's taken control of that from the beginning.  But now, really, I can't start new systems.
The main thing, my role at this point, is knowing how everything works.  That's kind of the thing.  People need someone who knows, when they want to do X, where would I even start?  What area of the code am I even going to?  That's my main thing at this point.  All day, I'm getting people walking into my office asking questions.  Which isn't really that conducive to doing long, hard projects.  So it really makes it hard for me to concentrate on something like, for example, the whole desync fix thing would have been very difficult to do now, for me, because I get so many people walking into my office every day, and that required a serious amount of face-to-the-wall work.  [All laugh]

ProjectPT: You have so much of the base information that, since everyone needs to go through you, you're going to be interrupted way too often to realistically handle one of those projects.

Johnathan_GGG: There are still things that I do work on.  We haven't announced what trade stuff we're doing is, but I am working on some trade improvements right now, and I still do -- I'd say most of the work I do is mostly on the backend.  We've only got one other backend programmer, and he's completely swamped with work for the China release.  Backend programmers are very difficult, because they have, more than anyone else, the ability to completely fuck up the realm.  [All laugh]  Basically, they need to be 100 percent certain everything they're doing is good.

ProjectPT: Could you elaborate a little bit more to help most people, what does the backend programming entail?  Just in general, and then GGG.

Johnathan_GGG: Basically, we have what you might call the frontend, which is the place where all the gameplay happens.  In the parlance of Path of Exile, those are instance servers.  Basically, that's the server you play the actual game on.  But, behind all of those servers is what you call the backend, which is basically the database system which has all of your items and that kind of stuff in it; as well as, for example, the party manager, the thing that handles partying up; there's chat; and guilds; and basically all those kinds of features that require communication across different instances.  So that's basically what that stuff is.  Generally speaking, most of what you do when you're playing, like fighting monsters and so on, doesn't touch the backend at all.  Every now and then, there's a save, which saves the backend, which saves all of your progress recently.  That's basically what the backend is all about.  Which is true in most game companies, there's generally that thing.  You really don't want people connecting directly to your backend, because that's a potential problem for security.  Having an extra layer between the backend and the frontend is just really useful to prevent any kind of security problems.

ProjectPT: Are you saying it's not to your advantage if players can figure out exactly how items are generated and all that fun stuff?

Johnathan_GGG: Certainly, yeah!  We're very careful about that kind of thing.  One of the big things that I was really serious about when we started the company was security, because the economic security is a cornerstone of people's -- You're only going to value an item if you think no one else can -- If people are just duping uniques, you're not going to value the uniques, right?  If we want to sell on people that items are valuable, then we have to make sure the economy is very robust.  To prevent that, we have to make sure there aren't any economic exploits.  Most of the time, the exploits we have are not so much dupe methods, but more along the lines of someone made a gameplay mistake, which leads to someone being able to farm something really efficiently.  There was that case recently, I can't remember the exact details, but it was something along the lines of, there was some prophecy which you could -- There was some way to get a drop from a prophecy more than once?


[10:00]


ProjectPT: The Kintsugi one?

Johnathan_GGG: I don't remember the exact name of the thing involved.

ProjectPT: Fair enough.

Johnathan_GGG: Basically, there was a thing like that, and that's the kind of exploit that -- Those kinds of exploits are really hard to nail down, because they can be generated by gameplay programmers, whereas things like dupe methods typically are more in the core, and we're looking more closely at that code.  Also, the way that stuff has been architected has been designed to be as foolproof as possible for anyone interacting with it.  We're really careful about things like -- Really common cases of exploits are when people are able to --

ProjectPT: Rollback the servers, right?

Johnathan_GGG: Well, that's a different class.  The rollback thing is actually only an issue if you can find out the result of something that you're going to make the decision of whether to rollback or not.  We actually inserted a save now in chaos orbs and exalts, and stuff like that, so you can't do rollback crafting anymore.  The cost there is, we're causing more server load by saving more often, but in exchange for that we have rollback protection against the really expensive crafting options.

ProjectPT: I didn't actually know that was implemented.

[All talking]

Johnathan_GGG: We don't really announce that kind of stuff.  [All laugh]  It's only a problem if you've already been exploiting the system.

ProjectPT: But it's interesting.

Johnathan_GGG: That's one of those things that we -- It was a few months back, I think.  Anyway, where was I?  That's one class, is rollback stuff.  A huge class of exploits that we don't have is people addressing things in the database that they shouldn't have permission to access.  The way that we do that is we make sure that every kind of query that you can do is always relative to your user.  To give you an example of that, rather than saying "I want to pick up item with ID 15", we say "I want pick up the item onto my cursor the item with X-Y location 5-7."  It's very much harder for someone to exploit.  In Diablo II, for example, there were tons of exploits of that nature.  You can buy an armor off someone else's body.  That kind of stuff.  [HegemonyTV: Feels good.]  Really, really dumb stuff like that.  That's all because of the fact that they weren't being careful with their IDs, making sure you'd only be able to address stuff that you're supposed to be able to address.  Things like account IDs and item IDs and all that stuff, you never see any of it.  It's not even in the protocol between the client and server, because we want to be really careful with that kind of stuff.  That's one of the security layers that we have.
An example of other security layers like that is, you can't modify an item while it's on your body.  In the backend, the way an item is modified, it's lifted, and then modified, and then put back down again.  That's the sort of thing that prevents huge numbers of things.  A lot of games, you'll get problems with -- You'll find an exploit that lets you get an infinite amount of some particular stat because you've got some way of fooling the thing into only adding it and not removing it when you have some interaction with the button.  That's the kind of thing that when we were starting the game, we were really really careful to get all that stuff right, and make sure it's just really hard for another programmer to accidentally make a mistake that causes those kinds of issues.  I like that we're pretty good on that front.  As I said, you still can randomly get things like the double item drop.  Some gameplay programmer just didn't realize that that was a possibility to res that guy.

HegemonyTV: With new leagues, and a lot of new content, things slip by, like the Atziri's Disfavour card at the very beginning of the league.  Like you guys said, you are very protective of that.  Is it because of those things that those things slip?  Are you expecting to lose sleep when a launch happens because you're just anticipating Chris being, "Johnathan, we have a problem!  Fix it right away."


[15:00]


Johnathan_GGG: Absolutely.  On the first day of a launch, we're monitoring the forums and reddit as much as we can.  The moment something appears on the new queue, like someone talking about an exploit like that --.  It helped a lot once we had the ability to deploy server-side hotfixes without restarting the realm, which we've had for quite a number of years now.  Before that point, it's like, "Do we fix this kind of minor economic thing, and kick off the fifty thousand players who are online?" [All laugh]  You kind of have to make that kind of call.  So it's much better now that we can just deploy stuff wtihout having to worry about that at least.  It means we can do pretty fast turnarounds.  It's annoying, once again, as the company and as the codebase has been getting bigger, it's getting harder and harder to get those fast turnaround times.  The compile times of the servers have started to go -- A full recompile of the servers takes around three hours.  If there's a change to something that's used by a lot of things -- For example, if we add a stat, that requires a nearly full recompile as almost everything touches stats.  Probably 90 percent of the codebase either touches stats or inherits from something that touches stats.  That sort of thing takes three full hours to compile, just to compile.

HegemonyTV: How would you compare that to the start?

Johnathan_GGG: It's just linearly scaled.  The beginning is like seconds, and now it's hours.  Something recently has really slowed down the compile times in Linux.  Our servers are on Linux.  On Windows, the servers take quite a lot faster.  So in Windows, I think you can compile the servers in probably half an hour from scratch, maybe a little less.  On Linux it's gone to about three hours.  We need to work on that.  This is just the constant struggle that we have.  Over time, if you just leave it, everything will fall apart eventually.  All of the assumptions that were made were made based on things that were smaller scale.  You just have to keep on continuously working and working and working at it and getting things faster.  At some point soon, we're probably going to declare that the build taking this long is a bit of a crisis, and we need to move a bunch of engineers onto it to work it out, and I'll probably walk out there and talk to Thomas who is our server, "We need to get our build down to 20 minutes." [All laugh.  HegemonyTV: You're armed with a gun.]  Yeah, exactly.  He'll eventually get there.

ProjectPT: With your security things that you've done, obviously they've overcomplicated something that would have been easy.  Can you give us an example?

Johnathan_GGG: An example I've used before is the whole addressing of IDs thing.  One complication there is that on the client you can't tell if an item is the same item as before.  To give you an idea of what I mean by that, it's like -- If you have any kind of interaction where it's like, "I want to do this thing to an item, and then after it's done, I want to put up a bit of UI related with regard to that item." Because of the fact that the item has to be lifted and placed back down again, and because of the fact that there's no item IDs that are consistent, that means that all that the client has to go on is like, "I know there's an item that's in the same place as the item that was there before, but I don't technically know that it's the same item."  That's the kind of annoyingness you get from having to jump through those kinds of hoops that we have done in the name of security.  You can do sort of hack-y things, "Is there an item in the same place that has the same itemtype?"  Yeah there is, but I can't actually be 100 percent certain that's the same item.  That's the kind of annoying thing that we get on the client side when you have this kind of thing.   On the server, we know the item's the same item because on the server we have all of the IDs and everything.  Certainly, security makes things complicated.  The other thing is, whenever I'm working on any kind of system, the initial version is super easy and the code is really clean, and then you start to think about what can fail here, and what do I have to do to prevent that, or prevent the system from getting into a bad state.  By the time you finish with all of those cases, the code looks like an eldritch nightmare.  [All laugh]
It's kind of terrifying.  And it really sucks doing the kind of distibuted system that we have with Path of Exile.  It's not like it's just one server, one database server that the game runs on; there's a collection of servers.  When you've got a distributed system like that, there's three things that can happen with any operation.  One of them is that it can succeed and you know it succeeded.  There's the case where it fails and you know it failed.  And there's the case where it succeeded but you thought it failed.  That's what happens when the message that comes back from the thing that told you it succeeded got lost.  That case there is what really keeps us awake at night because obviously the thing on the other end is going to make decisions based on the idea that that failed, and so the whole thing just gets really nightmarish very quickly.  We try to work out all the repurcussions of that and you have to work on back and you think, "Oh God, this thing's gonna happen this way, but if this thing succeeded but I thought it failed then blah," it's just, yeah.  Gets crazy.  That's the nightmare of distributed system programming.


[20:00]


ProjectPT: You mentioned a little bit earlier about desync.  When did you [GGG] decide that this needed to be addressed?  That it was waiting too long.

Johnathan_GGG: We always kind of thought that this was a problem, but we kind of were in the mindset of, "We just need to work around all the problems."  In D2, they had the same model as we used to have before lockstep.  It wasn't a problem back then, but I think the reason it wasn't a problem back then was player expectations.  Back then, when Diablo II came out, the expectation of network play was not the same as it is now.  There came a point, I think it was about five months before The Awakening came out, where I was like, "We're not going to be able to make progress going down the direction we're going down, we need a massive change.  It's going to take a long time, and it's going to be horrible, and I"m going to have to stall a pile of crap that I gotta work on, but the network system needs to be rewritten from the ground up."  I think it took me about three and a half months solid of work to get from the point where I had the idea of "why don't we literally just change our entire network model from one thing to another" to the point where it was actually releaseable into the Beta of The Awakening.  That was, yeah, that was pretty crazy.  That process was probably very important for us because it really showed us the value of actually going back and fixing something the hard way.  So in other words, taking all the code we had, and rearranging it completely.  Which is something we have to do again with the perfomance stuff we've been doing in multiple ways.  the threading thing is pretty crazy, the amount work that has to be done just to get the engine to a state where we can do that.  Of course right now, we have a multi-engineer project of porting the game to 64-bit DX11.  But really, the important thing is having the mindset of looking to the longterm.  At the end of the day, you have to look at these games and think, these things last -- Look at something like Eve Online, they've been running for how long, like a decade?  At least, more that a decade.  A very long time.  If we want to be in that category of game, where we're an evergreen continuing on forever type of game, then we have to address these types of problems.  The earlier we fix them, the earlier, the more players will enjoy it.

LiftingNerdBro: There's people who would say, "I'm still not playing Path of Exile cuz there's desync in the game."

Johnathan_GGG: Exactly, exactly.  We want to make sure there isn't anyone -- Let's put it this way, if you don't like the game, I want the reason you don't like it to be because you don't like the game as opposed to some technical problem.  That's why we have to make sure we do this sort of stuff.  But it is slow.  It takes a long time, and the really annoying part is that people -- It seems like the problems get worse while we're in the middle of solving it.  There was a point where the community was really complaining about performance a lot, and I'm like, "We've got these projects already, we already know about it, it sucks that we can't talk about what we're doing."  Especially with performance, you can't promise anything you don't have working because you might get to the end and just find that it doesn't actually work as well as you had thought.  So I'm always really cautious about promising performance improvements.  Especially because there's also this really annoying phenomenon with performance improvements where every change you make, it gets better for 95 percent of people and gets worse for five percent of people. [All laugh]  It's really annoying, it really sucks.  I think that when we first did some of these changes we really had to just deal with the fact that it's going to be a little like that, and we can just clean up the five percent afterwards.  Because we can't possibly know what problems a given person's machine is going to have.  People have some really seriously terrible computer issues, and then they'll say, "It's fine in every other game."  That might very well be true; some other game may not happen to stress the exact same components or whatever that we do.  They may have gone through the gauntlet of finding out that everyone else has this problem and then fixing it afterwards.  That's a real problem with that stuff.


[25:00]


ProjectPT: You brought up DirectX11, and that was something we want to talk about.  I think that segues into a question for Lifting, and we should get that out of the way because chat's really excited.

LiftingNerdBro: I was very curious about, with Act 5 -- No, PT, you ask this because I want to save this.  [ProjectPT: Alright.]  I butchered that.   I completely butchered that.

ProjectPT: Yeah.  Act 5, what?  You mentioned this DirectX11, this 64-bit client, and it's gonna change the performance improvements.  What do you want to say about DirectX as of now?  What should players be able to expect?  Do you know what should the five percent that their performance is going to get worse expect?

Johnathan_GGG: Well, I really hope we don't get too many of those people.  DX11, the main benefit of it technically is that the performance of the actual API on the CPU side is improved.  That is to say, the time that the engine spends telling the graphics card what to do will be reduced.  Currently, that's about 50 percent of the total time the client spends doing anything.  About 50 percent of the time is spent telling the GPU what to do, and 50 percent of that time will be gone when we go to DX11.  So that makes, if you're counting, the framerate about 25 percent faster.  Graphics programmers never look at things in terms of FPS, they look at things in terms of frametime.  If your frametime was 10 milliseconds before, now it's 7.5 milliseconds, and you can work out what the framerate is from that.  That's some of what we're getting right now.  However, there are in DX11 more ability to use multithreading within the render itself.  Right now, we have multithreading in all the code that we run CPU side.  That is to say, things like particles and animation and trails and all that kind of stuff, all the graphical effects that we simulate on the CPU.  But, on DX11, you can have multiple threads that are rendering at the same time.  On the GPU, usually they won't actually be rendering at the same time, but the thing is is that it's the CPU side, it's the "telling the GPU what to do" part.  So, for example, we'll be able to render shadows concurrently with other shadows, and we'll be able to render certain parts of the scene that are rendered to temporary buffers.  For example a bunch of our particles render to a different resolution than the rest of the scene, so we'll be able to render all of those particles at the same time we're rendering some other parts of the scene.  Basically, we can just look at all the different parts and what the dependencies are and we can work out what parts can we do at the same time, and then we can do all those parts at the same time.  Basically, it's going to be more multithreading in the engine.  And I'm hoping -- Vincent is always like, "I want to get it to 2x."  Every time there's a performance improvement, he's like "I want to double the framerate."  Obviously, under certain situations.  [LiftingNerdBro: It's a noble goal.]  I don't know if we'll get there, but as of right now, we're kind of like 1.5x, 25 percent shorter frametime.  Is that right?  I'm doing math wrong, whatever.  In any case, we've cut about 25 percent off the frametime right now, and I think we can do better than that once we get the multithreading working.
Actually, there's a lot of improvements to the multithreaded system as well as the DX11 port.  That'll probably come out in 2.4.3 as well.  He's really packed those threads tight.  I don't know if you guys saw the performance video I did where I talked about multithreading of the frame.  In that video with the performance profile, there were a lot of gaps between each of the different stages of doing the threading.  And so he's managed to get a pretty large performance improvement from just working on that system over time and getting the threads packed together.  It'll consume a lot more of your CPU.  The more of your CPU will be able to be used.  I'm pulling this number out of my ass: If we can currently use 60 percent of your CPU, then maybe that number is going up to 90 precent of your CPU after the change.  Those numbers are just fabricated, but that's the kind of thing we're talking about.  That translates to more work done faster, so your framerate goes up.


[30:00]


LiftingNerdBro: For players who are having issues right now with performance in the game, what's your advice?  What do they need to do, what do they need to upgrade the most?

Johnathan_GGG: It really depends on what is slow.  I think we need to do some improvements on servicing that information to the player.  Because right now people are very confused about is it -- Just to give you an example of a few things it could be: one thing is stuttering from loading assets that aren't loaded.  That could be what you call a cache miss.  We went to go get a resource and we found, "Hey look, it wasn't actually loaded!"  and we thought it was supposed to be loaded.  Then you have to load off the disk.  Ideally, you would never, ever have to load from disk during gameplay at all.  Or, rather, it would only happen in the background, you would never have to wait for the disk.  Unfortunately, there are cases where, just due to preloading mistakes, or for example there was a video on reddit might have been yesterday where someone was playing the Hall of Grandmasters and had a massive lag spike.  That almost certainly was hitting the disk for an asset that should have been loaded and wasn't.  We've got code that's supposed to load all the stuff that the Grandmasters need, but apparently, it isn't working.  There's obviously a mistake there, and so I'll put an engineer on that and he'll fix it.  We actually do need to fix some stuff Hall of Grandmasters anyway, there are a few issues with loading that need sorted out.
But regardless, that's one type of issue.  That's the one where I think we've made the most strides.  We made a lot of strides on that during the 2.3.0 release cycle.  2.3.0 came out and it wasn't totally -- There were a lot of problems still when 2.3.0 came out.  But after a few patches of 2.3 we mostly got that sorted out, so I don't think that's a huge problem anymore.  The other thing that people often mistake for performance problems is just network lag.  When you're in lockstep, if you have a hiccup to the server in some place between you and the server, that looks like a freeze, because obviously the simulation freezes.  The way you can actually tell that that's like that, that that is a lockstep lag frame rather than another thing, is that all the particles will keep moving.  Well, no, the particles don't move, but the effects on the particles -- So for example, there's kind of these wobbles and other random stuff.

ProjectPT: Your auras rotating around you would be a good sign.

Johnathan_GGG: Yeah, exactly.  Anything that's done on the GPU, any vertex shader effect will keep running.  Basically what it looks like is when we do the pausing in our trailers that we do, where some stuff is moving still, that's what it looks like.  The way that we implement that, by the way, is that we literally just pause, we just freeze the server, and then the lag causes the client to stay frozen like that.  It's literally simulating the exact thing that happens if you get nailed with lag.

LiftingNerdBro: That's creative thinking, man.

ProjectPT: So, Shaper is the lag boss.  Confirmed.

[All laugh]

Johnathan_GGG: Lockstep actually gave us the ability to freeze the simulation.  Actually, that turned out to be really useful for debugging.  When you're running a debug build, you can actually press equals and it will freeze the simulation so you can just see it, and if you go ctrl-equals, then it will move forward frame by frame and you can just inspect things very slowly.  Anyway I have totally gone off what I was --
Oh yes, people mistake that for lag, for framerate issues.  And the thing especially if you've got a juttering performance where you've got a very uneven ping and feel like the game is lagging.  A lot of the reason for that is just because peopole have bad connections via wireless.  A huge percentage of people who approach customer support and say they have this kind of problem, we say, "Can you try with a cable," and they say, "Oh it's fine now."  Because people just have crappy wireless setups in their homes, and that's kind of really unfortunate.

LiftingNerdBro: Sometimes, during the beginning of the league, especially, well only when you're playing with lockstep, it feels like you're having FPS drops.  What's that?  Is that in relation to that?

Johnathan_GGG: As I said, people do feel that that stuff can be a framedrop sometimes.  I don't think it should be related to the beginning of the league.

LiftingNerdBro: I was thinking maybe server overload or something like that.


[35:00]


Johnathan_GGG: With regards to servers, we have a very strict limit of the number of instances that can go on a given server.  A lot of people think it's just like, people pile onto the servers and it just gets slower and slower and slower, but that's not really how it is.  We know exactly how many instances can go onto a server before there are performance problems, and then we load exactly that number on and no more will go on once it hits that number.  If that starts happening, players will start getting the error which is -- I don't actually remember exactly what the text of it is, but basically, you can't join new areas once that happens.  And you've also got the login queue at the start.  It shouldn't ever really be the case that you start to get those kinds of issues.  That being said, there always could be bugs.  Also, performance sometimes regresses, especially when there's a big expansion that changes a lot of tech stuff.  We monitor that stuff and try to fix those performance problems.
That said, I think the next big performance push will actually be serverside, and that's going to fix a lot of the problems that you get with things like, there's that map mod that makes it so everything gets charges like frenzy charges every amount of seconds, that's the kind of thing that has massive -- The reason you get lag there is entirely because the server is recalculating stats on all the monsters on literally the entire level.  And that introduces lag.  This is one of those things where we need to speed up the performance of stat recalculation, and that will fix a pile of things including players when they join; there can be a little lag sometimes when a new player joins the instance, and that's because it's calculating all the stats of the player and that kind of stuff.
And then of course there's the infamous minion stat recalculation problem, once again I can't remember what the unique was called. [ProjectPT: Femur of the Saints.]  Yes.  I believe it was that one.  That's the kind of thing where, once again, stat recalculation is our bane on the server and it causes performance problems on the client.  Before we had lockstep, you wouldn't even notice that kind of thing because it would just be lag that would be absorbed into the -- It would cause desync, but it wouldn't cause obvious lag.  Sometimes we could get away with that.  With lockstep, we can't get away with it.  We need to absolutely process every frame within 33 milliseconds or you're causing lag on the client.

ProjectPT: To give you a little bit of a difficult question because it's related to Femur of the Saints, how do you -- Why do skills like this happen?  You know that's an issue, and when we see like Extra Gore on Blade Vortex and the Femur of the Saints stuff, it's not, to the people who have been paying attention, they know that a high calculations with mob hitting is quite problematic.  Why do these things get designed?

Johnathan_GGG: I can't necessarily talk for the designers, but what I can tell you is that they are always trying to push the boundaries number one.  And number two is when QA tests them, they don't always have their eye on performance as much as they should, and they don't necessarily -- They try to test extreme cases, but sometimes the players just come up with more extreme cases than we can imagine as well.  When yuo're talking about Extra Gore on Blade Vortex --

ProjectPT: Or anything?  [Laughs]

Johnathan_GGG: Extra Gore has been in the game for a very long time.  When Extra Gore was added, we had not nearly the ability to have attack speeds that skills can have today.  Unfortunately, the designers often aren't thinking about things like Extra Gore because it's a microtransaction.  When they're designing a skill, they just want to make the skill as they make it, and they'll make it and test all the gameplay stuff consequences of it.  They're not thinking "Oh yeah, Extra Gore is going to make this terrible."  The correct solution there is for us to just fix things like that from becoming points of causing lag in the first place.  It sucks, I understand, it's a pain.  That's just one of those things -- Honestly, if I could halve everyone's attack and cast speeds, I would totally do that, because everything, everything!, in the game that is bad scales with attack and cast speed.  Server lag -- The number of hits that people can do now per second is just -- If you had told me five years ago how that would have gotten, I would have probably not even architected the system like it is now. [sigh] People have way too fast attack and cast speed.  I sometimes say to Chris, "What if we literally just said, one day, we're going to make the next league, we're just going to have a [monitor?], which is half attack and cast speed" and just see what happens.  I don't think we could do that because players would be revolted.

ProjectPT: Chilled ground temp chain league, guys.  It's coming.


[40:00]


Johnathan_GGG: It's one of those things -- As a technical director, it's my job to take what the designers make and make it work.  We have been getting these things better.  Blade Vortex was out of control.  Blade Vortex was actually out of control, so they had to change it to have less stacks.  The case of the way that we changed, what do you call them, Cast on X skills.  [Group trying to figure out the names.]  The things you could control the speed of.  That stuff was motivated mostly by performance.  Creating so many skills in one frame -- Normally a player can only do one skill per frame.  That's an actual rule.  The server runs at 30 frames per second and you can only do 30 skills per second.  Though if you have multistrike and stuff, that can happen in one frame.  One actual invocation of a skill per server frame.  And so, having the ability to create so many player skills in one frame with Cast on Crit was just a nightmare.  It was causing massive server-side lag and all that sort of stuff.  It was just a huge pain, so that was a huge part of the motivation to changing the way Cast on Crit worked so that it spreads it out over several frames, so that you don't have so much crap happening at one time.  That was what was going on there.

HegemonyTV: There seems to be a lot of choke, and I don't know if it's the network problem, but when you encounter the undying mod and you don't manage to kill it and you just attack into it let's say a slayer who has AoE who's constantly recalculating --

Johnathan_GGG: Right.  Those are the kinds of things we're going to do.  I think the next big performance push is going to be serverside performance.  I'll stick Vincent on that after we're done with the current batch.  After the DX11 client is out, I think we'll start looking at that a lot more seriously.  It will help a lot.

HegemonyTV: In regards to the CPU that you were mentioning earlier, do you guys actually collect hardware data to see what kinds of players with what kinds of hardware --

Johnathan_GGG: We actually did start doing that fairly recently.  I think it was probably two months ago.  I honestly haven't looked at the data too much since then.  We added it because we wanted to see what the effects of the multithreading was, so it must have been a bit longer than that actually.  It was added around the same time as we added the multithreading, because we wanted to see the difference between people who had it on versus off, and basically collect some data around that, what the framerates were and so on.  We also added collection for what CPU and GPU you have, in order to -- The main reason we do that is because we want to see -- We only want to know -- To compare results, we need to know does the person have the same hardware?  What we want to do is go like, "This guy with this graphics card and this CPU had this thing, and this guy with the same GPU and CPU had this performance."  The annoying part is that there's actually a huge amount of variability just from people's -- Things like just having a virus checker installed is something that can randomly kill performance, or God forbid you have multiple virus checkers installed and they're fighting with each other, that's the kind of thing you get, and that's when you start to get really bad performance.
I don't understand how there are people who seem to be able to have multiple-second long -- You go to the disk and it takes multiple seconds to get a single file off the disk.  I don't understand how someone can have a system that can possibly be that slow.  While still having a very fast graphics card and CPU.  I don't understand it sometimes.  I sometimes cry at the ridiculous performance problems some people seem to be able to have.  I just don't understand it.

ProjectPT: My last major death, I want to point out, I have a solid state drive, GTX 970, and I think I had a good four-sceond freeze against a warband pack with a Beacon mob.

Johnathan_GGG: Okay, but was that server-side or client-side though?  Because that's --

ProjectPT: I don't know!

Johnathan_GGG: Exactly.  But that's the thing, right.  What I want to do is add stuff to the client so that it will tell you what type of freeze you are having right now.  For example, I want an icon of a disk that flashes up if we're waiting on disk and I want an icon of a network socket if we're waiting on network.

ProjectPT: I would really like that.

Johnathan_GGG: Exactly.  I've been meaning to do it for a long time and it really wouldn't be that hard, it's just been gone by the wayside unfortunately.  I really do want to add that, and that would mean that -- Also once we do that, then we can add that information how long people have freezes for to the telemetry we're taking from the client and that means we can start to think about, "Are there specific areas that people are getting disk freezes on?  Are there specific -- Or never freeze."  That kind of stuff.  Just getting more data about what causes it, and then we can start to work more on --

LiftingNerdBro: You basically have a system that says, "Guys, it's your shitty system that's the cause of this.  This is not our fault."


[45:00]


Johnathan_GGG: If everyone gets the same problem on the same area, then we know this is on us.  Whereas if it's just a general background radiation, then it's like -- Or if it's just one guy who gets it all the time, or certain people --

ProjectPT: It saves you diagnosis time.  You don't have to spend your time looking, confirming that it's your issue.

Johnathan_GGG: I would like to get more telemetry there, but it's always hard.  With a company this size -- I set the DX11 port rolling and that takes a long time to filter out of the system before we can start working on these systems.  I should mention as well with the DX11 by the way is, one thing we've really noticed is that there's way less mysterious graphics driver freezes with DX11, and our mysterious graphics driver freezes aren't long, they're like on the order of five milliseconds, but they cause the stuttering framerate that really suck, especially when entering -- The thing we noticed was shaders.  Every time you load a shader, especially on ATI cards, you just get mysterious freezes of about five milliseconds.  And if your frametime is like 15 milliseconds, where 16 milliseconds is the cutoff for 60 FPS, and you have a five millisecond freeze, you're dropping down to 30 just like microstutters.  This microstuttering problem seems to have gone away with DX11, which I'm really happy about.  I think DX11 is really going to help with that, too.  I think people are really going to be happy with that.

HegemonyTV: I wanted to ask this question, and it's mainly regarding the network code and serverside controls.  A lot of hardcore players, they're willing to deal with a lot of the imperfections with the crashing and the freezing -- Are there any issues with giving players control what they're disconnect time to the server; you know, when you're no longer sending packets.  Currently, we've tested it, and it's like eight seconds.

Johnathan_GGG: Yeah, I understand.  I think it's actually six, but I could be wrong.  In any case, let me tell you the problem with this.  If we give you the ability to set that really low, then all the streamers will do it, and all the streamers will tell the players "Hey you should do this," and then all the players will do it even though their connections aren't up to it, and then they'll continuously get disconnected, and then they'll bitch at us.  This is honestly the problem.  It entirely comes down to player psychology.  If you give someone a lever, and all the hardcore people, all the highest level people are pulling that lever, then they will influence the community to all pull it as well.

ProjectPT: Are you now placing the blame on us, the streamers?

Johnathan_GGG: What I'm saying is, as much as anyone would say, "Hey set this lower if your connection is good enough that you don't have dropped packets" and stuff like that, people just will not listen.  They'll be like, "Ah, this guy does it, so why shouldn't I be able to do it?"  That kind of thing.  It'll just increase the number of disconnections significantly.  That's kind of a thing that sucks.  [sigh]  I can understand the frustration, and we've talked in the past like maybe on hardcore we should lower the time and that kind of stuff.  Chris, I know, has a somewhat of a D2 style attitude to this, which is, "Oh, in D2, if you're playing hardcore, just make sure you never put yourself in a situation where six seconds of lag is going to kill you."

ProjectPT: That's called "offline mode."

Johnathan_GGG: That's his attitude about this, which I know isn't particularly satisfactory.  I'll tell you what, Chris on D2 -- The dirty secret about Chris playing hardcore in D2 is that he never actually did any damage.  [All laugh]  He'd stand there in a party and basically just be leeching.  He looks like he's attacking, but he's specced entirely into life and not damage at all, and he's just leeching from everybody.  Chris was the most paranoid person in hardcore.  He specs defensive.

HegemonyTV: What's funny is I asked Chris when he was playing hardcore last league to link me his gear.  He linked me his gear, he was using like a one-handed mace, and he was playing cyclone sword and board.  I was like, "Oh my God, he must be doing no damage!"

Johnathan_GGG: He only cares about defensiveness.  He's ultimately paranoid about dying.

LiftingNerdBro: In a bit, Johnathan is mysteriously going to disappear from this podcast.  [Laughs]

ProjectPT: You were talking a little bit earlier about some of the shortcuts, not the shortcuts, the attack speed in the current game and how you would have done things differently in your original design.  So what are some of these shortcuts or the design implementations that you did early on that, let's say, you didn't necessarily regret, but you didn't expect the game to be where it is and that's causing some massive problems?


[50:00]


Johnathan_GGG: I would definitely say that most of those problems are just caused by things scaling to a level we never really expected.  Attack speed is a really great example of that.  If I was starting again, I definitely would put way more controls on getting up attack and cast speed.  One of the things that really annoys me about it is, well, it looks really dumb when you see a high level person playing, [All laugh] just to someone who doesn't play Path of Exile.  If you show a new player "Oh, this is Path of Exile" and you're starting there with 30 attacks a second boom boom boom boom boom, you're like, "This doesn't look reasonable."  We have this problem on press tours as well, where we want to show journalists.  They're not going to understand these crazy -- When we were doing the Atlas, I was like, "I need builds that don't have ridiculous attack speed" in order to actually show journalists, because it looks ridiculous.

ProjectPT: So that's why Earthquake is the strongest melee skill!  Because it doesn't do the cha-cha when it attacks.  [All laugh]

Johnathan_GGG: It is true that we used Earthquake in the press tour videos a lot because it was an in-game build that didn't rely on huge attack speed.  You still have pretty fast attack speed, but it wasn't crazy like it can be with some of those builds.  It is annoying.  There are things that are probably -- There were moments in the past where we had forks in the road.  Obviously, one of the big things is the skill tree.  I persnoally really love the fact that giant skill tree that's really awesome.  But it does do, to some extent, scare off a few players.  That is an unfortunate thing.  It annoys me a little bit because I think that the scariness of the tree is not really real.  It's more perceived than it is actual, because yeah, it's really complicated in that you can make really complicated builds, but you don't have to do that to play your first character.  Just play and look at stuff!  People have this mindset, "If I see a tree like that, I absolutely have to look up a build, I'm not going to trust myself."  I really wish people weren't like that.
We actually had this idea when we were still in closed Beta I think we were thinking we could change the tree so that it isn't all available initially.  We were thinking maybe so that it's locked and you have to unlock parts of it.  [HegemonyTV: Kind of like the Atlas.]  Yeah, kind of like that.  So this would be a thing where in order to play a certain way you might need to do some various -- I think initially we were thinking, "What if there were achievements of some kind where, if you unlock this achievement it unlocks this part of the tree," so each part of the tree has a thing in it like, "Do this thing, this goal and it unlocks this part of the tree" or whatever.  The idea was, for new players, you wouldn't be able to see the whole tree, so it would be less scary at the start.  What you would see is just the nearest goals that you had to unlock more of it.  But then we thought, the problem is, players will just go online and they'll see what the whole tree is, and they'll still want to do the build, only now they won't be able to do the build they want to do because now they have to do some added challenges they don't want to do.  That was the reason we didn't go that route.  That's the kind of thing where looking back, that might have been kind of cool, maybe the whole thing would have played out.  I don't know, it's still an interesting idea.  Obviously, we can't change it now.  Obviously, we can't restrict the tree right now for people.

ProjectPT: Well, if you want everybody to leave.

Johnathan_GGG: Yeah, I know, exactly.  I'm not saying it was necessarily a good idea, it's just another interesting story about stuff that we thought about in the past.

ProjectPT: You had a question Lifting?

LiftingNerdBro: I just wanted to follow up on that.  Because of that, you mentioned stuff that you've been wanting to do.  Do you feel like your programming team -- Does it feel understaffed in that department, in that regard?  Is there stuff you'd really like to do, but you can't do, because of the size of your team?


[55:00]


Johnathan_GGG: Most of those things are not really gameplay things.  They're more technical fixing.  We've talked a pile about things that I would like to do to fix in terms of performance stuff and that's all just manpower.  I don't necessarily think that just adding more programmers would fix that, though.  The problem is, every programmer you add -- If I just added ten programmers immediately, then it would be a nightmare because they all wouldn't know the codebase.  I'd have all these people training.  When someone's a new programmer, they have to come to me more often, so I'd be sitting there trying to train people when I don't have time to do that.  We need to add programmers slowly, they need to get onboard with the codebase and understand things work and so on, and then after about six months, maybe even a bit longer than that, that's the point where they actually know enough that they don't need much help.  You really need to get programmers onboard slowly.  When GameLoft folded in New Zealand, we hired like a bunch of programmers all at once, so that was really tricky to try to get everyone integrated.  Now we've kind of got that, now they're all kind of in very well, and everything is going very smoothly, but that kind of thing is really tough.  That's why we can't say, "Let's hire ten more programmers, it'll fix everything!"  They won't be able to fix stuff without help.  You can't just get a programmer off the street and say, "Hey go fix serverside lockstep lag."  They'll go like, "What?"  They won't understand, they need a lot of help and direction.
As I said, the big things are technical.  I think in terms of gameplay, we mostly -- We design an expansion and we design all the gameplay stuff that needs to happen in that expansion, and then that stuff, because that is what the expansion is according to the players, that stuff takes more priority, the game systems.  I think that we pretty much always deliver the gameplay code that we wanted.  We've got four gameplay programmers -- I think it's four, I'm pretty sure it's four.  They're fulltime just adding stats and stuff like that.  There's a huge amount of stuff that gets added.

HegemonyTV: Mark is the lead of that team, right?

Johnathan_GGG: Yes he is.  Mark one.

HegemonyTV: With regards to expansions, we're well aware Act 5 is coming along.  Are you guys planning on running a prolonged Beta like you did with Act 4?

Johnathan_GGG: Absolutely.  And the reason why --

ProjectPT: No!

[All laugh]

Johnathan_GGG: I know you feel -- Chris told me beforehand, "He's going to care about the prolonged Beta thing."  Here's the thing.  Honestly, 3.0.0 is so crazy.  The amount of shit that's going on in 3.0.0, I just like, I'm not -- Obviously I can't say anything about it, but man, the amount of content and stuff that's happening there is very high, and we just couldn't do it without a Beta.  We've already talked about how Cruel is not a thing.

ProjectPT: We know about that.  You wanna add anything about that?

Johnathan_GGG: I don't want to add anything about that.

ProjectPT: Are you sure?

[All laugh]

Johnathan_GGG: Absolutely sure.  Even just something like that.  When you screw around with the leveling curve, it just screws up the balance in ways that are really hard to predict.  Just from a balance perspective alone.  But there's tons of [sigh]  There's a lot of re-laying out of everything that has to happen in order to add an act and remove a difficulty and stuff like that, so that's a pain.  There's lots of other features and stuff that are going to be a big.  The other thing as well is, one thing that would be crappy if we did deploy Act 5 right into the production realm, people would finish it in Standard with a level 100 character in like 15 minutes or something like that.  People will rush through and go like, "Oh yeah, was that Act 5?  It was shitty."  Seriously.  Someone would do it immediately.  We were able to avoid that with the Shaper fight because it was actually just hard enough that at least took a couple of days before someone --

ProjectPT: It took two days!  Well done guys.  [Applause]

Johnathan_GGG: Exactly.  That is a win.  That is actually a win.  That's Act 5.  You could just go to Normal Act 5, and playthrough it with a level 100 character in 15 minutes.  [Laughs]  That's not the experience that I want to see on stream on day one.  That is one advantage of doing a Beta, is at least we get to see people play through the content properly.

ProjectPT: You could piss everybody off, and make it so that you disable the permanent leagues for a month.  [Laughs]

Johnathan_GGG: Well, that's actually, we seriously talked about -- When we were discussing the Act 4 launch, we seriously considered that as an option.

[All laugh]

LiftingNerdBro: I would just love to see the ramifications of that.

Johnathan_GGG: Yeah, well, not -- I shouldn't say "seriously," they were never really serious.  [All laugh]  But we actually did discuss that as an idea.  Anyway.

ProjectPT: What is something you learned from the 2.0 Act 4 launch with the Beta that's going to make the process smoother?  I'm going to hate the Beta no matter what.  I can live with that.  Is it going to be a smoother process?


[1:00:00]


Johnathan_GGG: I suppose that depends on your perspective.  One thing I want to definitely do is add more integration between the Beta and non-Beta realms, so more of your -- How do I put it, more things that you want shared between the two realms are shared.  That's one thing.  The second thing is, people had problems with services being unavailable in the Beta.  Certain things like is there something you can do about trade. [HegemonyTV: And microtransactions.]  Exactly, and microtransactions.  Stuff like that.  I think that we'll do better next time.  Adding more realm integration, so that they're not quite so separate.  We can make it so that -- Put it this way, wouldn't it be nice if the items in the Beta realm just showed up as a different league according to the public item tracker?

All: Yeah, definitely.

Johnathan_GGG: That's what I'm talking about.  Stuff like that.  Just making sure that you're not missing out on stuff like that.  That sort of stuff I think we'll be able to do better, and reason it was like that on the other realm was that they really were separate realms with a terrible hack in order to get the accounts to be the same.  I think we'll be able to do better next time.

LiftingNerdBro: Did you mention if it was going to be a closed or open Beta?

Johnathan_GGG: Closed.  [LiftingNerdBro: Closed, okay.]  Thank you for that.  The feedback we get on closed Betas is much better.

LiftingNerdBro: Are we talking like in the same duration as the one we saw for -- ?

Johnathan_GGG: I can't predict a duration.  I know we're hoping for a bit shorter, but ultimately the reason the Beta for Act 4 kept going for so long was the game wasn't finished.  We had to finish the Malachai fight and all that stuff at the end there.  Honestly, I hope everything is completely finished by the time we hit into the Beta.  Reality is that that's never the case; I'm sure there will be stuff that isn't done, so it's going to depend a little on that.  We have a bigger team now, we can produce content way faster, this expansion will be a much bigger expansion than 2.0.0 was.  And it will be much more finished.  Consider the teamsize.  Right now, we have 90 people.  The teamsize when we launched 2.0.0 was, how much was it, 50 maybe?  I don't even remember.  Lots more.  Just consider that.  There's a lot more going in.

ProjectPT: I did want to say we have reached about the hour mark that is all we had promised of your time.  Did you want to do a little bit more?

Johnathan_GGG: I can do a little more, sure, why not.

ProjectPT: I'm going to throw you a wild card and see if you're willing to ask this -- Do you do anything in terms of backend with the microtransactions stuff?

Johnathan_GGG: I did originally do the backend for microtransactions, and I will say I am planning on completely replacing the way that microtransactions stash works.  That is a project that I have.  It's not going to be soon, I'm going to warn you there.  It's probably going to be next year.  I'm hoping quarter one.  But once again, this isn't even started, I can't make any promises about timeline.  I understand, the microtransaction stash is shitty [Laugh].  It's because we never really thought -- Once again, it's a scaling thing.  We never really realized the amount of microtransactions people would get and just the way things would evolve and so on.  The microtransaction stash was completely fine when all you have is one tab, right.  It's not a problem in that situation.
Modern games, when you buy things like mystery boxes and stuff like that, they all have way more stuff going on with the shop.  You know, "Whee!" and then the chest open up, and you do animations and stuff.  We really need to get with the times and get all that kind of stuff going, just the experience of visiting a shop.  It's kind of ridiculous that the things that we make our money from are the things that we spend the least time working on it.

All: It is a bit ridiculous, it's kind of weird.

Johnathan_GGG: Look, that's honestly because we prioritize gameplay, and that's always been our company's thing, is we always prioritize the game over the monetization model.  So it's a good thing you guys are actually paying us money because that actually does keep us alive.

ProjectPT: The reason I brought it up -- It was an interesting point when you talked about that they were different servers or different instances entirely with no connection when you had the Beta.  One thing that was weird to me is you have new people playing the Beta, and they can't have the same --


[1:05:00]


Johnathan_GGG: Yeah, that was a terrible thing.  We basically said we don't have a way of doing this.  The reason why, is because microtransactions are currently stored in the items database.  They're basically requested at the same time when you get all of the items in your stashes and stuff like that.  They're just a stash full of items, except that you can't move them around and stuff.  What that means is, because the items database was completely separated in the Beta, basically all of those tables to do with character data were all separated out, the microtransactions couldn't be shared between them unfortunately.  That was a thing.  The new model for microtransactions, and I really shouldn't talk too much about this, because, once again, we don't want to promise anything we don't actually implement, but my ideal model is where microtransactions won't really be items the same way other items are, they'll be like a thing on your account; you have this, and then you can apply it to an item.  We might even get to the point where you apply it to your body armour slot rather than to an item, and that way when you change body armours, the microtransaction is still in the slot. [ProjectPT: Holy shit, I would love that.] It would be a lot better.  Once again, this is far away.

ProjectPT: This is just spitballing ideas.  No promises are being made.

Johnathan_GGG: The soonest that would happen would be 3.0.0.  Maybe it'll happen then.  It's definitely something I want to fix, I'll put it that way.

ProjectPT: I'm waiting to hear Chris yelling at you now at the end of this.  [Laughs]

Johnathan_GGG: I'm not in his office, so he can't hear me.

ProjectPT: He'll find out in a few hours.

LiftingNerdBro: He's furiously driving there now.

Johnathan_GGG: He knows.  We are planning on doing that.  I'm just hoping we'll get time before 3.0.0 releases.

ProjectPT: How do you do a priority -- Sorry, Hege, go for it.

HegemonyTV: I was going to ask about the priority -- How do you determine and decide what power gets put where, with the DX11 coming out, once that's done, what's next on the docket?  And does stuff like this, like the overhauls and just overall enhancements of the game, does that push back really janky ideas of content or new systems?

Johnathan_GGG: I would say we have different programmers working on different things.  We have three programmers working on the engine, and they're unlikely to be moving over to gameplay once they're done.  So I expect it's kind of gonna be a case of them continually working on improvements and so on.  People definitely have special areas; the gameplay programmers work on gameplay, and the graphics programmers work on graphics stuff.  I think you'll start to see, and we already have some of the stuff in progress but I'm not going to talk about it, you'll start to see some pretty cool engine tick features after the performance stuff has been dealt with.  We're going to have soem graphics programmers lying around at that point who we can use.  I want to really start work on -- There's some really basic stuff we could add in, features that you can just turn on like SSAO and maybe better lighting things, cool stuff like that we could work on; you'll start to see stuff like that I suspect.  Those programmers just don't know about gameplay and teaching them about gameplay would take another six months again.  [ProjectPT: It's a specialization.]  It really is a specialization.  Learning about how -- It really does take six months for someone to be really good at working on gameplay code because there are so many moving parts, so many stats, and some of the skills, there's so many items, it's just crazy.  And once again, working on graphics, it takes quite a long time as well to get familiar with everything, and just understanding stuff like that.  The programmers don't move around between specializations too much, and so because of that I wouldn't say that one thing is delaying another thing, it's more a question of my time training them in a certain area.  Once I've trained them to be a graphics programmer, they're probably going to be a graphics programmer for a long time.

ProjectPT: Do you want to mention anything about a particle slider?  I'm sure you've heard about this.

Johnathan_GGG: People talk about that kind of thing.  It's not ideal, and I also don't -- You can't just turn off a few percent of the particles and expect a thing to look the same as it did before, or even vaguely the same.  I think honestly that particle simulation is not nearly as much of a problem as it was before, now that we've optimized that a shit ton.  I haven't really heard people calling for that as much recently as I did before.  After DX11 comes out, I think we'll see what people say.  It's not ideal, let's just say that.  I actually really dislike -- This is another one of those things where, if people -- This is like shadows.  All streamers turn shadows off and it makes the game look really fucking ugly, and it really annoys me and that's the kind of thing that's like, "Okay --"


[1:10:00]


ProjectPT: I actually recently turned shadows on after not playing with it, and the shadows are amazing when I went through the labyrinth.  I was like, "Holy shit!  This is good!"

Johnathan_GGG: Way better, right?  Seriously!  [sigh]  There's this whole RuneScape mode thing.

ProjectPT: Yep.  I've done that.  When we had flashback league, I just RuneScaped potato-mode claymation land.

Johnathan_GGG: The thing that really annoys me about that is that beyond maybe one level below what we let you set it to in the UI, maybe you can make an argument for performance for that one, but anything below that is completely irrelevant.  It does not change anything about performance.  The only thing that lowering it does is reduce the amount of GPU RAM you're consuming.  And every time you lower it another level, it cuts off 50 percent of the RAM.  So you halve it, then you halve it again -- You halve it down to Medium, then you halve it down to Low, the equivalent of whatever Low would be that we don't expose because we don't feel like it's really necessary.  After that point, halving it again is doing nothing for you.  [HegemonyTV: Diminishing returns.]  Right.  You're going from like 200 meg of GPU RAM consumed to 100 and it's like, okay, well you've got like two gigs in your graphics card probably, so what are you even achieving here?  You're achieving nothing.  And it honestly doesn't affect performance in any way because GPU bandwidth is already going to be really low at that point.  It just trolls me to say it.  I would be tempted to prevent people from doing it, if it wasn't for the fact that people would get needlessly annoyed; it's just going to anger people with no real benefit to me

[All laugh]

ProjectPT: We're not programmers.  I've set mine to one desperately trying to get any type of performance going into potato mode.

Johnathan_GGG: It honestly does nothing.  You can maybe make an argument, although I honestly don't think it's going to help going one down from where we have it, beyond that it's just a total wash.  It just doesn't work.

ProjectPT: It is a situation where the answer we've gotten from GGG before is you want to solve the problems, and particle sliders would be this last-ditch effort.

Johnathan_GGG: Right, exactly.  It's not my preferred solution.  Once you've added that, you can't take it away later.  I would much prefer be in the situation where you get the graphical fidelity without having to make any sacrifices.  Imagine if we had added that before now, then all these people would have set it to the lowest setting possible, and then now their performance would be for many people totally fine now to just have it [all out?], but because they're used to having it at a lower level, they would just have it low.  That's one of those things where it's really shitty to set people's expectations too low.  That's the annoying thing about it.

HegemonyTV: It's gotta be painful to see your creation and people are playing it in toaster mode.  People are like, "What game do you make?"  "Oh, this game that someone's playing with Minesweeper."  [ProjectPT: RuneScape!] [Laughs] RuneScape.

LiftingNerdBro: Alright guys.  I don't think we should take too much more time of Johnathan's time.  Do you have anything else you want to add here near the end, Johnathan?

Johnathan_GGG: Eh, not really.  It was great being on the podcast.

LiftingNerdBro: We appreciate it.

ProjectPT: I did have a quick question, a fun question.  Do you talk this fast in real life, or are you just slightly nervous or excited?  Because --

Johnathan_GGG: No, I think I normally talk this fast.  Sorry, I didn't even realize I was talking fast.  It just is my normal pace.

LiftingNerdBro: You're doing really good, you're a natural.

Johnathan_GGG: Hopefully I didn't have too many um's and ah's, that kind of thing.

ProjectPT: Uhhhh, don't listen to the podcast again.

Johnathan_GGG: I cannot listen to myself in press interviews.

ProjectPT: Then you won't notice it, and you're perfectly fine.  Don't worry too much [Laughs]

LiftingNerdBro: You won't become a YouTuber then.  Thank you very much for joining up, Johnathan.  Very enlightening.  Thank you very much, we appreciate that.
If you want to see the next podcast on Sunday, make sure you follow the channel.  If you didn't catch the entire podcast, it will be uploaded to Hege's YouTube, you can find the VoD there.  Once again, thank you very much for joining us.

All: That was fun.