Tackling the Mighty PDF with Anvil Transcription

M2: Hello, everyone, my name is Ben. I'm here today to talk to you about tackling the mighty EPRDF, so we'll start with a little bit of context about the PDF and then we'll get into some code stuff. Let's talk about the challenges of working with kids and some of the challenges that we faced. And then I'll demo some things that hopefully make working with PDF a lot easier. So co-founder, CTO of Anvil, a little bit of background. I have been a developer for a long time. Previous to Anvil. I worked at GitHub on the Adam Ed and before that I founded a design tool startup. So let's get started. So the PDF kind of needs no introduction, but I want to give a little bit of context before we dove into the code things. So. So you've done this before. You've had an email from somebody that with a PDF that's asking for like all of your most personal information. And so then you fill that out with like a random app and you email it back and you weren't sure exactly where your data was going to end up. So I know probably not a lot of people like doing this. I sure don't. But it's it's super common. Turns out most email attachments are a PDF like most non cat picture email attachment or PDF, and they're everywhere. If you had to guess how many PDF were out there, what would you guess? Let's say would you say like a billion? Ha ha. Would that be exaggerating? Turns out that there's two and a half trillion PDF. That's not necessarily unique PDF. But but it's a lot of it's a lot of PDF. There's 20 billion just in Dropbox. So there are some really good uses of PEDs aside from forms. So like resume's e-books, House plans we submit to these talks, slide's via a PDF academic papers. And one of my favorite uses that I just I just learned about is fashion specs. So all of these.
M7: All of these have a couple of things in common, so what do they have in common? They are all for consumption and sharing.
M2: So kids are really good at four four things that you don't need to edit. You're not editing the PDF. You are not you're not filling some something out in the PDF. All you're doing is you're reading them. So I think we can we can say that pets are really good for screens and for printers, and this makes a lot of sense because this is what kids were originally built for. They were built to display information exactly the same across all devices and also be shareable. So that way you can send your underwear PDF to the underwear factory and get them made exactly like you want them to be, because you don't have to worry about them having the font or not, it'll look exactly the same as you in exactly the way that you intended. So but the most common way that people have interacted with with kids, certainly for me is, is by filling out forms. Forms are really annoying for a couple of reasons. One is that there's a lot of repetition, so you have to fill out your name six times or fill out several CDs with the exact same information. There's also a lot of kids that have logic, so you have to read and understand part of this PDF and then make decisions about your situation, so. This is this is a four one K tax form, so if you want to take money out of your for K, there are tax implications, but it depends on your situation. So this is obviously hyper complicated, right. It turns out that humans are really bad at this. They're really bad at filling out a filling out PDF, so professional data entry persons error rate is four percent, but I know that my error rate is a lot higher unless I'm really, really paying attention. And we've heard from our customers and potential customers that their customers make a lot of mistakes filling out CDs. So perhaps a really good four screens and four printers, but maybe not so great for humans because filling out filling out these forms is is is pretty cumbersome. OK, so what would be better than having humans do it? I mean, why are we all here? Computers obviously would be a lot better than having a human fill it out. So computers are really good at logic and repetition. So it makes sense that that they could they could sort of make these decisions. We could ask the user, what's your name and your birthday in your address? And then we can basically with a couple of pieces of information, fill out all of those all of those checklist boxes and fill out the right the right fields. So this is what we want. We don't want people to be interacting with these with these forms. So so how do we abstract away the PDF? We can't get rid of the PDF because there's two and a half billion of them and behavior change is is really hard, but I think that we can make it so that no human has to deal with Wikipedia's. All right, so I guess the question is, how do we interact with kids as developers? Do we go and read the spec? Probably not. I don't think anyone's going to do that. I certainly haven't read all the spec or the e-mails back and I write a fair amount of e-mail. So it's huge. It's complicated. What what do we do? We go, we go and we find a library on the Internet. So when we were starting and this is exactly what we did, is we went to the Internet and we're like PDF filling libraries and what we learned is that developer attention to a technology is important. So some technologies have a lot of attention from developers to take like Web technologies.
M3: It was created sort of for the same reasons as PDF for display and consumption and that kind of thing, for sharing academic papers, but since then it's had like insane developer adoption. There's literally millions of developers building things like for the Web on the Web. They're building libraries. There's people complaining and issues. There's people fixing those bugs. And what all of this all of this collaboration results in is a bunch of really well maintained high level libraries. They have good docs. They they're well, they're updated all the time. And so in this world, you're never interacting with the DOM APIs. You're interacting with like react and sass and type scripts and all of these all of these cool technologies that people have have built on top of Webjet. So as a developer, I've become super accustomed to this kind of thing. Like I got a problem. I can go to the GitHub and I can find I can find a library that does mostly what I want. But PDF tech isn't really there yet compared to web development. It's it's there's there's very few developers looking at so like every human has to deal with PDF, but there aren't so many developers looking at this. There are libraries, though, and they generally fall into two categories, they're complete or incomplete. And so the complete libraries can do everything that you would maybe want to do with a PDF. You can read them, you can write them, you can update them. And on the incomplete side, they usually just do like one tiny thing. And often these incomplete libraries aren't well maintained that haven't been touched in five years. You can also run Headless Chrome to generate PDF, but you can't update a PDF with with Chrome. So the complete side seems really good, right? This is the route that we went down, but like the dorm epis there, they're pretty low level, like like the dumps. So you have to know a lot about the path and how it's put together to to read at it, create them. So so if you're saying we want to do something simple, like something seemingly simple, like creating an invoice that can lead to like a lot of overhead to set a whole thing up to create this invoice. So now kind of into the code part of things, if we are working with low level APIs, we have to know a lot about these PDF details. What are some of the challenges? So I'll take you through some of the challenges that we face when we were when we were building our PDF service. And so these are some of the smaller challenges. So no one is layout, so if you want to create a new PDF or you want to add something to a PDF, you have to deal with with layout. Everything in a PDF is absolutely positioned, fixed sizing and there's no text wrapping. So there's not a lot of niceties for a content creator. So this logic that that would wrap things and whatever is like pushed up to the sort of application layer. So coming from HTML, this might be a little weird e-mails declarative. So this is like here's what we want to display and then the browser will make decisions based on based on that, it will be like, well, how wide is the screen? Like, let me shrink it down and wrap stuff. You might, you might you can also give the browser guidelines with success by saying, OK, I want to use flex blocks and I want this column to be this wide. But the browser might ignore you.
M1: The browser is making these decisions on how to on how to display things. So the PDF is opposite that where you are specifying exactly how to draw the document. There's a bunch of little drawing commands in here saying, OK, I want to start text, I want to pick this font, I want to move the cursor here and then I want to paint the text from that cursor position. So e-mail will automatically wrap stuff, but if we want to do that, we have to split up the lines manually.
M6: So here is an example where where we did that, we you measure the box, the bounding box, and then you measure each character and then you figure out how many words will fit into that and you break it into lines and then you have to paint all the lines yourself.
M1: So this was a challenge for us. There are some libraries you could put on top of some of the libraries, but but it was it was a challenge.
M3: So the second thing I want to talk about was Font's, it seems like it should be really simple, but but with PDF or not. And I think one of the reasons is because kids are supposed to be self-contained and there weren't really built to be edited. So there's no fallback font, the font glyphs like the specific characters to render that in that font, to render a character in that font is embedded inside of the PDF and you can't mix fonts in a text block. You can't say like, here's a sentence bobe this word. You have to, like, draw those things independently. So it is going back to that this is this is a lot simpler, right? Like you just specify a whole bunch of fonts in the browser will figure it out. And if we use a character that our font doesn't support the browser, we'll find a font that does have a character and display it.
M1: So this is magic. But fonsi, the PDF are a little bit more complicated. So there's a lot going on in this slide.
M4: First of all, my I picked this phone call called Notar Noto from Google and it doesn't have the character, so I can't use it. It won't render in the PDF. And the second thing is that I'm drawing the regular and the bold word independently. And on the right there, we're embedding all of the glyphs for that specific for those for those specific words.
M1: So from the regular, we're getting the yellow and then from the bold, we're getting all of the world characters. So if we were to say, use a lowercase H, then, then it wouldn't display because this doesn't have that font. And so if you're generating a new PDF, you have to be really careful about what characters are coming in because it has to be in the font. So that was a challenge.
M6: The next thing is rotation's, so you look at this and you're like, that seems fine, right? Seems fine, but then you go to print it and it's rotated. OK, cool. So this happens a lot with scan documents. They'll scan them upside down and then something in the scanner software will set a rotation on the pages to be one eighty. So it looks to the user like it's like it's not rotated, but under the hood it is. And so this is hard because if you want to read the coordinates of something or you want to add something back into the PDF, you need to sort of back out all of these rotations and figure out what exactly is happening so that you can orient your thing that way. So it looks right to the user. So. So, like, you have to know if there's rotation's on the page. There's this thing called the content stream. It can be rotated, there can be rotations on foreign fields. And so this is a challenge to sort of work out like, OK, we're going to add this thing into a content stream, has the page, has the content stream, and then we have to sort of orient our coordinate system to try to match that.
M1: OK, so that's just a couple of the sort of challenges that we that we ran into. There's a whole bunch of other things that were that were hard as well, like dealing with forms can be complicated depending on how you want to interact with them. Signatures are like their own talk, their crazy town. And then extracting content can also be hard because because of the absolute positioning. So the last line on a page could actually be first in the file. So extracting content can be can be difficult. There's a whole bunch of other things to that that can be hard. So. All right, so we've established that perhaps are good for screens and printers, not so great for humans because it's cumbersome, error prone to fill out PDF, fill out forms, and it's also pretty hard for developers to set up and maintain something even for simple, simple use case. So so we're like, can we fix this? If a computer fills out a PDF. Then the Phillips is just consumable. It's no different than, say, an ebook or a resume or something like that, just a piece of information that you can reference later. So then it becomes good for humans. So the last piece of this then is trying to make it so that developers can fill out all these PDAF so that humans don't have to deal with it. And so that's what we've been working on, we've been working on and we released recently is this PDF API that does E signatures PDF filling in PDF generation with Mark on right now. So our goal is to make these things super easy to use. OK, so now I'll get into the demo, hopefully we're all good.
M8: Oh, I see. OK, I hope everyone can see that.
M7: So I'm going to take you through the three different things that we that we do. So we have a generate PDF. What we're going to do is going to generate a brand new PDF and generate this invoice. And so we are able to send a Jason Pela to a single endpoint and it will respond with the the PDF bytes. So we're going to be using our Noad client, which is just for off help, it has has come off of authentication helpers and then also some helpers for actually calling the endpoints. We're going to build up our data here, and then all we're going to do is we create the client and send and then generate the PDF with our data. So each one of these is like each one of these objects is a section on the PDF and they're separated by a little bit of padding here. They support markdown. And then we have some helpers for creating tables because that's a common use case. So this thing's already generated. So I'll show you a little bit. So we'll add some more stuff here. We'll add Will Ipsum. We will capitalize this thing, let's make this left aligned.
M5: Right, sweet, so.
M7: And then we generate it and it magically works, and so we just automatically get markdown support here and table support and all that. So that's it for generating APIs. OK, so. Moving on to the filling APIs, it's similar to the API, except that we have a template, so we have like an existing. We have an existing template that we have already created and uploaded into our system, and the boxes were found when we uploaded it. We did. And so each one of these boxes has a has an ID on it. So we are able to send data to each one of these IDs and it will fill out the template so we can generate this W4.
M8: I'll pick the other. Terminal.
M7: OK, so this thing works really similar to the first generation, how about the. So we got it, we filled it out awesome, it works really similar to the generation endpoint where we just have a palette of data and then we create the client and we send the payload with the template ID to the system and it responds with the PDF bytes. We save it to disk and that's it. So it's pretty simple to add new things, so we'll go over here and we'll add a box here and we'll call this call this thing's. Now that we've done that, we have a new things, it supports this this new box so that we can go in here and add this anywhere. We generate generated. They jumped for always, OK, so then it and it worked cool, so that's the the filling piece and you have some some options here to set up text color and font size and all that. And so the last thing that I wanted to talk about was the signature packet. So this is a little bit more complicated than the last two because there's just a lot more going on. So this example, what we're going to do is we're going to create a a packet for onboarding a new employee. So a new employee needs to fill out this W-4 and then we're going to have them. We're going to have an NDA generated for them and then we're going to have them sign. The employee is going to sign both documents and the employer is going to sign one document.
M1: So to PDF two signers. So first thing we do is we set up the email addresses.
M9: Now.
M7: For this for purposes of this example and simplicity is Annville is going to handle this, the signing process, it will send out emails to all of the signers when it's their turn to sign, but that's totally configurable. So you can set up a packet where everything is handled in your application. So it's embedded into your application. We don't send your users any any emails. OK, so what? We're going to use the email, the email setup. So we set up emails so that the names this works semi's similar to the the other ones in that it's just one endpoint. So this is a graphical endpoint. We're going to call we're going to call a mutation with a bunch of variables to set up the packet and it's going to respond with some packet information. So in the variables, we set up the files we want to use our our PDF or our W4, we want to upload a new NDA and we're going to draw the boxes. And then we specify the data that we want to go on to each one of these these files. So that's the W4 one. Here's the NDA one. And then we set up the signers as well. So that she says, like which files, who signs, which files? OK, cool. So we are going to go to this other terminal first. And they will create the Hacket.
M4: All right. So we get this little URL here that it's the details. So this is a URL for our.
M7: Is the details yourself for our new packett. So you can go onto your dashboard and you can see the status of the packet and see who's signed and who hasn't and when all those things. So also I should have an email to sign these documents. So Sally is the employee.
M8: So we go through and I'm going to sign. Except my signature. Start signing.
M7: And so you can see that all of this data has been filled, so the W4 has been filled with the data that we specified and the NDA has been filled with the data that we specified. So all we need to do is then sign this.
M8: And it's waiting then for the employer. So we've got a new email.
M3: So this is the employer side of things.
M7: And then we can see all of the the signatures so we can see Sally's signature here and then we can see her signature there, we sign it.
M8: And it's all done. So that. Everyone gets a completed email and also we can see it on the.
M7: On the dashboard. OK, great, that's all if there's any question.