BlitzBASIC 2 PlayBASIC Conversion Tool WIP - Episode 04 - Parsing Variable Detection

 BlitzBASIC 2 PlayBASIC Conversion Tool WIP - Episode 04 - Parsing Variable Detection
 https://www.youtube.com/watch?v=Ie1LPSVwPS0&t=9s&ab_channel=PlayBasic
 https://www.underwaredesign.com/forums/index.php?topic=4625.msg30567#msg30567
  ---------------------------------------------------------------------------------------------

hello welcome back here we are looking

at another

one of these

blitz to play basic translation videos

so today i was working on

the identification of variables uh

the implicit ones in particular

so it'll pick up dim statements it'll

pick up global and local statements

but also has to pick up

variables throughout the code

just pull up this

pretty rudimentary example so we've got

our global statements we can phrase

those

uh i don't think we've got list support

at the moment but i think we're picking

up the keyword

and just storing this for later

locals the same

we just want to raise as well but i'm

not working on those at the moment

just

been working on variables

so i've got global scope which is this

initial part of the program they'll be

all tagged as global scope

and

a function here

and we've got a

bit of code inside here we've got some

array stuff there we'll just pull it out

so we don't confuse ourselves


now things here to notice really are

our usages of our variables so i've got

local

declared here my float my string

my float has the hash symbol on it my

string

it's the first usage

so it has dollar sign at the end of it

and a loop here has

the integer

tag which is the percent symbol

input

but as we move through you'll notice

that uh

i've tagged a as an integer

i've used loop without its

postfix without

suffix b the same these are all used

without them

and down here we've got a string

and then we're using my string

without its

dollar sign

and same goes to float so previously i

was talking about wanting it to pick

these up

and do those translations for us and so

which is something you really can't just

do with like a replacement

things like the percent sign here

if you just pull in a bunch of code

into you know notepad or whatever just

and just do a raw replacement

that would probably get you a long way

to fixing those problems but um

you have a you'll have collisions as

well inside strings and comments etc but

but for the most part you wouldn't no

one would care

here's another bit of code which

actually dragged across from

from facebook just before

to demonstrate this in a broader

spectrum so we've got

in that language all variables were

inherently float

so here i've just added a local

statement with the variables in it

but left all of the occurrences

throughout the program

and without the

hash symbol on them

i've done that intentionally because i

want the converter to pick them up and

and do that work for us

so i'll just hit save on that and close

it

go back to here back to play basic now

this is an example we're loading and

running

running debug so we can see the output

you know in debug is taking 91

milliseconds

most that's just text airport to the

console

here's the function that's that's output

for this last piece here

just send that in a little bit

you can use the scroll function too with

text boxes


you'll notice that all of the variables

that follow

here

so after a declaration up here

the original code hand

was just width etc

all of these are now

read and read and run pretty much in pv

which is pretty good

that saves a lot of messing around

i won't say it's going to be perfect

there's probably some situation where it

will do the wrong thing or whatever

at the moment that's more likely than

not but

that will give us curry that we almost

run in pv as it stands

almost

[Music]

this is actually using radians rather

than angles so we have to convert these

two

that might be something we need to do

um

in blitz as well i'm not quite sure if

the cos and sine functions are actually

if we need to wrap them or not

i haven't tried this wasn't blips code

by the way this was on something else

but

i use that as a good example of

how we can get to

add the postfix symbols do our variables

for us

on mass

uh this is the code that was that's the

converted code so this is a

it won't

probably won't compile in pv we'll

probably get some functions in there

that we're not using

but we should come pretty close um

up here

this is the list of

scopes we've found so we found

what we're kind of like calling global

scope

which is the

the entire program that's not inside a

function

from top to bottom

and this is my function which is

the source position there is wrong

but these are the variables that are

known to be inside it

um some of these are getting or

incorrect is picking up some common

commands that it shouldn't

um

we might do a secondary search on those

and make sure that they're

they're

blips keywords they should already be

classified but i think there's

i think that classification's not quite

working probably

anyway so

you know a function that we dragged

across before we've got all of these

variables widths etc

and they're all tagged as

being a type 2 which is a float and it

gives us an indication of how frequently

the variables are used

and that's kind of that's handy because

we might be able to

use that to give hints about

uh

you know in certain scopes there are

variables that are only used once

that you're not you know maybe it's a

typo maybe it's something else

give warnings possibly

so you could add you can insert a bit of

a header in front of a function let's

say

that's it with a comment that says you

know this variable is not used

more than once or something like that

possibly even remove it

the problem with that would be of course

is is if we've misclassified something

and it turns out to be a constant

somewhere else or something like that

which we don't actually support at the

moment

come to think of it

so we'll have to add

support for constants to make sure and

they'll have precedence over uh

variables as well

the variables are kind of the lowest

thing on the pecking order

but our output here so

this just gives me a bit of idea of what

variables that's found inside the scope

they all seem to be okay if you look

down the bottom here what's

in this scope here

it's picked up things like swap

do

ellipse it thinks they're variables

they're not

of course not

but you know

we're getting somewhere

uh

i'm getting somewhere with this

at the top here

pick up us

our skypes i'm not sure if our

about searching is uh

our scope searching it's working

as per blitz

but i can update that later if it's not

quite quite correct

as we're inside a function is going to

work out what version of a variable

you're using

um

you know if the thing's declared

that's global uh above it

then we need to go okay use the global

occurrence of this so if the global was

declared

let's say it was a floating point

variable and then later on using it

without the hash symbol it must update

that as well

rather than think that's a new variable

inside the scope

but i think actually

even though we're not doing a full

compile

we're just picking through with this

logic we should be able to get a lot of

that stuff done

um

to be horrified if we run it in

on the on the large code base

this takes about 10 seconds to run this

well i'm sorry i can pull compile and

run it to straight

with no debug

so

this is the

9206 line

block of code

so it's been converted in about 900

milliseconds just under a second

that's pretty good i'm actually

i'm happy about that i wasn't expecting

it to do that well

i thought it might take you know five

ten seconds or something for code bases

that were large

run debug this takes a while because

it's dumping all of that crap to the

console and the console is just a

buffered window

so the thing looks like it's died

as you can see it's a long

lot of code

yeah now i can

pick through the scopes and and see what

um

what we've uncovered so this function

gy wind but skin

has these variables in it

some of these might not be variables

because it at the moment doesn't

understand types and the field uh

separators and types so might be picking

up field field separators as variables

here we've got some

references to these things here we're

looking for that in the code base

let's find the global scope

all right that's global scope there

got a bunch of just gar materials at the

front

gui screen width

see that's surprising

saying that screenwrites is only used

once the skin variables are used

that one's used 10 times

so that includes its initial occurrence

so it's declaration

i wonder if that's true

gee i scream with hmm

i don't believe that for a second

so gy screen

width

occurs

yeah it's more like it

was gonna say that that can't be right

so clearly we're not we're not fetching

um

hmm

we're not skipping over something there

we're not searching and finding those

correctly for us

good

what have you learnt well we're kind of

working we're kind of not working so

that's pretty much what we expect to

find isn't it

uh if i try this array thing actually i

might try the array

one here function scopes

so dimension

uh

dim

and i call it uh strings

a hundred

i'll use the example always do it the

other day so four

from the pickles naught to 100

i just want to see if it picks that up

actually sorry print

string

no

dollar sign

loop

thank you let's go

and we didn't pick that up

oh

yeah i know why there's no

trapping of the bracket

if it sees

this keyword and then a bracket

following it

you might know what to do and just

assume that's an integer

and strings without that will just be

integer so

in that scope

my function one there should be a thing

called where are we

yes yet

hmm

that's some some more oranges there

we're not getting both of those we

should have two strings if that's how it

was to seeing it

i had some really weird problems with

this this afternoon actually

weird is not saying the least and i was

at the point where i thought oh no

i found another weird bug in pv

it was right at that moment where i

uncovered oh hang on

it's not a bug in pb it's just i haven't

initialized

okay so we've opened

yeah

uh here's the logic here so if we scan

through we find a word

which is just any group of characters

that's been classified

so we'll see a block of character like

that

there is no white space the block of

characters is alphanumeric so you can

have lowercase uppercase and

numbers

and lower score character as well but i

think it can it has to start with it

with a underscore or

has to start with a

letter

anyway so we're skimming through we find

our word

we go okay

grab the token that's our variable name

we'll

need that to search for it in a second

we grab

whatever's after this thing

if this is an open bra which means that

the character after this so i just

just here for example

um

with the actual card again

so if we go let's zoom off a bit

so we're looking at this

keyword here

uh

it'll grab the token next to it which is

going to be the bracket token now what i

can actually do and get away with is you

have white space between these things

i think in pb generally it doesn't like

the white space to be there but in a lot

of languages that are pre

tokenized and then translated

which means that the white space can

just be thrown away so it doesn't matter

if there's white space between stuff

which is kind of weird

if people can't really do that well

i'm not sure enough to talk ahead but

anyway so we said this keyword here this

strings keyword

and since the next token is an unknown

stop character it's not a hash it's not

it doesn't know what to do if this is

presenting as an integer array

that's why it's failing so

what do here

sounds pretty crazy it's just um

just

jump to check this variable

that's how lazy i am yep

so look

and there we go we fixed our fix one

little shortcoming so

strings has been picked up as being

that is the array because it's saying

the next character is here or that's the

assumption it's going to make

it's pretty good assumption

um

and we're looking for that in the

current skype and found it so it's happy

to to promote it

here we have

we've got our strings array

and we found two occurrences of it

that's what we wanted

happy days

isn't that terrific so it's pretty much

just bumbling logic to fall through and

go okay this is our best guess for what

this keyword is

uh if we were building a full-on

compiler we actually have

as i was saying before you

you grab

as you found each keyword you then

dissect what the keyword actually is if

is it a

structural statement like a for

statement or a go-to or you know or a

compare you know where it's going to be

and then

uh they all have a certain pattern

so a for loop has okay after the full

statement we have a declaration then we

have

an assignment and then we have an

expression

a two statement and an expression

that's that's the pattern that i have

and if statement has

you know it's just an if and it's an

expression

uh

what we have to do is we have to break

the code down

at least then if we did that we could be

type aware this is not type aware

what i mean by that is is it

is it's not aware if you

like if you go hey here's my string

right here and i add

an integer value to it

pv is going to go what the hell are you

doing you know it might have a clue

but if we

wrote this as a true compiler we could

run through here

uh

promote this is to be the correct array

we wouldn't even need to do that

actually

uh and then we have a string we have

this replacement

we have um

conceptually half we've seen the

expression the first time

we pull the fullest result see it we end

up with like result

string plus this integer then we have a

bit of logic there going okay

you

since it's a string mixed with integers

we're going to want to treat the string

like an and a number so i have the value

of the string

yeah you could do that stuff

in a translator for you

and life would go on but in our version

here we can't do that

because i don't know

what this is

right or what the parameters in here are

i have no idea

i'm not matching types i'm not matching

the patterns of expressions nothing

so

so we can get you know get away with a

very preliminary um

comparison comparative sort of

replacement but we're not going to have

things that are

you know if your input code makes no

sense and your apple code won't make any

sense and in places

the output code here white making sense

in pb because you'll have

i noticed there's a thing in blitz where

you can do you can do type conversions

like

you know

a equals

so you have a literal string

and have it converted for you to an

integer value

um

yeah it just saves having you know a

function

val for example

you know

this is a shortcut they've added

anyway

is the this is the basic logic we're

just picking through here

um

just to select a slight case

with if the next the character after our

current current word is this

and it's most likely this thing

yeah so i have an example statement eg

this word bracket

if for the percent

it would be this word

percent

hash dollar sign etc

um there should be a you know

another version here that handles um

the trapping of

fields within types

but that's not something it even looks

at the moment

we'll try and get this to work pretty

well and then we'll worry about

stepping up for types

make sure it can pick up the

blitz commands correctly output them you

know into a

wrapped function a format and

then try and

make some simple programs

in blitz and try and get them to run

so

isn't that fun

anyway uh thanks thanks for hanging

around and listening to me ramble on

and i'll see you next time bye