MSX Basic Tokenizer. Should I?

Resident (39)

30-07-2019, 01:51

So,
The newest version of Basic Dignified imposed certain constrains that maybe, perhaps, who knows, could make it easier for me to try to create a tokenizer in Python. It wold certainly speed up the build and run process.
I have Two questions:
1. Is there already a cross platform, command line tool for that?
2. Is there any interest on the community for something like that?
It is not something I can do right now (moving again) but I certainly can begin the research.
So?

Por Latok

msx guru (3938)

30-07-2019, 08:03

I'd say if you like doing it, do it! Don't do it for others...

Por Grauw

Ascended (10768)

30-07-2019, 09:24

What Latok says is really the most important thing in my opinion .

Although as an assembly programmer it's not something I would use, I can see how it would be interesting to research and implement, and I think it's definitely good to have a more structured and less restricted version of Basic which compiles down to minimized MSX-Basic. I would make sure it integrates well with Basic-kun as well.

Por farique

Resident (39)

30-07-2019, 17:25

Oh yeah, sure, the challenge is the motivation; even I will not use it beyond some toying. If it already existed, however, the allure would be much smaller.
And the question about the community is to mesure the scales on letting it go or enduring when the problems arrive.
Thank for the feedback.

Por brnomade

Supporter (7)

30-07-2019, 20:45

This project implements a tokenizer but not in python. Maybe you can write a wrapper to reuse the existing implementation? Just as a suggestion.
https://launchpad.net/msxbas2asm

Por PingPong

Enlighted (4136)

31-07-2019, 10:05

brnomade wrote:

This project implements a tokenizer but not in python. Maybe you can write a wrapper to reuse the existing implementation? Just as a suggestion.
https://launchpad.net/msxbas2asm

Wow! This is practically the core of a basic compiler. Can serve as a basis to create a good msxbasic compiler

Por zPasi

Champion (499)

31-07-2019, 19:37

brnomade wrote:

https://launchpad.net/msxbas2asm

That certainly looks promising.

I managed to compile that on my Mac. It required some tweaks:

On Makefile:
CFLAGS = -Wall -fexceptions -std=c++11
On main.cpp:
#include <sys/malloc.h>

Por farique

Resident (39)

01-08-2019, 04:15

brnomade wrote:

https://launchpad.net/msxbas2asm

That looks amazing!
Shame it is way beyond my pay grade for any meaningful repurpose (not that I haven't put it on my reference list).
And anyway, where is the fun in making a wrapper :P

On a happier note, I managed to find some time and a quick proof of concept is already churning workable little pieces of tokenized code.

Por farique

Resident (39)

02-08-2019, 20:18

Well, it was easier than I thought.
Of course there should be a ton of fringe (and not so fringe) cases not covered but it passed some tests.
No Basic-Kun or even Dignified integration yet, it is still very preliminary.

Here:
MSX-Basic-Tokenizer

Por Manuel

Ascended (19462)

02-08-2019, 20:31

Could be massively tested using a script that saves a basic program to ASCII and then tokenize it back and compare the results.

Por farique

Resident (39)

02-08-2019, 20:49

Great idea.
Is there any easy source of varied and tested basic programs in good quantity I could download?