Dot Commands - the final word

[Alcoholics Anonymous]

This seems like a topic that never ends But I think the end is here now as the main facilities of NextZXOS are finalizing.

This post is about dot commands in general and dot command generation by z88dk in particular. There are other posts here talking about dot command generation with z88dk but this one will supersede those and I'll edit the others to point here.

There are a number of issues that come up with dot commands - how to parse the command line, how to report errors back to basic, how to check core and nextzxos versions if needed, how to ensure basic is not contaminated, how to write large dot commands of any size. So this will be mentioned in the course of this post.

But first, a review of what a dot command is.

Dot commands are a means to execute a small program that returns to basic cleanly without interfering with basic's state. They are loaded and run from a divmmc page overlapping the rom at address 0x2000-0x3fff. While the dot command runs, the lower 8k is occupied by the esxdos page. So the rom is completely invisible and the dot command itself sits below main ram so it doesn't interfere with basic at all.

Below is a somewhat technical discussion of some things. You can skip forward to --> if you like.

Since esxdos is present in the lower 8k, dot commands can use the esxdos api as normal through "rst 8". However, where parameters are documented to be passed using IX, these are instead passed through HL. When the basic rom is present, IX must be used to pass parameters because code used to bring the esxdos page into the 0-8k area uses HL so what happens is after esxdos is brought in, IX is copied into HL so that the esxdos routines see the parameter in HL and not IX. While the dot command runs, the esxdos page is already there so those parameters are expected in HL. Most people put those parameters in both HL and IX so their generic routines always work. This is the case for the z88dk's c esxdos api as well.

Because the 48k spectrum rom is not visible while a dot command runs, you can't call rom routines directly. Instead there are two special rst to help with interaction with the rom. One is "rst 0x10" which will print the ascii code in register A to screen just like the standard rom's "rst 0x10". The esxdos page contains code at address 0x10 to do the necessary banking to call the rom 0x10 and return properly to the dot command. The second special rst is "rst 0x18" which allows the dot command to call the address specified after the rst. So eg "rst 0x18, defw ROM_ADDR" would call ROM_ADDR in the rom. esxdos has code at address 0x18 to handle all the necessary banking. One caveat - the rom routine cannot see your dot command's memory space so if memory addresses are passed to the rom routine, those must be in main ram. You can create space on the stack that doesn't interfere with basic and another way is to use the rom routine BC_SPACES to create space underneath the program. The advantage of the latter is that the basic system will automatically free the space reserved by BC_SPACES when the dot command returns to basic. Note that if BC_SPACES fails due to insufficient memory, the basic system will generate a "4 Out of memory" error and will not return to your dot command.

-->

Because dot commands are intended to provide facilities available from basic, they are not normally meant to change the existing basic environment. So it is normal to use "rst 0x10" to print information so that it works seemlessly with basic. On exit, the dot command reports an error to basic. No error, a canned esxdos error, or a custom error string (with final char having bit 7 set) can be returned. There are a few lines documenting the exit condition here ( https://github.com/z88dk/z88dk/blob/mas ... sm.m4#L864 ). An error returned will result in a basic error with the appropriate error message printed at the bottom of the screen.

Dot commands are so-called because they are started with a leading dot. A typical invoke might look like:

.test This is the command line

These can also be put into basic statements:

10 .test look at my command line: PRINT "hello"

When the leading dot is seen by basic, the dot command will be launched. In the examples above the binary "TEST" in directory /bin (esxdos) or /dot (nextzxos) will be loaded to address 0x2000 in divmmc memory and executed. Notice the different directories used by esxdos and nextzxos - this is something new with NextZXOSv1.99I - and this allows esxdos and nextzxos dot commands to be separated so there is no issue with esxdos-only or nextzxos-only dot commands being accidentally executed in the other operating system, possibly leading to a crash. esxdos dot commands wanted in the nextzxos environment should be copied into the /dot directory.

The dot command is started with a pointer to the command line. Both esxdos and nextzxos will pass a pointer to the first non-whitespace character following the dot command name in HL. In the cases above, HL will point to the letter "T" or "l" after ".test".

Notice that HL is pointing into the basic area. This means you must not modify the command line so that the basic area is not corrupted. Because this is a basic area, the end of the command line can be marked by 0, ':' or 0x0d. (HL itself may also be zero if there is no command line).

Nextzxos dot commands are also passed the complete command line including the current dot name in register BC. So in the above examples, BC points at the "t" in ".test". (Again BC may be zero if there is no command line).

Under esxdos, dot commands are limited in size to about 7k. The exact number varies with esxdos version because esxdos places some data at the top of the 8k divmmc page. Dot commands that are too large will crash or behave improperly.

Under nextzxos, dot commands can occupy the entire 8k space.

So those are the basics of dot commands as created by esxdos.

NextZXOS adds a new element. Because nextzxos manages allocation of memory pages, it is possible to create dot commands of any size without interfering with basic. z88dk calls these dotn commands and it will automatically generate these. This will be discussed in a following post.

Why the attention to both esxdos and nextzxos? Both will be used on the zx next. The nextzxos esx api (which is an expanded esxdos api) is only present when the system boots up as a zx next. If you boot up as a 48k or 128k spectrum, nextzxos is not there but esxdos is. Similary, when new cores appear, for example there may be uno or bare 48k/128k cores that come along, all these systems will use esxdos. But the zx next is a much more capable machine and using the nextzxos esx api in ways incompatible with esxdos (and other machines running esxdos) will be a thing to make the most use of the machine.

[Alcoholics Anonymous]

STANDARD ESXDOS DOT COMMANDS (7k/8k)

Dot commands limited in size to about 7k (esxdos) or a full divmmc page 8k (nextzxos) require no special attention. The start up and exit conditions are as documented in the first post of this thread.

For esxdos dot commands (and this is compatible with nextzxos dot commands):

1. org is 0x2000
2. HL points into the command line at the first non-whitespace char following the dot command itself. (HL = 0 for no cl)
3. on exit, the contents of A,HL and carry flag determine if an error is to be generated.
4. "rst 0x10" is preferred for text output for seamless integration with basic.

That's all there is to it. However, there are some caveats.

Using "rst 0x10" for text output means a "scroll?" prompt can come up while the dot is running. If the user breaks out of the dot, then the dot command is abandoned and the break message is printed. This means your dot command will not be able to clean up file handles or restore other state it has changed. (This behaviour occurs if any basic error happens, such as might happen from a "rst 0x18" to call a basic rom routine).

esxdos will clean up file handles opened by the dot but I don't think nextzxos will. However, nextzxos has a solution for this.

nextzxos supplies as esx api function "M_ERRH" that allows the dot command to register a function that is called if a basic error occurs. In this way your dot command will be returned to after a basic error so that it can clean up before erroring to basic.

Since this functionality is not available to esxdos, it would be part of a nextzxos-only dot command stored in the /dot directory. Dot commands can also test if they are running on esxdos or nextzxos through esx api call "M_DOSVERSION" and a dot command intended to work on both systems could check if nextzxos is there and optionally register a cleanup function if so. Such a dot command would work on both systems but the esxdos version would not be able to do any cleanup if basic errors occur.

On the subject of interrupts, ideally interrupts in im1 mode should continue to operate under nextzxos. NextZXOS can have drivers installed like the uart and esp network that empty buffers during the interrupt to avoid buffer overruns. Note that nextzxos does not require the value of IY to be preserved for the rom isr.

On esxdos if im1 interrupts continue, the value of IY must be preserved as basic's isr uses its value to poke into the systems variables area. So your program must either disable interrupts (not a loss on a standard spectrum where esxdos is running), not use IY, or intercept the interrupt and restore IY before jumping to basic's isr. Note that if you disable interrupts or do not run the rom isr, printing via "rst 0x10" may hang the computer on a "scroll?" prompt.

For nextzxos dot commands:

1. org is 0x2000
2a. HL points into the command line at the first non-whitespace char following the dot command itself. (HL = 0 for no cl)
2b. BC points into the command line at the first letter of the dot command's name. (BC = 0 for no cl)
3. on exit, the contents of A,HL and carry flag determine if an error is to be generated.
4. "rst 0x10" is preferred for text output for seamless integration with basic.
5. optional test for nextzxos presence or version can be made to avoid running under esxdos if nextzxos-specific functionality is used.
6. an error handler should be registered so that clean up can occur if basic generates an error.
7. an optional core version check can be added if next-specific code is present that requires a minimum next core version.

The core has already been frozen for the cased nexts (and I think delivered to sms). I believe the core version is 1.10.51. However, this will not be the last core version delivered by the team. Development will continue for a while yet so any new features added may require core versions after the initial 1.10.51. Such programs will need to check that the minimum core version is present and inform the user if he needs to upgrade to run the program.


GENERATING STANDARD ESXDOS DOT PROGRAMS USING Z88DK

A. An ESXDOS dot program

z88dk supplies a plain dot crt that runs before your program does and ensures basic can be returned to cleanly (see https://github.com/z88dk/z88dk/blob/mas ... asm.m4#L59 ). This crt optionally adds code that deals with the issues raised above. What is in there is controlled by pragmas.

Common controlling pragmas:

* CRT_ENABLE_COMMANDLINE (default = 3)

0 = The program does not get a command line
1 = The program gets an empty command line in bc=argc, hl=argv
2 = The program gets the unprocessed command line bc=len, hl=cl
The length is computed in bc by the crt
3 = The program gets the command line on the stack parsed into bc=argc, hl=argv
The command line parsing understands quoting and stops at unquoted <, >, | for future dev
Because the command line is copied onto the stack it is safe for the program to modify it
4 = The program gets a copy of the unprocessed command line on the stack bc=len, hl=cl
Because this command line is copied to ram, it can be modified.
The copy is zero terminated.

Where applicable, the command line is written to basic's stack and are limited to about 128 bytes.

* CRT_ENABLE_COMMANDLINE_EX (default = 0)

0x80 (bit 7 set) = use nextzxos's full command line as provided in BC instead of HL.

* ESXDOS_VERSION (default = 0)

Currently esxdos does not return version information but you can set this to one to have the crt check if esxdos is present as opposed to nextzxos.

* NEXTOS_VERSION (default = 0)

If non-zero add a nextos version check. NEXTOS_VERSION should be set to the minimum decimal nextzxos version required, eg "199". This also implies that a function will be created to intercept a basic error and properly perform cleanup (via atexit()) if a basic error occurs.

* CRT_CORE_VERSION (default = 0)

If non-zero add a core version check. CRT_CORE_VERSION should be set to the minimum decimal core version required, eg "110051". One or more digits for major "1", two digits for minor "10" and three digits for sub-verson "051".

The crt can generate some basic errors depending on what pragmas above are active:

"Requires Core v1.10.51, 0:1"
"Requires NextZXOS 128k, 0:1"
"Requires NextZXOS 128k 1.99, 0:1"
"Requires ESXDOS, 0:1"
"D BREAK - no repeat, 0:1"

B. Compiling a C ESXDOS Dot Program (subtype=dot, zx spectrum compatible only)

There is no difference from any other program except it must be limited in size to about 7k for esxdos.

test.c

Code: Select all

#include <stdio.h>
#include <stdlib.h>

void cleanup(void)
{
   printf("\nFini!\n");
}

int main(int argc, char **argv)
{
   atexit(cleanup);
	
   printf("\nTESTING 1, 2, 3\n");
   printf("\nCommand line contains %u words\n\n", argc);
	
   for (unsigned char i = 0; i < (unsigned char)argc; ++i)
      printf("argv[%u] = \"%s\"\n", i, argv[i]);
	
   return 0;  // 0 indicates no error
}

In this example, I've registered an unnecessary cleanup() function with atexit(). Functions registered with atexit() will be executed in reverse order of their registration whenever the program exits. This is how c cleans up after itself and is a good place to put things like closing open file handles. As mentioned earlier, if a basic error occurs an esxdos dot command will exit immediately so these atexit() functions will not run. But under a nextzxos dot command, they will always run even if a basic error occurs.

zpragma.inc

Code: Select all

// limit the size of printf

#pragma printf = "%u %s"

// make space for one atexit function

#pragma output CLIB_EXIT_STACK_SIZE = 1

// defaults keep binary size at minimum already
// see https://www.z88dk.org/wiki/doku.php?id=libnew:target_embedded#crt_configuration
// and https://github.com/z88dk/z88dk/blob/master/libsrc/_DEVELOPMENT/target/zxn/crt_config.inc#L69

The pragmas control what the crt does before your program is started.

This test program simply prints out the words in the command line it is given. It's using the default "CRT_ENABLE_COMMANDLINE = 3" so the crt will have parsed the command line into words in the argv array.

In z88dk, startup values control what drivers are attached to stdin,stdout,stderr. There are several of them for timex 64 columns, timex 128 columns, spectrum 32 column, spectrum 64 column, fzx proportional fonts, etc. The ones most pressing for dot commands are "startup=30" to select the rom rst driver and "startup=31" to have nothing on stdin,stdout,stderr (perhaps for printing another way besides through c's streams).

The other bother is that this is an esxdos dot that should run with interrupts enabled which means IY must be preserved. To do that in C, we must avoid using IY. sccz80 will all the time and zsdcc will almost all the time (enough you don't have to worry about it unless you create large data structures on the stack).

Compile commands:

[zsdcc, smaller faster code]
zcc +zx -v -startup=30 -clib=sdcc_ix --reserve-regs-iy -SO3 --max-allocs-per-node200000 --opt-code-size test.c -o test -pragma-include:zpragma.inc -subtype=dot -Cz"--clean" -create-app

Note: the recommended zsdcc compile uses "-clib=sdcc_iy" but we can't do that here because the library will use IY for stdio functions in that mode.

[sccz80, faster compile guaranteed no IY]
zcc +zx -v -startup=30 -clib=new test.c -o test -pragma-include:zpragma.inc -subtype=dot -Cz"--clean" -create-app

The 'subtype=dot -Cz"--clean" -create-app' sequence chooses dot command as output form, asks z88dk to clean up after itself and causes z88dk to built the dot command automatically. The output will be a binary "TEST" that must be less than about 7k to work with esxdos (or 8k for nextzxos). This binary can be copied to /bin for esxdos and/or /dot to run under nextzxos.

You can try to break the command line parsing by trying some tricky invokes:

.test hello there
.test hello "one whole sentence" there
.test hello"there"people
.test hello there "ladies and gentlemen

Did you notice something in the compile commands? The target chosen was "+zx" (for regular spectrum) and not "+zxn" (for zx next). This is because dot commands in the esxdos directory are most likely to be run by a regular spectrum and not a zx next by cores other than the zx next core. A compile with "+zxn" would have the c compiler inserting zx next opcodes which cannot run on a regular zx spectrum.

So, dot commands for native esxdos in the /bin directory are for the zx spectrum only. zx next dot commands should use the compile method in the next section. Once the dot commands in /bin are committed to regular zx spectrum only, you can again consider running them with interrupts disabled to remove the restrictions on IY.


C. Compiling a C NextZXOS Dot Command (subtype=dot-n, zx next only)

We'll use the same source code and just compile it differently:

[zsdcc, smaller faster code]
zcc +zxn -v -startup=30 -clib=sdcc_iy -SO3 --max-allocs-per-node200000 --opt-code-size test.c -o test -pragma-include:zpragma.inc -subtype=dot-n -Cz"--clean" -create-app

Note: we're back to the recommended "-clib=sdcc_iy" for better code output. nextzxos does not require us to save IY while im1 is running.

[sccz80, faster compile]
zcc +zxn -v -startup=30 -clib=new test.c -o test -pragma-include:zpragma.inc -subtype=dot-n -Cz"--clean" -create-app

The output binary will again be "TEST" but this time copy it to /dot (and check it's < 8192 bytes in size). The dot command may not be runnable under esxdos but if you mistakenly put it into /bin, the dot command will just stop with an error message if nextzxos is not detected.

The dot command type has changed to "-subtype=dot-n". What this does is define "__NEXTOS_DOT_COMMAND" and you can see how it impacts the crt here ( https://github.com/z88dk/z88dk/blob/mas ... sm.m4#L104 ). In short, some code will be added to check if nextzxos is present and it will register a basic error handler to guarantee that the cleanup atexit() code will run even if a basic error occurs.

Since the target is "+zxn", zxn opcodes can be used by the compiler. The program can also use zx next features.


D. Compiling an ASM NextZXOS Dot Command (subtype=dot-n, zx next only)

This can also be done with the native esxdos dot command in part A but I'll limit this to one example. In this example, we'll get the crt to do its usual things but we'll turn off c's streams and leave it all to unassisted asm. The topic of using asm in z88dk is much larger because there is a library of about 1000 asm functions available for use and z88dk defines all io ports as well as nextreg but this is not the place to describe this. So we'll just pretend we're writing asm with nothing but our wits and rom at our disposal.

The crt is equally applicable to asm; some people have hangups connected to it because crt stands for "c runtime" but actually it has little to do with c itself. It's just a means to perform some initialization and cleanup around the program. In this case, the crt is going to take care of returning to basic and doing the other things previously discussed like core version checks, command line parsing and so on.

test.asm

Code: Select all

SECTION code_user

PUBLIC _main

_main:

   ; bc = length of command line (0 if no cl)
   ; hl = address of start of command line (0 if no cl)

   ld hl,message
   ret

message:

   defm "I made this", '.' + 0x80

zpragma.inc

Code: Select all

// pass the unprocessed command line

#pragma output CRT_ENABLE_COMMANDLINE = 2

This program does nothing really except return with an error message. The return to the crt is through HL with 0 for success, 1-255 for a canned esxdos message and >=256 for the address of a custom error string. The error string must terminate with the last char having bit 7 set.

I chose "CRT_ENABLE_COMMANDLINE = 2" in the pragmas so that the crt will deliver the command line as esxdos delivers it. The crt passes one additional bit of information - the command line's length. You could use other values for "CRT_ENABLE_COMMANDLINE" including the default 3, in which case the crt will parse the command line into argc,argv and take care of quoting. Then BC=argc and HL=argv when your program starts.

The entry point of your program must be "_main" because the crt contains a "call _main" to start. You must also assign your code to a section that is part of the memory map. In the absence of explanation "code_user" is fine. z88dk works like commercial tools, not like common assemblers in the zx scene so it defines a memory map into which code and data is placed by section name. This has enormous advantages which become clear when you start to write programs spreading across memory banks and building those into standard forms (in z88dk it's just a change to "subtype" to automatically generate, sna, snx, nex, and so on).

Anyway the compile line is identical to that used for c:

[zsdcc]
zcc +zxn -v -startup=31 -clib=sdcc_iy test.asm -o test -pragma-include:zpragma.inc -subtype=dot-n -Cz"--clean" -create-app

[sccz80]
zcc +zxn -v -startup=31 -clib=new test.asm -o test -pragma-include:zpragma.inc -subtype=dot-n -Cz"--clean" -create-app

"startup=31" has been chosen so no c streams are created.

The compiler choice only matters if your project contains parts written in c. The c compiler will never be invoked if there are no c files to compile. No additional code will show up, just the crt as in https://github.com/z88dk/z88dk/blob/mas ... asm.m4#L59 and your asm.

The "clib=*" lines are choosing which version of the zxn library to link against. If your program uses library functions, they will be pulled out automatically and as needed from the library version chosen.

An extra note about argc and argv for asm programs. When "CRT_ENABLE_COMMANDLINE = 3" (the default) the crt will parse the esxdos command line into zero terminated words and push them onto the stack (the parser can be seen here https://github.com/z88dk/z88dk/blob/mas ... rse.asm#L9 ). In memory they will be laid at as shown:

.test hello there world

test\0hello\0there\0world\0

(esxdos will have nothing in place of "test" as it doesn't supply a pointer that includes the dot command name)

argc = BC = 4
argv[0] = word at HL = address of "test\0"
argv[1] = word at HL+2 = address of "hello\0"
argv[2] = word at HL+4 = address of "there\0"
argv[3] = word at HL+6 = address of "world\0"

A "LD E,(HL) ; INC HL; LD D,(HL)" will have DE pointing at the "t" of "test\0hello\0there\0world\0"


Next post dotx commands, then dotn. -->

[Timbucus]

I chose "CRT_ENABLE_COMMANDLINE = 2" in the pragmas so that the crt will deliver the command line as esxdos delivers it. The crt passes one additional bit of information - the command line's length. You could use other values for "CRT_ENABLE_COMMANDLINE" including the default 3, in which case the crt will parse the command line into argc,argv and take care of quoting. Then BC=argc and HL=argv when your program starts.

Excellent article - should have read that yesterday! Can I just ask if you use the CRT option 2 for a raw command line surely setting the length in BC destroys the pointer to my command name under NextZXOS? Could that be copied to DE or something?

[Stefan123]

Very useful and well-written information Will this make it into an appendix in the Spectrum Next manual? Would be nice to have as much as possible in one place.

[Alcoholics Anonymous]

I chose "CRT_ENABLE_COMMANDLINE = 2" in the pragmas so that the crt will deliver the command line as esxdos delivers it. The crt passes one additional bit of information - the command line's length. You could use other values for "CRT_ENABLE_COMMANDLINE" including the default 3, in which case the crt will parse the command line into argc,argv and take care of quoting. Then BC=argc and HL=argv when your program starts.

Excellent article - should have read that yesterday! Can I just ask if you use the CRT option 2 for a raw command line surely setting the length in BC destroys the pointer to my command name under NextZXOS? Could that be copied to DE or something?

For dot/dot-n and dotx/dotx-n (not written about yet) the default is to give you the esxdos command line without the dot name. For dotn (not written about yet) the default is to give you the nextos command line which means the dot name is there.

You can specifically ask for the the nextos command line by setting "CRT_ENABLE_COMMANDLINE_EX = 0x80" (or setting to 0 to get the esxdos command line). Then even in "CRT_ENABLE_COMMANDLINE = 2" the command line you get will include the dot name. But make sure you are running under nextos otherwise this will be garbage in esxdos.

Note there was a bug found last night as I'd forgotten to change the length computation to consider quoting. The fix is in git now and will be in the next z88dk build Aug 26.

Very useful and well-written information Will this make it into an appendix in the Spectrum Next manual? Would be nice to have as much as possible in one place.

Cheers and still much to say especially around dotn because that one is the holy grail There are still some things being feeled out as some have identified areas where how things are done could be improved in the past day and I will update here. It's mainly about how to coexist with esxdos, nextos in 48k mode and regular nextos.

I think Pheobus is looking for a section on dot commands but at the end of this it will be a bit lengthy. Maybe the salient points can be extracted.

[TheSMoG]

Very useful and well-written information Will this make it into an appendix in the Spectrum Next manual? Would be nice to have as much as possible in one place.

Cheers and still much to say especially around dotn because that one is the holy grail There are still some things being feeled out as some have identified areas where how things are done could be improved in the past day and I will update here. It's mainly about how to coexist with esxdos, nextos in 48k mode and regular nextos.

I think Pheobus is looking for a section on dot commands but at the end of this it will be a bit lengthy. Maybe the salient points can be extracted.

Did anyone call my name?
Well, it all depends on how much space we have. Right now, there's an Appendix devoted to the dot commands included in the distribution and a bit about dot commands in the NextZXOS chapter. But I have a hard limit of 300 pages and the choices are hard...

[sol_hsa]

This thread is most of the information I've found about dot commands. Would it be possible to get this information into the wiki? Also, what about the bigger-than-8k dot commands? is that a z88dk crt feature or does the dot command loader in next do something funny?

[SevenFFF]

The general technique is that NextZXOS loads the first 8K, and leaves the handle open. There is an API call to retrieve the handle, and another one to allocate free 8K banks to load the remaining code into.

All this must be done by the dot command, and undone/deallocated before exit. Z88dk has a mechanist to make this completely automatic for dotx targets, but the same techniques are available for use with plain asm commands.

You can look at the source of .ls or .find in the distro for z88dk examples, and look at the source of .espupdate for an asm example.

[Ped7g]

"Would it be possible to get this information into the wiki?"

It already is possible (except some pages like front one, anyone can edit wiki).

Feel free to add it (for example) here: https://wiki.specnext.dev/File_Formats in the similar way as other are done.

[Cthutu]

The information AA provides is amazing and should not be left hidden in the forums. It should be a blog post on specnext.dev!