Generating Huge Dot Commands with Z88DK

Discuss game and other programming topics not specifically covered in another forum

Moderator: Programming Moderators

Alcoholics Anonymous
Posts: 463
Joined: Mon May 29, 2017 7:00 pm

Generating Huge Dot Commands with Z88DK

Postby Alcoholics Anonymous » Wed Jun 20, 2018 3:06 pm

(The June 21 build of z88dk will contain them see for when it is finished)

If you don't know what they are, dot commands are utilities that sit in the /bin directory and can be called from the basic prompt by prefacing their name with a dot. For example ".ls". They can be run at the basic prompt or from a program and are passed a command line that the dot command can parse as an option list. ".ls /bin" would list all the dot commands in the bin directory with "/bin" being parsed by the dot command itself. Dot commands are loaded into an 8k space that overlays the rom in the 0x2000-0x3fff area and have access to all of main memory while they run.

Z88DK can generate two kinds of dot commands at the moment (and number three is what will be described below). The first is the plain dot type that is defined by original esxdos. These dot commands are limited to 8k in NextOS and around 7k in esxdos because they must fit in a single divmmc page ORGed at 0x2000. 8k can be limiting as dot commands become more complicated so a second type was introduced called dotx (extended dot command). This one splits the program into two parts - one part in the 7k/8k divmmc space and a second part in main memory. This one checks if ramtop is set low enough for the second part before loading so that it doesn't interfere with basic. Dotx commands work in esxdos as well as nextos.

You can see a few examples of these types of dot commands here: ... ot-command

The new huge dot command type is called dotn (nextos dot command). This one takes advantage of a NextOS page allocation facility. The api function IDE_BANK is an 8k page allocator that acts as memory manager and can give out pages that are not used by basic or other programs loaded into memory. This makes it possible to create huge dotn commands that get pages from NextOS and load into them before running. This way the program can be as large as you want without interfering with basic, running drivers or anything else loaded into ram.

How does it work

Z88DK provides a crt ( ... asm.m4#L69 )
that runs before the dot command starts. It does the following:

On start:

1. Verify that NextOS is present and error if not.
2. Record current memory and banking state for return.
3. Load the dot command into allocated memory pages.
4. Construct a table of logical to physical page mapping.
5. Optionally parse the command line. The command line is copied into divmmc memory so that it is always available.
6. Register a basic error intercept so that cleanup will still occur if user presses break during scroll? prompt (eg).
7. Set up the main memory bank (what you see in the top 48k when the program starts)
8. Call your program

On exit:

1. Run the exit stack if you have one (functions you register to be run if the program terminates).
2. Run library and user termination code inserted into section code_crt_exit
3. Restore the original memory and banking state
4. Free any allocated memory pages back to NextOS
5. Safely return to basic with proper error code returned by program.

This code sits in the 8k divmmc page along with some other library code that (if used) will be placed here to try to fill up part of the 8k. You can also place some of your code here to pad it close to 8k if there is space (about 200 bytes are needed at the top to hold the command line). One advantage of placing code or data here is that it will always be visible no matter what banking your program might be doing in the top 48k.

Programming model

The programming model is unchanged from an extended sna ( viewtopic.php?f=13&t=1209 ) or a typical spectrum program.

That is, your main code is loaded into the typical (logical) 16k bank 5,2,0 arrangement in 0x4000-0xffff and you place extra code and data into PAGEs or BANKs. The only difference in the program is that before paging in new memory, the page number must be passed through a table that does logical to physical page mapping and then this physical page must be used as the actual page number for paging. This table is built when the dot command is loaded and before your program starts, and it contains the physical page numbers that NextOS has allocated for your program. The table is located in divmmc memory and is always available no matter what the banking arrangement is in the top 48k.

Assuming you've done this, z88dk will automatically build a big dotn command for you. Copy the dotn command to /bin and run it with ".name command line parameters here"


Save these two files:


Code: Select all


defs 128, 0
defs 128, 30
defs 128, 0xff


defs 128, 0
defs 128, 41
defs 128, 0xff


defs 128, 0
defs 128, 125
defs 128, 0xff


defm "DIV1", 0

Code: Select all

#pragma output DOTN_EXTRA_PAGES = 10
#pragma output DOTN_LAST_PAGE = 125

#include <stdio.h>
#include <stdlib.h>
#include <arch/zxn.h>

extern unsigned char _z_page_table_sz;
extern unsigned char _z_page_table[];

extern unsigned char _z_page_extra_sz;
extern unsigned char _z_page_extra[];

void print_page(unsigned char page)
   printf("Page %u\n\n", page);
   for (unsigned char *p = 0x6000; p != 0x6180; ++p)
      printf("%02x", *p);

int main(int argc, char **argv)
	printf("Command line contains %u words\n\n", argc);
	for (unsigned char i = 0; i < argc; ++i)
		printf("%s\n", argv[i]);
   printf("\nPAGE TABLE SIZE = %u\n\n", _z_page_table_sz);
   for (unsigned char i = 0; i != _z_page_table_sz; ++i)
      printf("log page %03u = phy page %03u\n", i, _z_page_table[i]);
   printf("\nEXTRA PAGES = %u\n\n", _z_page_extra_sz);
   for (unsigned char i = 0; i != _z_page_extra_sz; ++i)
      printf("log page %03u = phy page %03u\n", i, _z_page_extra[i]);

   return 0;
Compile with:

zcc +zxn -v -startup=30 -clib=sdcc_iy -SO3 --max-allocs-per-node200000 test.c mem.asm -o dotn -subtype=dotn -Cz"--clean --fullsize" -create-app

The output is "DOTN" which is the actual dot command that should be copied to /bin.

The dotn output type is selected with "-subtype=dotn" and the "--clean --fullsize" options passed to appmake delete binaries used to create the dotn and ensure that pages are padded to 8k when necessary. Any remaining binary files after the compile are not in the output and if you need them you must get them into memory somehow. "-startup=30" is choosing the rst$10 driver for stdout (and printf).

"main.asm" places items into pages 30, 41, 125 and divmmc page 1. "test.c" is compiled into main memory (the 64k you see in 16k banks 5,2,0). It is important to note that these are logical page numbers that your program refers to. These logical page numbers must be translated to physical page numbers just before any banking occurs.

The compile will automatically build the dotn command with all these pages part of the binary and the dotn will automatically allocate pages and load all the dotn pages in the compile before running your program. The divmmc page 1 is only there for testing. divmmc memory is not made part of the output binary and will instead be output in a separate bin file as you will see shortly.

The compile generates this information:

Code: Select all

Notice: Main binary occupies [32768,33402]
Notice: Space to end of dot is 5242 bytes
Notice: Main bank allocation mask is 0xf0
Notice: Main bank load mask is 0x10
Page 0, main bank allocated
Page 1, main bank allocated
Page 4, main bank allocated
Page 5, main bank allocated
Page 30, 7808 tail bytes free
Page 41, 7808 tail bytes free
Page 125, 7808 tail bytes free
Creating dotn__DIV_001.bin (org 0x2000, 8187 tail bytes free)
The main binary is shown to occupy addresses 32768-33402. This will cover the lowest address in the main bank to the highest. It does not contain items normally external to the binary in z88dk builds - this would be the stack and heap (a note on this below when talking about the allocation mask).

In addition to this there is the loader and a portion of the code in the 8k divmmc memory at address 0x2000. The notice says there are still 5242 bytes available. This extra space must be at least around 150 bytes to hold the command line.

The main bank allocation mask indicates which 8k slots in the z80's 64k memory space will be allocated to hold the dot command's main binary (the 5,2,0 configuration). The mask is 0xf0 with bit 0 corresponding to mmu0 and bit 7 corresponding to mmu7. This mask shows that the loader will allocate pages for addresses 32k and up. This might seem excessive as the main binary only occupies 32768-33402 so allocating one page at mmu4 would be enough. However, this has to do with how z88dk normally treats the stack and heap. Usually these items are placed in memory outside the main binary. The default in dotn commands is to place the stack at the top of memory (setting it to 0) and it will grow down. The heap (which is disabled by default) is normally set to take the space between the end of the main binary and the bottom of the stack. So you can see, normally the program would occupy everything from the lowest code address to the top of memory. And that's what the default main bank allocation mask will always be set to.

The main bank load mask indicates which main bank mmu slots will be loaded by the dotn loader. The mask is 0x10 indicating only mmu4 will be loaded and that makes sense since the program only occupies the one page in address range 32768-33402.

Then information follows for each page added to the compile.

The final indicator mentions that a separate binary is produced to hold the divmmc page. divmmc memory is not made part of the output and if you needed it, your program would have to get it into memory somehow. It's only used here to verify that the dotn creation tool is working as intended.

The file used to read the dotn command contents is left open and is closed on program exit. You can get hold of the file handle with c function esx_m_gethandle() ( ... dos.h#L161 ) or via the M_GETHANDLE api function in asm. The file pointer will point just past all the parts loaded into memory, so you are free to append your own data to the dotn command and seek / load that at runtime if you have extra data for your program.

The following files are generated:

Code: Select all

65,536 DOTN
     0 dotn_UNASSIGNED.bin
 8,192 dotn__DIV_001.bin
"DOTN" is the output dotn command that is copied to /bin. It's 64k because it contains all the pages used by the program. The "*_UNASSIGNED.bin" should always be 0 size in z88dk compiles. This file will be non-zero in size only in error situations where something was not assigned to a section in the compile. The last file is the 8k divmmc page 1 created by the program. As mentioned, its presence means it was not added to the dot command itself.

How is the banking handled

The program is written like a normal one using any page numbers it likes. These page numbers are logical page numbers. z88dk knows what pages your program uses and creates a dotn command that will allocate physical pages for each page required. It constructs a table that translates logical page numbers to physical page numbers. So when your program wants to bank a new page into memory, it must pass this logical page number through the table to get a physical page number to do the actual banking. This is done in function "print_page()" in the example program:

Code: Select all

void print_page(unsigned char page)
   printf("Page %u\n\n", page);
   for (unsigned char *p = 0x6000; p != 0x6180; ++p)
      printf("%02x", *p);
The program wants to place logical page "page" into mmu3 (address range 0x4000-0x5fff). Instead of using "page" directly it passes it through the "_z_page_table[]" to get the corresponding physical page. On exit, logical page 11 is restored by again passing 11 through the table to get the physical page number.

Technical Tweaks

A number of compile details can be controlled with pragmas:

CRT_ORG_MAIN (default 0x8000)
The org in the main memory bank

DOTN_REGISTER_SP (default 0x4000)
The stack location used while inside divmmc memory. Normally you wouldn't change this.

DOTN_LAST_PAGE (default 223)
The last page in the logical to physical page translation table. The default is to create a translation for all pages in the zx next but you can shrink that to the last page your program uses. A check will be made by appmake at build time to ensure the page is big enough.

DOTN_EXTRA_PAGES (default 0)
You can ask to allocate additional workspace pages with nothing in them. If you need extra memory you can then get it at load time rather than having to request it inside your program at runtime.

You can tell appmake you want to overlay parts of the main memory bank with allocated pages. Bits 0 & 1 are ignored (these correspond to address range 0-16383 where the dot command lives), bit 2 corresponds to mmu2... bit 7 to mmu7. A set bit will cause appmake to allocate a page to cover the underlying memory in the main bank.

Again bits 2..7 correspond to mmu2 through mmu7. A set bit indicates that the dot should not allocate a page for those pages, instead loading into physical memory for those pages.

CRT_ENABLE_COMMANDLINE (default 3, same settings as other dot commands)
0 = no command line
1 = empty command line. argc = 1, argv[0] = ""
2 = unprocessed command line. A pointer to the esxdos command line zero terminated and length is passed to the program.
3 = parse the command line into words per normal with argc, argv passed.

REGISTER_SP (default 0 meaning top of memory)
Stack location while your program runs

CLIB_MALLOC_HEAP_SIZE (default 0, disabled)
The program's heap size for malloc() etc. Set to -1 to take the space between the end of your main bank and the bottom of the stack.

The following variables are created in divmmc memory so that they are always visible no matter what banking your program does in the top 48k:

* the command line accessed via argc/argv or raw pointer.

The size of the page translation table and the page translation table itself:

* extern unsigned char _z_page_table_sz;
* extern unsigned char _z_page_table[];

The size of the extra pages table (as indicated by DOTN_EXTRA_PAGES) and the extra pages physical page lookup table indexed from 0:

* extern unsigned char _z_page_extra_sz;
* extern unsigned char _z_page_extra[];

This _z_page_extra[] table *always* immediately follows z_page_table[], which means you can index the extra pages by giving a big index to z_page_table.


Now you can make huge dot commands, hundreds of k or megabyte in size that will not interfere with basic or other programs co-existing on the zx next. Imagine tools you can exit and re-enter (see upcoming functionality with environment variables to support this behaviour) or tools that are much more sophisticated than can be done in 7-8k.

Z88DK supports both C and asm so you can generate dotn commands that are pure C, pure asm or a mix of C and asm.
Last edited by Alcoholics Anonymous on Sat Jun 23, 2018 1:52 am, edited 1 time in total.

Alcoholics Anonymous
Posts: 463
Joined: Mon May 29, 2017 7:00 pm

Re: Generating Huge Dot Commands with Z88DK

Postby Alcoholics Anonymous » Sat Jun 23, 2018 1:23 am

Answering a question here:

The main bank is the 64k of memory that a program sees when it is loaded. On the spectrum this is normally 16k banks 5,2,0. If you LOAD"" or run an sna, the main program is loaded into 16k banks 5,2,0. 16k Bank 5 covers the range 16k-32k, Bank 2 covers 32k-48k and bank 0 covers 48k-64k. As 8k page numbers these correspond to pages 10/11, 4/5 and 0/1.

This model does not change for dot commands. For dotn commands, however, these 16k banks 5,2,0 are *logical* not physical. Your program calls them 16k banks 5,2,0 internally but the dot command will allocate physical pages that cover up the real 16k banks 5,2,0.

If the first byte in main memory that you use is at address 32768 - the start of mmu4 - then z88dk assumes that your program will occupy addresses 32768-65535, right to the top of memory. It will generate an "allocation mask" internally of 0xf0 where these set bits indicate that mmu4 through mmu7 are to be overlaid by your program in the main bank. The loader will allocate pages for mmu4-mmu7 and page them into the 32768-65535 address range before your program is loaded. This address range covers 16k banks 2,0 so the page table will have entries for pages 4/5, 0/1 that translate the logical 16k banks 2,0 to new physical page numbers. Ie - your program thinks it's using 16k banks 2,0 but really it's not.

You can alter this "allocation mask" for the main bank with pragmas.

The first pragma is DOTN_MAIN_OVERLAY_MASK. This value is ORed with the z88dk's internal allocation mask so that a set bit will get z88dk to allocate a page for the corresponding mmu slot in the main bank. In the example above, the internal allocation mask was 0xf0 so that mmu4 through mmu7 will be allocated. Suppose you want to move the stack just under 32768 (set REGISTER_SP=32768). To avoid writing on top of basic, you'd want to place an allocated page there in mmu3. So you can set DOTN_MAIN_OVERLAY_MASK = 0x08 and have z88dk allocate mmu3 as well.

The second pragma is DOTN_MAIN_ABSOLUTE_MASK. This mask is applied last to the effective allocation mask. Set bits indicate you do not want a particular mmu in the main bank to be overlaid with an allocated page. In the example given in the last post, the program only occupied addresses [32768,33402] - ie part of mmu4. The stack is at the top of memory by default. So really we only need to allocate pages for mmu4 and mmu7 to keep things nice with basic. So we can tell z88dk not to allocate pages for the rest of the mmu slots by setting DOTN_MAIN_ABSOLUTE_MASK = 0x6f. Originally z88dk was allocating for an allocation mask of 0xf0 = four pages in the main bank. With this mask, the allocation mask would be reduced to 0xf0 & ~0x6f = 0x90 or two pages in the main bank. The dot command now requests two fewer pages when run which means it is less demanding on memory resources.

z88dk will also be using this model for loading large programs that coexist with nextos.

While I remember, although layer 2 can seem to occupy any six consecutive 8k pages by using nextreg 0x12 ( to indicate which 16k bank layer 2 starts in (and notice nextreg 0x12 takes a 16k bank number and not a page number), layer 2 is actually restricted to the first 512k ram chip only. This means it must be entirely contained in 16k banks 0-15 (or 8k pages 0-31). Since NextOS itself occupies 16k banks 0-8, this leaves little room for layer 2 to be moved around. It's best if it's left where NextOS puts it as it's unlikely you will have available pages to move it elsewhere. You can find out where layer 2 is by reading nextreg 0x12.

Alcoholics Anonymous
Posts: 463
Joined: Mon May 29, 2017 7:00 pm

Re: Generating Huge Dot Commands with Z88DK

Postby Alcoholics Anonymous » Sat Jun 23, 2018 4:03 pm

Another question:

A dotn is still a dot command. This means while it runs, the rom is not present in the bottom 16k. Instead esxdos (or nextos-esxdos) will be present in a divmmc page in the 0-8k range and the dot portion of the dotn command will be in another divmmc page in the 8k-16k area.

Because esxdos duplicates rst$10 in its divmmc page, you can still use rst$10 to print and this is what z88dk's rst$10 driver for printf does when using startup=30. The rst$10 jumps to the esxdos rom and the esxdos rom does what is necessary to run rst$10 in ROM3.

To call a ROM3 (the 48k rom) subroutine at address x you must use "rst$18; defw x". rst$18 in the esxdos rom is another special subroutine that will do what is necessary to call x in ROM3 and return to the dot command.

Note that while using rom routines, including rst$10, the system variables must be visible in the memory map. This means mmu2 must contain physical page 10 (the first part of 16k bank 5). Because ROM3 is present in 0-16k while rom routines run, you can't refer to data from divmmc memory while running a rom routine. Be careful with this if you place some of your program's data into the dot command's 8k page. Also, if you use layer2 write-only mapping in the bottom 16k, that must be switched off while making rom calls or esxdos calls. Otherwise esxdos will not be able to write into its data structures in the bottom 16k

Alcoholics Anonymous
Posts: 463
Joined: Mon May 29, 2017 7:00 pm

Re: Generating Huge Dot Commands with Z88DK

Postby Alcoholics Anonymous » Tue Jun 26, 2018 7:25 pm

Another question:

DOTN_EXTRA_PAGES allows you to add additional workspace pages to the compile so that z88dk will allocate these for you before your program starts and it will deallocate these for you on program exit. So you can statically add additional pages if you know you'll need them ahead of time.

If you want to dynamically allocate pages at runtime, ie your program decides it wants another page, you must allocate those yourself using the IDE_BANK api and you must make sure you deallocate those pages before your program exits or these pages will remain unavailable until the machine is reset. You can get the crt provided by z88dk to run exit code when the program terminates and that is an ideal place to put your page deallocation code into.

There are two ways to have a function run on program exit.

One is by registering it using atexit() ( ... texit.html ) . It's part of the c standard library but this is all written in assembly language inside z88dk and works equally well with asm functions. You will have to change CLIB_EXIT_STACK_SIZE via pragma to the number of atexit() functions you want to register as it is set by default to 0. See ... figuration for details on other pragmas.

The other is by inserting asm code that calls your deallocation function into "section crt_code_exit". Any code placed into that section will be inserted into the crt at a point like this: ... sm.m4#L497 As you can see, this code will run after the program returns or after a basic error.

If further details on how to do these things are needed, just ask. I don't really want to write pages of documentation unless it's needed :)

Alcoholics Anonymous
Posts: 463
Joined: Mon May 29, 2017 7:00 pm

Re: Generating Huge Dot Commands with Z88DK

Postby Alcoholics Anonymous » Tue Jun 26, 2018 7:45 pm

Basic and the IY register.

While dot commands run, you want basic's interrupt routine to remain enabled. The reason is NextOS allows drivers to be installed that may have code that needs to run periodically. In the current NextOS, there is already a uart driver, a mouse driver and a kempston joystick driver that run during the interrupt.

What this means is your program cannot modify the IY register with im1 enabled. The dotn command generated by z88dk (and soon the other dot types) start with interrupts enabled by default to support the basic system. If you don't want basic's isr to run you can choose to start with interrupts disabled via pragma (CRT_ENABLE_EIDI see ... figuration ) or to change interrupt mode.

The problem with IY is that some library functions in z88dk can modify IY and zsdcc can use IY even if told not to in some circumstances. That causes problems with basic's isr. Most of the time IY will not be used and problems don't exist but if you want to guarantee there will be no problems, you must implement an im2 interrupt routine that restores IY and jumps to the basic isr at 0x38.

Creating an im2 routine can be easily done statically by a single asm file in a z88dk project. The hardest part is probably deciding where to place it and its vector table in memory.

Using the example in the top post, we have the main program occupying [32768,33402] and the stack at the top of memory. We'd like to leave as much space for the program to grow as possible so we try to place im2-related things as high up in memory as possible but also keeping in mind we want at least around 256 bytes for the stack. A possible candidate is to have the 257-byte im2 vector table occupy address range [0xfe00,0xff00]. This will leave 256 bytes for the stack which grows down from 0x10000 (initially set to 0). Placing all 0xfd bytes into that range will have the z80 jump to address 0xfdfd to service an interrupt. There's three bytes there underneath 0xfe00 which is enough space to jump to the real interrupt routine. This is set up by the following file:


Code: Select all

; Set up IM2 interrupt service routine:
; Put Z80 in IM2 mode with a 257-byte interrupt vector table located
; at 0xFE00 (after the program) filled with 0xFD bytes. Place a jump
; to the interrupt service routine at 0xFDFD which sits just underneath
; the vector table.

; create 257-byte im2 vector table

org 0xfe00

defs 257, 0xfd

; create interrupt service routine

org 0xfdfd

jp isr_basic

; initialize im2 mode inside crt before main is called

SECTION code_crt_init


ld a,__VECTOR_TABLE_head / 256
ld i,a
im 2

; Special interrupt routine that jumps to basic's isr at 0x38

SECTION code_user



   push iy
   ld iy,__SYS_IY   ; basic expects this value for IY, defined for zxn target

   call 0x0038        ; call basic isr, ints are enabled on return

   pop iy
Add "interrupt.asm" to your compile line and your dotn command will now be operating in im2 mode and calling basic's isr with the proper value of IY.

In this arrangement, im2 related structures and the stack occupy all addresses 0xfdfd and up. If your program uses the heap, you must inform the crt, again via pragma, that the heap's top address is 0xfdfc.

It may be worthwhile to add -Cz"--main-fence 0xfdfd" to the zcc compile line. This will tell appmake to check that your main binary does not exceed this address and it will emit a warning if it does.
Last edited by Alcoholics Anonymous on Tue Jun 26, 2018 11:55 pm, edited 1 time in total.

User avatar
Posts: 167
Joined: Fri Jun 23, 2017 1:13 pm
Location: Albuquerque, NM USA

Re: Generating Huge Dot Commands with Z88DK

Postby varmfskii » Tue Jun 26, 2018 8:04 pm

my understanding is that (at least for the original spectrum) the basic interrupt routines preserved all registers.
Backer #2741 - TS2068, Byte, ZX Evolution

User avatar
Posts: 189
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Generating Huge Dot Commands with Z88DK

Postby SevenFFF » Tue Jun 26, 2018 8:15 pm

BASIC expected IY to remain pointing at the sysvars, though, and didn't explicitly set IY in the ROM IM1 handler.

NextOS's 48K ROM is pretty close to the original 48K ROM, in terms of functionality. Most of the differences are in the 128/+3 ROMs and the divMMC ROM.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
Spectron 2084blog

Who is online

Users browsing this forum: No registered users and 2 guests