Simple hex dumper with source

Show us your work, thrill and amaze us :)

Moderator: Programming Moderators

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Simple hex dumper with source

Postby sol_hsa » Fri Apr 24, 2020 7:38 am

To try out making a dot command, I wrote a simple hex dumper.
Image
Compiles with z88dk 2.0 (at least) with:

Code: Select all

zcc +zxn -v -startup=4 -clib=sdcc_iy hexdump.c -o hexdump -subtype=dotn -create-app
I figured going printf-free would yield a smaller binary but either I haven't used the magic command line parameters to do so, or z88dk is unable to make binaries smaller than 16kB. Anyway, the compilation is horribly slow even without optimization flags on; I don't know what exactly is going on there. My other (much larger) sdcc projects compile faster.

However, since I want to make dot commands that use the esxdos in c, I don't know a better option right now, but I guess I could research things some more..

Code: Select all

#include <errno.h>
#include <stdio.h>
#include <arch/zxn.h>
#include <arch/zxn/esxdos.h>

static uint32_t filelen(unsigned char fh)
{
    struct esx_stat es;
    if (esx_f_fstat(fh, (struct esx_stat *)&es)) 
    {
        return 0;
    }
    return es.size;
}

static const char hex[17] = "0123456789ABCDEF";

void hexout2(char d)
{
    putchar(hex[(d >> 4) & 0xf]);
    putchar(hex[(d) & 0xf]);
}

void hexout32(uint32_t d)
{
    putchar(hex[(d >> 28) & 0xf]);
    putchar(hex[(d >> 24) & 0xf]);
    putchar(hex[(d >> 20) & 0xf]);
    putchar(hex[(d >> 16) & 0xf]);
    putchar(hex[(d >> 12) & 0xf]);
    putchar(hex[(d >> 8) & 0xf]);
    putchar(hex[(d >> 4) & 0xf]);
    putchar(hex[(d) & 0xf]);
}

// "puts" adds newline which we don't want
void myputs(char *s)
{
    while (*s)
    {
        putchar(*s);
        s++;
    }
}

int main(int argc, char *argv[])
{
    uint32_t len = 0;
    uint32_t p = 0; 
    unsigned char i;
    unsigned char f = 0;
    unsigned char b[16];
    zx_cls(PAPER_WHITE);
    if (argc < 2)
    {
        myputs("Hexdump usage:\nGive a filename to dump\n");
        return 0;
    }
    
    errno = 0;
    f = esx_f_open(argv[1], ESX_MODE_READ);
    if (errno != 0)
    {
        myputs("Unable to open ");
        myputs(argv[1]);        
        myputs("\n\n");
        return 0;
    }
    len = filelen(f);
    
    myputs("Hex dump of ");
    myputs(argv[1]);
    myputs(" - ");
    hexout32(len);
    myputs(" bytes\n");

    // 1234567890123456789012345678901234567890123456789012345678901234
    // 00000000  00010203 04050607  08090A0B 0C0D0E0F  0123456789ABCDEF
    
    while (p < len)
    {
        // Clear the buffer to spaces so the last line doesn't bug
        for (i = 0; i < 16; i++)
        {
            b[i] = ' ';
        }
        esx_f_read(f, b, 16);
        hexout32(p);
        putchar(' ');
        putchar(' ');
        for (i = 0; i < 4; i++, p++) 
        {
            if (p <= len)
            {
                hexout2(b[i]);
            }
            else
            {
                putchar(' ');
                putchar(' ');
            }
        }
        putchar(' ');
        for (i = 4; i < 8; i++, p++) 
        {
            if (p <= len)
            {
                hexout2(b[i]);
            }
            else
            {
                putchar(' ');
                putchar(' ');
            }
        }
        putchar(' ');
        putchar(' ');
        for (i = 8; i < 12; i++, p++) 
        {
            if (p <= len)
            {
                hexout2(b[i]);
            }
            else
            {
                putchar(' ');
                putchar(' ');
            }
        }
        putchar(' ');
        for (i = 12; i < 16; i++, p++) 
        {
            if (p <= len)
            {
                hexout2(b[i]);
            }
            else
            {
                putchar(' ');
                putchar(' ');
            }
        }
        putchar(' ');
        putchar(' ');
        for (i = 0; i < 16; i++) 
        {            
            if (b[i] >= ' ' && b[i] <= 127)
            {
                putchar(b[i]);
            }
            else
            {
                putchar('.');
            }
        }
    }
    
    // enough newlines to make sure speccy doesn't wipe our bottom rows
    myputs("\n\n\n\n");
    esx_f_close(f);    
    return 0;
}

Alcoholics Anonymous
Posts: 781
Joined: Mon May 29, 2017 7:00 pm

Re: Simple hex dumper with source

Postby Alcoholics Anonymous » Fri Apr 24, 2020 2:13 pm

You can see some existing dot commands made with z88dk in the repository:
https://gitlab.com/thesmog358/tbblue/-/ ... otCommands

The 8k ones are cowsay, cp, extract, mv, run, strings. There are a few other ones for which source is not available eg more.

Extract has a simple hex viewer built in that adjusts the output according to the currently selected number of columns in basic (32/64/85). But I can see garry's done something in nextzxos so that is not so reachable anymore for dot commands -- I will have to have a look.

.extract foo

Your output looks a little nicer. Something that allows hex editing would be useful.

--

z88dk has a much more sophisticated library than sdcc. It has a file implementation modelled on unix so there are underlying streams in the system. It's preferable to use the built-in functions puts() etc as they are written in asm and take advantage of the stream primitives (write one char, write one char many times, write a string, etc). You do have to decide whether you want printf or scanf as they will pull in the printf/scanf core. However the situation is not as bad as sdcc as these things are also written in asm and you can control which printf/scanf converters you want included in the compile. Eg, you can choose to have "%s" only so that these things can be made relatively compact. The examples above all use printf for reasons that will be explained below and most of them limit the printf converters to what is wanted with suitable pragma in an associated file "zpragma.inc" or similar.

In addition to this, z88dk's model has drivers at the end of streams to receive bytes being read or written. When you're using startup=4, you are getting a canned configuration that has a pseudo-64 column driver on stdout and stderr and an edit line facility on stdin. The startup=4 driver has window and scrolling capability and includes a 4-bit font in the compile since it is using the 256x192 screen with 4-bit wide chars (there is another startup that uses the timex 64 column mode with the normal rom font which will be smaller). The stdin side you are not using. It is line oriented by default so the user can edit a line before the result is sent to the program as is normal in any standard terminal (scanf, gets, getline, etc). So now you should see why it is larger than an sdcc compile -- there are windowed drivers, some kind of terminal emulation (special codes to set print positon, colour, etc), a font and edit line facility being put in the binary.

Dot commands are a special case of program. They can be invoked from basic and can be part of the basic program. Normally you want the dot command not to change anything about the running program's display, or at minimum return without having changed anything. So normally you want the driver to be the rom routine and when you select that (startup=30, no stdin) rom code via rst$10 will be used for all output. Nextzxos has more in it than the zx spectrum rom. You can change the display resolution of rst$10 and can send the same print codes as is understood from basic and is described in the manual. When doing things this way all the print code is now in the rom and the size of the dot command can be considerably reduced.

I can return later to show you a compile using your example code above.

Ped7g
Posts: 256
Joined: Mon Jul 16, 2018 7:11 pm

Re: Simple hex dumper with source

Postby Ped7g » Fri Apr 24, 2020 3:52 pm

One of the things which is lingering in my mind is hexa editor, as one of the first things to try to do in 80x42 tilemode, if I finally release it in the shape I will be happy about (it's now taking months.. omg).

But I will surely write it in asm-only, so this is kinda off-topic comment in this thread. :)

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Simple hex dumper with source

Postby sol_hsa » Fri Apr 24, 2020 5:41 pm

I understand z88dk does way more than what I'd need (like handling malloc and all sorts of things). The build times are rather slow, though, so I'll probably try to rig my own toolchain based on sdcc.. I'm a bit of an odd bird that way that on one hand I don't want to write direct assembly but on the other hand z88dk does a bit too much =)

The hex dumper isn't meant to compete with anything, it was just a test to see how the tools work.

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Simple hex dumper with source

Postby varmfskii » Fri Apr 24, 2020 10:04 pm

With some of the app targets for the zx next, malloc in z88dk doesn't work. The only time build times are slow is when you use aggressive optimization options.
Backer #2741 - TS2068, Byte, ZX Evolution

Alcoholics Anonymous
Posts: 781
Joined: Mon May 29, 2017 7:00 pm

Re: Simple hex dumper with source

Postby Alcoholics Anonymous » Sat Apr 25, 2020 6:33 am

sol_hsa wrote:
Fri Apr 24, 2020 5:41 pm
I understand z88dk does way more than what I'd need (like handling malloc and all sorts of things). The build times are rather slow, though, so I'll probably try to rig my own toolchain based on sdcc.. I'm a bit of an odd bird that way that on one hand I don't want to write direct assembly but on the other hand z88dk does a bit too much =)
You have the freedom to use as little or as much as you want in z88dk -- it is highly configurable in its builds. By selecting startup=4 you are choosing to use certain functionality.

Have you turned on optimization in your sdcc builds? It is slow in the extreme and dominates the time of any build. I have stuff that takes 30-40 minutes to build and it is all sdcc but that only happens if you turn up the optimization.

z88dk doesn't just use sdcc, it attempts to fix its bugs and correct its poor code generation when it generates poor code. The quality of output is higher, generating both faster and smaller code. Because the libraries are written in asm, the advantage there is around 3-4x in size and speed, nevermind the difference in scope in the two libraries

Beyond that, z88dk understands its targets. For the zx next in particular, it knows about the bankswitched memory model (both legacy 128k and new mmus), it knows there is a z80n processor so that the new opcodes are allowed, it knows about hardware like the copper and dma and can perform simple checks on copper and dma programs, it defines constants for all the zx next hardware, and it knows about all the output formats so it can automatically build dot, dotn, tap, nex, etc.

For z80-related projects there is no advantage in using sdcc directly. You are free to use what you want but sdcc on its own will never be on par :) This should be expected from a tool that specializes in zilog-related cpus compared to one that is meant to be portable across several different cpus.
varmfskii wrote: With some of the app targets for the zx next, malloc in z88dk doesn't work.
It will always work but you may have to define where the malloced memory comes from. The default is to take it from the end of the program to the bottom of the stack and that makes sense for most programs and makes it very easy to use in most cases. But in some program types, that space is not available so you will have to take more control of where the heap is.

The malloc implementation can support many heaps, not just one, and that can help in a bankswitched environment as well where you might be using different heaps for different memory configurations.

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Simple hex dumper with source

Postby sol_hsa » Sat Apr 25, 2020 9:51 am

Alcoholics Anonymous wrote:
Sat Apr 25, 2020 6:33 am
Have you turned on optimization in your sdcc builds? It is slow in the extreme and dominates the time of any build. I have stuff that takes 30-40 minutes to build and it is all sdcc but that only happens if you turn up the optimization.
Yes, I found that out quite fast. Still, the builds even without optimization enabled can be slow. Building the source code at the first post with the command line in the first post (which doesn't include optimization options, unless those are hidden behind other options) takes about 16 seconds on this ryzen7 system, which is pretty much forever considering what it's doing.
Alcoholics Anonymous wrote:
Sat Apr 25, 2020 6:33 am
z88dk doesn't just use sdcc, it attempts to fix its bugs and correct its poor code generation when it generates poor code. The quality of output is higher, generating both faster and smaller code. Because the libraries are written in asm, the advantage there is around 3-4x in size and speed, nevermind the difference in scope in the two libraries

Beyond that, z88dk understands its targets. For the zx next in particular, it knows about the bankswitched memory model (both legacy 128k and new mmus), it knows there is a z80n processor so that the new opcodes are allowed, it knows about hardware like the copper and dma and can perform simple checks on copper and dma programs, it defines constants for all the zx next hardware, and it knows about all the output formats so it can automatically build dot, dotn, tap, nex, etc.

For z80-related projects there is no advantage in using sdcc directly. You are free to use what you want but sdcc on its own will never be on par :) This should be expected from a tool that specializes in zilog-related cpus compared to one that is meant to be portable across several different cpus.
I understand a lot of work has gone into z88dk and it's great at what it does. And yes, I've had plenty of grief using sdcc, at some version they broke something I had been using and when I complained the answer was that the feature had never worked, so maybe I was hallucinating or something. Regardless, sdcc produces code that I can use and which can be used as a starting point for hand-optimized assembly. If only sdcc's assember itself wasn't broken and incompatible format with others.. =)

What attracted me to making hobby projects for the 48k spectrum is that it's possible for me to understand everything that's going on. However, I'd rather not write everything in assembly. So, I feel that sdcc is a nice halfway point there. Not perfect by any means. z88dk solves a lot of problems, but it also adds a lot of stuff I don't know about.

dom
Posts: 4
Joined: Fri Aug 18, 2017 9:47 pm

Re: Simple hex dumper with source

Postby dom » Sat Apr 25, 2020 5:40 pm

This is a curious issue, but I now understand why it's never really been spotted before.

Running on a virtualised windows 10 (i5-6260U), the command supplied runs in about 14 seconds.

Compiling with -clib=new (so using sccz80) takes about 9 seconds.

Compiling with zcc +zxn -clib=classic hexdump.c -o hexdump -subtype=dot -create-app -lesxdos -v - so using the classic library to create a dot file compiles in 3.8 seconds.

Running on my ancient MacBook (i7-3557U), the times are: 3.8, 1.8 and 0.75 seconds respectively which is altogether much more acceptable.

Looking at the log as it's printing to screen, it appears that most of the time is being spent at the linking stage. What I suspect is happening is that z80asm is searching for library routines by opening the library file, scanning through for the routine, closing the file and doing it again for the next library routine. On *nix this behaviour goes un-noticed, but because (in my limited experience) Windows is horrendously slow at file IO and newlib is decomposed into lots of small routines it becomes really noticeable.

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Simple hex dumper with source

Postby varmfskii » Sun Apr 26, 2020 1:23 pm

Alcoholics Anonymous wrote:
Sat Apr 25, 2020 6:33 am
sol_hsa wrote:
Fri Apr 24, 2020 5:41 pm
varmfskii wrote: With some of the app targets for the zx next, malloc in z88dk doesn't work.
It will always work but you may have to define where the malloced memory comes from. The default is to take it from the end of the program to the bottom of the stack and that makes sense for most programs and makes it very easy to use in most cases. But in some program types, that space is not available so you will have to take more control of where the heap is.

The malloc implementation can support many heaps, not just one, and that can help in a bankswitched environment as well where you might be using different heaps for different memory configurations.
It may have changed as z88dk is in a constant state of change (this is a good thing), but I have written code that when using malloc, would compile and work fine with some app targets and would not compile with other app targets. If I removed the allocation and simply used a statically allocated chunk of memory it would also work on all app targets. So, for some definition of doesn't work, malloc didn't work for all app targets.
Backer #2741 - TS2068, Byte, ZX Evolution

dom
Posts: 4
Joined: Fri Aug 18, 2017 9:47 pm

Re: Simple hex dumper with source

Postby dom » Thu May 14, 2020 5:24 pm

Paulo has merged in a change to improve z80asm linking performance, and the improvements are quite dramatic:

Running on the machine specs I mentioned above, the results are in:

Windows:

-clib=sdcc_iy: 6.4s was 14s
-clib=new: 2.2s was 9s
-clib=classic: 1.6s was 3.8s

MacOS:

-clib=sdcc_iy: 3.5s was 3.8s
-clib=new: 1.2s was 1.8s
-clib=classic: 0.6s was 0.75s

So for windows, compilation times, especially for sccz80 builds now seem to be reasonable.


Who is online

Users browsing this forum: No registered users and 1 guest