Feasibility of wifi file sync

Discuss game and other programming topics not specifically covered in another forum

Moderator: Programming Moderators

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Feasibility of wifi file sync

Postby sol_hsa » Thu May 14, 2020 5:27 pm

I've seen a few topics on the subject of using the wifi to transfer files to the next. These threads generally end up with "get flashair".

I started looking into the idea; my concept would be to have a python server on the host machine and a .sync command on the next. When .sync is called, it contacts the python server, which then checks for changed files in a specific directory and sends those (and only those) to the next. That's the basic idea.

When looking a bit deeper I started to ponder how feasible this would actually be. The optimal transfer rate is about 14kB/s (given uart speed of 115200/8=14400). Add handshaking and error checks, possible delays on writing to sd card, and whatnot, I'm not sure what the actual rate would end up being, possibly not even half of that.

If we're pessimistic and say the effective transfer rate was 4kB/s, it would take 12 seconds to transfer a 48k file. Still borderline usable assuming the transferred files aren't huge. If the transfer time goes much higher than that, however, it's probably faster to physically move the sd card around.

There's some ways to make things faster, like doing binary comparison on the host and only sending data from the changed bytes onwards; or the data could be compressed with zx7 before transmission (assuming the data wasn't compressed to begin with), but these are just tricky hacks.

Maybe that flashair is a better option after all..

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Feasibility of wifi file sync

Postby varmfskii » Thu May 14, 2020 5:35 pm

I can manage faster than that using zxnftp. Of course I can also use faster data rates than 115,200 (even up to 2M). That said it does not transfer anywhere near the theoretical limits.
Backer #2741 - TS2068, Byte, ZX Evolution

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Thu May 14, 2020 6:52 pm

2,000,000 baud is pretty reliable. Switching bauds complicates matters because both the UART and the ESP must be switched to the same baud in order to communicate. A good ESP citizen program should probably put the ESP back to 115200 before exiting.

You can always run:

Code: Select all

.uart -f 
to "find" the ESP, which systematically tries every baud rate, and resets it back to 115200. This should recover the ESP even if it was permanently put to a nonstandard baud line 1,234,567.

To use the ESP in anything other than VGA0 mode, your program should set the UARTs baud using the prescaler. This applies even if you are sticking to 115200. Since the program author can't control the video mode their software might be run in (unless it's a personal build), they should always set the baud using the prescaler.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Thu May 14, 2020 8:00 pm

It's good to know the speed can be increased, so that's not a blocker.

I wonder why nobody has done this yet? I know there's a lot of unknowns and as such I'm not horribly confident that I could do it, but I know others might find it easier..

So far I've managed to connect the next to a echo server using the terminal, so I know it can be done.

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Thu May 14, 2020 8:51 pm

varmfskii allows zxnftp to run at 2meg, I think. I'm not sure if nihirash does with uGophy.

I chose not to with NXtel and NXTP, as the payloads are small. In NXtel it already takes many times longer to render a 960 byte page in layer 2 than it does to receive it at 115200. NXTP only sends and receives a handful of bytes anyway.

I did experiment with >115200 in ESPUPDATE, but I didn't get great reliability in low-level programming mode. Since anything >115200 in programming mode requires changing the baud rate half way through after switching from the ROM stub to the uploaded stub, I stuck with 115200 for maximum simplicity and reliability. You really want maximum reliability when flashing firmware. It wouldn't permanently brick it since there's a ROM loader, but it's still a pain to have a flash fail.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Sun May 17, 2020 11:38 am

Image

It's a start..

I just had a sd card corruption while copying files back and forth with the next and my pc, which just reminds me why I want to have this kind of tool.

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Sun May 17, 2020 12:49 pm

While doing this I noticed that the post https://www.specnext.com/the-next-on-the-network/ has obsolete uart speed setup information. This page is referred to from several places, so it might be a good idea to revise it..

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Sun May 17, 2020 7:12 pm

I've made some steady progress. Not transferring files yet, but filenames and -sizes go across fine.

I've kept most of the routines in C so far to ease development, and as such figured I might as well boost the clock to speed up the text rendering, and found that suddenly I can't talk to the uart anymore.

This might be a case of my timeouts being too short or something, or it might be something else. I have no idea.

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Mon May 18, 2020 5:59 am

The problem with 28MHz was that I managed to drain the uart fifo and stopped receiving before the transfer was complete. Added a little timeout to the receive and things run (more) smoothly now.

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Mon May 18, 2020 1:50 pm

Did you see there are ready flags for both send and receive. It's ok to have timeouts, but you must use check these flags before sending and receiving every byte.

Code: Select all

0x133B UART Tx
(R)
bit 2 = 1 if the read buffer is full
bit 1 = 1 if the transmitter is busy sending a byte
bit 0 = 1 if the read buffer contains received bytes
(W)
Send a byte to the connected device.  There is no Tx buffer so the program must make sure the last transmission is completed before sending another byte.
It's also a good idea to design your protocol so that the data packet length is sent before the data packet, then you can semantically detect a good receive without needing to rely on timeouts.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins


Who is online

Users browsing this forum: No registered users and 1 guest