Feasibility of wifi file sync

Discuss game and other programming topics not specifically covered in another forum

Moderator: Programming Moderators

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Mon May 18, 2020 2:34 pm

SevenFFF wrote:
Mon May 18, 2020 1:50 pm
Did you see there are ready flags for both send and receive. It's ok to have timeouts, but you must use check these flags before sending and receiving every byte.
Yes, I'm already watching those flags. It seems that in the 28MHz mode it's possible to drain the uart fifo (i.e, bit 0 here) while data is being fetched.
SevenFFF wrote:
Mon May 18, 2020 1:50 pm
It's also a good idea to design your protocol so that the data packet length is sent before the data packet, then you can semantically detect a good receive without needing to rely on timeouts.
Sounds like a good idea.

I've managed to transfer the first file successfully. Yayy!

Before letting other people try it I'm still missing configuration of the host; it's currently hardcoded. I'm thinking of saving this to a file so that the config is (typically) a single-time thing. What would be the place to put this file? 'c:/sys' seems to have env.cfg, and 'c:/sys/config' has a couple .cfg files at least.

The bottleneck for transfer doesn't seem to be the uart speed, but the sd write speed. Over half of the transfer time goes into writing to disk. I'm using 1k chunks which should be pretty much optimal for sd writes. It's possible that I'm not using the fastest sd card on the planet, though.

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Mon May 18, 2020 2:39 pm

Okay, that was weird. Now that I tried again, in a subdirectory of the sd card, the write was fast. So I guess I should investigate that uart speed change at some point..

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Mon May 18, 2020 3:51 pm

Ahh, being able to use the sync to develop the sync is soooo much nicer than having to move the sd card back and forth. I'm so happy I decided to do this project before working on the .nex file project.

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Mon May 18, 2020 6:11 pm

The tool is starting to be feature complete. I couldn't get 2000000 bps to work reliably, but 1152000 is plenty fast. I still have a few tweaks to do, and then I get to writing documentation..

User avatar
dundarach
Posts: 3
Joined: Tue May 30, 2017 7:41 am

Re: Feasibility of wifi file sync

Postby dundarach » Mon May 18, 2020 6:59 pm

This looks really interesting. It would be great to sync over wifi and cheaper than buying flashair cards!! :)

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Feasibility of wifi file sync

Postby varmfskii » Mon May 18, 2020 10:35 pm

When I send or receive a file, I let the other end know how bit the file is, then send fixed size chunks with acknowledgement in between with the final block being short. This way, I don't risk over filling the RX buffer and I know exactly how much data to expect. What I haven't done is implement any sort of recovery protocol. That means that if something goes wrong, it currently falls apart. The next version (whenever I get the time to do it) will have some sort of resiliency built in, even if it means a reduction in throughput.
Backer #2741 - TS2068, Byte, ZX Evolution

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Tue May 19, 2020 12:00 am

I meant to say this earlier, but I forgot. The ESP receive buffer is 2KB, and the UART receive buffer is 512 bytes. You won’t get any indication the ESP buffer has overflowed. The UART flag will tell you when the UART buffer has overflowed, but by that time data will have already been lost.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Tue May 19, 2020 3:53 am

Release 0.5 here: https://www.specnext.com/forum/viewtopi ... =17&t=1715
which seems to work.

Protocol:

Code: Select all

Handshake:
next: "Sync1"
server:"NextSync1"
- To check that we're talking the same language

Next file:
next: "Next"
server: [file length 4 bytes big endian][filename length 1 byte][filename]
- Request the next file to be synced
- If no files left, packet of five zeroes

Next chunk:
next: "Get"
server: [up to 1024 bytes][xor checksum 1 byte]
- The next chunk of data, followed by xor checksum for this chunk 
  (all bytes xor:ed together)
- If file has ended, single zero byte is sent

Retry:
next:"Retry"
server:"Ok"
- Server rewinds file to the start

Otherwise:
next:???
server:"Error"
- For other requests, server returns "Error"
It's a rather simple protocol. I'll revise it at some point to include packet size at the start of the server packets to make the timeout handling easier on the next side. Also a bit less aggressive retry (i.e, only retry the latest chunk) might be nice.

I picked the chunk size of 1024 bytes because that seems to fit nicely in a wifi tcp packet, sd cards tend to write in chunks of 512 bytes, it fits into the wifi buffer and I can get away with a reasonable buffers in software.

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Tue May 19, 2020 4:00 pm

It doesn’t fit in the UART buffer though, so you will have to be very timely in emptying that. I don’t know if this is why you had reports of unreliability. The buffer could easily be full before you start, so you should drain it first.

It would be better to use a chunk size which fits in the UART buffer.

Since transferred files can probably can be larger than the entire Next RAM, you are really at the mercy of the SD chunk save speed, which can vary with environmental factors as you discovered. So a protocol which uses chunks <=512 and which delays the send of the next chunk until save is complete is the only way you can be completely robust. This is probably why you had issues with 2meg yourself.

Congrats on the release :)
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Tue May 19, 2020 5:12 pm

SevenFFF wrote:
Tue May 19, 2020 4:00 pm
It doesn’t fit in the UART buffer though, so you will have to be very timely in emptying that. I don’t know if this is why you had reports of unreliability. The buffer could easily be full before you start, so you should drain it first.

It would be better to use a chunk size which fits in the UART buffer.

Since transferred files can probably can be larger than the entire Next RAM, you are really at the mercy of the SD chunk save speed, which can vary with environmental factors as you discovered. So a protocol which uses chunks <=512 and which delays the send of the next chunk until save is complete is the only way you can be completely robust. This is probably why you had issues with 2meg yourself.

Congrats on the release :)
The whole protocol is next-driven, so the server doesn't send anything unless the next asks for the next bit of data, so the SD card speed doesn't matter.

I can see that there may be problems in two places: either the uart buffer is overrun, or we stop waiting for data to arrive too early for some reason. Maybe the wifi packet is fragmented and parts of it arrive too late, or something.

When I upped the z80 clock and things broke, it was a case of emptying the uart fifo too fast and giving up on receiving. Adding more delays made things work. That was with the default bit rate, though.

Upping the bit rate and breaking does sound like the buffer may be overrun. At 2 megabits we're talking 250 000 bytes per second. At 28MHz that means we have around 100 clocks to process a byte, which isn't a lot. I think you're correct!

What's still a mystery to me is what the video clock have to do with any of this. I'm using the prescaler calculation code from .uart source so that part should be dealt with.

And why does this work for me (with vga4 50hz as far as I recall), but other people are having trouble.


Who is online

Users browsing this forum: No registered users and 3 guests