Feasibility of wifi file sync

Discuss game and other programming topics not specifically covered in another forum

Moderator: Programming Moderators

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Tue May 19, 2020 5:18 pm

Sync2 protocol:

Code: Select all

All server sent packages are encapsulated with
[2 bytes big endian whole packet size][payload][checksums, 2 bytes]

Where checksum1 and checksum2 are calculated by:
checksum1=0
checksum2 = 0
for every x in payload:
  checksum1 = checksum1 xor x
  checksum2 = checksum2 + checksum1

Handshake:
next: "Sync2"
server:"NextSync2"
- To check that we're talking the same language

Next file:
next: "Next"
server: [file length 4 bytes big endian][filename length 1 byte][filename]
- Request the next file to be synced
- If no files left, payload of five zeroes

Next chunk:
next: "Get"
server: [up to 1024 bytes]
- If file has ended, payload is empty

Retry:
next:"Retry"
server: resend previous payload

Otherwise:
next:???
server:"Error"
- For other requests, server returns "Error"
Changes:
- All packets handled the same way, with size field and checksums.
- Retry doesn't start file over, but retries the most recent packet.
- More robust checksums

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Feasibility of wifi file sync

Postby varmfskii » Tue May 19, 2020 5:48 pm

how are you poling the UART?
Are you checking if the buffer contains data for every byte read?
Backer #2741 - TS2068, Byte, ZX Evolution

User avatar
SevenFFF
Posts: 565
Joined: Mon Jun 05, 2017 5:30 pm
Location: USA

Re: Feasibility of wifi file sync

Postby SevenFFF » Tue May 19, 2020 5:49 pm

sol_hsa wrote:
Tue May 19, 2020 5:12 pm
The whole protocol is next-driven, so the server doesn't send anything unless the next asks for the next bit of data, so the SD card speed doesn't matter.
Agreed, provided you flush the RX buffer before you start. In particular, any ESP reset will fill the buffer with a huge load of crap and some init text. Or the user may have hit Next reset during a previous data transfer, and that continues to fill the buffer without any receiving code running.
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
I can see that there may be problems in two places: either the uart buffer is overrun, or we stop waiting for data to arrive too early for some reason. Maybe the wifi packet is fragmented and parts of it arrive too late, or something.
Agreed. Sending the length as part of the protocol is a better way to deal with this, as I mentioned. Then you can have much longer timeout while you are still waiting for data, and still finish immediately when the length is reached. If you don't send the length, then your long timeout will still be triggered after receiving is complete but before you know it is.
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
When I upped the z80 clock and things broke, it was a case of emptying the uart fifo too fast and giving up on receiving. Adding more delays made things work. That was with the default bit rate, though.
I hit this issue with both NXTP and ESPUPDATE. I would write code with fixed delays which worked 100% great on my own board, but failed when other users ran the same code. I think there are physical differences between ESP hardware, and also between ESP firmware versions. I only really solved this with lots of trial and error, and by making my timeouts VERY long. See above note about completing early if you know the length.
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
What's still a mystery to me is what the video clock have to do with any of this. I'm using the prescaler calculation code from .uart source so that part should be dealt with.

And why does this work for me (with vga4 50hz as far as I recall), but other people are having trouble.
I agree video mode shouldn't matter if you handle it correctly. However I know the .uart code is out of date, and doesn't yet write to port 0x153B where it should:

Code: Select all

0x153B UART control
(R/W)
bit 6 = 0 to select the esp uart, 1 to select the pi uart **
bit 4 = 1 if the bits 2:0 are being written
bits 2:0 = most significant bits of the 17-bit prescalar setting baud rate
* pi gpio must be configured for uart, see nextreg 0xa0
** either uart can be redirected to the joystick ports, see port 0x37
So you should write 0 to bit 6, 1 to bit 4, and 000 to bits 2:0, ie. %00010000. Somebody else may have previously set bits 2:0 to a different value if they used a very low baud (unlikely) or they may have previously confgured the UART for the Pi (very likely).
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Tue May 19, 2020 7:29 pm

I'm currently working on a rather large rewrite of the uart i/o stuff and naturally everything has blown up in my face. There I was, thinking this was a small change..
varmfskii wrote:
Tue May 19, 2020 5:48 pm
how are you poling the UART?
Are you checking if the buffer contains data for every byte read?
While uart has data, read to buffer. Once it doesn't, look what's in the buffer. If the buffer doesn't contain what we want, try reading more from uart.
SevenFFF wrote:
Tue May 19, 2020 5:49 pm
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
The whole protocol is next-driven, so the server doesn't send anything unless the next asks for the next bit of data, so the SD card speed doesn't matter.
Agreed, provided you flush the RX buffer before you start. In particular, any ESP reset will fill the buffer with a huge load of crap and some init text. Or the user may have hit Next reset during a previous data transfer, and that continues to fill the buffer without any receiving code running.
If that's true, I've probably gotten really lucky since things worked fine for me. I wasn't pre-flushing the buffer by any means.
SevenFFF wrote:
Tue May 19, 2020 5:49 pm
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
When I upped the z80 clock and things broke, it was a case of emptying the uart fifo too fast and giving up on receiving. Adding more delays made things work. That was with the default bit rate, though.
I hit this issue with both NXTP and ESPUPDATE. I would write code with fixed delays which worked 100% great on my own board, but failed when other users ran the same code. I think there are physical differences between ESP hardware, and also between ESP firmware versions. I only really solved this with lots of trial and error, and by making my timeouts VERY long. See above note about completing early if you know the length.
Ugh.
SevenFFF wrote:
Tue May 19, 2020 5:49 pm
sol_hsa wrote:
Tue May 19, 2020 5:12 pm
What's still a mystery to me is what the video clock have to do with any of this. I'm using the prescaler calculation code from .uart source so that part should be dealt with.

And why does this work for me (with vga4 50hz as far as I recall), but other people are having trouble.
I agree video mode shouldn't matter if you handle it correctly. However I know the .uart code is out of date, and doesn't yet write to port 0x153B where it should:

Code: Select all

0x153B UART control
(R/W)
bit 6 = 0 to select the esp uart, 1 to select the pi uart **
bit 4 = 1 if the bits 2:0 are being written
bits 2:0 = most significant bits of the 17-bit prescalar setting baud rate
* pi gpio must be configured for uart, see nextreg 0xa0
** either uart can be redirected to the joystick ports, see port 0x37
So you should write 0 to bit 6, 1 to bit 4, and 000 to bits 2:0, ie. %00010000. Somebody else may have previously set bits 2:0 to a different value if they used a very low baud (unlikely) or they may have previously confgured the UART for the Pi (very likely).
Added that write just in case. Okay, I doubt the video timings matter at all as long as the application can read from the uart, i.e, either the application works or completely doesn't re: video timings.

User avatar
varmfskii
Posts: 287
Joined: Fri Jun 23, 2017 1:13 pm
Location: Stone Mountain, GA USA

Re: Feasibility of wifi file sync

Postby varmfskii » Tue May 19, 2020 7:36 pm

sol_hsa wrote:
Tue May 19, 2020 7:29 pm
I'm currently working on a rather large rewrite of the uart i/o stuff and naturally everything has blown up in my face. There I was, thinking this was a small change..
varmfskii wrote:
Tue May 19, 2020 5:48 pm
how are you poling the UART?
Are you checking if the buffer contains data for every byte read?
While uart has data, read to buffer. Once it doesn't, look what's in the buffer. If the buffer doesn't contain what we want, try reading more from uart.
Ok, sounds like you are doing the right thing.
multibyte:
loop: check uart for data
no data goto loop
read data
write data in buffer
increment count
if count=total return
goto loop
single byte
loop: check uart for data
if no data goto loop
read uart
return
Backer #2741 - TS2068, Byte, ZX Evolution

User avatar
sol_hsa
Posts: 273
Joined: Fri Jun 02, 2017 10:10 am
Location: Finland
Contact:

Re: Feasibility of wifi file sync

Postby sol_hsa » Wed May 20, 2020 3:05 pm

I rewrote the data receive function in assembler to make sure it runs fast enough. The 0.6 release still uses the 1152000 bit rate, but I've tried with 2000000 and it worked without problems. I'm waiting for reports on whether this version works before making any decisions on that front.


Who is online

Users browsing this forum: No registered users and 4 guests