These boards have a built in ESP8266, complete, even with it's own pin header if you want to use it. However, in my case, I am communicating between the 2560 and 8266 wuth a provided hardware serial link. I have a choice of which Serial device I can use on the Mega, and have all my switches set to Serial3
So I have a nice fast 115,200bps Serial connection between the two, and they write data back and forth to each other. Some of that data are requests for data, so for example, the 2560 will request a copy of it's configuration from the 8266, since the 8266 has WiFi, and a MySql Database Connector.
So the 2560 sends a request to the 8266, which generates the appropriate sql statement, executes it on the MySql server via the MySql Connector, reads back the data from the database, repackages it for the 2560's needs, and then fires it back over the serial port to the 2560.
Another scenario is when the Sensor Module scans all the sensors, and then sends all the data at once to the 8266, which has to parse through it, repackage it, do some lookups for more data, then generate sql statements to write the data to a log table, as well as update a "current" readings table, and then reread the log data to ensure it was written.
Once it has completed all the database updates, the 8266 loops though the array of sensor data, and sends a list back to the 2560 with the database record id included, this acts as confirmation that the records were properly written to the database.
The 2560 reads this list (array) and then displays it on the Serial output.
Now, these "packets" of data consist of XML records, well, not XML to spec, but a bastardized version..
The "payload" is the name of the "table" in XML... so for example, we may send a request for the module configuration, and include the module Id with it. This "packet" would look like this:
The first <MODULEREQ> is the payload indicator, it is a single xml tag, alone on a line, ending with a NewLine. When the parser encounters this, it will save this name in a "payload" field. When it encounters the same text, but in a closing tag, as in </MODULEREQ> it then sees the end of the record.
While it has a "payload" name, the parser will look for predefined fields, <MODULEID> for example... Fields are defines with both opening and closing tags on the same line with the data between them, and a newline at the end of the line.
Special fields include a set of tags specifying that we are done with this record, <DONE></DONE> Both opening and closing tags on the same line, ending with a newline. There is no data associated with these tags. If a payload contains multiple records, it will look like the following, notice each record has a <DONE> tag, but at the very end, there is an <ALLDONE></ALLDONE> tag set, this signifies no more records...
It's all very easy to parse the incoming data this way.
You will notice, there most of the size of this data packet consists of a description of the data record, ie, field names in the opening and closing tags... In order to be meaningful while reading the code, the names must be long enough to make sense or it will be impossible to fix any bugs...
So to send 14 bytes of data, we actually sent 169 characters plus 9 newlines... Not very efficient..
To top this off, there is no flow control on the serial line between the two boards, so while a lot of data can lead to issues, the problem was occuring even when there was not a lot of data, but it was data that required work on the receivers part, and therefore it was not fast enough to keep up., and because of lack of flow control, the data was just being discarded...
Ok, so here's what was happening...
2560 sends sensor readings to 8266
8266 takes 60-90 seconds to complete it's processing, meanwhile, the 2560 is monitoring the serial line for a confirmation packet... The 8266 sends this packet, and the 2560 goes off to display it on the serial monitor line, and by the time it comes back, more data has arrived, followed by updates fired off from timers, like the current WiFi Signal Strength, etc... These can fire off before the previous data has been read, overflowing the buffer, and causing loss of data, which means if the 2560 requests its module configuration, but the data gets "jumped on" by say the RSSI data, then even if the 2560 read 95% of the record, it won't see the <DONE> tags, and won't finish processing it, and will report a timeout receiving data... This is what was confusing me, I knew it was sending the data, but didn't really realize the implications of not having proper flow control, or, to be honest, not realizing that it wasn't implemented in the libraries.
My decision to use the above methods were made very early in the project, actually, even before it was a project... I found an XML library, and found it interesting the way he parsed the data, it was so simple... I implemented my own version, although it was entirely based on that existing code. I wish I still had the link to it so I could give proper credit, but I never thought to save it... I generally paste links right into my code when I "borrow" ideas from others, but it's not very often... At the time, there was very little data to be sent back and forth, and almost no other processing going on on either board, so the problems I'm having now did not manifest themselves, in fact, it was not until I was fully invested in using this method that small problems started popping up. I fixed them with bandaids, adding delays, and flushes, anything I could that would make it work...
Now, I completely 100% understand the problem, and the breakdown of what is happening. I know that I need to implement flow control, it doesn't have to be fancy, just a way to say, Ok, Send Me Some Data, or Hold On a Sec... Whether I will continue to use XML or not, I'm not sure, I think if I can make flow control work, I suspect the amount of data is not the issue, and knowing when it is ok to send will make all the difference in the world.
I'm going to spend some time playing with this, and see what changes I'd like to make... do some experimenting... I'll let you know what I find :)