September 2019 Update

New! We're moving to the new forum, eventually it will replace this blog. Please join us there.

View the Most Recent Feature Sheet by clicking HERE!

Tuesday, July 16, 2019

Sensor Module update - confirmation of database inserts

Nope, don't need em...


Oh, you'd like to know why?

I'm still looking into how to better manage sending data between the Mega 2560 and the ESP8266 which share a circuit board and hardware serial interface, but certainly any decrease in the amount of data sent back and forth will help.

Originally, when the Sensor Module booted up, the 2560 would request a copy of the module configuration from the 8266 by sending an XML request string over the serial port. The 8266 would query the mysql database over the WiFi connection, and return the configuration data to the 2560 over the serial port, again, in XML strings.

Then the Sensor Module requests a copy of the Sensor Configuration, a list of all it's sensors... the 8266 again, gathers the data and sends it back over the serial port. 

The 2560 scans each of the sensors to get its reading, and sends it all back to the 8266 again, which writes the data to the mysql database. The 8266 then sends the confirmation list back to the 2560 with the record id from the database for each record inserted.  Nothing is done with that information other than displaying it on the 2560 serial monitor output...  That's a lot of back and forth just to save the sensor readings.

Each time the module does a sensor scan it also reloads the module and sensor configurations....

With all this back and forth, it was hard to tell where data was being lost when some data was not being written.

First off, I added an Acknowledgement function on the 8266, so when the 2560 sends data, it can, in certain instances, expect an acknowledgement.   

So now, the Sensor Module loads it's Module Configuration ONCE at boot up.  Unless I change the configuration, reloading it is a complete waste of bandwidth. Originally, I didn't expect the system to be as complex as it is,  and this never would have been an issue. I will let the 8266 check the last update timestamp every 10 minutes or so, and if it finds an updated record, it can just go ahead and send it to the 2560 as required.

I THINK I'll do the same with the Sensor Configuration...   I say think, because I'm not certain yet... Changes to the sensor configuration is more likely to change than the module, and checking a last update timestamp on every sensor all the time would be a resource hog as well, so some thinking to do on this one still...

Now, sending the data from the 2560 to the 8266, and sending confirmations back.  Here's where I made the biggest change.  Because the 2560 sends all the sensor data to the 8266 BEFORE the 8266 starts writing to the database, I was never guaranteed it got all the data...  I know that on occasion, records got scrambled or lost...  Now, each sensor data record is sent, and then the 2560 waits for a specific acknowledgement from the 8266 for that sensor. If it does not receive the ack within 5 seconds, it will resend the data for that one sensor. It will retry up to 5 times. Once it receives the ack, it moves on to the next record, and sends the data for the next sensor...

The 8266 now has all the sensor data, so it goes off and writes it to the database, and that's it, there is no confirmation response back to the 2560, since the 8266 already acknowledged receiving each record, and is therefore responsible for ensuring it gets saved. 

Now, because there is no indication of when the 8266 finishes writing all the data from a scan, the possibility exists that the 2560 will attempt to send more data before the 8266 is ready. In this case, there will be no acknowledgement, so the 2560 will simply retry sending it...

Right now I've got this new code running on Sensor Module 2, which controls the sensors in the Flower Tent, and out of the first 100 readings, all 100 made it to the database...


No comments:

Post a Comment

Any comments deemed off topic or offensive will be removed