September 2019 Update

New! We're moving to the new forum, eventually it will replace this blog. Please join us there.

View the Most Recent Feature Sheet by clicking HERE!

Thursday, August 15, 2019

Sensor Module Rewrite Update #1

Posted: 10:45pm

If you've read my previous posts, you know I'm in the middle of a complete rewrite of the Sensor Module. This includes both the ESP8266 and Mega 2560 firmware.

I'm also making a huge change in the way it works. The old version (V1) has the Mega 2560 orchestrating everything, and using the ESP8266 as a gateway to the database and wifi,

This new version (V2) has the ESP8266 orchestrating everything, and only using the Mega 2560 to read sensors attached to it's pins.

Also part of this rewrite involved an overhaul of the way data is sent between the two modules, which are hooked directly together using a hardware serial line, on the same shared circuit board.



This is the Wemos Mega 2560 with the built in ESP8266.  The ESP chip is circled in yellow, so you can see, the serial connection between them is short!  We can pump data at 115,200 bps back and forth without problems....

I'll be doing updates on this post through the night as I progress with the rewrite.


Update: 11:45pm

Another benefit of changing things up and putting the ESP in charge of orchestrating everything, I can easily update the firmware using OTA (Over The Air) updates using WiFi, and the code on the 2560 should be minimal, and not require as many updates.

One of the goals here is to clean up the code, and this involves removing some features, ones that seemed so important when I added them.  I guess remove isn't the right word, just not adding them to this version.


Update: 5:00am


I have it reading analog sensors now. Here's what I've got so far:

The SENDER sends data over the serial link, packaged much like XML. I hesitate to call it XML any more, so I'll use that term loosely :)  The PAYLOAD is sent, along with a CRC32 checksum so now when the  RECEIVER reads the data, it can know with certainty that it received the entire payload and it was intact.  If the CRC32 calculated by the RECEIVER doesn't match the value calculated by the SENDER, then it sends a <NACK> (Negative ACKnowledgement), if it does match, it sends an <ACK>  The SENDER will wait for this response, and if none comes, it will be treated as a <NACK>.  The SENDER can then resend the payload, how many times it will retry is dependent on the importance of the data being sent.


The ESP boots up, fetches the time from the database using a SELECT NOW() and passes the result to the Mega 2560, which uses the timer class to keep time. Every 10 minutes, the ESP8266 fetches the current date/time and sends it to the Mega 2560. The Sensor Module only uses the time for display on the log shown on the serial monitor. All time and dates saved to the database are generated by the database itself.

Once it receives confirmation that the Mega received the time correctly, the ESP8266
fetches the Module Configuration from the database, this tells the Sensor Module what its moduleId is, how often to scan the sensors, how many sensors are attached, and some other information about this module. The ESP sets a Blynk timer to fire at the interval specified in the module configuration. This will be used to do the sensor scans.

Now the ESP sends information about the current network connection for the
Mega to display at boot up. The ESP then sets a timer to fire just once, 5 seconds in the future. This will trigger the first scan of all sensors. (A Module level scan, as opposed to a Sensor Level scan where a specific sensor has it's own schedule). 

On the ESP, the timer fires, and firsdt, it refreshes its list of active sensors attached to this module. At this point, it only loads the sensorId for each.

The ESP now loops through the list of sensorIds, and for each sensor, it fetches the sensor configuration record, and sends a limited number of fields to the Mega. It only sends the bare minimum data for the Mega to be able to read the sensor.



The Mega acknowledges the payload, then performs the sensor read, which can involved doing a power off/on test to ensure a sensor is actually present and working, then reading the sensor 100 times to get an average reading, and it sends the reading back using a limited number of fields. The ESP acknowledges receiving the reading.

This is the point I'm at now... 

The next step is to save the reading to the database, but quite a bit of work needs to be done first, such as looking up more information that is needed for the log.  When saving data to the log, we don't want to save an id for something, such as a Location, because in the future, if we reconfigure the layout, the location id may no longer point to the correct location. So for any data we will want to look at in the future, we save the name of the location. There are other things looked up and saved with the record in the log table.

Of course, before we can save the record, we also have to validate the data and do any calculations on the raw readings to arrive at useful data, such as calculating the temperature from a thermister reading, or the soil moisture as a percentage. In the previous version the calculations were based on the sensor type, and were all hard-coded.

In this new version, rather than doing the calculation based on the sensor type, we have a table describing all the possible calculations, which are still hard-coded in the firmware, but each calculation has an Id, so that when you set up a sensor, you assign a calculation to it, so the module knows which calculation to use.  This mean two identical sensors could do two different calculations. An example might be two thermisters, one reporting degrees F and the other degrees C.

Once all the data validation and calculations are complete, the ESP generates a SQL insert statement, and inserts the data into the log table, as well as another table with only the current entry from the log table (tbl_log_sensor_readings and tbl_latest_sensor_readings)

The ESP then moves on to the next sensor in its list, and repeats the above until it finishes them all.

At the next sensor scan interval, it dies it all over again...

No comments:

Post a Comment

Any comments deemed off topic or offensive will be removed