August 2019 Update

While this site is still somewhat new, and still under construction, this project has been underway since roughly August 2018. Please excuse any unfinished pages, I'm still catching up!

New!
View the Feature Sheet by clicking HERE!

New to the site?

Project information can be found on the "Grow By Wire Site Links" panel. The right side of the screen consists of my blog posts, a window into my thought process as I work on the project.

Tuesday, August 13, 2019

We're gonna have fun for a few nights!

It's always nice when you get the opportunity to just completely rewrite something from scratch.  That's not something that ever occurred in real life as a professional programmer :)  

I began my programming career as a shareware programmer, and back then, this was something I did on a somewhat regular basis. In the real world, it's always a matter of finding someone to pay for the rewrite, but when it's just me, I can afford to do it. Sometimes I can't afford NOT to :)

The benefit of a complete rewrite is twofold, first, a refresher on just what it does, and how, and second, the opportunity to make improvements. In this case, the Sensor Module was the very first piece of Grow By Wire, written when I was still learning c++, the arduino, and not really having a clear picture of where I was going with the system.  This meant a ton of modifications as the system grew, changes in the way things worked, you get the idea...  So the code changed so much, there are pieces of old code still there, some commented out, some not. So I may think something is done a certain way, but maybe I made a change and it's different now... There's no way I can remember all the changes, everywhere in the system, since day one!  A complete rewrite lets me streamline the code so only code that is used is present, this alone will make it easier to come back and see how something works.

The second, and I think an even better reason to rewrite the code from scratch, I have learned so much in the past year, I have a much deeper understanding of the hardware, the software, libraries, shared code, you name it... This means that I can take advantage of more complex concepts, giving me more ways to accomplish things.

For example, when I first started using the Wemos Mega with the built in ESP, I needed a way to package data to send it back and forth on the serial line between the two modules. Being old school, I was familiar with XML, so went that route. Being somewhat unfamiliar with c++ and the arduino, I opted to copy some code from an XML library. I made some changes, but the concept was the same, and I managed to get it working, and have been using that same code  every since.

Today while rewriting the software, I was able to make more sense of it, in fact, make improvements, and streamline the code which processes the XML. I will have better control over preventing cross-talk where a new XML record suddenly appears in the middle of an existing one which is still being processed.

Initially I figured the rewrite could be done in a day... now I'm thinking a week :)

Update: 10:30pm

Everything is progressing well, no more troubles, although I did spend a lot of time on the routines that pass data back and forth between the 2560 and the 8266. I was able to beef up the XML parsing, and I get to fix something else that is just messy...

When I send the module configuration to the Mega 2560 from the ESP8266, it is sent in XML strings which make parsing the data simple.

For example, if I want to send a text string to display on the serial output, I have a method called sendText("hello world");

This generates the following XML

<MSG>
    <TXT>Hello World</TXT>
</MSG>

The first line is the PAYLOAD TAG, or name of the record.
The second line contains a TAG, some DATA, and a CLOSE TAG
The last line contains the PAYLOAD CLOSE TAG

The 8266 sends these three lines over the serial interface, and the 2560 reads them one character at a time, and parses the data as follows:

It sees the opening <  in <MSG> and knows it starts a TAG, so starts saving the string as a TAG.
As it adds each character to the string, it compares it with a list of PAYLOAD names that it has been coded to look for. In this case, <MSG> is a valid PAYLOAD, so it is caught by this line:

if (strcmp_P(xmlTag, PSTR("<MSG>")) == 0) setPayload();


On the second line, it finds a TAG <TXT>, again, it is specifically looking for this TAG, in this PAYLOAD.  This allows using the same TAG names in different PAYLOADS.

This is caught with:


It sees that the current PAYLOAD is <MSG>, and looks for the TAG <TXT> specifically. If it finds it, it can act on it, in this case, simple print the DATA portion to the Serial port so I can read it on the serial monitor.


That's a simple payload, only one tag, and one action.

Here is the Module Configuration PAYLOAD


In addition to having multiple lines, it also includes two sets of TAGS at the bottom, <DONE> and <ALLDONE>. These are used when there is more than one line of data, this is because the design allows the data to come in any order, as long as it is enclosed in the proper TAG

A better example would be Sensor Configuration data. There are many fields for each Sensor, such as it's ID, it's Name, the Type, etc, long list...  So the <DONE> signifies that we have read all the data for that sensor. When we see the <DONE> TAG, we know we can go and do whatever processing we need to do on that data, because it is complete.  Then we can process another sensor, and another, until we see <ALLDONE> which means we are finished ALL sensors...

Each of the TAG names is the same as the field name in the database, which is the same as the variable name in the code, everything matches, so we can simplify the code which sends the XML.

The above Module Configurations is sent using this code:



You may notice, I use a view in the database, this lets me perform all the joins and field renaming right on the server, making this code very streamlined.

The old way was to read all the data from the database, then loop through the data and write each field out, but using a different TAG name. Originally I never though of doing it the way I am changing to now. Here's the original code:

First it saves the data in a variable

if (strcmp(cols->fields[f]->name, "sensorName") == 0)
{
  snprintf(sensorName, sizeof(sensorName), "%s", row->values[f]);
}


It does this for each field it reads from the database. Once it has loaded all the fields for a record, and all the records for a query, it then spits them out as XML enclosed strings...

Serial.print("<SENSORNAME>");
Serial.print(sensorName);
Serial.println("</SENSORNAME>");

So now the name of the TAG is hardcoded, and is not consistent, and as you can see, requires a lot more code, remember, I'm only showing you the code for one field, the sensor name... There could be 50 or more fields for each sensor...

Changing it to use the same field and TAG name throughout means a lot less code, which means using less memory, which on an Arduino is akin to finding cash in an old jacket pocket! 

It also means less chance for mistyping a TAG name, and the smaller code is much easier to understand.

Seems to me that once I finish rewriting the Sensor Module code, I'll want to do the Switch Module and Automatic Watering Module as well...  

Perhaps it makes sense to move some of this code into a library, that will make rewriting the other modules a lot faster, but slow down the rewrite of this one.

ok, my 10:30 update has turned into 11:30, I best quit the jabbering and get back to work...




No comments:

Post a Comment