August 2019 Update

While this site is still somewhat new, and still under construction, this project has been underway since roughly August 2018. Please excuse any unfinished pages, I'm still catching up!

New!
View the Feature Sheet by clicking HERE!

New to the site?

Project information can be found on the "Grow By Wire Site Links" panel. The right side of the screen consists of my blog posts, a window into my thought process as I work on the project.

Tuesday, August 20, 2019

Cool new gadget... OLED Screen for Modules

In this previous post I mentioned that part of rewriting the Sensor Module was to remove bloated code that isn't used any more. Because for the most part, I'm learning as I go, sometimes I discover something that I just have to include.

I'm afraid that may never end...


It's a 0.96" OLED screen, and it plugs right into the I2C buss, runs on 5v, and contrary to the poor quality of the picture, the screen is extremely easy to read under any light conditions. It was under $3 at AliExpress, I grabbed two, but will be grabbing more now that I see how easy they are to use.

I haven't figured out what I need it for yet, just that I need it :)


Dazed and Confused...

I'm stumped...  The new firmware is working extremely well, except for two things...

The module is configured to scan all attached sensors at specific intervals. When it's time, it fetches a lit of sensorId's from the database, and then loops through that liat and loads the configuration for each sensor.  

For some reason, the first sensor configuration is loaded ok, but the rest are not, the records are empty, or in some cases contain weird data that appears to be left over from a previous query, like field names...  I can't put my finger on the problem...  My quick solution at the time was to simple close the database after each use, which has the effect of clearing out the memory buffers used, so the query works now, however, the whole process is slowed down as it needs to keep connecting to the database every query.

The second issue just popped up last night. When a sensor fails the Sensor Test, three things happen:


  1. Send a warning to the Serial Output
  2. Send a notification to my phone using Blynk
  3. Write an entry to the debug log database table


For whatever reason, it will not write the database entry (while every other read and write is working fine, including many writes to the debug log table) and causes the ESP8266 to reboot.  If I remove the call to write the debug entry, it works just fine.

While I have a workaround for both issues, they must be a symptom of something being quite wrong. Whatever is causing this could be causing all sorts of unseen issues, and those can be the most dangerous bugs :)

So, I've tried all sorts of things, and if I can't get a handle on this soon, I guess I'll start over. That won't be so bad, I'll aim to get the sensor test code done first, and see if it still has the same problem, if not, it will be a simple matter of adding the rest of the code back in until it fails...

It looks like I have a game plan, it's certainly going to set me back timewise, but I'm sure I'll learn a lesson here somewhere. :)

Monday, August 19, 2019

The "Mega Shield" Part 3

Link to Part 2

Posted: 10:30pm

A complete rewrite of the Sensor Module firmware deserves a hardware upgrade as well doesn't it?

Introducing the "Sensor Module Backpack" AKA "The Mega Shield"



I've added the 12v to 5v converter, and the Analog and Digital pins will be powered from this power supply which is capable of providing up to 3A.  3A is overkill, but power for the sensors won't be a problem :)  I'm also powering the I2C bus from this as well.

I took that picture before I plugged it in, just in case :)  Good news, no smoke, no bad smells, just what appears to be a functioning unit.   Now I'll take some voltage measurements on the pins just to be sure I hooked it up to the right side of the converter :)  If all goes well, I'll plug some sensors in and see if they work using the transistors to switch the power on and off...


Sunday, August 18, 2019

Effortless Clones - finally!

I honestly think this is the ultimate cloning method. Last year, after years of very successful cloning using peat pellets, a humidity dome, and a spray bottle, disaster struck, and I couldn't get clones to root no matter what I did.

For some reason, I was getting 1 out of 15 or more that would root, that was it. Nothing changed in my methods.  I even tried a DWC clone machine, but after one or two good runs, it was a flop. It was so bad I was having to reveg my plants after harvest, or I would have run out :(

Then I created the "Mister Clone Clone Mister" - I'm kidding about the name. lol...

The Automated Clone Mister project, it's purpose, to ensure the clones are misted regularly to keep them properly moist at all times.  I've now done 3 batches of clones, and hit 90-95% sucess rates, al rooted within 2 weeks. This batch, I have 6 rooted at 10 days out of 20. Very very pleased, PLUS I don't have to do anything, check on them every couple DAYS, not hours...  just make sure  the water jug is full, and at one week, I like to change the water and wash the trays etc.

Here's the setup...

I have a plastic shelf unit, with only 3 shelves, and the space between the 2nd and 3rd shelf is surrounded with Panda Plastic, with the white side inside. There is a LED light inside, the kind you might find in your hallway... works good for the clones and seedlings in a protected area...



On the bottom is a heated seedling mat. I have not hooked this up to a relay yet, but if I needed to turn it on and off I could. As it is, leaving it plugged in all the time was perfect.

Over the heated mat are some 1/4" square dowels, just to lift the seedling tray up off the mat so there is an air gap, to prevent too much heat...





The seedling tray sits on the wooden  pieces, and is filled with water, just enough to float the smaller seedling tray inside. This allows the heat to warm the water, which keeps the upper tray warm, but also provides lots of humidity, which the dome traps inside.  Note the multiple trays... I'm paranoid about leaking water :)






The dome is set on an angle, this allows air flow, plus it aligns the big hole in the end with the spray bottle, so when it sprays, it mists the entire inside of the dome, and all the plants.  With two of the smaller trays floating inside, I was able to fit 20 cuttings inside at once all in peat pellets.




Initially for the first 3 or 4 days, I have it set to spray 5 times every 5 minutes, then every 10 minutes, then every 20 as time passes and the cuttings start making roots.

Once I see roots poking through the peat pellets, I plant them in the red Solo cups, and move them under the 250W MH light. 









Saturday, August 17, 2019

An afternoon ramble...

When I embarked on this project, I was quite a rookie. Sure, I'd been a programmer for nearly 40 years, and that experience was invaluable, but not directly useful.

First off, I had just been through the worst part of my life, from the tragic loss of my Mother and Brother, the night before Christmas to a Carbon Monoxide leak, to a near addiction to opioids, not street drugs mind you, these were prescribed like smarties by the "pain clinics", to a near-complete mental breakdown.

This project was meant to give me something to focus on as I struggled to quit the pharmaceuticals and get ON the pot, so to speak... I had just started growing my own, my first legal grow, and had just completed my first build with an arduino. It was a carbon monoxide alarm, how fitting eh?  I actually made it for the car, since we used to be into car camping...

Anyhow, after struggling through that build, I saw the potential to have an arduino monitor the soil moisture level, and there it was, the beginning of an idea...

My point in this rather long rambling introduction is that it was all new to me. And I love to explore new stuff...  Every time I discovered a nifty feature, I had to include it in the project...  Originally all the modules had LCD Screens, Realtime Clocks, multiple LED's, etc.  That's just hardware... the software was so bloated with "cool things" that it started causing problems.  Mainly lack of memory... So I removed the LCD, and the RTC, each having required a library to be loaded.

So now that I'm at the point of doing a complete rewrite, I'm able to pick and choose what will actually go back into the code.  There are also places where I moved the task to another module, such as when I created the maintenance module, or the web server module. That left chucks of code that really don't need to be there, just taking up memory...

Oh look, it's 4:20! Gotta run...

In an orderly fashion...

Posted: 2:30pm


Things are looking pretty darn good right now (knock on wood!).  I don't know how to describe it, but watching it run is like watching a precision military marching band whereas in the past, it was like watching recruits wearing clown shoes...

There is no more overlapping data being sent over the serial port, everything is based on acknowledgements, and knowing what happens next.

For example:

The 8266 sends a scan request to the 2560, and waits for an acknowledgement
The 2560 receives the scan request, and acknowledges to the 8266
The 2560 scans the specified sensor
The 2560 sends the response to the 8266 and waits for an acknowledgement
The 8266 receives the response and sends an acknowledgement to the 2560


That by itself works perfectly... which is how I tested it while writing it, one sensor...

When it loops through a list of sensors, something happens...

Once the 8266 sends the scan request, and gets the acknowledgement, that's it, that is considered one transaction, a complete one...

The fact that the result came back was like a pleasant surprise.... but one the 8266 was waiting for, it had nothing else to do....

But, if once it sends the first scan request, and receives the acknowledgement, it merrily goes off to send the scan request for the next sensor.  Now we have colliding data again.

Lets look at the above statement again...

There is no more overlapping data being sent over the serial port, everything is based on acknowledgements, and knowing what happens next.

You can see the acknowledgments in action in the above scenario, but the knowing what happens next is the new part today...

When we send the scan command, we need to have it acknowledged so we know it was received complete. But we also know that's not the end of the conversation, we are expecting it to do something, and send back a response, and because we know this, and are expecting it for a scan command, we can anticipate it, and wait for it before we move on, just like we wait for an acknowledgement. That's the concept, and it works, I handled it a little differently thoug. Since I want to wait until I have not only received the results payload, but have finished processing it, and saved it to the database, I am using a flag that I set on the sensorCfg record when I have finished with it. 


Friday, August 16, 2019

Ever have one of those days?

Posted: 10:05pm

I've been struggling all day with a problem in the new code.

The ESP8266 has an array which holds the sensorId for any attached sensors. When it is time for the module to scan the sensors, it first fetches this list of sensorIds from the database.

Then it loops through the array, and for each sensor, it loads the configuration. 

Like this

SELECT sensorId from vw_active_sensors where moduleId = 3
Loop the results and add into array sensorIdList[x]

The array can hold 20, there is only one sensor....

Once it has loaded the list of id's, it then loads the config for each, then sends the SCAN payload to the 2560 so it will read that specific sensor and send back the result.

That process occurs repeatedly at a configurable interval, I've set it to one minute for my testing...


So the first time through, it loads the sensorId (#55)

Then it loops the array, and loads the configuration for sensor #55

The request is sent to the 2560 to scan the sensor, and it reports back with the results.

(I don't have it saving to the database yet)

That's it for the first cycle, but it will repeat in one minute, fired up by the Blynk Timer.

The second time through, and all subsequent times, it cannot load the list of sensorIds, no results are loaded, or a whole bunch of fields show up, but they are from the sensor Configuration query?


SELECT `sensorId` FROM `growbywire`.`vw_active_sensor_info` WHERE `moduleId`=3

During the read, I print out the field names and their values, and look at this


FieldName     : Field Value
---------------------------------
sensorTypeId : def
sensorName   :growbywire
moduleId       :vw_active_sensor_info


If I comment out the actual execute of the sql in the configuration load, then the list load works fine. 

It took me all day to get to the point where I could actually pinpoint where it was happening, repeatably...

All I can think is some sort of memory issue, since it still had all the field names from the previous query, but I've been using this library since day one, and never have seen anything remotely like this.

I even thought maybe it was because I was calling this from the blynk timer event handler, but I just called it from loop() every 30 seconds, same problem...

At this point, I'm happy to know where the problem is, and that I can get around it. All I need to do is close the database connection and reconnecting before I try loading the list, and it's fine. Closing the database releases the buffers according to the author, really adding weight to my theory of a memory problem.

I wonder if this has happened in other places, but I'm just not aware of it? It was so repeatable, I can't see how I'd miss it.


Update: 3:05am

I ended up closing the database after every time I access it.  I did see this occur when it was trying to fetch the current date/time from the database using a SELECT NOW();  I wonder if this has been an ongoing thing, or did I do something to suddenly cause this?  Closing the database after each use has solved it, I let it run while I had a nap :)

I initially decided to leave the database open all the time, since I was accessing it so much. The decision was mainly for speed.

Thursday, August 15, 2019

Sensor Module Rewrite Update #1

Posted: 10:45pm

If you've read my previous posts, you know I'm in the middle of a complete rewrite of the Sensor Module. This includes both the ESP8266 and Mega 2560 firmware.

I'm also making a huge change in the way it works. The old version (V1) has the Mega 2560 orchestrating everything, and using the ESP8266 as a gateway to the database and wifi,

This new version (V2) has the ESP8266 orchestrating everything, and only using the Mega 2560 to read sensors attached to it's pins.

Also part of this rewrite involved an overhaul of the way data is sent between the two modules, which are hooked directly together using a hardware serial line, on the same shared circuit board.



This is the Wemos Mega 2560 with the built in ESP8266.  The ESP chip is circled in yellow, so you can see, the serial connection between them is short!  We can pump data at 115,200 bps back and forth without problems....

I'll be doing updates on this post through the night as I progress with the rewrite.


Update: 11:45pm

Another benefit of changing things up and putting the ESP in charge of orchestrating everything, I can easily update the firmware using OTA (Over The Air) updates using WiFi, and the code on the 2560 should be minimal, and not require as many updates.

One of the goals here is to clean up the code, and this involves removing some features, ones that seemed so important when I added them.  I guess remove isn't the right word, just not adding them to this version.


Update: 5:00am


I have it reading analog sensors now. Here's what I've got so far:

The SENDER sends data over the serial link, packaged much like XML. I hesitate to call it XML any more, so I'll use that term loosely :)  The PAYLOAD is sent, along with a CRC32 checksum so now when the  RECEIVER reads the data, it can know with certainty that it received the entire payload and it was intact.  If the CRC32 calculated by the RECEIVER doesn't match the value calculated by the SENDER, then it sends a <NACK> (Negative ACKnowledgement), if it does match, it sends an <ACK>  The SENDER will wait for this response, and if none comes, it will be treated as a <NACK>.  The SENDER can then resend the payload, how many times it will retry is dependent on the importance of the data being sent.


The ESP boots up, fetches the time from the database using a SELECT NOW() and passes the result to the Mega 2560, which uses the timer class to keep time. Every 10 minutes, the ESP8266 fetches the current date/time and sends it to the Mega 2560. The Sensor Module only uses the time for display on the log shown on the serial monitor. All time and dates saved to the database are generated by the database itself.

Once it receives confirmation that the Mega received the time correctly, the ESP8266
fetches the Module Configuration from the database, this tells the Sensor Module what its moduleId is, how often to scan the sensors, how many sensors are attached, and some other information about this module. The ESP sets a Blynk timer to fire at the interval specified in the module configuration. This will be used to do the sensor scans.

Now the ESP sends information about the current network connection for the
Mega to display at boot up. The ESP then sets a timer to fire just once, 5 seconds in the future. This will trigger the first scan of all sensors. (A Module level scan, as opposed to a Sensor Level scan where a specific sensor has it's own schedule). 

On the ESP, the timer fires, and firsdt, it refreshes its list of active sensors attached to this module. At this point, it only loads the sensorId for each.

The ESP now loops through the list of sensorIds, and for each sensor, it fetches the sensor configuration record, and sends a limited number of fields to the Mega. It only sends the bare minimum data for the Mega to be able to read the sensor.



The Mega acknowledges the payload, then performs the sensor read, which can involved doing a power off/on test to ensure a sensor is actually present and working, then reading the sensor 100 times to get an average reading, and it sends the reading back using a limited number of fields. The ESP acknowledges receiving the reading.

This is the point I'm at now... 

The next step is to save the reading to the database, but quite a bit of work needs to be done first, such as looking up more information that is needed for the log.  When saving data to the log, we don't want to save an id for something, such as a Location, because in the future, if we reconfigure the layout, the location id may no longer point to the correct location. So for any data we will want to look at in the future, we save the name of the location. There are other things looked up and saved with the record in the log table.

Of course, before we can save the record, we also have to validate the data and do any calculations on the raw readings to arrive at useful data, such as calculating the temperature from a thermister reading, or the soil moisture as a percentage. In the previous version the calculations were based on the sensor type, and were all hard-coded.

In this new version, rather than doing the calculation based on the sensor type, we have a table describing all the possible calculations, which are still hard-coded in the firmware, but each calculation has an Id, so that when you set up a sensor, you assign a calculation to it, so the module knows which calculation to use.  This mean two identical sensors could do two different calculations. An example might be two thermisters, one reporting degrees F and the other degrees C.

Once all the data validation and calculations are complete, the ESP generates a SQL insert statement, and inserts the data into the log table, as well as another table with only the current entry from the log table (tbl_log_sensor_readings and tbl_latest_sensor_readings)

The ESP then moves on to the next sensor in its list, and repeats the above until it finishes them all.

At the next sensor scan interval, it dies it all over again...

Are we having fun yet?

Posted: 11:30pm


I sure am! :)

If this wasn't fun, I wouldn't be doing it.

This complete rewrite of the Sensor Module has given me the freedom to not only fix bugs, but to do some major enhancements.  

Last night I decided to add CRC32 checksums to all data sent back and forth between the Mega and ESP on the serial line. After struggling to get the CRC code to spit out the same value on the ESP and Mega (yeah, it was my fault) I was finally able to get it working, but by then, I'd been at it for almost 24 hours...

I had some sleep today, and got started about 9pm, and it's 11:30 now, and I have one "transaction" complete.  It's a simple one, the ESP sends the current time to the Mega. The ESP gets the time from the database with a  query as neither the ESP nor the Mega have a real time clock. I originally had one on the Mega, but it was a cheap one that drifted all the time, and I ended up having to fetch the current time from the database to keep updating the clock.  Why bother, this works just fine. Remember, the modules only use the timestamps for display on the Serial Monitor. All timestamps on data being inserted or updated in the database are generated by the database, ensuring the entire system is operating on a synchronised time.

The complete transaction includes:

ESP8266 sends DateTime to Mega 2560 along with the CRC32 of everything up to the end of the closing ALLDONE tag.


The Mega 2560 parses the incoming characters building the XML strings, and doing a running CRC32 calculation. When it encounters the closing ALLDONE tag, it also grabs the value it has in its running tally, and saves it, because the next line is the CRC32 value from the ESP8266. The two values are compared, and either an ACK, or a NACK is sent back to the ESP which has been waiting for one or the other, but also will time out if nothing is received.


If a NACK is received from the Mega, the ESP8266 can resend the entire payload.

I envision that the ESP8266 could adjust the speed it is sending the data if it is not being received properly, but first I want to see just how often it fails...

The real purpose of doing the crc32 check is to determine if there are issues with the data fidelity, and if so, how to correct it.  The current versions of modules which talk between a Mega and ESP all exhibit "timeouts" when waiting for data, and I'm not sure if the problem is getting data from the database (not likely) or sending it over the serial port (again, should be reliable, the two modules are on the same circuit board, and it uses hardware serial ports on both). I have always suspected the problem is due to one payload being clobbered by the arrival of another one before the current one is processed.

By being able to certify that we have received a complete unaltered payload, we can now acknowledge this back to the ESP with an ACK.  If the CRC32 values do not match, we can send a Negative ACKnowledgement (NACK)

So the ESP sends the data, and now it waits for an acknowledgement, preventing it from sending any more data until this payload has been dealt with, so it solves the clobbering issue if that's what was happening.

Now I can try a larger payload, like the WiFi info...

Update: 1:30am

This is awesome. The Sensor Module is booting up very reliably. I can see the initial sending of the date and time is occasionally corrupt, it looks like a buffer problem, however, the Mega detects the corruption now, and sends a NACK (Negative Acknowledgement) so the ESP simply resends it.

I'm coding the send methods so you pass the number of retries. For example, sending the date and time is more important at boot up, when the Mega doesn't know the time, and less important for the updates which are sent every 10 minutes. So at boot up, I pass a 5 for the number of retries and 1 for the updates.

Once I have the code finalized, I'll take all those variable pieces and save them in the database, such as # of retries for each type of payload, and even if it should wait for an acknowledgement, and how long, etc.

So, I'm excited, and still having fun!


Update: 2:30am

When I started out on this rewrite, I acknowledged that the communication between the 2560 and 8266 was probably my weak link, although I didn't really know why that was so.  Perhaps I still don't really know what causes it, but I think I've fixed it :)

All payloads will require acknowledgements, except, obviously, acknowledgements themselves. That could create quite a bouncy loop...

There are two acknowledgements, ACK and NACK.

  1. ACK means the crc32 checksum matched
  2. NACK means the crc32 checksum did not match
The only other response which the ESP may encounter is no response at all. While waiting for an acknowledgement, there is a timeout value, and if this is reached before receiving a response, it will just treat it as a NACK.


I'm going to take a little break here, regroup and figure out what's next.  I think I might work on getting it to read a sensor, but I want to make sure I think it through and write code I can use, not just rush to make something work.  I know it will work, so no proof of concept is needed. What I do need is a good design so I'm not wanting to rewrite this again in my lifetime :)


Wednesday, August 14, 2019

Do-Overs Rock!

At almost 60, I don't sleep a lot, so I spend my time working on Grow By Wire :)

I took a break, and was sitting here smoking a joint, and having a rum and egg nog (yeah, in August) in the middle of the night, with Metallica cranked to 100% in my ear buds...

Anyhow, I've mentioned my "Thoughts" file, where I keep the text file open in Visual Studio so I can jot down ideas before the pot makes me forget them. Gotta be quick sometimes!

So as I sat here thinking of the possibilities during a Do-Over, I made these notes... 



So off I went, I usually do a proof of concept, and here it is...




The extra text for the DONE and ALLDONE lines as well as the actual CRC32 line has more than doubled the size of the payload, but it standardizes EVERY payload that will be sent back and forth.  I downloaded a CRC32 library to use, since that's a little out of my skill set, plus it is a tiny library, the .h and the .cpp each fit on the screen....  

The CRC32 is calculated on the fly, and will be done the same on the other end.

This will enable me to know with certainty that the data is good or bad and act accordingly.

Did I say a week?


Here is a link to the library I used: https://github.com/bakercp/CRC32


Update: 11:15am

This may not work...  For some reason the library calculates a different CRC32 on the Mega than it does on the ESP. I used the same test string on both...

"This is a test to see if the ESP calculates the same CRC as the Mega"
Mega=[98EC]
ESP=[541198EC]

That's gonna make it hard to validate the data is intact :)

Time to look for alternatives...

Update: 2:30pm

If I had looked closer at the results, I would have noticed something...  did you?

"This is a test to see if the ESP calculates the same CRC as the Mega"
Mega=[    98EC]
ESP =[541198EC]

Notice now?

These lines are displayed using sprintf like so

snprintf(tmpBuffer, sizeof(tmpBuffer), "Mega=[%X]", crc_finalize());


I replaced the code with another routine to calculate crc32, and it actually came up with different values, but still they were consistent between the two boards, just the Mega was missing the first 4 digits. Both tests were done on the same string.

"This is a test to see if the ESP calculates the same CRC as the Mega"
Mega =     [6713]
ESP  = [ABEE6713]

So I decided to print out the actual crc as an unsigned long, it's native type as well as the HEX.

"This is a test to see if the ESP calculates the same CRC as the Mega"
Mega =     [6713] = 2884527891
ESP  = [ABEE6713] = 2884527891

So the actual crc IS calculated correctly, just not displayed correctly...

After some research, I discovered this post: https://forum.arduino.cc/index.php?topic=443334.0
with this advice:



I modified my code to include this

snprintf(tmpBuffer, sizeof(tmpBuffer), "Mega=[%08lX]", crc_finalize());

and now get

"This is a test to see if the ESP calculates the same CRC as the Mega"
Mega = [ABEE6713] = 2884527891
ESP  = [ABEE6713] = 2884527891


Yah!

Uhm, wait, not so fast...

<DATETIME>
<TIME>2019-08-14 14:22:44</TIME>
<DONE></DONE>
<ALLDONE></ALLDONE>
<CRC32>9561525A:82</CRC32>
</DATETIME>

The ESP8266 calculated the CRC on the highlighted characters, including a cr/lf at the end of each line. It calculated 9561525A, using 82 characters

The Mega 2560 on the other hand, calculated FF797376 using the same 82 Characters.

Since my "proof of concept" proves they both generate the correct CRC, I must suspect something in my code which feeds these characters into the crc calculator...

So, back to it....

Update: 4:15pm

I finally found it!

On the Mega, it counted the characters correctly, but due to the way I detect a PAYLOAD, I was "catching up" with my crc updates. I don't know that there is a PAYLOAD defined until it has real all the characters in the first line  <DATETIME>

Now it knows we need to keep a running CRC, so it actually loops through the PAYLOAD name and adds each character to the crc_update method, and update the character count for each.

In addition to the PAYLOAD name though, there is a CR/LF at the end of the line, and these were included in the character count, but not added using the crc_update method.

Skipping 2 characters when calculating the CRC will definately cause problems :)

Ok, onward we go!

Tuesday, August 13, 2019

We're gonna have fun for a few nights!

It's always nice when you get the opportunity to just completely rewrite something from scratch.  That's not something that ever occurred in real life as a professional programmer :)  

I began my programming career as a shareware programmer, and back then, this was something I did on a somewhat regular basis. In the real world, it's always a matter of finding someone to pay for the rewrite, but when it's just me, I can afford to do it. Sometimes I can't afford NOT to :)

The benefit of a complete rewrite is twofold, first, a refresher on just what it does, and how, and second, the opportunity to make improvements. In this case, the Sensor Module was the very first piece of Grow By Wire, written when I was still learning c++, the arduino, and not really having a clear picture of where I was going with the system.  This meant a ton of modifications as the system grew, changes in the way things worked, you get the idea...  So the code changed so much, there are pieces of old code still there, some commented out, some not. So I may think something is done a certain way, but maybe I made a change and it's different now... There's no way I can remember all the changes, everywhere in the system, since day one!  A complete rewrite lets me streamline the code so only code that is used is present, this alone will make it easier to come back and see how something works.

The second, and I think an even better reason to rewrite the code from scratch, I have learned so much in the past year, I have a much deeper understanding of the hardware, the software, libraries, shared code, you name it... This means that I can take advantage of more complex concepts, giving me more ways to accomplish things.

For example, when I first started using the Wemos Mega with the built in ESP, I needed a way to package data to send it back and forth on the serial line between the two modules. Being old school, I was familiar with XML, so went that route. Being somewhat unfamiliar with c++ and the arduino, I opted to copy some code from an XML library. I made some changes, but the concept was the same, and I managed to get it working, and have been using that same code  every since.

Today while rewriting the software, I was able to make more sense of it, in fact, make improvements, and streamline the code which processes the XML. I will have better control over preventing cross-talk where a new XML record suddenly appears in the middle of an existing one which is still being processed.

Initially I figured the rewrite could be done in a day... now I'm thinking a week :)

Update: 10:30pm

Everything is progressing well, no more troubles, although I did spend a lot of time on the routines that pass data back and forth between the 2560 and the 8266. I was able to beef up the XML parsing, and I get to fix something else that is just messy...

When I send the module configuration to the Mega 2560 from the ESP8266, it is sent in XML strings which make parsing the data simple.

For example, if I want to send a text string to display on the serial output, I have a method called sendText("hello world");

This generates the following XML

<MSG>
    <TXT>Hello World</TXT>
</MSG>

The first line is the PAYLOAD TAG, or name of the record.
The second line contains a TAG, some DATA, and a CLOSE TAG
The last line contains the PAYLOAD CLOSE TAG

The 8266 sends these three lines over the serial interface, and the 2560 reads them one character at a time, and parses the data as follows:

It sees the opening <  in <MSG> and knows it starts a TAG, so starts saving the string as a TAG.
As it adds each character to the string, it compares it with a list of PAYLOAD names that it has been coded to look for. In this case, <MSG> is a valid PAYLOAD, so it is caught by this line:

if (strcmp_P(xmlTag, PSTR("<MSG>")) == 0) setPayload();


On the second line, it finds a TAG <TXT>, again, it is specifically looking for this TAG, in this PAYLOAD.  This allows using the same TAG names in different PAYLOADS.

This is caught with:


It sees that the current PAYLOAD is <MSG>, and looks for the TAG <TXT> specifically. If it finds it, it can act on it, in this case, simple print the DATA portion to the Serial port so I can read it on the serial monitor.


That's a simple payload, only one tag, and one action.

Here is the Module Configuration PAYLOAD


In addition to having multiple lines, it also includes two sets of TAGS at the bottom, <DONE> and <ALLDONE>. These are used when there is more than one line of data, this is because the design allows the data to come in any order, as long as it is enclosed in the proper TAG

A better example would be Sensor Configuration data. There are many fields for each Sensor, such as it's ID, it's Name, the Type, etc, long list...  So the <DONE> signifies that we have read all the data for that sensor. When we see the <DONE> TAG, we know we can go and do whatever processing we need to do on that data, because it is complete.  Then we can process another sensor, and another, until we see <ALLDONE> which means we are finished ALL sensors...

Each of the TAG names is the same as the field name in the database, which is the same as the variable name in the code, everything matches, so we can simplify the code which sends the XML.

The above Module Configurations is sent using this code:



You may notice, I use a view in the database, this lets me perform all the joins and field renaming right on the server, making this code very streamlined.

The old way was to read all the data from the database, then loop through the data and write each field out, but using a different TAG name. Originally I never though of doing it the way I am changing to now. Here's the original code:

First it saves the data in a variable

if (strcmp(cols->fields[f]->name, "sensorName") == 0)
{
  snprintf(sensorName, sizeof(sensorName), "%s", row->values[f]);
}


It does this for each field it reads from the database. Once it has loaded all the fields for a record, and all the records for a query, it then spits them out as XML enclosed strings...

Serial.print("<SENSORNAME>");
Serial.print(sensorName);
Serial.println("</SENSORNAME>");

So now the name of the TAG is hardcoded, and is not consistent, and as you can see, requires a lot more code, remember, I'm only showing you the code for one field, the sensor name... There could be 50 or more fields for each sensor...

Changing it to use the same field and TAG name throughout means a lot less code, which means using less memory, which on an Arduino is akin to finding cash in an old jacket pocket! 

It also means less chance for mistyping a TAG name, and the smaller code is much easier to understand.

Seems to me that once I finish rewriting the Sensor Module code, I'll want to do the Switch Module and Automatic Watering Module as well...  

Perhaps it makes sense to move some of this code into a library, that will make rewriting the other modules a lot faster, but slow down the rewrite of this one.

ok, my 10:30 update has turned into 11:30, I best quit the jabbering and get back to work...




Monday, August 12, 2019

We're gonna have fun tonight!

Probably the biggest weakness of the Sensor Module design is the amount of data being sent back and forth between the 8266 WiFi module and the 2560 MCU.  While it is a hardware serial line and both systems share the same motherboard, overlapping data still occurs, and this messes up the XML structure.

Currently the 2560 orchestrates everything, but that means it has to maintain a lot of configuration information, and keep it up to date. Every time it scans the sensors, it reloads the configuration for all of them, it needs to reflect any possible changes that might be made to the configuration...

So it wakes up and requests the config for all sensors, then loops through the list, and tests, and then reads each sensor, does any data manipulation (convert to % or degrees, etc), then sends a bunch of data back to the 8266, which then does a few lookups, then writes the sensor reading to the database.

Once it has finished all the sensors, the 8266 sends a total of how many records it received, skipped, and wrote to the database.


I've been toying with an idea in my head, why not let the 8266 be the brains...  When it wants a sensor reading, it sends a simple request to the 2560 over the serial port, and gets a simple response.... We don't need to send all that configuration information back and forth. Using this method, we don't need acknowledgements of data being received, since the 8266 will either get a response, or not... that's acknowledgement enough eh?

Obviously this is a HUGE HUGE HUGE change, so I'm just going to start a completely new project and have at it...

I've got my music playlist updated with some rocking tunes, a couple of Monster Energy Drinks in the fridge, and half a mason jar of nicely dried buds...

Let the fun begin! 


Update: 9:45pm

Whew!  I finally have the esp8266 connecting to wifi , loading data from the database, and taking OTA updates!  I thought I was going to wear out the cheap little dip switch while I was having to change it back and forth from upload to run... 

Overall things are progressing about as I expected, slowly :)

Update: 4:30am


I uploaded the code, and the esp8266 went crazy, rebooting constantly... Extremely frustrating, I was right back to when I was a rookie with these... and I was lost...


I have what I call blank sketches for all my boards, so in a case like this, I upload the blank sketch, and in the case of the 8266, it has WiFi code, and the OTA code as well, but that's it... It was fine, it connected to WiFi, and did not reboot...



I then installed my previous version of the Sensor Module code for the 8266, and it was rebooting too! WTH, this was working code???



Ok, back to the new version...
In the code that connects to the WiFi, it does a loop while checking to see if it is connected, and in this loop, there is a delay(1000). So, I decided to change it to a while millis() < startTime + 1000; instead. I'm not really sure where I picked this up, but it allows for finer control over delays, I can actually do something in the loop if I need to while waiting...



Now, when I was first playing with these ESP8266 modules, I had this problem a lot, and while it eventually went away, I never knew exactly why it had occurred...



Now I know... The ESP8266 has a built in Watchdog Timer, I knew this... and I knew this was what was causing the reboots... I just didn't realize HOW I had inadvertently fixed it...



The watchdog timer is there so that if your code gets into an infinite loop, the watchdog will reboot the module after, I think it's 2 seconds or something. My loop waiting for the WiFi connection looked like an infinite loop to the watchdog timer, so it kept rebooting. So the code needs a way to notify the watchdog timer that it's ok, we're still doing what we were designed for... and it is done in the Delay(x) method. Apparently a few things happen inside the delay method, but this is the one of interest now. 



I discovered this as being the culprit when I first removed the loop altogether, and it worked fine... When I added the loop back, it rebooted.



I kept the new code for the delay, but I just added a delay(0); inside the loop. This is a delay for 0 milliseconds, so it really won't add any delay, but it will do it's internal work, including telling the watchdog timer to buzz off!



So while I could have simply cut and paste the old code in, I would not have actually learned the true cause.



Oh, why did the original Sensor Module code not work any more? It turns out that when I removed the delay and replaced it with the while loop, I decided to make that same change in the original code :( You know, pot and short term memory and all...




So the moral of this story is: 
Always include a delay(x); in loops on the esp8266
even if x = 0

Now that this is solved, I can continue...   More later....


After more digging...

From: esp8266/Arduino

Within the delay method, it calls the esp_yield();method, which is where the timer gets reset. You could call esp_yield(); directly I suppose.


More Power, More Clutter...



If you look at the previous post, you'll see a nice neat and tidy board.

I've added the dc/dc converter and the resistors for our transistors, and started adding the transistors, and it's starting to get messy!





I have to stack the transistors so I can fit them in. This will make things fragile, and put stress on the solder joints. I'm not even sure there's enough holes to string the wire I need...

I will have to rethink this...


Sunday, August 11, 2019

More power part 2

Part 1 of this post, click here...


I finally got the dc-dc converter installed, and also added a small green LED so I know when power is plugged in. The LED is powered off the 5v side of the converter, and has a 330 ohm current limiting resistor in series.


The wall wart plugs into the 12v In connector, which powers the dc-dc converter. The converter outputs 5vdc which powers the small green LED through a 330 ohm current limiting resistor. The 5vdc will also be distributed to all analog and digital ports. I say ports, because a port consists of three pins, ground, power, and signal. Signal is the one connected to the analog or digital pins so we can read a sensor. We are using the transistor to switch ground in and out, either complete, or break the circuit. Being on the ground side, we don't have to worry about the voltage drop across the transistor. I have a schematic in Part 1 of this post...

I added a row of 5 pins, positive and negative, to allow me to make use of the 5vdc, but mainly to make it easy for me to measure the voltage while I adjusted the pot on the converter.

At this point, I still need to distribute the positive 5v to the proper pins (replace all those nice neat red wires) and then add the rest of the transistors and their associated resistors...



Saturday, August 10, 2019

Down the rabbit hole!

Spent the day chasing bugs that weren't anywhere close to where I was looking :(

Everything was running great last night, all the data was being saved properly. I even added a comment to my blog post abouit the DHT22 fixes I did to reset them properly, saying that the reset worked every time it ran...

Almost immediately after posting that, it failed three times in about an hour, and Sensor Module 1 was loading either extra sensors (with duplicates) or not enough sensors, so I naturally assumed it was the new code I wrote for loading the sensors. I spent all day today analyzing, changing, adding debug logging, and pulling out what little hair I have left...  No luck, couldn't figure it out...

Then I decided to do the one thing I always do when something like this happens...

Dump the sql statement to the serial monitor, and copy and paste it into the MySql Workbench, and check the results...   yup, there's the duplicate records alright... 

It turns out that I had moved a couple plants from Veg into Flower, and the code somehow wrote two records to a database table (actuially, 2 tables, each had two copies of the relevant rtecord for the same sensor) Because of these duplicates, it cause the database view to return 4 copies of that record.

Removed the duplicate records, and everything seems to be running fine...
\
As for the DHT22, I added another reset in case the first one fails, hopefully that might help.

Night Watchman

A mysterious bug popped up while I was fiddling with code early tonight, suddenly it was going haywire when loading the list of sensors for a Sensor Module.  I'm not sure the circumstances that kick off the error, but it had to do with me loading the sensor data in two parts because of a limitation of the MySql library (memory related)

I opted to first load a list of sensor Id's only, into an array of INT[maxNumberOfSensors]
then I would loop this array and load the first bunch of sensor configuration fields for a sensor, then another query to load the second group of fields for a sensor, and simply populate one struct with the results, leaving me with a complete sensor configuration record, and I then send it to the 2560 over the serial port, enclosed withing XML tags. 

I used the same database cursor for all three queries, figuring in each step, I was done with it, and didn't need the result sets any more, I had already grabbed what I needed.  This was new in my changes from the other night, as in the past, it was two completely separate calls to load the sensor config, and completely separate sql queries.

It was loading a complete list of sensorId into the array, but when it queried for a specific sensor, if simply failed to get any results, seems random...  The code made an assumption (ok, the programmer made an assumption) ((OK, IT WAS ME!!))  that it would find the record, afterall, it got the id in the first query...


When it failed to find the record, and my lack of error checking (honest, I'm usually pretty good about error checking) exposed another failure on my part... I knew it seemed too easy...
Between each call to populate the sensor configuration data structure, I SHOULD clear out any data that is already in therem because in this case, when it failed to load a record, it merrily went on it's way thinking the data already in the record was correct, and sent it to the 2560 again as another senor, but it was a duplicate...

The fix, simply add a call to the clearSensorConfig method, and create new cursors for each of the three queries.  




So far, we're batting 1000!

So here I sit, watching the screen, listening to Metallica blasting away full blast in my ear buds... Smoking joints, just chilling tonight...

I'll post an update later in the morning...