September 2019 Update

New! We're moving to the new forum!

ALL new Blog posts will be on the forum.

Please join us there.

This Blog will no longer be updated...

Wednesday, July 24, 2019

Slowing Down...

Well, I've finally reached that point, the entire system is functioning, it does everything I set out to  do, and much, much more.  Sure there are still lots of loose ends, but I've run out of ideas for new features, or enhancements. 

I suppose the next step would be to make it available to a select few who would help me get it into shape so that it could be released as Open Source. The problem is that it was built in place, growing all the time, changing all the time.  That means I'm not even sure how to do a fresh install, we'd have to figure all that out.  Contact me if you are interested in helping out...

I'm going to take a break from this project for a bit (I think) and work on some other small projects, mostly related to R/C toys...  I'm sure I'll find something that needs my attention though, and will update here when I do.

So, if you're looking for me in the mean time, I'll be over at my other blog...

Saturday, July 20, 2019

The "Mega Shield!" part 2

Link to Part 1

I've pretty well finished the first Sensor Module Backpack, although I've left room to add new connections.  

I started by soldering the wires onto the pins I wanted to access on the Arduino.

Then I threaded the wire around the board trying to keep it somewhat neat and tidy, and easy to follow... 

Here's the top view of the completed board, I've added some tags so you can see the connections.

The 16 analog ports are completely configurable as to which Analog Pins they use, and if/which digital pin is used for power. Using a digital pin for power to the sensor means it is not sitting in wet soil with power applied for long periods of time, corroding away... By setting the digital pin high or low, we either apply 5v power, or not.

As for the available ports, there are, as you can see, 16 analog and 4 digital ports.  For the pins, I found that on the current modules, trying to plug 16 sensors in all in a row, close together, can be a real pain in the ass, there is so little room.  By removing every second set of pins, this will leave plenty of room to easily plug in sensors.

I've also added 4 sets of I2C pins, plus a "header" female connector to plug the I2C Realtime Clock into.

Last but not least, a 5mm RGB LED.  When things are running smoothly, the green is lit up, and when something is wrong, red.  While it is receiving or sending data, the blue led will flash.

Friday, July 19, 2019

The "Mega Shield!"

Tonight I'm making a new "Mega Shield"  Well, it's sort of like a backpack board for the Wemos Mega 2560 which lets me create exactly the pinouts I need for a specific application.

I start with a blank piece of perfboard, these are 3.5" x 6", a perfect size for the Mega, and add the pins which are lined up to the holes on the Mega. 

On the other side, I can solder pins on anywhere, and in any configuration that suits my needs. I use thin "wire-wrapping" wire to connect pins together, it's small enough it can be threaded through the holes. Eventually once I have the design and layout locked down, I'll probably have some nice custom etched boards made.

As an example, here is the Automatic Watering Module, you can see the board sitting on top of the Mega on the right with all the buttons, led's, pins and connectors...

The board I'm working on tonight will be for the Sensor Modules, I want to make it so all the sensors will plug into a row of pins angled off a small daughterboard which plugs into this board.  This will save me a ton of space, and make the layout much neater, while still allowing me to disconnect all the sensors easily, yet keep them in order.

It's currently 2am, I tend to sleep in short hour or two long naps, maybe 2 or 3 times a day, otherwise I'm here at the computer for the most part.  I especially like the night time.  I've had tremors in my hands my whole life, and as I get older, it's been getting worse.  But, at night, it's like a miracle, they are almost non existent. I do almost all my soldering at night simply because during the day, the iron is all over the place, yet at night, I can solder stuff so small I need my magnifying light to even see it...  well, that's another story, my eyesight... lol... lets just say, thank god for magnifying lamps! and eye glasses!


Thursday, July 18, 2019

GBW Serial Terminal - Update #7

I don't know why, but I just keep coming up with new ideas for the terminal program...

As you would expect, it lets you follow the Serial output of a module when it is hooked up to a USB port on a computer. This is very useful so you can follow along with what the module is doing.  I also have a debug log table in the database, a table which I've been using as a general debugging tool, I can write to it, and even if the module is not on a usb port, I can follow along in the database. The Serial output and the database log are two separate streams of data.

So, today's new feature for the terminal, the ability to follow the debug log from the database while viewing the Serial output.  The Debug log view is refreshed every 5 seconds, so it's pretty well real time.

Checking the checkbox "Tail debug table" while connected to a module will open the bottom panel, and refresh it every 5 seconds with the 10 most recent entries from the debug log table.  Unchecking will hide the bottom panel.

Tuesday, July 16, 2019

DIY Photo Sensors

Added a page describing my DIY Photo Sensors.

Sensor Module update - confirmation of database inserts

Nope, don't need em...

Oh, you'd like to know why?

I'm still looking into how to better manage sending data between the Mega 2560 and the ESP8266 which share a circuit board and hardware serial interface, but certainly any decrease in the amount of data sent back and forth will help.

Originally, when the Sensor Module booted up, the 2560 would request a copy of the module configuration from the 8266 by sending an XML request string over the serial port. The 8266 would query the mysql database over the WiFi connection, and return the configuration data to the 2560 over the serial port, again, in XML strings.

Then the Sensor Module requests a copy of the Sensor Configuration, a list of all it's sensors... the 8266 again, gathers the data and sends it back over the serial port. 

The 2560 scans each of the sensors to get its reading, and sends it all back to the 8266 again, which writes the data to the mysql database. The 8266 then sends the confirmation list back to the 2560 with the record id from the database for each record inserted.  Nothing is done with that information other than displaying it on the 2560 serial monitor output...  That's a lot of back and forth just to save the sensor readings.

Each time the module does a sensor scan it also reloads the module and sensor configurations....

With all this back and forth, it was hard to tell where data was being lost when some data was not being written.

First off, I added an Acknowledgement function on the 8266, so when the 2560 sends data, it can, in certain instances, expect an acknowledgement.   

So now, the Sensor Module loads it's Module Configuration ONCE at boot up.  Unless I change the configuration, reloading it is a complete waste of bandwidth. Originally, I didn't expect the system to be as complex as it is,  and this never would have been an issue. I will let the 8266 check the last update timestamp every 10 minutes or so, and if it finds an updated record, it can just go ahead and send it to the 2560 as required.

I THINK I'll do the same with the Sensor Configuration...   I say think, because I'm not certain yet... Changes to the sensor configuration is more likely to change than the module, and checking a last update timestamp on every sensor all the time would be a resource hog as well, so some thinking to do on this one still...

Now, sending the data from the 2560 to the 8266, and sending confirmations back.  Here's where I made the biggest change.  Because the 2560 sends all the sensor data to the 8266 BEFORE the 8266 starts writing to the database, I was never guaranteed it got all the data...  I know that on occasion, records got scrambled or lost...  Now, each sensor data record is sent, and then the 2560 waits for a specific acknowledgement from the 8266 for that sensor. If it does not receive the ack within 5 seconds, it will resend the data for that one sensor. It will retry up to 5 times. Once it receives the ack, it moves on to the next record, and sends the data for the next sensor...

The 8266 now has all the sensor data, so it goes off and writes it to the database, and that's it, there is no confirmation response back to the 2560, since the 8266 already acknowledged receiving each record, and is therefore responsible for ensuring it gets saved. 

Now, because there is no indication of when the 8266 finishes writing all the data from a scan, the possibility exists that the 2560 will attempt to send more data before the 8266 is ready. In this case, there will be no acknowledgement, so the 2560 will simply retry sending it...

Right now I've got this new code running on Sensor Module 2, which controls the sensors in the Flower Tent, and out of the first 100 readings, all 100 made it to the database...

Sunday, July 14, 2019

Quiet Time?

As you can see, I'm taking a bit of a break here...

I still haven't decided how to handle the inter board communications yet.

I've had a couple ideas on how to approach this, but because I already made a bad less than ideal choice in how it works now (discussed in another post) I want to make sure I find the best solution, so I'm in no rush... 

Right now, everything is running as well as it ever has, I'm really very happy with it, and in fact, it's getting extremely hard to find new ideas and slip them in. Seriously, I never check plants to see if they need watering, and I never see droopy plants any more.... Once I can afford the pumps etc, I'll be doing completely automated watering, but financially, it's not a top priority.

When I get in a situation like this where sitting here thinking about it isn't getting me anywhere, I just walk away from it, stop thinking about it... First, I sit down and define the problem as best I can, in writing (point form) in a file called thoughts.txt which I edit right in the IDE. Here, I organize, yep, my thoughts... Once I've defined the problem, I start outlining any possible solutions... They down't have to be complete solutions, and can be outside the box so to speak, since I'm just gathering my thoughts at this point. Then I forget about it, move on to something completely different, or go take a nap, or anything to get it out of my head. While I'm otherwise engaged, there's a little part of my brain that, in a background thread, taps into the cosmic knowledge base and sifts through it, and if and when it finds an answer, it will fire an interrupt in my brain, and I'll go "Eureka!"

Ok, it doesn't always work like that... but it has served me well. If I get an idea, I go back to my thoughts.txt file, add it, and review what's already there... If I still don't see a clear answer, I go away again, do something else...

So, it's been a week now, my documents is growing, but still haven't decided yet. No big deal, I'm not in any rush, like I said, it's running fine now, but I do want to make changes more for future capacity...

To keep busy, I've gone back to my R/C cars. My 1:6 scale Beetle is a full FPV rig, meaning I drive it using a video screen and a camera mounted pointing out the front window. I also have a Mustang and a Camaro, each smaller than the other. These are all TOY radio control cars that I've converted to using proper 2.4Ghz radio equipment and modern servos. The radio gear and servos are all left over from my R/C hobby days, so this has been an out of the drawer type project. Anyhow, the Beetle has an OSD, or On Screen Display, module which overlays text on the video from the camera, so when I've driving, I can see things like GPS position, direction, speed, battery voltage, current draw, anything I deem important enough... This OSD module is an Arduino Pro Mini and a Max2456 breakout board, and software I wrote last summer. The Camaro has no Arduinos in it, but the Mustang does. I didn't have another ESC (Speed controller) for the brushed motor, but I did have an L289N H-Bridge Motor Driver board, so I hooked it up to a Pro Mini, and wrote the code to read the R/C receiver channel for the throttle, and to send the proper commands to the L289N to control the motors speed and direction based on the transmitter commands. I also created a voltage sensor for 12v, and monitor the battery voltage so if it gets low, it will limit your speed so it doesn't damage the lipo batteries.

Anyhow, that's what I've been up to, and why there have been no new posts...


Sunday, July 7, 2019

Serial Communication on the Wemos Mega 2560 modules

These boards have a built in ESP8266, complete, even with it's own pin header if you want to use it. However, in my case, I am communicating between the 2560 and 8266 wuth a provided hardware serial link. I have a choice of which Serial device I can use on the Mega, and have all my switches set to Serial3

So I have a nice fast 115,200bps Serial connection between the two, and they write data back and forth to each other.  Some of that data are requests for data, so for example, the 2560 will request a copy of it's configuration from the 8266, since the 8266 has WiFi, and a MySql Database Connector.

So the 2560 sends a request to the 8266, which generates the appropriate sql statement, executes it on the MySql server via the MySql Connector, reads back the data from the database, repackages it for the 2560's needs, and then fires it back over the serial port to the 2560.

Another scenario is when the Sensor Module scans all the sensors, and then sends all the data at once to the 8266, which has to parse through it, repackage it, do some lookups for more data, then generate sql statements to write the data to a log table, as well as update a "current" readings table, and then reread the log data to ensure it was written. 

Once it has completed all the database updates, the 8266 loops though the array of sensor data, and sends a list back to the 2560 with the database record id included, this acts as confirmation that the records were properly written to the database.

The 2560 reads this list (array) and then displays it on the Serial output.

Now, these "packets" of data consist of XML records, well, not XML to spec, but a bastardized version..

The "payload" is the name of the "table" in XML... so for example, we may send a request for the module configuration, and include the module Id with it. This "packet" would look like this:


The first <MODULEREQ> is the payload indicator, it is a single xml tag, alone on a line, ending with a NewLine. When the parser encounters this, it will save this name in a "payload" field. When it encounters the same text, but in a closing tag, as in </MODULEREQ> it then sees the end of the record.

While it has a "payload" name, the parser will look for predefined fields, <MODULEID> for example...  Fields are defines with both opening and closing tags on the same line with the data between them, and a newline at the end of the line.

Special fields include a set of tags specifying that we are done with this record, <DONE></DONE> Both opening and closing tags on the same line, ending with a newline. There is no data associated with these tags.   If a payload contains multiple records, it will look like the following, notice each record has a <DONE> tag, but at the very end, there is an <ALLDONE></ALLDONE> tag set, this signifies no more records...


It's all very easy to parse the incoming data this way.   

You will notice, there most of the size of this data packet consists of a description of the data record, ie, field names in the opening and closing tags...  In order to be meaningful while reading the code, the names must be long enough to make sense or it will be impossible to fix any bugs...

So to send 14 bytes of data, we actually sent 169 characters plus 9 newlines...  Not very efficient..
To top this off, there is no flow control on the serial line between the two boards, so while a lot of data can lead to issues, the problem was occuring even when there was not a lot of data, but it was data that required work on the receivers part, and therefore it was not fast enough to keep up., and because of lack of flow control, the data was just being discarded...   

Ok, so here's what was happening...

2560 sends sensor readings to 8266

8266 takes 60-90 seconds to complete it's processing, meanwhile, the 2560 is monitoring the serial line for a confirmation packet...   The 8266 sends this packet, and the 2560 goes off to display it on the serial monitor line, and by the time it comes back, more data has arrived, followed by updates fired off from timers, like the current WiFi Signal Strength, etc... These can fire off before the previous data has been read, overflowing the buffer, and causing loss of data, which means if the 2560 requests its module configuration, but the data gets "jumped on" by say the RSSI data, then even if the 2560 read 95% of the record, it won't see the <DONE> tags, and won't finish processing it, and will report a timeout receiving data...  This is what was confusing me, I knew it was sending the data, but didn't really realize the implications of not having proper flow control, or, to be honest, not realizing that it wasn't implemented in the libraries.

My decision to use the above methods were made very early in the project, actually, even before it was a project...  I found an XML library, and found it interesting the way he parsed the data, it was so simple... I implemented my own version, although it was entirely based on that existing code. I wish I still had the link to it so I could give proper credit, but I never thought to save it... I generally paste links right into my code when I "borrow" ideas from others, but it's not very often... At the time, there was very little data to be sent back and forth, and almost no other processing going on on either board, so the problems I'm having now did not manifest themselves, in fact, it was not until I was fully invested in using this method that small problems started popping up.  I fixed them with bandaids, adding delays, and flushes, anything I could that would make it work...

Now, I completely 100% understand the problem, and the breakdown of what is happening. I know that I need to implement flow control, it doesn't have to be fancy, just a way to say, Ok, Send Me Some Data, or Hold On a Sec...  Whether I will continue to use XML or not, I'm not sure, I think if I can make flow control work, I suspect the amount of data is not the issue, and knowing when it is ok to send will make all the difference in the world.

I'm going to spend some time playing with this, and see what changes I'd like to make... do some experimenting...  I'll let you know what I find :)

Saturday, July 6, 2019

Sensor Modules - Do I need Confirmation?

Each time a Sensor Module reads a sensor, and sends the data to the database, it also reads the record back to get it's record Id in the database as confirmation the record was written.

The steps involved are kinda complex...

Module is the code running on the Mega 2560
Server is the code running on the ESP 8266

Both are on the same board, connected by a hardware serial port.

The Server has the WiFi connection, and the Database code.

Here's the current process....

The Module requests the Sensor Configuration from the server via XML on the serial line. It then monitors the Serial line for a response...

The Server parses the incoming XML, and sees the request. It then does a SQL query to get a list of all sensors assigned to this module, and their configuration, and then sends it via the serial encoded in XML to the Module.

The Module, reads the response, and stores the configuration information in an array.

The module loops through each sensor in the array, and scans it, taking a reading.

Once it finishes scanning all the sensors, the Module then sends the data, one record at a time, via XML over the Serial line, to the Server, and then waits for a confirmation from the Server that each record was written.

The Server then loops through the sensors and writes the data to the database. After writing the record to the log, the Server attempts to read it back to get the record id, verifying that the record was written.

Once all the records are written, the Server sends a lits of Sensors and the record Id's back to the module

The Module displays these confirmations, but that's it, it doesn't actually do anything with the information...

Whew...   Ok, so here's what I've learned...

I was always seeing "Timeout waiting for confirmation..." and presumed there was an issue saving the data and responding on time.

However, it you look at the status lines I added to the terminal (grayed out) you will see that the number of module readings is equal to the number received by the server, and all of them were saved successfully.  The only issue seems to be the confirmation, somewhere around 20% of these confirmations are not coming back to the module properly even though they were saved properly.

This makes me question the point of even returning the confirmation...  I think now, having the status lines is more than enough for me to know whats happening. The information is not even being used, just displayed...  It causes a lot of traffic on the Serial Line as well.

If I remove it, I'll need to still wait for some confirmation that the Server has finishes it's tasks saving the data before the Module starts sending more data, or requesting something from the Server.

With all the traffic on the Serial line now, it will only be beneficial to reduce it as much as possible, especially if it really serves no useful purpose any more.

Friday, July 5, 2019

GBW Serial Terminal - Update #6

Seems I can't just leave this alone...

Two new features here today.

First, the modules all synchronize their clocks with the MySql Server, which is synchronised with all the other computers on the network. So my development machine will have the same time within a second or so of any of the modules...

The first new feature is an upgrade to an old one. The terminal has been displaying the clock time of the module it is connected to, the idea was that you could compare it to the current time, and you'd know that it was still alive.  Then I found a way to be even lazier...  The Terminal can monitor the time that the module sends, compare it against the computers clock, and if they are more than a minute apart, the text turns from DarkGreen to OrangeRed, and more than 5 minutes, white on a red background, you can't miss it, even across the room...

Hmm, perhaps an audible warning too? I'll throw that on the todo list.... :)

The next new feature, 5 status lines...  These are lines on the display (Just under the RSSI) that each module can use for anything it wants. There are 5 lines available, and using a {{STATUSLINE:1}} tag it can write whatever follows, up to end of line, to Status Line 1.  In the screen shot below, the Sensor Module is updating Status Line 1 with the total number of sensor readings taken, the number successfully written to the database, and the number that failed...  These numbers are maintained by the module itself, updated when it wants, and in this case, they are updated after every sensor scan, and reset when the module is reset, so this status line will give me a good idea how things are running...

Wednesday, July 3, 2019

Software Integration...

In order to do the firmware updates for modules connected via USB cables, I need to know the COM Port  assigned to it.  For the most part, it doesn't change unless you move the cable, although it can, and eventually will change... Keeping this in a configuration table means having to go update it whenever the com port does change.

So I had an idea....  since I'm using the Serial Terminal to monitor the modules, I have to select a com port in order to connect with a specific module, so, when they do connect, the module passes it's Id to the terminal, and the terminal then updates the database with the current, and correct COM Port for that specific module.  If for some reason the port changes, and the firmware update fails, all you need to do is load the terminal, pick the new COM Port, connect and verify it's the right one, and that's it, the terminal has already updated the configuration for the Firmware Updater. The terminal also checks for the IP Address of the machine its running on, and looks it up in a table to get a computer Id, so it updates the module with a Computer AND Com Port...  The Firmware updater will need that info...

So today, to bring this full circle, I had the idea that since I can grab a list of modules which are plugged into "this" computer (whatever computer I run the terminal on) I populate a combo box with a list of modules by name...  No more having to remember which module is on which com port, just select the module you want to connect to, and it sets the proper com port and baud rate, and connects...

Monday, July 1, 2019

Keeping my cool...

Building management said we are not allowed to have air conditioners hanging out windows above the first floor... 

Summer temps have arrived, and air conditioning IS required in the grow room, so the air con just butts up against the screen, and is enclosed to allow proper operation...

Working like a charm...

Firmware Updater - OTA Working!

OTA (Over The Air) firmware uploads are working...

My original code left you without any feedback during the upload, you just waited, wondering if it was uploading...   So, I rewrote it to send the data in chunks, and having the upload class raise an event every 1% so I can update the progress bar.  Works very well, and looks awesome :)

I also added the section at the top of the screen to show you what you are about to do...  Once you press a button on the left, then you will see this screen.  It gives you one chance to back out in case you hit the wrong button.

The buttons on the left are generated at runtime based on the modules configuration...

All that's left is sending updates via the RPC Server on another computer...