Add tag persistence

[pollockm]

Add ability to persist most recent (within decimation amount) value of certain critical tags.

Proposed approach:
Use SQLite library to expose a simple database.

Editor:
Add a column to the Tags editor page that exposes a Persist? checkbox. Add an item to the engine settings editor page that determines persistence decimation (how often to save persisted values).

Config:
Add a config item for persisted tag list to the engine

Runtime:
At engine init, add a check for each tag for whether that tag is on the persisted list. If so, check if a persisted value exists at the predefined database location. Database location/name should be tied to the engine name, IMO, so we can handle multiple concurrent engines. If value exists, init tag to that value. If no persisted value exists, use the default value for that tag like we do now.

At the designated decimation value of loop iterations, copy the latest value of each tag in the persisted list to the database file. It would be nice if the engine also kept the latest timestamp of the data, but that is not currently there.

Initial tests are promising - library already works on Windows and Linux, and is reasonably performant.

[pollockm]

Use cases: total hours of runtime, keeping track of how many times a valve has been opened, etc. This was a requirement for a lead user.

[theSloopJohnB]

Why not write an output module to send data to a database?

[pollockm]

We would need the engine support to pull the persisted value out at init time.

I think it is a very good idea to have a module to do this as well for additional data.

[Beazurt]

Overall this feature looks great to me. A few questions though:
Would the database writes happen inline with the main engine, or would the engine do this in a background thread?
Does the database automatically provide the protection needed from a power cycle in the middle of a write? Would the entire tag bus be a write transaction or would each tag be its own unique transaction?
I don't know if engine names currently have to be unique. I like the name for readability, but the GUID would be a lot safer without additional protections around the name. Would a user ever care what the database is called?
Does this change require additional source code dependencies or would it be done with native LabVIEW functions?

[pollockm]

Would the database writes happen inline with the main engine, or would the engine do this in a background thread?

• Not sure yet. I'm toying with the idea of it being essentially a hidden output module that always runs last, and we provide an option for async execution or not.

Does the database automatically provide the protection needed from a power cycle in the middle of a write?
-Yep. SQLite looks pretty awesome in that regard. Other options may have similar capabilities.

Would the entire tag bus be a write transaction or would each tag be its own unique transaction?
-I would not expect the entire tag bus to be persisted, just a subset. I would expect that there would be a single transaction, but possibly multiple calls. We would want either the whole data set or none to get committed.

Names:
-We would need a unique idenitifier. Name, GUID, combination - whatever works. I don't think a user should have to care.

Dependencies:
There already exists a LV SQLite library: vipm://drjdpowell_lib_sqlite_labview?repo_url=http://ftp.ni.com/evaluation/labview/lvtn/vipm

It is updated regularly and supports LV 2011 and later. This would require also installing the SQLite package on the Linux target (no additional install needed on Windows). SQLite is included in the NI feed, but not part of the default image. I prefer to ask users to install SQLite via opkg rather than us pushing a fixed .so version over.

[DCAF-Builder]

Update: I have a prototype of a SQLite module up and running here: https://github.com/LabVIEW-DCAF/SQLite

Still needs a lot of love, but has the database interface looking fairly decent.

[DCAF-Builder]

If that module works out, should be fairly trivial to repurpose some of it for a persistence feature.

[jdeantx]

Any updates on this? The feature is really handy for some of our test systems that go into safe state or unexpectedly go down. Also, we have thresholding modules that are tag based, and it would be nice not to have to reconfigure the thresholds every time.

[becega]

@jdeantx, we haven't had time to work on this. and we don't have it in the roadmap at the moment.