This defaults the channel to stalled and forces the system to take action to mark it as not stalled.  In a bad state we want to default to bad, not a bad state hidden because the producer wasn't run.

This adds code to the shmem receiver to have it signal when it has taken 2s to look for new data.

This is meant to be a simple flag to show when the producer is 'stuck' waiting for data.

Add a parameter to the daqd 'DUMP_BROADCAST_FRAME' which keeps the broadcast frame that has gps % 60 = 0 around for 1 minute.

This is actually a checksum of he data block checksums, so that we only compute the checksum once (when we verify the data) for the data.

The data is being exported as {PREFIX}PRDCR_DATA_CRC

Ensure that when the terrabyte option is passed in it is treated as a numeric (float) value instead of a string.  Multiplying by a string crashes the program.

 * Allow _C- for raw/commissioning frames
 * Remove an '+' that was left in when converting from string concatenation to path joins.

 * Convert to using os.path instead of manual path creation
 * Estimate the number of TB of data per day and use that as a default amount of free space to hold to.

These are utility scripts used when running the daqd.

checkdaqconfig - freezes the config, archiving it by time & contents
daq_wiper - this deletes old frames to keep disks from filling on a frame writer

This is the python3 script used at LHO for frame wipping.  Moving it from userapps to the advligorts repository.

This script is responsible for running every hour and freeing up space on the disk system for new frames.

PRDCR_UNIQUE_DCU_REPORTED_PER_S -> the number of dcus that reported at least once in the last second
PRDCR_TOTAL_DCU_REPORTED_PER_S -> the count of dcus that reported in the last second.  In good conditions this should be 16*PRDCR_UNIQUE_DCU_REPORTED_PER_S

PRDCR_TOTAL_DATA_RATE_KB_PER_S -> the total data rate over the last second in kb (excludes < 1kb of headers a second)
PRDCR_TP_DATA_RATE_KB_PER_S -> the data rate of test point data over the last second in kb
PRDCR_MODEL_DATA_RATE_KB_PER_S -> the data rate of the regular model data over the last second in kb

PRDCR_OPEN_TP_COUNT -> the count of test points opened sampled once per second

added some unit tests for checkdaqconfig.py.  Now accepts hardcoded paths in master template, but will look in expected paths if the paths in the file are dummies (template variables for example).

Erik von Reis authored 4 years ago and Jonathan Hanks committed 4 years ago

adding script for archiving daq configs when using multiple DAQs.  Script uses a hash to dermine when to update archives.  Manages separte pointers for each DAQ.

Update the test to check if daqd stops on a timestamp jump to give a proper return value on success.

The stopping_now_ flag should be set to true to signal trender threads to stop.

Also move some stack based variable sized arrays onto the heap in std::vectors to remove a place that the stack can be overflowed.

Added a test to ensure the daqd stops when there is a jump in the data, and updated the live test daqdrc to start the trender thread to highlight issues with stopping.

Historically, there have been two main types of broadcasts in daqd, the DC data stream (the 10Gb udp broadcast) and the GDS frame transfer.  When DATA_CONCENTRATOR was defined only the DC data stream is used.  This mode has been removed from the daq architecture.  So the only remaining broadcast mode should be the GDS frame transfer.

The big difference between the two broadcast methods is the data source.  For the DC stream the source is the main circular buffer.  For the GDS the data source is a frame file that the framer_io function mmaps and pushes as a block of memory.

There were several modes of operation for the daqd, and DATA_CONCENTRATOR was picked when collapsing the daqd down to one binary.  So remove support for the DC data stream.
 * Removes support for broadcasting testpoints
 * Removes the udp broadcast logic for the DC stream
 * Removes a number of #if DATA_CONCENTOR macro guards
 * Removes a number of #if GDS_TESTPOINT macro guards

 The final flow of the broadcast data is now as follows:

 the parser sees the request for the net-writer with a broadcast address.
 This triggers a net writer to be created, a producer thread to start, a circular buffer to be created.  The producer copies data from the main circular buffer and unlocks the main circular buffer it puts into the new circular buffer with a nowait put operation (which is allowed to fail).  Nothing has ever emptied this, so a consumer thread is not launched (it didn't do anything anyways).

Work to remove slave from the code.  This is not a complete cleanup, but it goes through

 * src/daqd
 * src/gds
 * src/gdsConfig
 * src/mx_stream

Change an || to an &&, so the loop that waits for work or a stop signal exits when one of those conditions is met.

 * Spelling fix, changing to stop_function_t from stop_funtion_t.
 * Corrected the timed circular buffer get to timeout.

Move all the trender threads to being tracked by a thread handler and having a uniform stop signal.

Add a timed_get function to the circular buffer which will allow queries to the circualr buffers to time out.  This is used to allow the trender to see a stop request.

Converting it to use atomics.

This is done as part of expanding the wrapping of the pthread... calls with C++.  Starting on a interface to better track threads.

Fix the error handling code to work with how nice has been implemented for a while.  Change to clearing errno and checking it after the call as proscribed by the man page.

The daqd has carried code for at least 9 years to set the GPS leap seconds in the frameH structure that is not used, too small, and the configs where never updated.

Remove the code.  Putting the correct leap second value into the FrameH structure is done by FrameCPP in the call to SetGTime.  There is no need to keep this old code around.

* For now turning the set gps-leaps into a no-op in the config.
* Removing the leap seconds field from the circular buffer
* Removing some member variables and a helper function from daqd.

these are code copies of an install script from another program that
are not needed afaict

Update the daq parser to consume a port number as well as a broadcast interface name.

* This introduces a address:port parsing call into the net_writer code to allow it to handle both the port information as well.
* Removes a minor memory leak, by moving from strdup w/o a free to a std::string.
* As we no longer transmit data via daqd data concentrators this removes the DC code path from the net_writer broadcast section.
* Also removes to unneeded constants.

Changing the defaults for shmem_input/size to good defaults so they do not need to be mentioned in the daqdrc.

This will detect a missing libfl-dev package on Debian.  Find it at configure time, not build time.