LHO-Distributed-NDS-Install.md

@@ -5,12 +5,12 @@ Install Date: June/July 2021
 
 ## System Nodes ##
 
-| Node | HA-Proxy | NDS Metadata | NDS IO Node | Memcache node | Notes |
-| ------ | ------ | ------ | ------ | ------ | ------ |
-| nds-1.dcs | X | X | X | X | Public interface |
-| nds-2.dcs | | | X | X | |
-| nds-3.dcs | | | X | X | |
-| daq1.dcs | | | | X | dev/test system, runs its own metadata and nds io node |
+| Node | HA-Proxy | NDS Metadata | NDS IO Node | Memcache node | daqd_stream | Notes |
+| ------ | ------ | ------ | ------ | ------ | ------ | ------ |
+| nds-1.dcs | X | X | X | X | X | Public interface |
+| nds-2.dcs | | | X | X | X | |
+| nds-3.dcs | | | X | X | X | |
+| daq1.dcs | | | | X | X | dev/test system, runs its own metadata and nds io node |
 
 
 Additional systems required to run
@@ -30,6 +30,10 @@ The data path from h1daqgds1 to a nds node in LDAS.
 
 ![Rough_data_flow_of_low_latency_data_to_NDS.svg](uploads/ed2e3a81eec8d8ca48a51b6e2dc35826/Rough_data_flow_of_low_latency_data_to_NDS.svg)
 
+The data path for a typical NDS request
+
+![Rough_data_flow_for_a_NDS_request.svg](uploads/ae8419b08fb52d6549d79f6c8d2d2ef3/Rough_data_flow_for_a_NDS_request.svg)
+
 ## Services ##
 
 The nodes in the NDS cluster run a number of services.
@@ -80,6 +84,29 @@ This service can be restarted as needed.  Restarting will only interrupt current
 
 Requires the diskcache server to be running.  The service will query disk cache for the general existence of frames for each frame type.
 
+#### Channel list updates ####
+
+The channel list information is stored in /var/lib/ndsmetadata.  There are two classes of updates to be concerned with at this time.
+
+##### Online Channel Changes #####
+
+A ad-hoc service is run daqd-stream-live-data-updater.service which is a simple python service which periodically checks the channel list in the daqd_stream memory buffer.  If it has changed it regenerates the /var/lib/ndsmetadata/online/Lho_Online-ChanList.txt.gz file and restarts the nds metadata service.
+
+##### Offline (frame) Channel Changes #####
+
+The nds-metadata server only tracks the contents of the frames, not their location.  So it uses a smaller set of data than the monolithic nds2 server.  Tracking the location of frames is delegated to the disk cache system.
+
+The channel lists are tracked as flat files, however a new channel list updater has not yet been completed.  Until this is completed the cronjobs, programs, scripts on nds.dcs are still required to run to update the *Chanlist.txt files.  Nds.dcs must be kept running for now.
+
+There is another ad-hoc service nds_chanlist_sync.service which calculates the checksum of each Chanlist.txt file in /var/lib/ndsmetadata/channel-lists/ and checks with another ad-hoc server on nds.dcs.  If the checksums have changed a new copy of the channel list is brought over and the nds metadata server is restarted.
+
+To add a frame type:
+ * Add the frame type as normal to nds.dcs
+ * touch the frame type's channel list.txt(.gz) file in /var/lib/ndsmetadata/channel-lists/
+ * restart the nds_chanlist_sync service
+ * add the new frame type to the frame type list in the puppet, running puppet
+   * this will regenerate the config to include the new frame type and restart the metadata server.
+
 ### NDS2 IO Node ###
 
 | item | description |
@@ -99,3 +126,39 @@ Memcached is not authenticated.  The contents of the cache are protected via fir
 | ------ | ------ |
 | systemd unit | memcached |
 | role | cache the frame data in memory |
+
+## OS, Repositories, and Packages ##
+
+All of these systems are Debian 10, tracking development of the CDS software.  The software is currently automatically packaged by the cds packaging system.
+
+All the software will build with a new enough C++ and Go compiler.  It is expected to take minimal porting work to build on the next generation LDAS OS.
+
+The packages are pulled from the CDS buster-experiment repository.
+
+ * Install the base setup package, the latest when this was written is https://apt.ligo-wa.caltech.edu/debian/pool/buster/cdssoft-release-buster/cdssoft-release-buster_1.0.5_all.deb
+ * Install the experimental repo
+ * apt-get update
+
+<code>
+deb [signed-by=/usr/share/keyrings/cdssoft-unstable-archive-keyring.gpg] http://apt.ligo-wa.caltech.edu/debian buster-experimental main
+</code>
+
+### Packages ###
+
+ * nds-ha-proxy
+ * nds-metadata-server
+ * nds2-io-node
+ * daqd-stream-server
+
+## Puppet Configuration ##
+
+A local puppet policy has been created to configure the nds-* systems.  It is put in /root/puppet/modules/cds_data_stream.  It is then applied with 'puppet apply --modulepath /root/puppet/modules /root/puppet/modules/cds_data_stream/nodes/nodes.pp --verbose'.
+
+Very little to no manual work should be needed to setup a node.  Mainly installing the keytab (proxy system only) and creating empty files in /var/lib/ndsmetadata/channel-lists (metadata server only, could be automated).
+
+Typicall work has been to commit the puppet changes to git, pull the puppet onto daq1.dcs, then use a script to push the puppet files out to the nds-* boxes.  Then run a script to apply the puppet to each of the nds-* systems.
+
+Scripts:
+ * /root/local_puppet.sh
+ * /root/pp.sh   (push puppet)
+ * /root/rc.sh   (run commands on each of the nds-* systems)
\ No newline at end of file