LHO-Distributed-NDS-Install.md

@@ -118,14 +118,32 @@ The nds2-io-node is a python process which speaks the NDS2 protocol.  It can tak
 
 ### Memcache ###
 
+| item | description |
+| ------ | ------ |
+| systemd unit | memcached |
+| role | cache the frame data in memory |
+
 The memcached distributed network cache is used to cache recently used frame data.  It is split over several nodes.  The presence of the memcached instance is purely an optimization, the system will function with reduced speed if the cache is not available.
 
 Memcached is not authenticated.  The contents of the cache are protected via firewall rules which restrict access to the nds nodes.
 
+### daqd_stream ###
+
 | item | description |
 | ------ | ------ |
-| systemd unit | memcached |
-| role | cache the frame data in memory |
+| systemd unit | daqd-stream-server |
+| role | receive the CDS data stream and metadata and combine them in a shared memory buffer |
+| writes to | /dev/shm/daqd_stream |
+
+This service simply needs to know how to receive the data stream and the metadata.  It is running as:
+
+<code>/usr/bin/daqd_stream_srv --meta-data cds://10.29.100.1:9010 --data-source tcp://10.29.100.1:9000</code>
+
+This will consume about 66MB/s steady state.  When metadata is pulled it is about 43MB of data.  This should only happen at startup or when there is a channel list change in CDS.  The system is designed to auto reconfigure itself, so minimal intervention should be needed.
+
+If there are issues stop it, nds2-io-node, delete the entry in /dev/shm/daqd_stream.  Then restart daqd-stream-server and then nds2-io-node.
+
+By the current design, clients that consume daqd_stream will not block this service.  If the system is heavily loaded the clients may not be able to read all the data in time.  Presently data is kept for 5s.
 
 ## OS, Repositories, and Packages ##
 
@@ -200,3 +218,12 @@ The above descriptions of the services should provide help in localizing issues.
  * On all nodes
    * start nds2-io-node
    * start memcached
+
+## Known Limitations/Issues/Future work/Notes ##
+
+* Online streaming of the dmt frames is not possible.  The online source for the nds2 servers is the CDS data stream.  This is being worked on, and will require the server to serve from low latency frames and the CDS data stream.
+* The proxy is a bottle neck.
+  * On the cluster we can run the nds2-io-node on any cluster node (after packaging), this is easy.  However distributing the full ifo data to all those nodes is probably prohibitive.
+* The metadata server cannot be configured via the proxy like the nds2-io-node can be
+* To help with scaling the metadata server can replicate its channel data to other metadata servers.
+* As this setup is duplicated in the other clusters (and CDS) the metadata server is able to share its channel lists and provide a global view of all the channels with the ability to proxy requests to remote servers (slow and proof of concept only at this stage).
\ No newline at end of file