| ... | ... | @@ -10,12 +10,12 @@ Install Date: June/July 2021 |
|
|
|
|
|
|
|
## System Nodes ##
|
|
|
|
|
|
|
|
| Node | HA-Proxy | NDS Metadata | NDS IO Node | Memcache node | daqd_stream | Notes |
|
|
|
|
| ------ | ------ | ------ | ------ | ------ | ------ | ------ |
|
|
|
|
| nds-1.dcs | X | X | X | X | X | Public interface |
|
|
|
|
| nds-2.dcs | | | X | X | X | |
|
|
|
|
| nds-3.dcs | | | X | X | X | |
|
|
|
|
| daq1.dcs | | | | X | X | dev/test system, runs its own metadata and nds io node |
|
|
|
|
| Node | HA-Proxy | NDS Metadata | NDS IO Node | Memcache node | daqd_stream | HW | Notes |
|
|
|
|
| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
|
|
|
|
| nds-1.dcs | X | X | X | X | X | 10 Cores 16 GB ram | Public interface |
|
|
|
|
| nds-2.dcs | | | X | X | X | 10 Cores 16 GB ram | |
|
|
|
|
| nds-3.dcs | | | X | X | X | 10 Cores 16 GB ram | |
|
|
|
|
| daq1.dcs | | | | X | X | 8 Cores 32 GB ram | dev/test system, runs its own metadata and nds io node |
|
|
|
|
|
|
|
|
|
|
|
|
Additional systems required to run
|
| ... | ... | @@ -54,6 +54,7 @@ The nodes in the NDS cluster run a number of services. |
|
|
|
| config file | /etc/ndsproxy/ndsproxy.json |
|
|
|
|
| user acl list | /etc/ndsproxy/userfile.txt |
|
|
|
|
| number in cluster | 1 |
|
|
|
|
| system requirements | good network, a few cores, < 1GB memory |
|
|
|
|
|
|
|
|
Other notes.
|
|
|
|
|
| ... | ... | @@ -82,6 +83,7 @@ The authenticated interface of the proxy needs a kerberos keytab, and a user lis |
|
|
|
| systemd unit | nds-metadata-server |
|
|
|
|
| role | This service manages the list of channels, which frames contain them, and when |
|
|
|
|
| channel list directory | /var/lib/ndsmetadata |
|
|
|
|
| system requirements | good network, 8+ cores, 3-6 GB ram at the sites, likely 10+ GB ram at CIT |
|
|
|
|
|
|
|
|
Notes:
|
|
|
|
|
| ... | ... | @@ -89,6 +91,10 @@ This service can be restarted as needed. Restarting will only interrupt current |
|
|
|
|
|
|
|
Requires the diskcache server to be running. The service will query disk cache for the general existence of frames for each frame type.
|
|
|
|
|
|
|
|
The RAM usage (resident memory) of this service sits at around ~2.5-3GB once it has loaded and is running stable (it will have a VIRT space of closer to 8-9GB). However when initially loading the channel list it may take closer to 5-6GB.
|
|
|
|
|
|
|
|
The main data structures in this service are lock free for readers, and the per connection memory allocates are kept low. So the main limit is CPU resources when doing large/comples metadata queries.
|
|
|
|
|
|
|
|
#### Channel list updates ####
|
|
|
|
|
|
|
|
The channel list information is stored in /var/lib/ndsmetadata. There are two classes of updates to be concerned with at this time.
|
| ... | ... | @@ -118,15 +124,19 @@ To add a frame type: |
|
|
|
| ------ | ------ |
|
|
|
|
| systemd unit | nds2-io-node |
|
|
|
|
| role | This service speaks the full nds2 protocol and does the frame reading. It requires the use of a disk cache server to locate frames and a nds metadata server to query channel lists and availability. |
|
|
|
|
| system requirements | good network, 1+ cores, ~1 GB ram |
|
|
|
|
|
|
|
|
The nds2-io-node is a python process which speaks the NDS2 protocol. It can take its configuration via command line arguments, or be configured to pull its configuration from a central repository. The setup here pulls the configuration from the administrative interface on the proxy.
|
|
|
|
|
|
|
|
Regarding system requirements. The io node needs enough ram to open some frames and to buffer data for client requests.
|
|
|
|
|
|
|
|
### Memcache ###
|
|
|
|
|
|
|
|
| item | description |
|
|
|
|
| ------ | ------ |
|
|
|
|
| systemd unit | memcached |
|
|
|
|
| role | cache the frame data in memory |
|
|
|
|
| system requirements | good network, 1-4 cores, 1+ GB ram |
|
|
|
|
|
|
|
|
The memcached distributed network cache is used to cache recently used frame data. It is split over several nodes. The presence of the memcached instance is purely an optimization, the system will function with reduced speed if the cache is not available.
|
|
|
|
|
| ... | ... | @@ -139,6 +149,7 @@ Memcached is not authenticated. The contents of the cache are protected via fir |
|
|
|
| systemd unit | daqd-stream-server |
|
|
|
|
| role | receive the CDS data stream and metadata and combine them in a shared memory buffer |
|
|
|
|
| writes to | /dev/shm/daqd_stream |
|
|
|
|
| system requirements | 1-2 cores, 2+ GB ram |
|
|
|
|
|
|
|
|
This service simply needs to know how to receive the data stream and the metadata. It is running as:
|
|
|
|
|
| ... | ... | |
