The server rack has been built and various chains are up and running. I’ve been meaning to document what has been built as it has been running for a few months now (the rack was built towards the end of 2020 and hasn’t changed since). The specs:
Rack: StarTech 25U (130cm x 88cm x 47cm). It’s huge – its nickname is “the wine fridge”.
CPU: AMD Ryzen Threadripper 3970X (32 core/64 processor)
Memory: 256GB G.Skill F4-3600C18Q-128GVK (64GB as RAM drive)
SSD: Seagate FireCuda 520 2TB NVMe M.2 PCI4 (128GB as swap)
HDD Storage: 6x10TB (20TB Raid 10, 20TB Raid 0)
Cooling: Corsair iCUE H150i RGB PRO XT 360mm
Graphics: MSI GeForce RTX 3070 Ventus 3X OC 8G (5900 Cuda cores)
Motherboard: ASUS PRIME X570-P/CSM AM4 ATX
UPS: ATEN OL2000HV 2000VA (+ 4x18AHr AGM batteries)
Firewall: Ubiquiti Networks UniFi Dream Machine
Plus two 4U cases, PSU, NAS drives, modems, monitor/keyboard/mouse and the usual.
The main server is at the top, with a spare 4U unit which will be built at the end of this year. Another 1U server is planned for this year as well.
The 360mm cooler fit nicely with some minor mods to install the 3x120mm CPU cooler fans (in addition to the 6x80mm case fans). With a big case everything was quite easy to fit (this case can fit 14 HDD drives!). Some internal shots:
It is currenting running the following chains:
- Bitcoin full node (not mining)
- Ethereum 1.0
- Ethereum 2.0
- Cardano
- Solana
- Storj
Current CPU utilization is around 18-20% with Solana being the biggest usage of CPU. Without Solana CPU was around 3%. Memory usage is around 42GB and the network usage is quite busy (around 70Mb/s down and 20Mb/s up) – Solana is the big user here. Disk writes are around 45MB/s (again Solana is the main user).
The graphics card is quite busy as well due to Cuda cores being used by Solana. The temperatures are around 70°C. All the fans are quite busy, so if it wasn’t for Solana it would be a more peaceful machine…. (and 20°C cooler).
I like to get the fundamentals of a blockchain running within an hour. Bitcoin, Eth 1.0, Solana and Storj were extremely easy to install.
With Bitcoin, just install the entire block chain – with semi decent internet you’ll have the whole lot with half a day or less.
Solana did have one minor issue where it required 500000 file descriptors to be available which wasn’t well documented. It panics with a memory error without it. As for running Solana 24×7 I might have to turn it off periodically just to give everything a breather for a while (there is no penalty in turning it off). Plus it is affecting the home network with the bandwidth it is using.
Eth 2.0 took another few hours – you don’t stake 32Eth without being extra careful.
Cardano was a bit of an experience with a few false starts. Ultimately the CoinCashew site was the key to moving forward. I’ve got the relay and the block producer on the same hardware – ultimately it would be great to add another relay on another remote server.
The original spec was to install Filecoin, but that turned out to be quite a challenge. It was difficult to install and the miner was almost impossible to get started for a whole number of reasons (one which is here). Ultimately the project failed as it required 1TB free as the minimum requirement to pledge. The 2TB SSD was partitioned into 1TB (primary) + 750MB (secondary) + 128MB swap + boot/other and I just couldn’t get it started, even via HDD. Plus it was heavy duty on hardware with just the lotus daemon running. It just wasn’t worth spending more time (and money) getting it running.
Since Filecoin was abandoned, Storj was installed instead and this was very easy to get going. The returns are quite minimal – with 10TB allocated the return has been around 10c. An attempt was made to get MaidSoft to run, but the documentation didn’t match the latest release and failed to start. Not a great impression, maybe I’ll go back to see what is going on. I’ll also have a go at running Sia and see how that goes.
So not a great experience in sharing drive space yet.
As for staking, only the Eth 2.0 chain is returning anything. Cardano requires at lease 2 million coins to stand a chance and my measly few Solana won’t be doing much soon. There is no Bitcoin mining, so nothing there either.
As for monitoring a validator/stake pool, nothing beats Grafana/Prometheus and you can see an output of Cardano stats in the image at the top. Eth 2.0, Bitcoin, Cardano and the actual hardware stats are easy to show. It would be nice if Storj, Solana and other chains had similar.
On that point, it would be nice if Solana had better GUI tools as it is almost 100% CLI for staking at the moment which won’t appeal to many users.
The server runs on its own subnet, with no public SSH available and limited access even from within the network. Each chain runs as it’s own user with no chain able to read any other chain. A software firewall works in addition to the hardware firewall with port forwarding. Private keys are stored on offline devices. Most IOT devices are not on this network at all and all public ports have a rules associated with them.
Some final thoughts. This is quite an expensive piece of kit, including backup generators, beefed up UPS’s, firewalls and the rest. It is also time consuming looking after this gear. There’s still a bit to go including getting a backup network running (another saga there) and better redundancy. The intention was to not make a fortune out of it – the Eth 2.0 staking is enough to probably make a 100% ROI at present and that is enough to keep the missus happy with this wind tunnel device running in our house (yes, the fan noise is loud).
The next major piece of effort will be to do projects in Ethereum, Cardano and Solana and get those blockchain programming skills up 🙂 Stay tuned. And I can run them all locally now!
Crypto Cam Tech 