I published my next Bitcoin video tutorial. In this one, I demonstrate how to help combat Bitcoin mining centralization, by mining at home with a small, inexpensive ASIC, using Datum running on StartOS to construct your own block templates.
Hi, having recently installed Knots in addition to Core, I assumed that there was no need to download the entire chain. It seemed to be fine at first but now mempool seems to be downloading the transaction history. I am getting tight for memory space on the SSD and I am not sure if it will over-write the previous uploaded history or take up more space. The other alternative is that there is an issue with the Start9 setup and has run into an issue and not re-configuring correctly and so the health-checks are not completing. Any re-assurance or suggestions?
What size is the SSD?
The SSD is 1TB - I am wondering if I will end up with 2 copies of the chain
It won’t have two copies of the chain, but 1TB is probably cutting it close. One of my test servers has only Knots, Electrs, and Mempool installed on it, and fully synced it is using up about 907GB. I would definitely watch it to see how fast it is growing. You’ll probably want to upgrade to 2TB when you can, just to give yourself some breathing room.
Which service(s) are failing the health checks?
Looks like the Mempool is the main issues: Error: failing to sync but itd affecting the other apps. I have restarted but no change. I will just have to wait a while and see if it completes.
If you watch the Bitcoin Knots service for a while, do you see it periodically (every 10 minutes or so) indicating that the sync timed out and it is retrying?
Btc Knots just keep trying to start. The issue seems to be the Transaction Indexer in Mempool - messsage is curl: (6) Could not resolve host: bitcoind.embassy curl: (6) … Initial blockchain download still in progress …
SSD hard drive is 93.5% full.
Ok, let’s run the following process which should address some possible issues (you might want to create a backup first, if you are concerned about your lightning node failing if the space fills up):
-
Stop all services. Stop those with dependencies first, and make sure they each are fully shut down before proceeding to next one. For example, Core Lightning, then Mempool, then Electrs, then Knots.
-
Uninstall Mempool (we will reinstall later).
-
Restart the server
-
Give it at least 10 minutes to fully start up and settle.
-
Start Knots
Let me know if it gets this far, Knots starts, and health checks look good.
1 Like
Followed your instructions. Knots just keeps restarting on a cycling loop - mempool uninstalled.
Ok, and I suppose the disk space is almost full (System > Monitor > Disk > Percent Used). What is showing up in the logs (Services > Knots > Logs)?
An interesting observation – on my main server (with 22 services, likely including all the ones you are running) I am using just under 1TB (999.48 GB), so even though 1 TB is probably cutting it a little close, it should definitely be possible with a smaller set of services.
I think you have Sparrow running on there, right? Do you have your seed phrase backed up? If so and you are comfortable doing so (with not too much BTC on the line), then you could try uninstalling that service to see if it gives you any breathing room.
What is the list of all the services you have installed on there?
The logs are a long list of recurring items. Should i try re-installing mempool? I could remove Sparrow. Have you ever used the System Rebuild option? If I can get a stable S9 then I will look to transfer it to a larger sized drive.
What is the used disk space? I’m assuming a full disk is the problem, but without anything confirming that theory from the logs, it is just a guess.
My main concern is losing your lightning node (and your hot wallet from Sparrow if you don’t have the seed phrase written down). If you have a StartOS backup saved to a thumb drive or external hard drive (from System > Create Backup), then it would be more comfortable to explore the more aggressive options, like rebuilding.
Assuming disk space is the issue, my next recommendation is uninstalling some services to free up space, and reinstalling Knots.
For that, stop all the services. When all are fully stopped, uninstall Mempool, Electrs, Datum Gateway, and Knots. If you have your hot wallet seed phrase written down, then also uninstall Sparrow (otherwise leave it installed). Leave Core Lightning installed. Uninstall any other services besides Core Lightning you might have as well. You should now have only Core Lightning (and possibly Sparrow) installed but Stopped.
At this point, disk usage should be low. Check under System > Monitor > Disk to confirm. If usage is still high, then there is a more systemic problem happening.
Then reinstall Knots, and wait for the blockchain sync to fully complete (on typical hardware and internet speeds, should be around a day or two). If it gets all the way through syncing, then check if it gets back into a boot loop.
Next, if Knots is running stably and is fully synced, then reinstall, configure, and start Datum Gateway. Do not repeat the SSH steps-- those will still be installed. If Datum Gateway gets into a boot loop, uninstall and reinstall it again (go back and watch that part of the video if necessary).
If Knots and Datum Gateway are running stably, then launch Core Lightning. If these minimal set of three services are able to all be running stably (Knots, Datum Gateway, and Core Lightning), then take another backup.
Either way, let me know how the system looks at this point.
1 Like
It’s using about 911GB of disk space on a 2TB SSD. Everything seems to be running. (Mempool not installed yet).
Paul - this is one of the best tutorial videos out there. I hope you continue to make these kinds of videos - you are doing a real service to the Bitcoin community. Your video got me 100% of the way there in terms of setup.
I will say though that I’ve had nothing but problems getting CLN to work reliably on my Start9 system. At one point I was able to open a small channel with you, and do a Boltz swap to get some inbound liquidity, and I had mining going with Datum and was earning lightning payouts - just like your video walks through. However, when I tried to expand my CLN to include other channels, I wound up with endless payment routing issues and found it impossible to do any swaps and get any additional inbound liquidity or make any payments out of my node - or receive any substantial amount of sats through inbound liquidity (nothing beyond the tiny amount I’d made with your channel where I was getting a trickle of sats from my Bitaxe). Basically the node was completely unusable for anything else. I’ve since deleted it and tried LND and I can’t get that to work for anything either - so its even worse than CLN because I can’t even do lightning payouts from Ocean on LND.
Today I had an idea that maybe I could set up a Bolt12 offer on Ocean using my Phoenix lightning node and signing a message from HW wallet - but this failed also. Ocean does say “At the time of writing, only CLN is known to work, but we will update this documentation as more compatible implementations of BOLT12 become available.”
I’ve basically given up on Lightning for the moment, but I really want to get this working! Would it be possible to do a one-on-one with you to get some support so I can figure out what is going wrong? I’m not a coder/dev and the log files are meaningless to me. Happy to pay for a session!
Sure. Core Lightning has systemic routing issues, so it is unfortunate that it currently is the only implementation that properly supports BOLT12 offers. You can keep it going with some babying (periodically shutting it down, clearing the gossip store, starting it back up, and waiting several hours for the gossip store to re-build itself). The other issue with it is that it doesn’t provide a mechanism to increase your max routing fee, so when using things with high fees (like my node + Boltz + Liquiid) it will never find a route because the routing fees are too high.
I primarily use LND for Lightning (you can run both on the same server). Besides having much better routing logic than CLN, it also allows you to specify the fees you will accept when sending.
I’ve not had any routing issues with LND myself. Without further details, I would guess that maybe you were trying to use it too soon after setting it up? It takes several hours for the gossip store to populate once you’ve connected to your first peer. It also matters the size of transactions you are making relative to the liquidity of the nodes you have channels with (more channels with better-connected nodes means larger-scale transactions can be made)
I’ll send you a DM and we can set up a virtual meeting to discuss further if you like.
1 Like
I’m running Knots and CLN on an Umbrel Home. I’ve got a swarm of Bitaxe Gammas hashing to Ocean. Trying to create a Bolt 12 offer some how some way to pull my sats from Ocean into CLN Lightning wallet. I’ve a private channel with inbound via ACINQ. Can’t figure how to create the Bolt 12 offer for love nor money. Any chance of some help
This sounds interesting and relates to my challenges with CLN. Was thinking of loading Alby Hub via Umbrel store to see if I could create Bolt 12 offers. Alby won’t load without LND so I’ve held off. Do you know if I can run CLN and LND on my Umbrel? Can I create Bolt 12 offers via Alby Hub? How do I create Bolt 12 offers with CLN?