I cannot use Apple products...................

They have a restraining order against you?   

I gave up saving all my digital music. Now I just pay $7.95/month to stream from Amazon. Their music library is large and they have most of what I listen to.

I gave up saving all my digital music. Now I just pay $7.95/month to stream from Amazon. Their music library is large and they have most of what I listen to.

That's definitely an option, but I see three main reasons to have your own music library:

1.  If the PC police decide the music you listen to is hateful, or the artist is politically "dangerous", Amazon could bend the knee to the cancel culture morons and remove that content;

2. Most of us who spent the time and money collecting music that has meaning for us, we like listening to that music most often.  It takes us back to the times of our lives that the songs were associated with.  Why pay monthly for thousands of songs when 95+% of the time you only listen to your favorites?

3. You may own special edition releases or off-label recordings that are not available on any streaming services.

Nothing wrong with the streaming service if you just want a never-ending music library with great variety.  I have Spotify for free that was bundled with my AT&T plan.

I've tried the Amazon free/Prime and  the monthly fee subscriptions.  They let me upload some of my own music files for easy access through my Alexa devices, phone, FireTV, etc.  Best of both worlds, but the upload feature requires your music be in a format you can upload and use.  Back to having to store and organize your digital music files.   

I'm baaaaack!!   

After WEEKS of testing old hard drives, I finally have my unRAID server running under a temporary license.

Setting up the array was SUPER EASY.  The hard part was reading the recommendations on hardware, which unRAID options and add-ons  I wanted and the REALLY LONG process of preclearing the disks.

After playing musical chairs drives with my existing computers, I now have 9 storage devices (not counting the boot flash drive), which keeps me under the 12 drive limit for the 2nd tier license.  I still have some expansion room, too.

Last night I requested a 30-day trial license, started up my array, and it was off to the races. 

I configured:

a) 2 x 2TB drives for parity (better than 1 -- for redundancy);

b) 4 x 2TB drives for data storage;

c) 2 x 500GB SATA SSDs for caching; 

d) 1 x 500GB NVMe SSD "Unassigned", for Virtual machine storage, Plex metadata and caching, etc.

As a 2-drive SSD cache "pool", it provides 500GB of cache and a mirrored SSD for redundancy, so I don't lose anything stored there before it's written to the array.  There is a job scheduled nightly to move cached data to the array.  This is one way to compensate for the lower performance of the unRAID array design vs. a striped disk array.

It took 5+ hours for the cache and data drives to be formatted and for the 2 parity drives to synchronize.  Everything reports "healthy", so that's nice to see.

Now I need to configure a new login, change the default passwords, add my Media shares, and so on.

Once I have everything setup properly, and the trial expires in 30 days (plus 2 additional 15 day extensions if they'll let me), I'll have to pay for the license.  I like being able to test something for 2-3 months before committing financially.

My array provides 4 disks-worth of storage, so that's 8TB plus 4TB of parity "backup".   I could opt for 2TB parity and 10TB of data, but I prefer more redundancy should the parity drive die.  If the only parity drive dies, it can be rebuilt, but if another (data) drive dies, the array is lost.  I can recover most of the data from the individual drives, but the array can't be mounted.

My least-used drive only has 2 months of activity on it, and the oldest has over 7 years.  I'm using the next-to-newest and next-to-oldest drives for parity.  They happen to be the same model and have 5900RPM speeds -- better than the 5400RPM the rest have.

If you decide to use unRAID, and you plan to use spare drives of various sizes, I recommend not getting a license -- even a trial one -- until you've 100% tested and decided on the drives to use.  That can all be done without starting the array.  I tested 17 drives just to see if they were all still good.  3 had to be "retired".  3 smaller ones replaced 2 x 1TB surveillance recording drives, which I then used to replace 2 x 2TB drives in my main PC.  I couldn't justify having many TBs in the 2 workstations while running 8 smaller drives in the storage array.  I hope to use the RAIDs more now for direct storage rather than my local drive, both for protection and network accessibility.

If, however, you plan to use all new drives, then the process of deciding on which drives to use and preclearing them is more streamlined and should only take a few days at most, depending on the size of the drives purchased.

Stay tuned.  After I  get the "housekeeping" taken care of, it'll be time to finally install Plex...the #1 reason for this project.   

 Cool project. The last big project I worked on was fiber channel adapters for servers to arrays. I worked on so many different arrays we had to have multiple people get certified as admins for all the different arrays. The nice thing about all this was when a company retired their arrays and JBODs we got to take them home for personal use. I had a ton of stuff running 24/7 in my home office. All RAID 5. A little overkill for home use. 😂

Cool project. The last big project I worked on was fiber channel adapters for servers to arrays. I worked on so many different arrays we had to have multiple people get certified as admins for all the different arrays. The nice thing about all this was when a company retired their arrays and JBODs we got to take them home for personal use. I had a ton of stuff running 24/7 in my home office. All RAID 5. A little overkill for home use. 😂

I've had to care for and develop database backup plans for several RAID enclosures, including NetApp devices that were replicating on their own between SCIFs.

The servers users Fiber Channel to direct connect to the storage array, but the LAN connection was limited to 1GB speeds.  10GB paths were established for replication, though.

Most of the drives were SAS, connected to backplanes that were attached to a RAID controller in the Dell servers.

At my last job, the PMO sent us SATA drives that were later replaced with SAS as they originally intended to ship.  In an enterprise environment, storage speed is a major issue.

At home, until you start creating your own videos or storing and streaming movies, a mid-range NAS appliance is fine for most. 

This project highlights one of my, and I'm sure some others', problems of having older, lower-capacity drives with no real use for them.  A RAID is a great option for that, assuming the drives are not too limited in capacity.  At some point, buying a new multi-terabyte drive is a better solution than trying to gang together 8 drives to achieve the same capacity!   

Just leaned a cool/annoying lesson about unRAID tonight/this morning.

I wanted to try booting from an external USB-C SSD with Windows 10, just so I know I can for maintenance on that machine.  Sometimes Windows is the only OS for certain apps, like checking bad drives needing warranty work, and similar utilities.

So, the boot worked great.  Then, when I rebooted into unRAID, I just opened the MSI UEFI/BIOS, and dragged the flash to infront of the external SSD and booted.

It booted, but then I started having all kinds of issues with the array.  Cache disks were confused, the  parity drive wanted a resync, and who knows what else.  I stopped the array and checked the lsblk command.

The external SSD apparently didn't feel like sitting in the back of the buss (PC pun).  He took over as device sdda, which was the boot flash before.  That put every other device a drive letter higher.  When the array tried to start, it saw one cache drive, not 2 (one was the boot flash).

Then it went to the  next in line, the other actual cache drive, and thought it was disk 1 in my array -- basically the first parity drive.  It was a bad domino effect from there.

So, I stopped the array, rebooted with the external disk removed, and started the array again.  This time, the parity drives were happy, but the first cache drive was still showing no filesystem.  So, I reformatted that SSD, and the cache pool was happy again, too.

Lesson:  don't add devices to the PC with "Auto Start" enabled for the array.  If you add a device, disable the array from auto starting, reboot with the new drive, and verify nothing unexpected was changed -- like drive positions in the OS.

I also scheduled a daily parity check at 05:00, right after the scheduled job that moves the cached files to the array disks, just to make sure everything is being stored accurately.  It's a decent alternative to having ECC memory from what I've read. 

 

Update ...

Here's what my disk status info on the web GUI Desktop for unRAID looks like:



I now have the array and parity drives spinning down after 15 minutes.  Earlier this week, only disks 2, 3, and 4 were spinning down when idle.  That told me I changed something that was accessing the 2 parity disks plus Disk 1.

After doing research, I found that even though I configured everything I could to reside on the cache or unassigned disks, there are still some folders that are stored on the array no matter what.

I'd installed a new docker (app) called DuckDNS, a dynamic DNS application to automatically update my home's RoadRunner IP address on the DuckDNS,org site (free DNS account).   This docker on unRAID attempts an update every 5 minutes, which seems to keep resetting my 15 minute spin down clock for the first disk + 2 parity disks.

So, I found a good set of instructions for moving 2 related folders:  docker (which has the image docker.img) and libvert (the docker image libraries).  It's not as simple as moving the folders to the unassigned disk.  Services have to be stopped, config files have to be updated, etc.

Once that was all done, I watched the clock, and 15 minutes later, all 6 drives, plus the 2 cache SSDs, switched from "active" to "standby".  YAY!

This is a feature of unRAID I wanted to make sure worked, as it reduces power usage as well as prolongs the lives of the disk drives.  It can also slow response times as you have to pause while the drives spin up, but once what you need is cached, the drives can spin down without having to pause over and over again.  If everything you are doing can get done within the 15 minutes after spin up, you'll also have no need to pause more than once.

15 days left on my trial key -- plus another month if I extend it twice.  So far, so good, and I'm out no money for a license yet.  Plenty of time to make sure this is something I can use and that works before buying.   

A couple of things to consider if you're  thinking about trying unRAID:

1.  By default, there is no SSL/HTTPS protocol (port 443) installed for the basic unRAID server, nor for the add-ons you install (with a couple of obvious exceptions).

The server is not intended to be a production business data repository with both internal and external access and associated security features.  Having said that, you can install "Let's Encrrypt" with free SSL certificates registered to your domain (did I mention free SSL certs?  ), Reverse Proxy to route traffic coming in from the Internet to your specified servers & ports, thereby isolating the risk to a docker, VM or machine, and total hardware encryption for your storage drives to prevent physical theft and misuse of your data.

unRAID is designed and advertised to be a media storage solution, not a repository and fail-safe solution for your important and irreplaceable documents.  However, there's nothing that says if you keep your internal network secure and the port forwarding to the unRAID server to a bare minimum, that you can't make it secure.  In fact, you can install the unRAID docker container for OpenVNC and OpenSSL to tunnel into the server and encrypt your traffic, thereby creating a wrapper that mitigates the lack of built-in security for watching movies.

2.  I added the Let's Encrypt docker to setup secure connections to some other dockers, like NextCloud, and it broke my network connection to the PLEX media player.  Part of the Let's Encrypt template asks for the network connection type to use, and I'd created a separate custom connection that maps ports from the router to Let's Encrypt.  Originally I was using the BRIDGE network type for both PLEX and unRAID.  I had to change that setting to HOST.   

Were I not using multiple network NICs, that likely would not have been an issue, since there would not have been a BRIDGE option to begin with -- and HOST would have been the default.

Anyway, it's working again.

3.  Now that I have several docker containers installed, I am seeing a pattern.  Each docker that installs information in the appdata folder needs the default setting changed to add the "slave" option to the mount command..  If you left the appdata folder on the array as it is by default, this is not an issue.  I moved my appdata to an unassigned SSD to a) make the docker apps faster, and b) prevent the dockers from stopping the array disks from spinning down after the selected idle time.

4.  If you use the default settings for how balanced the array drives are filled with data to "High Water Mark, you'll see most of the initial data you copy, if not all of it, on Disk 1 exclusively (plus the parity drive/s).  Basically, the server takes the largest disk capacity in the array and divides by 2.  Once I use that much of Disk 1, it'll start saving to Disk 2, and so on.  That means that for now, I rarely see Disks 2, 3 and 4 spin up, and they are all pretty much  empty.  That helps prolong the disks' life, and it can save electricity cost.  Performance isn't much of a difference, since I cache pretty much everything to SSD, and it takes close to the same amount of delay to spin up one disk as it does four.  If you prefer, you can select other storage schemes that create a more balanced approach to where data is stored.  Or, if storing large video and photo files, you can set unRAID to store each sector of a file on the same disk to prevent having the file scattered across multiple disks.  If you ever need to restore your system from a crash, it's good to have the files self-contained on a single disk in case one disk can't be recovered.  That way you only have to rebuild the one disk from parity to see its data, and the rest of your files are immediately accessible via the working disks.   

That's all for now.  Later!   

Tip of the night:  If you want to learn how to use unRAID to do just about anything, the YouTuber "Spaceinvader One" has taken it upon himself to put out a bunch of videos over the years.

He moves rather quickly, so sometimes I have to press PAUSE every other sentence, or just use the YT player settings & slow him down to less than 100% playback speed.   

This is the first video I'm using for securing my server for outside access:



This is the second one:



I just connected to my server using a registered domain, a CNAME record for a subdomain, and a registered SSL certificate from Let's Encrypt (FREE!).  I also am using the Dynamic DNS service DuckDNS.org to bounce from the registered domain name to the RoadRunner IP, but I could have just as easily created an A record on my name server pointing to that IP address directly.  DuckDNS just keeps that IP updated in the event that it changes.

Next, I'll be diving into video #2 above to setup a wildcard SSL certificate (such as *.2ahawaii.com").  That makes the same certificate valid for any subdomain on your site as long as it's part of the same domain.  Most wildcard SSL certs allow unlimited subdomains.

I've done that at work several times.  Much less work (and expense) if you need an SSL certificate for more than a couple of servers, or if you do like me and create a bunch of subdomains that a single web host is configured to serve up a specific virtual web site based on the subdomain names. 

For example: 

(1) www.domain.com goes to the regular port 80/HTTP and 443/HTTPS web server, and that is located in the directory /var/httpd/www/.
(2) nextcloud.domain.com goes to the same ports, but the web server directs the traffic to /var/httpd/nextcloud/ and rewrites the URL (uri) you see in the browser address as "nextcloud.domain.com".
(3) photos.domain.com goes to the same ports, but the web server directs the traffic to /var/httpd/pics and rewrites the URL you see as "photos.domain.com".

This allows multiple web sites to exist on one physical server using the standard HTTP ports, while recognizing that the URL is asking for a specific site apart from the others.

The rewrite command for web servers is powerful.  I use it to force the incoming connection to only use HTTPS (SSL encrypted), to redirect links to appropriate locations, like if I rename a folder but don't want broken links on pages or favorites lists, and many other uses.

The most common rewrite is to remove the "www" from your domain in the address so it only really uses the domain named site as the default. 

If anyone tries to use the IP address instead of a DNS-registered subdomain/domain name, it'll be rewritten to a default URL I set, which might include a page of links to the available sites so you can get to where you really needed to go.

One thing I still need to do is set up a cron (scheduled) job to renew my Let's Encrypt SSL certs.  Apparently, the free certs expire between 2-3 months, but you can renew them with a script.     No problema.

Nevermind.  Renewing a Let's Encrypt certificate on unRAID is super easy ... barely and inconvenience!  Just restart the Let's Encrypt docker, and the certificates are automatically renewed for another couple of months.  As long as the unRAID server is rebooted, or the Let's Encrypt docker service is restarted beforehand, it never expires.  If it ever does happen, just restart the docker, and it's fixed.

Too easy.   

At first I was confused as to why I needed to re-home my domain DNS configuration away from my domain registrar's name servers and onto the CloudFlare servers.  I performed those steps to make sure I was following the video instructions, but now I'm analyzing the purpose for those steps for my own understanding.

Let's Encrypt takes advantage of an API call to request, install and renew your SSL certificates.  Those steps can get confusing, especially if you're bouncing between Linux, Windows, unRAID and Mac OSX servers having to install the certificates.  For example, If you send the request files to the SSL service in UNIX format, you have to convert the certificate files to install on a Windows OS.  That of course only applies to wildcard certs, as otherwise you would generate separate requests for each OS and not have to convert.  For wildcard SSL files, you have to copy them to each physical server that's part of the parent domain.  Still, it's a tedious, repetitive process that can introduce errors when changing web server and SSL configuration files.  The unRAID docker for Let's Encrypt does all of that for you as long as your domain is using API code and keys which both sides understand.

So, this is just one more step in the process that required a little additional education on my part .  Luckily I had it working before I asked the question.   

This is a lot of work. Thanks for sharing your research, FJ.

This is a lot of work. Thanks for sharing your research, FJ.

My pleasure.  It helps me also by keeping my progress and problems documented.   

Well, THAT was easy!   

I went through the steps in the video to configure a wildcard SSL certificate.  Basically:

1.  Log into my Cloudflare account and copy the API key.
2.  Edit the cloudflare.ini file for Let's Encrypt with my email address and that API key string.
3.  Edit 3 lines in the Let's Encrypt template settings: 
     a.  Replace subdomain list with the word wildcard,
     b. replace Validation method "http" with "dns", &
     c. add DNS-Plugin setting with "cloudflare".
4.  Restart the Let's Encrypt docker.

Now when I access my NextCloud server with my subdomain name,  my certificate says the SSL name is valid for *.<domain-name>,com instead of nextcloud.<domain-name>.com.

Why would I want to do this if I can get a bunch of specifically-named subdomain certs?

One reason might be that there are limits to this free service.  One is, you can only create certs for the same subdomain names 5 times per week.  If you keep creating duplicate-named certs, they lock out further creations.

So, if you're experimenting with the SSL setup, they have a sandbox (experimental environment) you can point to for that, then set it up for production when ready.  Unfortunately, sometimes it takes experimenting on the live setup to troubleshoot.  By then, you might have hit a limit.

https://letsencrypt.org/docs/rate-limits/

Another reason might be because when following tutorials, you tend to use the same labels and naming conventions in the tutorial.  Afterward, you might want to do some renaming or completely back out that setup to try another tutorial you found. Having wildcards means the SSL certs work no matter what subdomain names you configure, so the certs don't have to be recreated time and again.

From my limited use so far, I think CloudFlare is a very good service.  I'm hoping it'll remain free for at least the basic features.

The service is basically a replacement for your edge router -- the one sitting between your internal network and the Internet.  External traffic goes to CloudFlare, and it sends the data to your router after it's done its magic.

I set up a Page Rule that redirects all my <domain-name>.com/* page requests to HTTPS/SSL on port 443.  So I could effectively shut off port 80 mapping on my home router now, removing a security risk on my home network.   

CloudFlare also has a lot of website analytics and stats pages, one of which runs a series of tests to compare how much quicker your website runs with CloudFlare than without.

Of course, you have to wonder how reliably a site that's advertising faster load times would report on load time comparisons.   

From my limited use so far, I think CloudFlare is a very good service.  I'm hoping it'll remain free for at least the basic features.

The service is basically a replacement for your edge router -- the one sitting between your internal network and the Internet.  External traffic goes to CloudFlare, and it sends the data to your router after it's done its magic.

I set up a Page Rule that redirects all my <domain-name>.com/* page requests to HTTPS/SSL on port 443.  So I could effectively shut off port 80 mapping on my home router now, removing a security risk on my home network.   

CloudFlare also has a lot of website analytics and stats pages, one of which runs a series of tests to compare how much quicker your website runs with CloudFlare than without.

Of course, you have to wonder how reliably a site that's advertising faster load times would report on load time comparisons.   

You do know there only two or three of us on this thread that
understand your Jargon?

You do know there only two or three of us on this thread that
understand your Jargon?

the rest must be oppressed

You do know there only two or three of us on this thread that
understand your Jargon?

I was wondering why I was enjoying this thread so much!

    

the rest must be oppressed

You're tempting the devil.   

Well, yesterday I installed Ubuntu LTS 16.04 server on an unRAID VM.  Had to enable one of two CPU features in my BIOS (HyperThreading was on, but Vx-D was off).  Other than preparing by watching a video on tips and tricks for unRAID VMs, which mainly involved choosing which CPU physical and virtual cores to select, and a blog on what other options to select, I had it installed and working in 10 minutes not counting the 30 minutes to download the ISO image.

Today, I set up my remaining domain DNS records after moving all four of them to CloudFlare.

I found out the Let's Encrypt docker really doesn't work well with multiple domain names.  Even though there is a varialbe for EXTRA_DOMAINS that adds them to the SSL certificate, it doesn't quite set up the proxy to work with all of them.

So, I left the original domain I had working setup in Let's Encrypt, and then created CNAME entries to alias the others as that domain.  That way, the redirects are pointing to the one domain that Let's Encrypt is configured to properly route.

It took a few tries to get all the settings to mesh together, not to mention the TTL (time to live) delays that some settings required to expire before propagating.  In the end, I have 5 domains.  Four point to one as their redirect alias, and the alias points to another alias on DuckDNS,org.  The IP is resolved there, and the certs seem happy to accept any URL associated with it.

Part of the Linux installation includes a LAMP stack (Linux Apache MySQL PHP), so I can do web development.  Next step is to create my web repositories and figure out the best way to forward browser requests to them.  Right now, all of them point to the nginx web server as part of Let's Encrypt.  A simple ReWrite command to each domain on my Linux VM ought to work, but I won't be sure until I get it working.  The reverse proxy thing is rather new for me, so I can't be 100% sure of how it'll affect the redirects all on one physical server and 2 operating systems.