2aHawaii

General Topics => Off Topic => Topic started by: Flapp_Jackson on April 21, 2020, 04:41:02 PM

Title: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 04:41:02 PM
Not interested in storage media (CD, DVD, USB, iPod, LP, 45 ...). 

I've had time to organize my vast collection of music on my PC.  The problem is I haven't decided on a good library system.

iTunes used to be mandatory if you bought an iPod of any flavor.  Now, though, you can pretty much use anything and sync to anything else.

I used Amazon Prime Music service for a little and tried their cloud storage.  Not very good as a repository for a large collection, but useful for having the same playlists and songs on several devices.

I've found lots of reviews and recommendations for Music Bee.  Anyone have experience with that?

https://getmusicbee.com/

Any personal insight is appreciated.  Can't always trust the Internet.  Wait -- this is the Internet.   :crazy:
Title: Re: How do you keep your digital music files?
Post by: Jl808 on April 21, 2020, 07:51:04 PM
https://www.synology.com/en-us/dsm/feature/audio_station
Title: Re: How do you keep your digital music files?
Post by: Eric808 on April 21, 2020, 08:41:03 PM
Always have redundancies, backup of the files, on different medias and locations.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 08:48:40 PM
https://www.synology.com/en-us/dsm/feature/audio_station

Yeah, I looked into those already.  I already have 2 older NAS toasters.  Both are ReadyNAS models -- one from Infrant and one from Netgear after they bought Infrant.  Neither have the buffering needed for smooth high res video streaming, and running any kind of transcoding on them makes it that much worse.  Right now, I just do 720p streaming, and it's fine.  One of my NAS boxes is about 10 years old and running an ancient variant of Debian Linux called Sarge.   :shake:

My newer NAS can stream music fine, but I'm looking for an easy-to-use music library management application that's available across platforms (home cloud/server-based), no annual fees involved, and has basic auto-categorizing and smart playlist features.

I've been thinking of building a new storage server to make use of these 500GB - 2TB disks I have.  I could make a descent PLEX server for movies, a FreeNAS server for storage, a VM server, and so on.  With the M.2 NVME SSDs coming out at PCI-E 4.x speeds and Ryzen Thread Rippers with huge numbers of cores & threads for a few hundred bucks, one beefy server could be configured for around $1500 or so.

I have a tough time letting go of old computer parts that still work even when they don't have many uses compared to new hardware.  In the last 2 months, I've gotten rid of SO MUCH PC JUNK it's almost scary.  Why was I saving that Intel 1000GT network card for so long?  I don't think I'm ever going to need another PCI LAN adapter anytime soon!  Same for those PCI SCSI adapter cards.   :wacko:

AMD is making me think it's time to build that server.  Lots of sales this past month, and it's getting cheaper and cheaper.  Everyone was worried about the China supply chain making prices of computer parts spike, but so far it's not happening.  If anything, the demand for work-from-home systems is helping keep prices down from the sheer volume of sales.  Can't easily raise prices in a down economy if you have inventory to move before the new product lines come out this Summer and Fall.

I'll check out Synology again.  Maybe I can talk myself out of a project and into an off-the-shelf product.   :thumbsup: :geekdanc:

Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 08:57:23 PM
Always have redundancies, backup of the files, on different medias and locations.

That goes without saying.  I'm looking more for advice on software utilities or library management.

I've got a subscription for Spotify that's included with my AT&T cell plan.  I've also tried the Amazon Prime Unlimited Music plan.  While that can be fine, there are limitations, and the tools for uploading and organizing what I already own aren't that good.

If I like a song or group, I normally buy the music on Amazon or iTunes/Apple Store.  After buying, I add it to iTunes & playlists, then sync with portable players.

I'm abandoning iTunes as a library manager, so it's time to find something better.  I never really liked iTunes that much anyway, but it was pretty much required. 

I'm looking for input as I'm sure others have struggled with the same decision making process already.

 :shaka:
Title: Re: How do you keep your digital music files?
Post by: Jl808 on April 21, 2020, 09:16:03 PM
The Synology has some nice built-in packages that you can install. Drive redundancies and the ability to backup to USB drives or to other network storage or cloud storage, make it reliable in my mind.

 https://www.synology.com/en-us/dsm/packages
 
In addition, there are some nice apps already made by Synology that you can check out on the App Store to see how you will be using it within your LAN or across the Internet to your own private hosted cloud.

https://www.synology.com/en-global/knowledgebase/Mobile/help

Add RAM to the Synology NAS to max it out, use fast HHDs or SSDs (if you don’t mind spending) and call it a day. 
Title: Re: How do you keep your digital music files?
Post by: ren on April 21, 2020, 09:23:37 PM
I like my Synology NAS. While we on topic whats a good free CD ripper?
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 09:27:48 PM
I like my Synology NAS. While we on topic whats a good free CD ripper?

By CD, are you talking about music CDs, movie DVDs or BluRay discs?
Title: Re: How do you keep your digital music files?
Post by: Jl808 on April 21, 2020, 09:32:32 PM
ITunes for Music CDs.
Title: Re: How do you keep your digital music files?
Post by: ren on April 21, 2020, 09:33:06 PM
music
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 09:38:00 PM
music

I've tried many apps in the past, but most recently, I just used iTunes.  The default format for importing a CD is AAC, but I always changed the option to MP3.  Then I could burn a bunch of songs onto a single CD for playing on the go. 

Now I use an iPod in the truck, so no more homemade MP3 CDs.

Now that I'm looking for a new library management app, using iTunes for CD ripping might change for me. 
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 10:06:00 PM
The Synology has some nice built-in packages that you can install. Drive redundancies and the ability to backup to USB drives or to other network storage or cloud storage, make it reliable in my mind.

 https://www.synology.com/en-us/dsm/packages
 
In addition, there are some nice apps already made by Synology that you can check out on the App Store to see how you will be using it within your LAN or across the Internet to your own private hosted cloud.

https://www.synology.com/en-global/knowledgebase/Mobile/help

Add RAM to the Synology NAS to max it out, use fast HHDs or SSDs (if you don’t mind spending) and call it a day.

Back when I bought my NASes (??), RAID-5 was the "thing" to do.  ReadyNAS was one of the earliest to create a BTRFS type volume manage on the EXT3 Linux filesystems.

Now BTRFS is all the rage!  ReadyNAS called it X-RAID.  It took all the real work out of building a RAID volume.   Put in 1, 2, 3 or more drives, and the system decides whether it's RAID-0, 1, or 5.  If you want to increase the disk capacities later, just replace disk 1 with the larger disk, wait for it to be built, then replace 2, wait, then 3 ... etc.  Once the last disk is built, the volume automatically expands to fill the new physical disk capacity.  Pretty slick.

If I go with a new storage device, I'd opt for RAID 6.  With the capacities of NAS-specific drives what they are, and the cost I found ($160 for 6TB), I can see having 2 x 2 disk's capacity for storage and 2 x 1 disk's capacity for parity/spares.  That allows the volume to remain intact with 2 drives failing, versus just one failure for RAID 5.

My older NAS can handle 2TB disks x 4.  Newer one can use up to 6 TB x 4.  Right now I have 2TB disks in both.  The older NAS is an RSYNC mirror of the newer one.

BTW, Windows 10 kind of screwed over the older NAS.  There's a newer version of the SBM protocol now:  SMB2.  Windows 10 refuses to connect to any device using the "insecure" SMB1.  Guess what SMB version my old NAS runs?  And as old as it is, there's no updating it easily.  I could change the Windows registry to override the restriction, but I don't want to go that route.  I can still use AFS and CIFS on my MAC to see the shares, and of course sftp and ssh let me access the NAS as well.  One more reason the older NASes are becoming less useful.

I was toying with upgrading the disks in the new NAS to 6TB each, but then that messes up my backups.  If I put more than 5.5TB on it, the older NAS can't handle it.  Hence the plan to maybe build a new storage server.

As I was cleaning out 2 decades of backups  ??? I realized my files needed a better system of organizing, particularly my iTunes folders.  Eery time I replaced my iTunes repository drive for a larger disk, or to replace a dying one, I created a more recent backup of all that music.  I must have found 12 or more copies of iTunes repositories across my backup and NAS disks.  I had backups of backups!

I'm almost finished running folder comparisons and merges to make sure I'm left with one repository that contains all my songs.  That's when I need to have something create me a new library, preferably using an external database to clean up the album details and song titles.

When disks were less than a GB in capacity, and the costs were in the $300 range, deleing old unwanted files was easier.  Now, you can store the Library of Congress at your house multiple times, and good luck finding anything!
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 21, 2020, 10:08:42 PM
...
Title: Re: How do you keep your digital music files?
Post by: Jl808 on April 21, 2020, 10:26:35 PM
BTRFS is still the way to go with the Synology NAS as it provides the most flexibility. Synology calls it SHR (Synology Hybrid RAID). Some of the Synology packages (Active Backup) also require BTRFS and will not work on a RAID volume. Some DS models may also not have certain software packages so make sure to double check.

https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Storage/What_is_Synology_Hybrid_RAID_SHR

Attaching external USB drives as big or bigger than the NAS storage capacity is the way to go for backing up your music.  Store everything on the NAS and have a backup job to copy contents to an attached USB drive daily.  Cycling multiple USB drives gives you multiple backups easily with minimal brain damage as to figuring out what drive has what.

Unlike the ReadyNas, Synology seems to do a better job with keeping the software updated and doesn’t have an issue with Windows 10 requiring SMB2.

The DS Audio app has some playlist capabilities and can organize you your library by album, artists, composers, genre and folders, and playback to Airplay/DLNA/UPNP or Chromecast devices.
Title: Re: How do you keep your digital music files?
Post by: rklapp on April 21, 2020, 10:35:27 PM
Spotify <- mp3 <- CD <- cassette <- LP

Evolution of music...
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 22, 2020, 12:03:29 AM
BTRFS is still the way to go with the Synology NAS as it provides the most flexibility. Synology calls it SHR (Synology Hybrid RAID). Some of the Synology packages (Active Backup) also require BTRFS and will not work on a RAID volume. Some DS models may also not have certain software packages so make sure to double check.

https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Storage/What_is_Synology_Hybrid_RAID_SHR

Attaching external USB drives as big or bigger than the NAS storage capacity is the way to go for backing up your music.  Store everything on the NAS and have a backup job to copy contents to an attached USB drive daily.  Cycling multiple USB drives gives you multiple backups easily with minimal brain damage as to figuring out what drive has what.

Unlike the ReadyNas, Synology seems to do a better job with keeping the software updated and doesn’t have an issue with Windows 10 requiring SMB2.

The DS Audio app has some playlist capabilities and can organize you your library by album, artists, composers, genre and folders, and playback to Airplay/DLNA/UPNP or Chromecast devices.

Yeah, I had 4 external USB disks on my 2 NASes to perform nightly scheduled backups -- different shares backed up every other night incrementally to prevent the "Damn! I didn't mean to delete that folder!" data loss.

I bought a drive from Costco every time they had one on sale.  I had a 3TB, 4TB, 5TB and 6TB desktop backup-plus.  ALL 4 died or started having errors.  Eventually, I couldn't access any of them.  Luckily, Costco has a liberal return policy.  I got full refunds for all of them.  That was a great thing, since they were all beyond the warranties.   :thumbsup:

The 2 portable Seagates (4TB & 5TB) are having no problems.  Just the desktops did.  I took my last refund and bought an 8TB desktop model that was $50 off regular price.  So far, so good.  Using it to backup both NASes so I can wipe the old one completely (not with a rag, or something), and then set up the newer one to back up its files to the old one.  The movies have been backing up for a day now.

BTW, none of my NAS drives are SATA III 6GB/s drives.  They are all SATA II / ~5400rpm.  At the time, faster SATA III were very expensive by comparison, and the lower rpms didn't seem to matter if the bus speed was not 6GB/s.  That's another factor in my "itch" to upgrade.  My older NAS only supports up to SATA II, but the newer is SATA III.

Whatever I build/buy will become my streaming repository, and the newer NAS will be for its backup.  If I use both NASes as backup, I could start out with about 10+TB of total storage and still have room to back it up. Later, if the old NAS dies, I can bump the newer one up to ~18TB max capacity.

Know anyone who buys 5400RPM SATA II drives that are still working perfectly?   :wave:  I even have some matching spares I've been using in my workstation.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 22, 2020, 12:37:20 AM
I'm sure Synology is working fine with SMB2, as is my newer ReadyNAS.  it's my ReadyNAS NV+ on a SPARC CPU running Debian Sarge that's not keeping up.   :geekdanc:  This older NAS is my second NV+.  I bought it about 3 years before the first one died.  I used one NAS for streaming and the other for all my other files. Backups were on formerly NAS disks installed in external USB enclosures.

Jan 2015 -- Netgear ReaadyNAS RN314
Mar 2012 -- Netgear ReadyNAS NV+
Dec 2006 -- Infrant RaeadyNAS NV+ (worked for 8 years -- replaced with the RN314)

I don't even want to think about, much less list, the number of drives I've purchased.  I still have all the drives I upgraded from.

I guess 3 NASes over 14 years, and 2 of them still working, isn't too bad of a record!  I seem to be at that 5 year point again.  First one lasted 2006 - 2015 (do the math, man!  That's a long time!  :sleeping:).

Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 22, 2020, 12:39:43 AM
Spotify <- mp3 <- CD <- cassette <- LP

Evolution of music...

You missed 8 track ...

The first stereo I ever bought had an 8 track recorder.  Those things took up way too much room in the front passenger seat and floorboard!
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 25, 2020, 10:34:57 PM
Well, after much research and analysis, I came to a decision.......

this is going to be just like every other computer purchase!!   :wacko: :shake:  I'll research this to death for a month, then wind up buying what I wanted to start with.   :rofl:

So far, I'm leaning toward a NAS for the storage and a separate Home Theater PC (HTPC) to be the server.  It's just getting too difficult to justify $500 +/- $100 for a NAS that acts as a media server, web server, etc.  The standalone NAS devices, including Synology, use 3-5 year old technology for the CPU, RAM and motherboards -- at least those for home and small businesses do.  My current NASes have an Infrant Technologies branded SPARC CPU and an Intel ATOM.  Both are running Linux modified for their hardware. 

I think by building an HTPC server, I can leverage my existing hardware for storage and backups, reduce the load on each for non-storage tasks, and build the new HTPC with better specs than Synology offers at half the price.

I plan to upgrade the disks in my newer NAS, too.  Need to replace the 4 x 2TB 5400RPM SATA II 3g/s drives with at least 4TB 7200 RPM SATA III 6g/s drives.

I also found out Seagate and WD have recently admitted they've been marketing SMR-based drives as NAS-specific items.  SMR creates a write delay that can cause a timeout when a NAS attempts to rebuild a drive causing the rebuild to fail.  From my reading, SMR disks are not suited for NAS use at all. 

WD sent out a list of the SMR disks they offer at present, so NAS users can select CMR drives if desired.  Seagate hasn't offered that info yet, but probably will have to.

The real issue isn't the SMR itself, but that the drives are not sold with the SMR design disclosed before -- or after -- purchase.  Only when you can't get a NAS drive to finish rebuilding will you figure it out.

Intel seems to be the CPU for a good HTPC.  They have the Quick Sync Video technology that helps encode video to H.264/H.265.  AMD Ryzens can do the job, but they have to brute force the encoding at the normal CPU process level.  More cores can help, but by the time you get a good enough core count, you've spent as much on a Ryzen as you could have spent for a comparable Intel CPU.

I have a workstation with a very good i7-7700K CPU that would work.  So I also have the option of moving that over as a server and replace it with a new workstation.  This PC is 3 years old as of March, but after specing out parts, my CPU, motherboard and 1080 NVidia graphics card could be sold for more now than I paid in 2017!  I guess I bought the right stuff back then.   :thumbsup:


Not sure if you've seen XPENOLOGY, but I could use that to install Synology's DSM on my own hardware.  There are several other free NAS software programs to choose from.  FreeNAS seems promising.  The ZFR file system appears to have advantages over BTRFS.  ZFS was designed by Sun Microsystems, and FreeBSD is using it now.  FreeNAS is based on FreeBSD (go figure!).
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 26, 2020, 02:12:33 PM
I installed PLEX server on my ReadyNAS RN314 to see how it was for streaming video.  Very nice interface with lots of added info and artwork.

I wasn't positive it also streams music, so I just checked.  Looks like that is just as robust -- even plays over Amazon Echo devices.

So, back to the original reason for this thread:  I'm going to clean up my music files and folders in a format PLEX can easily use, then have it catalog and organize everything.  Should be a good alternative to iTunes for now.

As for video encoding, I usually rip discs down to a format that plays on a PS3, so re-encoding should not be an issue for the TVs.  Might have to do encoding if I decide to use a handheld device, though.

Thanks for getting me wrapped around the Synology axel, though.  I love a good system requirement analysis!   :geekdanc:  :thumbsup:  Tech cost, capabilities and compatibilities change all the time.  Have to stay current.  These research projects are always an opportunity to learn.
Title: Re: How do you keep your digital music files?
Post by: Jl808 on April 26, 2020, 09:40:49 PM
Thanks for sharing your research, Flapp!

Your last 2 posts have piqued my interest in building something, although it will likely have to take a backseat for when I have a bit more free time from work. No I haven’t heard of XPENOLOGY and will have to check it out including the other open source NAS options.  I’ve been pretty happy with the Synology and haven’t felt the need to look at other options, but it would be interesting to check our other options to see if they might be better. I’ll need to fire up a new VM on my server and check it out sometime.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 27, 2020, 03:14:30 AM
Thanks for sharing your research, Flapp!

Your last 2 posts have piqued my interest in building something, although it will likely have to take a backseat for when I have a bit more free time from work. No I haven’t heard of XPENOLOGY and will have to check it out including the other open source NAS options.  I’ve been pretty happy with the Synology and haven’t felt the need to look at other options, but it would be interesting to check our other options to see if they might be better. I’ll need to fire up a new VM on my server and check it out sometime.

One main thing I saw about the Xpenology DSM is that you can't do updates of the software from Synology, even if you already have a Synology NAS.  The machine has to be registered to allow updates.  So, any updates have to be performed the old fashioned way -- download and install over the old. 

The other thing thing I noticed is the drivers in the DSM installation don't always work with all non-Synology-based hardware.  Xpenology tried to help with that, but they don't have all NICs, motherboard chipsets, etc. tested with their drivers.  Synology is working with a closed hardware, so their software has a smaller target to hit.

I was thinking of using whatever new server I build as a VM server, too.  I really don't need something that fancy, as I just use the VMware Workstation and Fusion versions on my PC and MacBook.  I rarely fire up more than one VM at a time.

I'm in the process now of finally cleaning up all my iTunes files.  First step is to convert all the AAC/m4a files to MP3.  That's been running for 2 hours so far.  Next, copy, rename and organize in artist/album named folders all the MP3 files.  I'm using a free tool to bulk-rename files that need dashes added or artist names removed from the song titles.

I feel like the hoarder who's finally putting everything I have a use for in well labeled boxes and file folders.  :thumbsup:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 27, 2020, 03:22:36 AM
BTW, here are a few tidbits I've picked up from PLEX Server.

1.  The FireTV Android app for PLEX Client doesn't work on movies or TV shows that are split into multiple files ("stacked").  You can either make them all have non-stacked names and watch them individually, or append them to one another and make a single file.  I did the latter.

Stacked media files worked fine on my PC, Mac and ReadyNAS players.

2.  No need to group collections of movies into folders for easy binge watching.  Just use the Tags feature.  Select the movies in the group, and add them to a collection under Tags.  Then you can filter by changing the "ALL" filter button to "Collections."  Nice feature.

3.  There are separate articles on the PLEX site for movies, music, TV shows, and so on.  Each has a short tutorial on the best folder (directory) structure and file naming conventions to use for each type of media.  Finally!  I have a standard!!   :thumbsup: :rofl:  I've been struggling with that every time I tried to better organize my media files.  Never seemed to be as robust or useful as I'd hoped.

Lots to learn!   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on April 28, 2020, 05:26:20 PM
Well, I guess I'll be building a server.  Probably a good thing.  I watched a couple of movies on the ReadyNAS PLEX server yesterday, and about halfway into each movie it stuttered a bit.  Not too bad, but annoying.

Earlier I was wondering if I should get an AMD Ryzen for the core count or an Intel i5 for the Intel Quick Sync Video feature, which helps with transcoding to H.264/H.265.  Either choice would have cost around $300 -- just for the CPU.  Don't really want to get a less powerful CPU than my current i7-7700K workstation.

I checked Craig's List, and someone had an Intel CPU they'd just listed about 24 hours earlier.  I had looked at the i7-9700K on Newegg this week discounted for $397 + tax/shipping and decided against it based on price.

The i7-9700K I bought on CL was just $250.  I checked the Intel warranty page, and although there's no warranty available ("tray" CPU sold to OEM system builders), I at least verified it's genuine.  $150 less than a new i7-9700K and $50 less than the other options I considered.

This CPU is the next-to-top level CPU for the 300 series chipset motherboards -- the i9-9900K being the very top.  Performance-wise, they are almost identical until you need more than 8 cores.  It should be in demand for quite a while if I ever want to part with it.  My i7-7700K just turned 3 years old, and it's selling on ebay for $50 more than I paid new.  It's the best CPU available for its motherboard line and hard to find, so anyone looking to replace/upgrade their PC would be willing to pay a little more to avoid having to get an entirely new system. :thumbsup:

Now I need a motherboard and CPU water cooler. 

The board I am considering supports Intel HD video using this CPU, so no need for a discrete GPU card.  I have 32GB of 3200Ghz DDR4 DRAM I can split and use 1/2 in both PCs for now.  I also have 3 x 2.5" SSDs and 2 x NVME M.2 SSDs.  Plenty of parts to share.

Now if Newegg would just have the few parts I do need on sale this week!  My shopping cart comes to $300 right now. Add the case and CPU, and my cost is $650 -- about what I would have spent on a 4-5 Bay transcoding-capable NAS.  AND I can put some of the parts I have sitting in boxes to use.  Bonus!

Basically, the new CPU will become my workstation (with some RGB bling added  O0), and my old CPU will be "promoted" to server.   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 02, 2020, 12:44:58 AM
Friday night (opps!  Saturday morning!) update ...

Received my motherboard today.  Installed the Craig's List CPU and 1/2 my DDR4 RAM borrowed from my workstation, plugged in an HDMI cable to the TV, attached a brand new 1000W power supply (Warranty replacement for an 850W PS from Corsair that died), and added a spare keyboard and trackball. This was my bench test config -- with the MB sitting on it's cardboard box.

The "system" POSTed with no problems.  Very first order of business was to update the BIOS.  Only one version behind, so not bad out of the box.  I love being able to boot into a BIOS update utility instead of breaking out an old DOS boot disk/CD/Thumbdrive -- or after having to make a new one because I can't find the one I know is around here somewhere!   :wacko:

I turned on the CPU Game Boost feature on the MSI BIOS, and the CPU clock speed went from 3.6 ghz base speed to 4.7 ghz.  Memory is running at 3200 ghz, just like in the other machine -- overclocked using the Intel XMP setting (Extreme Memory Profile).

So, I guess the CPU works.   :thumbsup: :thumbsup:

My AIO Water Cooler for the CPU arrives Monday.  I also ordered a 32 GB RAM kit (2 x 16) today.  I can put all 4 sticks of my old RAM in my new workstation, and I'll have room to add another 32GB in the server if ever I need to.  A 64GB kit for that MB would have cost $400.  The 32GB I ordered was on sale today for $119.  This way, if I need more, it'll hopefully be cheaper in a year or so.  I was toying with buying faster clock speed RAM, but the cost/additional-performance just isn't there over 3200ghz RAM unless you're building a high-end AMD machine.  Those CPUs seem to need faster RAM to get the expected performance out of their CPUs.

Basically, I only need the cooler to get the build finished for both my new i7-9700K workstation and i7-7700K server.  Then I can see how well Plex does on a REAL computer.   8)  Right now, I have an old & partially corroded heat sink stuck to the CPU running 2 120mm fans.  Seems to still be good after some 13+ years.  Haven't used it for at least 8.  I figured 2 fans was prudent, since this heat sink wasn't really made for this CPU.

I considered running it on my workstation, but then I run the risk of messing up the server features when doing anything new, like upgrading hardware or install O/S updates and application software. 

Workstation  <->  Server <->  NAS  <->  Backup NAS

Looks like a plan!   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 02, 2020, 12:59:15 AM
BTW, I have one mystery to track down.  On my older MSI board, I was able to achieve more than the Intel-advertised Max Turbo Frequency of the i7-7700K by just turning on the Game Boost feature. 

i7-7700K
Base clock:  4.2 GHz
Max Trubo:  4.5 GHz
Game Boost Enabled:  4.8 GHz

However ..

i7-9700K
Base clock:  3.6 GHz
Max Trubo:  4.9 GHz
Game Boost Enabled:  4.7 GHz

I think the Game Boost should at least give the CPU setting for the Intel Max, or another 0.2 GHz.  I was expecting 5.0 off the bat, with MAYBE 4.9 as a minimum boost.

I know the new CPU is much faster than the old one.  But, there are those who see very stable settings running the 9700K at 5.0 GHz.

More fun!!   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Jl808 on May 02, 2020, 06:10:45 AM
Maybe you are having an apples-to-oranges comparison.

What other components are different besides the CPU?

Could the MB be making a difference?
Title: Re: How do you keep your digital music files?
Post by: jerry_03 on May 02, 2020, 09:35:27 AM
BTW, I have one mystery to track down.  On my older MSI board, I was able to achieve more than the Intel-advertised Max Turbo Frequency of the i7-7700K by just turning on the Game Boost feature. 

i7-7700K
Base clock:  4.2 GHz
Max Trubo:  4.5 GHz
Game Boost Enabled:  4.8 GHz

However ..

i7-9700K
Base clock:  3.6 GHz
Max Trubo:  4.9 GHz
Game Boost Enabled:  4.7 GHz

I think the Game Boost should at least give the CPU setting for the Intel Max, or another 0.2 GHz.  I was expecting 5.0 off the bat, with MAYBE 4.9 as a minimum boost.

I know the new CPU is much faster than the old one.  But, there are those who see very stable settings running the 9700K at 5.0 GHz.

More fun!!   :geekdanc:

What Operating system do you plan on running on your server?

I have a server too, but its a prebuilt Dell server with Xenon CPU, 64gb ECC RAM, etc.

Its a VM server, running VMWare's ESXi. If you just install ESXi the software is free to use but is neutered (doesnt allow vMotion, Host Clustering, etc, etc) vs the paid vSphere license.

My main VM that I run is Windows 2016 Server that hosts my Plex Server. I also run a couple of other VMs: pfSense firewall VM, Windows 10 VM that hosts my thin client image, Ubuntu VM, Kali Linux VM, and a bunch of other VMs that i run every now and then. I have all of these VM's cause i work in IT and use it as a lab to practice/learn new configurations and such. Plus im just a nerd and its fun to me.

I may actually build another server with an i7 or i9 CPU and dedicated GPU and put Windows 2019 Server on it as the hypervisor and run Hyper-V. Hyper-V allows you to passthru the GPU the VM's, so basically you can game on the VM. If I do go through with this build and QoS is good enough with the gaming, I may just get rid of my dedicated gaming PC and just run all my games off the VM hosted on the Hyper-V server, and have my thin client on my desk to connect to the VM.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 02, 2020, 11:24:23 AM
What Operating system do you plan on running on your server?

I'm going to start with Xen running with Ubuntu.  Seems to be a well supported and documented project, and it advertises as the only open source Type 1 (bare metal) hypervisor product out there.

Once I have that running, I'll definitely add a VM for Windows 10, a couple of Linux flavors (Ubuntu for sysadmin use, CentOS for dev/test),  and a FreeNAS VM of course (the whole reason for this "project").

The server's main purpose is to run PLEX Media Server.  It can run on Windows (but why?), Linux and even FreeNAS machines (FreeNAS is FreeBSD at its core). 

I'm trying the beef this machine up enough to run my main OS (hypervisor) and at least 2 VMs at once.  Loading PLEX and FreeNAS on one VM tackles two requirements in one instance. 

If I didn't want to run other OS flavors, too, I'd just load FreeNAS, with PLEX, and be done -- but I'm never done when it comes to computers.  Always something new to play around with.

I also have 2 licensed versions of VMware -- Workstation for the PC, and Fusion for the Mac.  I can use those machines for non-server type VM needs, which I do on the Mac daily.  I have a couple of Windows-only applications that I run in a VM on my older MacBook Air.  VMware had a very good promo for the holidays, so I finally upgraded both versions.  I'd been holding off because my kids no longer qualify for the student discounts  :(, and I didn't need to upgrade at full cost just to have the latest releases as long as all was working.  With Windows 10 and the latest Mac OS upgrade, I needed to upgrade VMware, too.  I was lucky I had a trial license to make sure the new Mac VMware fixed my issues.  The very week that trial ended, they had a sale.  That's how they get you!   >:D

I've never seen a need for ECC RAM.  Too expensive for current technology memory, and older tech isn't compatible with my hand-me-down PCs.  Most, if not all, of my MBs don't support ECC.  For home use, as long as I have lots of backups, a dropped bit isn't the end of the world.

This is all growing out of a new desire to simplify and take advantage of my media collection -- mostly music, but also movies.  So, I went down the PLEX rabbit hole.  My current NASes are not -- current.  Running streaming services on them wasn't a consideration 5 or 10 years ago.  I just wanted a NAS to store the files, and I would store transcoded movies to run on whatever streaming devices I had.  I used iTunes to support, well, Apple more than anything, but to use for iPods and access the iTunes store.  Not the case as much today.  I'll still have my media on the NAS, but the PLEX server will access the files, buffer it, transcode as needed, and provide real horsepower to make sure 2 or more people using it won't see any slow-downs.  I believe I can use RAM and SSDs to make buffering, transcoding and backup areas to increase performance and avoid the PLEX metadata being destroyed by HW failure or a brain fart.

I was getting an itch to upgrade my PC anyway.  Now I have a 9700K 9th Gen to replace my still very capable 7700K 7th Gen CPU, which is to become my new server once all the parts are segregated.  Just waiting on a Fractal Design Celsius S28+ water cooler.  I tried to stay away from the RGB fans -- too many additional wires, and the better fans are way more expensive than non-RGB.  I'll have enough pretty colored lights on my GPU, RAM, CPU Cooler pump, and motherboard.  :thumbsup:

Having said that, I have support on the new MSI board for both ARGB and RGB.  It's only a matter of time until I get suckered into this fad...   :shake:  My daughter bought me 2 LED strips powered by USB a few years ago.  Maybe that'll suffice for now.   :rofl:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 07, 2020, 08:33:32 PM
Well, I have 2 PCs up and running, so now I need to try migrating my workstation OS drive to the new CPU/MB.  It's that, or sitting for hours reinstalling everything .... again.  If Windows 10 plays nicely, it should load most, if not all, of the device drivers I need.  Otherwise, I should be able to come back and load just the drivers that show problems in Device Manager.  A clean install is always preferred, but I just went through that when my boot SSD died in Jan.  Don't like the idea of doing all that installation and setup again.  Although now I do have a working model (my old system) that I can use as a template this time if I go that route.

My old CPU/MB will be loaded with FreeNAS (FreeBSD) using the ZFS filesystem.  I just received 32GB of RAM which posted fine and runs at the same 3200 MHz as the other RAM I had.  This kit of 2 DIMMs leaves room for another 2, maxing out the MB at 64GB should I need it.

After inventorying my HDD collection, I see I have ZERO SATA III disks.  I do have a few that are 7200 RPM, so that's a plus.

I remembered I upgraded my MacBook Pro a few years ago to 16GB of RAM, and I still have the 8GB (2x4GB).  It so happens that RAM is the same form factor and speed as the 2GB single stick in my ReadyNAS RN314.  Since the max RAM for that thing is 4GB, I figured an upgrade was in order.  The memory test has been running for just over two hours without a single error.  I'll let it go for 3 hours as recommended and call it good.

While checking on my system's existing memory, I see that the Atom D2701 processor can't support ECC memory.  Seems odd that a NAS is not using ECC RAM, huh?

I've never cared about ECC RAM for home use.  I use incremental backups and snapshots on the RAIDs, so if I find a corrupted file, I can usually locate a copy, too.  I really can't think of a single instance when a file was corrupted by anything other than a power outage (before the affordable UPS came along) and/or a hard drive failure.I only buy top brand RAM with great reviews anyway.  Only had one set ever go bad, and that was the power supply's fault.  Before I figured it out, I'd lost a MB, 16GB RAM, a CPU AND the power supply.  I finally bought a PS tester and figured out who the culprit was.

There are some big shake-ups in the CPU market this week.  AMD now has a major advantage in the performance, workstation and gaming CPU segments now.  Their new Ryzen 3 3100 and 3300X 4 core/8 thread CPUs rival the 3-year old i7-7700K.  Intel's price new is $379 on Newegg.  The Ryzen will be selling for $100 and $120 respectively.  I think the best thing Intel has going for them still is better compatibility with operating system and driver software.  AMD isn't that far behind, though. 

Gotta love good ol' capitalist competition!  It drives innovation and keeps prices lower for the consumer.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 09, 2020, 01:18:16 AM
I opted to do a clean Win10 install on the new PC.  With the way MS has the system identity key embedded, I'm have to remove all the drivers from the old system I won't need, run sysprep to reset  the SID,  and move that media to the new PC. Then boot up the new PC and reinstall any drivers and hope I can reactivate Windows.

I figured it's easier and safer to bite the bullet and do a reinstall of everything ... again.  Downloading Office 365 now.

When the new workstation is completely setup, I need to install the OS for the PLEX server.  I've been planning to use FreeNAS on Ubuntu, but I've seen this YT channel where the guy is having great success using UnRAID.  After doing some research, I've decided FreeNAS (as in "costs nothing") not only suits my needs, but also has many features and capabilities that unraid lacks.

This is 2 years old, buit I found nothing to tell me it's changed significantly.

https://linuxhint.com/freenas_vs_unraid/

This paragraph alone from another article doesn't sound good to me relative to FreeNAS:

Quote
Traditional RAID systems use RAID 0, RAID 1, RAID 5, RAID 6 or some combination thereof. These RAID levels stripe data over
multiple disks and, for non-RAID 0 or RAID 1 systems, also distribute parity blocks across the member disks. unRAID is not like
these traditional RAID systems. The closest it can be compared to is RAID 4, a system in which data is striped across member
disks and parity is always written to a dedicated parity disk. In the case of unRAID, the data is never striped. A given file is written
to only one of the member disks. A dedicated parity disk enables recovery in case of a single disk failure. In addition, the disks can
be of different sizes, as long as the parity disks is the largest of the lot.

And this from the Wiki:

Quote
What are the disadvantages of unRAID compared to similar products?

No striping. So, although it performs well, better than many NAS solutions, it's generally slower than a RAID 0, RAID 5, RAID 6, or RAID 10, etc.

If performance is very important for an array you may be building, then one of those 4 RAID types will be a better choice than unRAID.

I do like the unraid "JBOD" design, in that you can use a collection of different size drives.  The parity data will be written to the drive with the largest capacity, but for the array itself, it uses all the space on all the other drives.  In a normal RAID array, the space available is limited to the size of the smallest drive if they are unequal.  So, 2x6TB and 2x8TB means you have basically 4x6TB drives less 1x6TB for parity in RAID 5.  Unraid would make one 8TB the parity drive and then allocated the additional 2TB on the remaining 8TB drive for storage, too.

Not sure that is worth the license fees.  unraid is including lots of other things like virtualization, too.  Seems like I'd be going down a stovepipe of only unraid-provided tools,  I much prefer integrating the "best of the best" when it comes to a Linux/Unix server.  Painting oneself into  a corner never seems to end in a happy place.   :rofl:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 18, 2020, 01:31:31 AM
The saga .... continues! 

Dun!  Dun!  Dunnnnnn!!

After much research, I've pretty much settled on unRAID.

The license isn't free, but you can start out cheaper and increase later (for an additional $9 upgrade "fee").  The license is based on how many disk devices are attached to the server.

Up to 6 devices:  $49
Upgrade from 6 devices to 12 for another $39
Upgrade from 6 devices  to unlimited for another $79

Up to 12 devices:  $89
Upgrade from 12 devices to unlimited for another $49

Unlimited devices:  $149

https://unraid.net/pricing

You can use the 30-day unlimited license to test your hardware, but then you need a paid license for the number of devices you'll make part of the array, part of the cache pool, unattached (non-array) storage, etc.

unRAID supports up to 30 devices in an array (28 data drives + 2 parity drives) plus another 24 devices to use as caching storage.  Additional devices can be added as "unassigned", meaning not part of a storage array, which can store anything, including VM files that probably don't need to be on the protected array anyway.  Performance matters more for VMs as long as you make a backup occasionally.

I guess the number one reason for choosing this over FreeNAS or the open source version of Synology's DSM is the way the filesystem manages the individual disks,  I've managed RAID arrays for decades, and they all have the same problem:  no matter how many spares you add to the array, there is always the chance of the array being lost if you have 2 drives fail. 

unRAID uses 1-2 parity drives (2 gives more protection). 

Let's say one data drive fails.  You replace it with a new disk, and the system starts rebuilding the new disk to replace the bad one.

Now let's say another drive fails.  The parity for the array is still being protected by 2 non-storage parity drives.  Not a problem.  Replace the bad drive #2, and it begins the rebuild along with bad disk #1.  And so on, and so on.  As long as one parity drive is present, the array can rebuild itself.

In a RAID 5 array, which has been the industry standard forever, that scenario would mean having to completely restore the array from the most recent backup (you do have a full backup, right?).  That's because the parity information is contained on the data storage drives.  If you lose a disk, there is sufficient parity information striped across the remaining drives to rebuild it onto a new disk.  But, once you lose disk #2, assuming #1 is still in the process of rebuilding, it's GAME OVER, MAN!

RAID 6, which nobody I ever worked with used or recommended, creates another disk to act as a failsafe against 2 bad drives.  That's okay for that array as long as 3 drives don't fail.  These things can, and do, happen close together.  My co-worker was demonstrating ON A LIVE PRODUCTION SERVER how he could remove a drive and not lose the array.  Unfortunately, he didn't plan on the offline spare in the RAID chassis being bad!  A little while after it started rebuilding, it went offline.  OOPS!

To make things worse, the good drive he pulled was still not part of the array, because he didn't know how to tell the RAID controller that drive bay was no longer "dead".  Each time he replaced the drive for a rebuild, it was marked "dead".  So, at that point, his spare disk and parity protection were offline.  He had zero data protection.  Had another drive failed, he'd have been up the proverbial creek.

The next day, he was finally able to contact the external RAID manufacturers, and they explained how the reset the status of a dead drive.  Super easy, but only if you know how.  There's no option in the GUI.  You change it, and it won't stick.  Instead, you have to change SOMETHING ELSE (like the setting for write-back operation) and save.  Then reverse the change to what it was and save again.  The first save makes the RAID re-poll the drives, and the status is magically set to ONLINE.  LOL!

He wound up spending the night in the facility doing a complete backup to tape -- twice, just in case -- and then reinitializing the RAID array and restoring the backup.  I was in San Diego and Vegas that whole time.  Poor guy.   :rofl:

With unRAID, every single storage disk in the array can be rebuilt, because there is no striping   And, if the parity is lost, any surviving storage drives can be mounted individually and the data extracted, unlike a striped array where the files are written across multiple volumes.

That whole concept of storing an entire file on a single disk makes too much sense.  You lose the performance gained from striping, but that can be compensated for by using SSD cache drives.

Right now, I'm going through all my old disks and running the unRAID "Pre-Clear" utility.  it stresses the disks and does full disk read-zero-read passes.  A 750GB drive on my PC took over 11 hours.  2TB drives took a day and a half.  All up to the speed of the PC, number of simultaneous disks being cleared, and the speed of the disk: SATA 2 or 3, hardware cache, and RPMs.  I'd doing 4 750GB disks right now. The tool gives more confidence that the drive is unlikely to fail soon.

unRAID boots from a USB flash drive.  It has to be that way, because the license is tied to the drive.

I've ordered an adapter card and a set of SAS-to-SATA connector cables.  The one card can support up to 8 SATA drives in HBA (Host Bus Adapter) mode as opposed to RAID mode,  All RAID functions will be software-based.  That, along with the 6 ports on the motherboard, puts me at 14 SATA devices.  Add the M.2 slots, and I'm at 16.  Looks like I'll be getting the unlimited license.  I can fit 8 drives in the 3.5" bays, 2 in the 5.25" bays, and 2 2.5" SSDs.  That's 12 storage devices that the case can handle plus the M.2 sticks.  Works out to where I have plenty of controller ports for the maximum number of SATA devices I can squeeze in my case.

At one point, I was ready to just install Linux and PLEX, then call it a day.  But, this unRAID product is interesting.  If it helps shore up my 12+ year old NAS storage as well as do PLEX media, VMs and anything else I throw at it, then this looks like a good long term way to go.

This is a lot more than I intended to type tonight, but I do like to share!   :thumbsup: :geekdanc:


Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 18, 2020, 01:50:05 AM
Almost forgot.  I read that only Intel network adapters should be trusted on unRAID and pretty much any other server.  But, unRAID had no problem with the Killer 2500 adapter on the motherboard,  Lots of people posted that was a "Windows-only" NIC.  I guess mine is different?

Anyway, I wanted to add some throughput for multiple client media streaming, so I ordered an add-on intel-based NIC.  I only needed one port, but 2 would be better.  Instead of 2 cards, I wanted one card w/2 ports.  But, I found a good deal on a new 4-port card instead,  incase one port dies, I'm covered!   :rofl:

Now I have 5 total NICs.  I intend to bond them as one IP address using a Round-Robin mode.  That'll allow the server to run at theoretically 5GB/s.  That assumes 5 files are being sent or received simultaneously.  I'll probably use 4 for the RAID and PLEX connections, and use the on-board NIC for a Windows or Linux VM.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 18, 2020, 02:12:56 AM
Man!  All of this so far is just leading up to having a system that I can start using. 

The parts are all hand-me-downs from my daily-use PC except for:

1. 4-port Network adapter
2. LSI 9211-8i SAS/SATA controller
3. Set of 2 cables for LSI controller
4. 500GB NVMe M.2 drive for unassigned storage (PLEX metadata, VMs, etc)
5. 32GB SDRAM (3200 MHz)
6.  unRAID licence

Not too bad.  I have a stack of 3.5" drives I removed each time I upgraded my NASes.  Hopefully this little project can put off the next NAS disk replacement until 6TB drives are selling for $40!
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 20, 2020, 10:34:57 PM
I know you all were dying for an update, so ...   :geekdanc:

Monday, I received my Host Bus Adapter (HBA SAS/SATA Controller) and the SAS-to-4XSATA cables.  Good timing between eBay and Amazon!

Hooked up 8 drives to the controller and started a Pre-Clear in unRAID on all 8 at in one process. 

One 500GB disk died right off the bat!  I think those are my only 7200RPM disks, so it's not surprising they are running 10+ degrees Celsius hotter than all the others.  Probably why they weren't in service all that long before I went with 750GB disks.  Of the 3 I tested, one failed early, and the other two are racking up some high SMART stats.  Only one is looking like it's likely to fail anytime soon, and the other looks fine.  However, they both are really slow considering they have the best platter speeds.  Hmm....  Might be good to set them aside and not waste time adding them into an array.

After several cycles, I can safely separate the disks into 3 groups:  bad, good but showing possible signs of potential problems, and Good w/99% confidence. 

Once I completed the tests on those disks, I updated the firmware on the controller.  It's an LSI 21XX-8i with a 2008 LSI controller chip.  That's probably the most common and well-liked non-Enterprise controller out there.  It's a 2008 design (hence the designator), but it's been getting FW and BIOS support up through 2016 by various companies that acquired LSI/Broadcom or by those who write drivers for servers like Dell, IBM, HP, etc.

Anyway, the card came with the IT mode already flashed onto it.  That's the non-RAID FW.  The RAID FW is the IR mode version. 

After researching, I erased the controller's P10 firmware completely, flashed the latest P20 onto it, and skipped flashing the UEFI and Legacy BIOSes.  Unless you're using the boot menu on the card to configure it, build RAID volumes, or boot from an OS device attached to it, leaving that part of the controller empty makes the boot process quicker -- like the controller isn't even there.  The OS driver only needs the firmware to work, so all's working fine.  The card was $38.88+tax on eBay, and a set of 2 8087 SAS-to-SATA cables was $11 on Amazon.  Controller card fits in any x8 or x16 PCI-E 2.0 or better expansion slot.

So, after all the Pre-Clear jobs finish, I hope to have at least 10TB for my array using 5 or more disks.

These Prep jobs take forever, but it could be worse.  I read it takes 3-4 days to Pre-Clear 10+ TB drives.

Now that I have 2 well-tested 750GB drives, I can use them to replace a 1TB surveillance video recording drive].  I noticed problems this weekend, and discovered one of 2 disks was dead.  I'll save my 2TB disks for the unRAID server.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 21, 2020, 06:39:44 PM
Just finished 3 passes of Pre-Clear on 4x320GB disks -- the original NAS disks I bought in 2006.    :shake:

The three Pre-Clear passes ran on all 4 disks simultaneously, and took just under 15 hours to finish without errors.   :thumbsup:

Only one drive is showing worrisome SMART stats.  I'm running a long self-test on all to see what that says.

I ran those in the NAS until 2008 when I upgraded to 500gb disks.  I then used the 320s in a Linux RAID config for a development machine.  After migrating to VMware, I no longer needed that, and the 320s have been stored -- I'm guessing about 10 years.

These disks are 7200 RPM, too.  Unfortunately they are SATA II - 3Gb/s, as are all my internal disks at the moment.  They run about 50-54 deg C, which is pushing todays thresholds, but I guess they were built back when components were less susceptible.

At some point, I'll upgrade my newer NAS to 4x8TB disks.  Then I can reuse its 4x2TB disks in unRAID.  My old NAS has been maxed out for awhile at 2TB disks.  8TB will max out the new NAS at 24TB useable space.

One day I'll ditch these old disks, but when they cost $100-$275+ each new, I'm reluctant to dispose of a working storage device. 

This year I finally got rid of a stack of EIDE and SCSI disks.  Some were dead, but most just had old bus connections, slow speeds and low capacities that made them next to useless.  Some of them had capacities less than a $10 USB flash today!  So, I'm getting better about letting go.   :wacko:

My 2 new USB flash drives arrived today.  I already made them unRAID boot devices.  I figured when I finally license unRAID, I need to use a new vs years-old USB stick, since you can only replace a USB boot stick once every 12 months without jumping through hoops.

Next step -- decide on the drives to use.  Then I can request a 30-day temp license to build an array and really start playing with unRAID.



Title: Re: How do you keep your digital music files?
Post by: robtmc on May 22, 2020, 04:30:58 PM
Mine are still on the CD they came on. 

How else would I play them in the stereo system?
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 22, 2020, 05:01:43 PM
Mine are still on the CD they came on. 

How else would I play them in the stereo system?

CDs don't last forever.  Most music CDs are lucky to be playable after 10 years if kept away from warm temps and sunlight.

I have vinyl that's almost 50 years old, and the only real problem is with some sleeves disintegrating, but very few with that.

There's a convenience factor with having your CD collection copied onto a media server.  You can then enjoy YOUR music on YOUR devices, such as your iPod and original iPhone.   :shaka:

That doesn't even bring in the potential for scratches, other damages or theft/loss preventing you from enjoying your CDs. 
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 23, 2020, 01:34:19 AM
Now that I've tested and cleared the 4 x 320GB drives, I'm going to be listing them on Craig's List.

Just seems counterproductive to take up 4 drive bays with barely 1 TB of usable space: 4*320 = 1,280 GB less whatever overhead.  Probably right at 1.1TB.  For $100, I can get a 4TB WD Red NAS drive and still have 3 bays and 3 sets of cables for other drives.

One YouTuber that does a ton of unRAID videos (Byte My Bits) uses "shucked" drives.  He buys the WD external USB drives when they go on sale for less than the same capacity internal drives cost new,   He then shucks them by removing the external case and, if needed, covering up a power pin that prevents those drives from working on a direct SATA connector.  The drives are almost all WD "White Label" disks with the same specs as Red Label NAS disks.

Prices are dropping on the lower capacity NAS disks (4TB or less).  I'm thinking of trying the shucking process if I see a nice sale on 8TB drives for my NAS device.  Then I can use the 2TB drives presently in that for the new unRAID box.   :thumbsup:

Sometimes I wish all these old drives would just crash at once, and I could refresh them with faster, larger capacity types.  I can't remember the last time I had a mechanical drive suddenly die.  Usually they present problems that worsen over a short/long time.  My Samsung 850 EVO 500GB SSD is another story.  One day, possibly right after a power outage that lasted a few minutes, it just stopped responding.  Samsung provided a warranty replacement, but still ...
Title: Re: How do you keep your digital music files?
Post by: robtmc on May 23, 2020, 08:50:28 AM
n YOUR devices, such as your iPod and original iPhone. 
I cannot use Apple products...................
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 23, 2020, 11:02:48 AM
I cannot use Apple products...................

They have a restraining order against you?   :rofl:
Title: Re: How do you keep your digital music files?
Post by: Inspector on May 26, 2020, 04:34:46 AM
I gave up saving all my digital music. Now I just pay $7.95/month to stream from Amazon. Their music library is large and they have most of what I listen to.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on May 26, 2020, 12:15:47 PM
I gave up saving all my digital music. Now I just pay $7.95/month to stream from Amazon. Their music library is large and they have most of what I listen to.

That's definitely an option, but I see three main reasons to have your own music library:

1.  If the PC police decide the music you listen to is hateful, or the artist is politically "dangerous", Amazon could bend the knee to the cancel culture morons and remove that content;

2. Most of us who spent the time and money collecting music that has meaning for us, we like listening to that music most often.  It takes us back to the times of our lives that the songs were associated with.  Why pay monthly for thousands of songs when 95+% of the time you only listen to your favorites?

3. You may own special edition releases or off-label recordings that are not available on any streaming services.

Nothing wrong with the streaming service if you just want a never-ending music library with great variety.  I have Spotify for free that was bundled with my AT&T plan.

I've tried the Amazon free/Prime and  the monthly fee subscriptions.  They let me upload some of my own music files for easy access through my Alexa devices, phone, FireTV, etc.  Best of both worlds, but the upload feature requires your music be in a format you can upload and use.  Back to having to store and organize your digital music files.   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 01, 2020, 12:40:58 PM
I'm baaaaack!!   :geekdanc:

After WEEKS of testing old hard drives, I finally have my unRAID server running under a temporary license.

Setting up the array was SUPER EASY.  The hard part was reading the recommendations on hardware, which unRAID options and add-ons  I wanted and the REALLY LONG process of preclearing the disks.

After playing musical chairs drives with my existing computers, I now have 9 storage devices (not counting the boot flash drive), which keeps me under the 12 drive limit for the 2nd tier license.  I still have some expansion room, too.

Last night I requested a 30-day trial license, started up my array, and it was off to the races. 

I configured:

a) 2 x 2TB drives for parity (better than 1 -- for redundancy);

b) 4 x 2TB drives for data storage;

c) 2 x 500GB SATA SSDs for caching; 

d) 1 x 500GB NVMe SSD "Unassigned", for Virtual machine storage, Plex metadata and caching, etc.

As a 2-drive SSD cache "pool", it provides 500GB of cache and a mirrored SSD for redundancy, so I don't lose anything stored there before it's written to the array.  There is a job scheduled nightly to move cached data to the array.  This is one way to compensate for the lower performance of the unRAID array design vs. a striped disk array.

It took 5+ hours for the cache and data drives to be formatted and for the 2 parity drives to synchronize.  Everything reports "healthy", so that's nice to see.

Now I need to configure a new login, change the default passwords, add my Media shares, and so on.

Once I have everything setup properly, and the trial expires in 30 days (plus 2 additional 15 day extensions if they'll let me), I'll have to pay for the license.  I like being able to test something for 2-3 months before committing financially.

My array provides 4 disks-worth of storage, so that's 8TB plus 4TB of parity "backup".   I could opt for 2TB parity and 10TB of data, but I prefer more redundancy should the parity drive die.  If the only parity drive dies, it can be rebuilt, but if another (data) drive dies, the array is lost.  I can recover most of the data from the individual drives, but the array can't be mounted.

My least-used drive only has 2 months of activity on it, and the oldest has over 7 years.  I'm using the next-to-newest and next-to-oldest drives for parity.  They happen to be the same model and have 5900RPM speeds -- better than the 5400RPM the rest have.

If you decide to use unRAID, and you plan to use spare drives of various sizes, I recommend not getting a license -- even a trial one -- until you've 100% tested and decided on the drives to use.  That can all be done without starting the array.  I tested 17 drives just to see if they were all still good.  3 had to be "retired".  3 smaller ones replaced 2 x 1TB surveillance recording drives, which I then used to replace 2 x 2TB drives in my main PC.  I couldn't justify having many TBs in the 2 workstations while running 8 smaller drives in the storage array.  I hope to use the RAIDs more now for direct storage rather than my local drive, both for protection and network accessibility.

If, however, you plan to use all new drives, then the process of deciding on which drives to use and preclearing them is more streamlined and should only take a few days at most, depending on the size of the drives purchased.

Stay tuned.  After I  get the "housekeeping" taken care of, it'll be time to finally install Plex...the #1 reason for this project.   :thumbsup:

(https://i.imgur.com/FtPYSbG.png?1)
Title: Re: How do you keep your digital music files?
Post by: Inspector on June 01, 2020, 01:12:11 PM
 Cool project. The last big project I worked on was fiber channel adapters for servers to arrays. I worked on so many different arrays we had to have multiple people get certified as admins for all the different arrays. The nice thing about all this was when a company retired their arrays and JBODs we got to take them home for personal use. I had a ton of stuff running 24/7 in my home office. All RAID 5. A little overkill for home use. 😂
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 01, 2020, 02:50:01 PM
Cool project. The last big project I worked on was fiber channel adapters for servers to arrays. I worked on so many different arrays we had to have multiple people get certified as admins for all the different arrays. The nice thing about all this was when a company retired their arrays and JBODs we got to take them home for personal use. I had a ton of stuff running 24/7 in my home office. All RAID 5. A little overkill for home use. 😂

I've had to care for and develop database backup plans for several RAID enclosures, including NetApp devices that were replicating on their own between SCIFs.

The servers users Fiber Channel to direct connect to the storage array, but the LAN connection was limited to 1GB speeds.  10GB paths were established for replication, though.

Most of the drives were SAS, connected to backplanes that were attached to a RAID controller in the Dell servers.

At my last job, the PMO sent us SATA drives that were later replaced with SAS as they originally intended to ship.  In an enterprise environment, storage speed is a major issue.

At home, until you start creating your own videos or storing and streaming movies, a mid-range NAS appliance is fine for most. 

This project highlights one of my, and I'm sure some others', problems of having older, lower-capacity drives with no real use for them.  A RAID is a great option for that, assuming the drives are not too limited in capacity.  At some point, buying a new multi-terabyte drive is a better solution than trying to gang together 8 drives to achieve the same capacity!   :rofl:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 02, 2020, 01:29:24 AM
Just leaned a cool/annoying lesson about unRAID tonight/this morning.

I wanted to try booting from an external USB-C SSD with Windows 10, just so I know I can for maintenance on that machine.  Sometimes Windows is the only OS for certain apps, like checking bad drives needing warranty work, and similar utilities.

So, the boot worked great.  Then, when I rebooted into unRAID, I just opened the MSI UEFI/BIOS, and dragged the flash to infront of the external SSD and booted.

It booted, but then I started having all kinds of issues with the array.  Cache disks were confused, the  parity drive wanted a resync, and who knows what else.  I stopped the array and checked the lsblk command.

The external SSD apparently didn't feel like sitting in the back of the buss (PC pun).  He took over as device sdda, which was the boot flash before.  That put every other device a drive letter higher.  When the array tried to start, it saw one cache drive, not 2 (one was the boot flash).

Then it went to the  next in line, the other actual cache drive, and thought it was disk 1 in my array -- basically the first parity drive.  It was a bad domino effect from there.

So, I stopped the array, rebooted with the external disk removed, and started the array again.  This time, the parity drives were happy, but the first cache drive was still showing no filesystem.  So, I reformatted that SSD, and the cache pool was happy again, too.

Lesson:  don't add devices to the PC with "Auto Start" enabled for the array.  If you add a device, disable the array from auto starting, reboot with the new drive, and verify nothing unexpected was changed -- like drive positions in the OS.

I also scheduled a daily parity check at 05:00, right after the scheduled job that moves the cached files to the array disks, just to make sure everything is being stored accurately.  It's a decent alternative to having ECC memory from what I've read. 

 :shaka:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 14, 2020, 12:33:26 PM
Update ...

Here's what my disk status info on the web GUI Desktop for unRAID looks like:

(https://i.imgur.com/J9FSxMN.png)

I now have the array and parity drives spinning down after 15 minutes.  Earlier this week, only disks 2, 3, and 4 were spinning down when idle.  That told me I changed something that was accessing the 2 parity disks plus Disk 1.

After doing research, I found that even though I configured everything I could to reside on the cache or unassigned disks, there are still some folders that are stored on the array no matter what.

I'd installed a new docker (app) called DuckDNS, a dynamic DNS application to automatically update my home's RoadRunner IP address on the DuckDNS,org site (free DNS account).   This docker on unRAID attempts an update every 5 minutes, which seems to keep resetting my 15 minute spin down clock for the first disk + 2 parity disks.

So, I found a good set of instructions for moving 2 related folders:  docker (which has the image docker.img) and libvert (the docker image libraries).  It's not as simple as moving the folders to the unassigned disk.  Services have to be stopped, config files have to be updated, etc.

Once that was all done, I watched the clock, and 15 minutes later, all 6 drives, plus the 2 cache SSDs, switched from "active" to "standby".  YAY!

This is a feature of unRAID I wanted to make sure worked, as it reduces power usage as well as prolongs the lives of the disk drives.  It can also slow response times as you have to pause while the drives spin up, but once what you need is cached, the drives can spin down without having to pause over and over again.  If everything you are doing can get done within the 15 minutes after spin up, you'll also have no need to pause more than once.

15 days left on my trial key -- plus another month if I extend it twice.  So far, so good, and I'm out no money for a license yet.  Plenty of time to make sure this is something I can use and that works before buying.   :thumbsup:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 17, 2020, 03:51:31 PM
A couple of things to consider if you're  thinking about trying unRAID:

1.  By default, there is no SSL/HTTPS protocol (port 443) installed for the basic unRAID server, nor for the add-ons you install (with a couple of obvious exceptions).

The server is not intended to be a production business data repository with both internal and external access and associated security features.  Having said that, you can install "Let's Encrrypt" with free SSL certificates registered to your domain (did I mention free SSL certs?  :thumbsup: ), Reverse Proxy to route traffic coming in from the Internet to your specified servers & ports, thereby isolating the risk to a docker, VM or machine, and total hardware encryption for your storage drives to prevent physical theft and misuse of your data.

unRAID is designed and advertised to be a media storage solution, not a repository and fail-safe solution for your important and irreplaceable documents.  However, there's nothing that says if you keep your internal network secure and the port forwarding to the unRAID server to a bare minimum, that you can't make it secure.  In fact, you can install the unRAID docker container for OpenVNC and OpenSSL to tunnel into the server and encrypt your traffic, thereby creating a wrapper that mitigates the lack of built-in security for watching movies.

2.  I added the Let's Encrypt docker to setup secure connections to some other dockers, like NextCloud, and it broke my network connection to the PLEX media player.  Part of the Let's Encrypt template asks for the network connection type to use, and I'd created a separate custom connection that maps ports from the router to Let's Encrypt.  Originally I was using the BRIDGE network type for both PLEX and unRAID.  I had to change that setting to HOST.   

Were I not using multiple network NICs, that likely would not have been an issue, since there would not have been a BRIDGE option to begin with -- and HOST would have been the default.

Anyway, it's working again.

3.  Now that I have several docker containers installed, I am seeing a pattern.  Each docker that installs information in the appdata folder needs the default setting changed to add the "slave" option to the mount command..  If you left the appdata folder on the array as it is by default, this is not an issue.  I moved my appdata to an unassigned SSD to a) make the docker apps faster, and b) prevent the dockers from stopping the array disks from spinning down after the selected idle time.

4.  If you use the default settings for how balanced the array drives are filled with data to "High Water Mark, you'll see most of the initial data you copy, if not all of it, on Disk 1 exclusively (plus the parity drive/s).  Basically, the server takes the largest disk capacity in the array and divides by 2.  Once I use that much of Disk 1, it'll start saving to Disk 2, and so on.  That means that for now, I rarely see Disks 2, 3 and 4 spin up, and they are all pretty much  empty.  That helps prolong the disks' life, and it can save electricity cost.  Performance isn't much of a difference, since I cache pretty much everything to SSD, and it takes close to the same amount of delay to spin up one disk as it does four.  If you prefer, you can select other storage schemes that create a more balanced approach to where data is stored.  Or, if storing large video and photo files, you can set unRAID to store each sector of a file on the same disk to prevent having the file scattered across multiple disks.  If you ever need to restore your system from a crash, it's good to have the files self-contained on a single disk in case one disk can't be recovered.  That way you only have to rebuild the one disk from parity to see its data, and the rest of your files are immediately accessible via the working disks.   :thumbsup:

That's all for now.  Later!   :wave:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 19, 2020, 01:38:59 AM
Tip of the night:  If you want to learn how to use unRAID to do just about anything, the YouTuber "Spaceinvader One" has taken it upon himself to put out a bunch of videos over the years.

He moves rather quickly, so sometimes I have to press PAUSE every other sentence, or just use the YT player settings & slow him down to less than 100% playback speed.   :thumbsup:

This is the first video I'm using for securing my server for outside access:

https://youtu.be/I0lhZc25Sro

This is the second one:

https://youtu.be/AS0HydTEuA4

I just connected to my server using a registered domain, a CNAME record for a subdomain, and a registered SSL certificate from Let's Encrypt (FREE!).  I also am using the Dynamic DNS service DuckDNS.org to bounce from the registered domain name to the RoadRunner IP, but I could have just as easily created an A record on my name server pointing to that IP address directly.  DuckDNS just keeps that IP updated in the event that it changes.

Next, I'll be diving into video #2 above to setup a wildcard SSL certificate (such as *.2ahawaii.com").  That makes the same certificate valid for any subdomain on your site as long as it's part of the same domain.  Most wildcard SSL certs allow unlimited subdomains.

I've done that at work several times.  Much less work (and expense) if you need an SSL certificate for more than a couple of servers, or if you do like me and create a bunch of subdomains that a single web host is configured to serve up a specific virtual web site based on the subdomain names. 

For example: 

(1) www.domain.com goes to the regular port 80/HTTP and 443/HTTPS web server, and that is located in the directory /var/httpd/www/.
(2) nextcloud.domain.com goes to the same ports, but the web server directs the traffic to /var/httpd/nextcloud/ and rewrites the URL (uri) you see in the browser address as "nextcloud.domain.com".
(3) photos.domain.com goes to the same ports, but the web server directs the traffic to /var/httpd/pics and rewrites the URL you see as "photos.domain.com".

This allows multiple web sites to exist on one physical server using the standard HTTP ports, while recognizing that the URL is asking for a specific site apart from the others.

The rewrite command for web servers is powerful.  I use it to force the incoming connection to only use HTTPS (SSL encrypted), to redirect links to appropriate locations, like if I rename a folder but don't want broken links on pages or favorites lists, and many other uses.

The most common rewrite is to remove the "www" from your domain in the address so it only really uses the domain named site as the default. 

If anyone tries to use the IP address instead of a DNS-registered subdomain/domain name, it'll be rewritten to a default URL I set, which might include a page of links to the available sites so you can get to where you really needed to go.

One thing I still need to do is set up a cron (scheduled) job to renew my Let's Encrypt SSL certs.  Apparently, the free certs expire between 2-3 months, but you can renew them with a script.   :thumbsup:  No problema.

Nevermind.  Renewing a Let's Encrypt certificate on unRAID is super easy ... barely and inconvenience!  Just restart the Let's Encrypt docker, and the certificates are automatically renewed for another couple of months.  As long as the unRAID server is rebooted, or the Let's Encrypt docker service is restarted beforehand, it never expires.  If it ever does happen, just restart the docker, and it's fixed.

Too easy.   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 19, 2020, 04:28:58 PM
At first I was confused as to why I needed to re-home my domain DNS configuration away from my domain registrar's name servers and onto the CloudFlare servers.  I performed those steps to make sure I was following the video instructions, but now I'm analyzing the purpose for those steps for my own understanding.

Let's Encrypt takes advantage of an API call to request, install and renew your SSL certificates.  Those steps can get confusing, especially if you're bouncing between Linux, Windows, unRAID and Mac OSX servers having to install the certificates.  For example, If you send the request files to the SSL service in UNIX format, you have to convert the certificate files to install on a Windows OS.  That of course only applies to wildcard certs, as otherwise you would generate separate requests for each OS and not have to convert.  For wildcard SSL files, you have to copy them to each physical server that's part of the parent domain.  Still, it's a tedious, repetitive process that can introduce errors when changing web server and SSL configuration files.  The unRAID docker for Let's Encrypt does all of that for you as long as your domain is using API code and keys which both sides understand.

So, this is just one more step in the process that required a little additional education on my part .  Luckily I had it working before I asked the question.   :thumbsup:
Title: Re: How do you keep your digital music files?
Post by: Jl808 on June 19, 2020, 04:59:48 PM
This is a lot of work. Thanks for sharing your research, FJ.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 19, 2020, 05:00:57 PM
This is a lot of work. Thanks for sharing your research, FJ.

My pleasure.  It helps me also by keeping my progress and problems documented.   :geekdanc:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 19, 2020, 08:06:11 PM
Well, THAT was easy!   :rofl:

I went through the steps in the video to configure a wildcard SSL certificate.  Basically:

1.  Log into my Cloudflare account and copy the API key.
2.  Edit the cloudflare.ini file for Let's Encrypt with my email address and that API key string.
3.  Edit 3 lines in the Let's Encrypt template settings: 
     a.  Replace subdomain list with the word wildcard,
     b. replace Validation method "http" with "dns", &
     c. add DNS-Plugin setting with "cloudflare".
4.  Restart the Let's Encrypt docker.

Now when I access my NextCloud server with my subdomain name,  my certificate says the SSL name is valid for *.<domain-name>,com instead of nextcloud.<domain-name>.com.

Why would I want to do this if I can get a bunch of specifically-named subdomain certs?

One reason might be that there are limits to this free service.  One is, you can only create certs for the same subdomain names 5 times per week.  If you keep creating duplicate-named certs, they lock out further creations.

So, if you're experimenting with the SSL setup, they have a sandbox (experimental environment) you can point to for that, then set it up for production when ready.  Unfortunately, sometimes it takes experimenting on the live setup to troubleshoot.  By then, you might have hit a limit.

https://letsencrypt.org/docs/rate-limits/

Another reason might be because when following tutorials, you tend to use the same labels and naming conventions in the tutorial.  Afterward, you might want to do some renaming or completely back out that setup to try another tutorial you found. Having wildcards means the SSL certs work no matter what subdomain names you configure, so the certs don't have to be recreated time and again.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 20, 2020, 12:28:39 AM
From my limited use so far, I think CloudFlare is a very good service.  I'm hoping it'll remain free for at least the basic features.

The service is basically a replacement for your edge router -- the one sitting between your internal network and the Internet.  External traffic goes to CloudFlare, and it sends the data to your router after it's done its magic.

I set up a Page Rule that redirects all my <domain-name>.com/* page requests to HTTPS/SSL on port 443.  So I could effectively shut off port 80 mapping on my home router now, removing a security risk on my home network.   :thumbsup:

CloudFlare also has a lot of website analytics and stats pages, one of which runs a series of tests to compare how much quicker your website runs with CloudFlare than without.

Of course, you have to wonder how reliably a site that's advertising faster load times would report on load time comparisons.   >:D
Title: Re: How do you keep your digital music files?
Post by: groveler on June 21, 2020, 11:39:37 AM
From my limited use so far, I think CloudFlare is a very good service.  I'm hoping it'll remain free for at least the basic features.

The service is basically a replacement for your edge router -- the one sitting between your internal network and the Internet.  External traffic goes to CloudFlare, and it sends the data to your router after it's done its magic.

I set up a Page Rule that redirects all my <domain-name>.com/* page requests to HTTPS/SSL on port 443.  So I could effectively shut off port 80 mapping on my home router now, removing a security risk on my home network.   :thumbsup:

CloudFlare also has a lot of website analytics and stats pages, one of which runs a series of tests to compare how much quicker your website runs with CloudFlare than without.

Of course, you have to wonder how reliably a site that's advertising faster load times would report on load time comparisons.   >:D
You do know there only two or three of us on this thread that
understand your Jargon?

Title: Re: How do you keep your digital music files?
Post by: ren on June 21, 2020, 12:53:58 PM
You do know there only two or three of us on this thread that
understand your Jargon?

the rest must be oppressed
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 21, 2020, 08:38:57 PM
You do know there only two or three of us on this thread that
understand your Jargon?

I was wondering why I was enjoying this thread so much!

 :geekdanc:   :thumbsup:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 21, 2020, 08:39:37 PM
the rest must be oppressed

You're tempting the devil.   :popcorn:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 21, 2020, 08:56:33 PM
Well, yesterday I installed Ubuntu LTS 16.04 server on an unRAID VM.  Had to enable one of two CPU features in my BIOS (HyperThreading was on, but Vx-D was off).  Other than preparing by watching a video on tips and tricks for unRAID VMs, which mainly involved choosing which CPU physical and virtual cores to select, and a blog on what other options to select, I had it installed and working in 10 minutes not counting the 30 minutes to download the ISO image.

Today, I set up my remaining domain DNS records after moving all four of them to CloudFlare.

I found out the Let's Encrypt docker really doesn't work well with multiple domain names.  Even though there is a varialbe for EXTRA_DOMAINS that adds them to the SSL certificate, it doesn't quite set up the proxy to work with all of them.

So, I left the original domain I had working setup in Let's Encrypt, and then created CNAME entries to alias the others as that domain.  That way, the redirects are pointing to the one domain that Let's Encrypt is configured to properly route.

It took a few tries to get all the settings to mesh together, not to mention the TTL (time to live) delays that some settings required to expire before propagating.  In the end, I have 5 domains.  Four point to one as their redirect alias, and the alias points to another alias on DuckDNS,org.  The IP is resolved there, and the certs seem happy to accept any URL associated with it.

Part of the Linux installation includes a LAMP stack (Linux Apache MySQL PHP), so I can do web development.  Next step is to create my web repositories and figure out the best way to forward browser requests to them.  Right now, all of them point to the nginx web server as part of Let's Encrypt.  A simple ReWrite command to each domain on my Linux VM ought to work, but I won't be sure until I get it working.  The reverse proxy thing is rather new for me, so I can't be 100% sure of how it'll affect the redirects all on one physical server and 2 operating systems.
Title: Re: How do you keep your digital music files?
Post by: Inspector on June 22, 2020, 01:21:50 AM
You do know there only two or three of us on this thread that
understand your Jargon?
Unfortunately, I have been so intrigued I have read every word of every post.  :shake: :shake: :shake:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 22, 2020, 01:54:27 AM
Unfortunately, I have been so intrigued I have read every word of every post.  :shake: :shake: :shake:

Well, I do have a way with words.   :shaka:
Title: Re: How do you keep your digital music files?
Post by: Inspector on June 22, 2020, 03:33:49 AM
Well, I do have a way with words.   :shaka:
I’m retired now. I don’t want to understand this stuff any more.  :P
Title: Re: How do you keep your digital music files?
Post by: groveler on June 22, 2020, 03:44:00 PM
Well, I do have a way with words.   :shaka:
I think i read you upgraded to Ubuntu 16.04 in a past comment.
I'm getting ready to go to 20.04.
I've got Win 7, 10, Mint, Ubuntu 14.04. 16.04, and a 10.04 around somewhere.
My problem is trying to remember all the idiocrancies of each system
and each computer.
You'd hate me but I recycled a Win 3 computer today.

Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 23, 2020, 12:38:28 AM
One more giant leap for web-kind!   :geekdanc:

Every time I take a long break from using or installing a major application or OS like Apache Web Server or Ubuntu Linux, it becomes a learning process, time that often exceeds the time saved by all the upgrades in the new versions.

I have to say, the Apache upgrades are mind-blowingly awesome!  If you've ever had to struggle with the Apache official documentation, you might agree that it's written in a fashion meant to keep IT professionals and consultants employed!

Tonight's task was to get Apache working on my VPS (hosted on the service A2Hosting.com).  Part of that process included adding the free SSL certificate for that subdomain -- just another "A" record in DNS with the subdomain name and the fixed IP address of my VPS.

After walking though a nicely written tutorial  on the BlueOcean website, I was shocked that the non-SSL virtual host was accessible on the first try.  That never happens! 

https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-16-04

The main difference from this version of Apache and the last one I configured is how compartmentalized the config files are.  I added the changes to one sites-available file, installed and ran the executable "a2ensite" to apply that to the Apache2 configuration, and POOF!  Caveman easy!

Then, I checked for an Ubuntu-compatible version of "certbot" -- the Linux client that does what "Let's Encrypt" on the unRAID docker does:  requesting, installing and renewing SSL certs.

Installed "Certbot", created a directory under root's home directory, added a cloudflare.ini file with the exact same info as the same file on unRAID at my house, and created a script to call the app with a few basic arguments.

I wound up having to run "certbot" twice.  The first time, it created the certificate for my subdomain virtual host, and the keyring for my public key.  Then I ran it with the "--apache" switch but left off the parameters telling where the files are to be stored.  That prompted me for my email address and whether I want to force all traffic coming in on HTTP to HTTPS (I chose yes).  This second run with the --apache switch configured my server for SSL automagically and restarted the server for me.

My VPS subdomain site is now protected by the CloudFlare proxy and secured by an SSL certificate that is properly signed.

These project developers are making great strides FINALLY at automating these redundant and mundane multi-step configuration tasks.  I have to say I'm impressed at how this is no longer a giant headache to figure out and get working.

I believe if I didn't already have years of experience having to do it all by hand while deciphering less-than-detailed instructions, tonight would have gone even faster!  I was waiting for the hammer to fall and require an hour of troubleshooting to make it work which never happened.

Now I have a few things to choose from to do next.  A few dockers for unRAID have my attention.  RADARR is a movie library management application -- sort of why I started down this road.  Then there's SONARR, almost the same function as RADARR, but supports fewer file formats.  There's a VPN docker to let you tunnel into your server, which mitigates the entire "PLEX can't do HTTPS" issue. 

I also want to install WordPress and Joomla! Content Management Systems (CMS) on my home-based Ubuntu server, which will be uploaded to my VPS as I get things ready for prime time.  I'm a huge fan of Joomla!, but I'd like to see if WordPress can't provide most of what I normally require on a website without all the additional plugins and components Joomla uses.

BTW, I left one of my domain names on my registrar's site yesterday, so I re-homed it onto CloudFlare's name servers today.  That makes 6 domains and a half-dozen subdomains working now.  Technically, it's 4 distinct domains.  Three domains were created with the same root name, but with .com, .net, and .org top level names.  The .org and .net domains are configured to redirect to the .com domain.  So wherever he goes, the others will follow!   :thumbsup:

I'm really liking "Let's Encrypt" and "CloudFlare" so far.  I think SSL certificates should now be included with every domain name you purchase anyway.  Unless you need support, the entire certificate process is automated.  The registrars would not need any real additional resources to issue them.  They could make their money on support services without charging everyone for a certificate that's signed and stored on a set of certificate servers.  Take the payment aspect away, and the process is even less complicated.

Charging for encryption and website verification is like paying a car dealer for seatbelts as an option.  SSL makes the entire web safer.  Why force website operators to choose if they want/need to spend for a signed certificate?  It would not surprise me if one day the InterNIC folks require an SSL certificate for all public-facing web and file servers.  At that point, free SSL certs will start to become standard issue.   :wave:
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 23, 2020, 01:42:40 AM
Oh, here's something I found out Sunday.

On most web browsers, when you get a warning that the website you're visiting doesn't have a valid SSL certificate, there's usually a method of accepting the risk and going to the site anyway.

When I started using Brave, a browser born from the Chrome browser code base, I started having problems bypassing that warning half the time -- normally when the certificate was self-signed without a way to validate it.

I'm sure Chrome had the same issue, but I was using Firefox forever.  Now that I'm using Brave, I've had to open Firefox to get past this problem and visit sites that have this issue.

Well, Sunday I finally did the research and found the bypass "code".  There's no link to click, certificate to download or option to select.  All you do is click anywhere on the error page and type:

thisisunsafe

-- 3 words all run together without spacing.

Hit <enter>, and you will be forwarded to the site.   :thumbsup:

I have a few devices at home that have web GUIs for managing the devices.  A couple only allow HTTPS, and the certificate is self-signed.  This trick allows me to use my one preferred browser again for everything.     :geekdanc:

Once I access the scary page, the browser normally allows me to return either with no or minimal links to acknowledge the risks.

FWIW

Title: Re: How do you keep your digital music files?
Post by: Inspector on June 23, 2020, 03:44:36 AM
Oh, here's something I found out Sunday.

On most web browsers, when you get a warning that the website you're visiting doesn't have a valid SSL certificate, there's usually a method of accepting the risk and going to the site anyway.

When I started using Brave, a browser born from the Chrome browser code base, I started having problems bypassing that warning half the time -- normally when the certificate was self-signed without a way to validate it.

I'm sure Chrome had the same issue, but I was using Firefox forever.  Now that I'm using Brave, I've had to open Firefox to get past this problem and visit sites that have this issue.

Well, Sunday I finally did the research and found the bypass "code".  There's no link to click, certificate to download or option to select.  All you do is click anywhere on the error page and type:

thisisunsafe

-- 3 words all run together without spacing.

Hit <enter>, and you will be forwarded to the site.   :thumbsup:

I have a few devices at home that have web GUIs for managing the devices.  A couple only allow HTTPS, and the certificate is self-signed.  This trick allows me to use my one preferred browser again for everything.     :geekdanc:

Once I access the scary page, the browser normally allows me to return either with no or minimal links to acknowledge the risks.

FWIW
Thanks for this. I have had the exact same problem with Chrome on occasion and didn’t know how to continue. Why it happens only occasionally I have no idea?  ???
Title: Re: How do you keep your digital music files?
Post by: drck1000 on June 23, 2020, 06:47:36 AM
The hard drive on my laptop crashed last week. Thankfully I had been regularly backing up music and photos on external hard drives for a while. Those are the only stuff that I really care about on my home laptop. Got the blue screen and couldn’t start/reboot after doing “checks”. Was able to get a full backup of data before it railed completely. Apparently my laptop consisted of a “spinning” drive and a small SSD. Replaced with a main SSD drive and computer is running well again.

Now I’m backing up my work laptop more regularly. So much data on that laptop that I do need. And our work IT doesn’t have enough drives to allow backup to servers or even allow individual departments to buy network drives with out own money.
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 23, 2020, 11:46:30 AM
The hard drive on my laptop crashed last week. Thankfully I had been regularly backing up music and photos on external hard drives for a while. Those are the only stuff that I really care about on my home laptop. Got the blue screen and couldn’t start/reboot after doing “checks”. Was able to get a full backup of data before it railed completely. Apparently my laptop consisted of a “spinning” drive and a small SSD. Replaced with a main SSD drive and computer is running well again.

Now I’m backing up my work laptop more regularly. So much data on that laptop that I do need. And our work IT doesn’t have enough drives to allow backup to servers or even allow individual departments to buy network drives with out own money.

The NextCloud application lets you create your own file sharing "cloud" at home.  That way you can sync your files between devices and the NextCloud repository without having to trust/pay a commercial site.

I installed the nextCloud client on my Mac.  Works great!  The PC client is almost identical. 

Might be something to consider so files get automatically backed up for you.   :thumbsup:
Title: Re: How do you keep your digital music files?
Post by: drck1000 on June 23, 2020, 11:48:09 AM
The NextCloud application lets you create your own file sharing "cloud" at home.  That way you can sync your files between devices and the NextCloud repository without having to trust/pay a commercial site.

I installed the nextCloud client on my Mac.  Works great!  The PC client is almost identical. 

Might be something to consider so files get automatically backed up for you.   :thumbsup:
What?   ??? Haha 

I'll get around to reading this thread and others more.   ;D
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on June 23, 2020, 12:01:50 PM
What?   ??? Haha 

I'll get around to reading this thread and others more.   ;D

If you've ever used iCloud from Apple, OneDrive from Microsoft, or any of the other "cloud storage" services like DropBox, NextCloud is the same.  Only thing different is you have total control over the files being stored and have to set it all up.

https://nextcloud.com/
Title: Re: How do you keep your digital music files?
Post by: drck1000 on June 23, 2020, 01:14:16 PM
If you've ever used iCloud from Apple, OneDrive from Microsoft, or any of the other "cloud storage" services like DropBox, NextCloud is the same.  Only thing different is you have total control over the files being stored and have to set it all up.

https://nextcloud.com/
I've used iCloud for some stuff, but not photos.  I'll look into NextCloud.  Thanks!  I have a bunch of external drives, but some of them are getting old and probably time to replace. 
Title: Re: How do you keep your digital music files?
Post by: Flapp_Jackson on July 04, 2020, 08:04:53 PM
Well, it's been over a week since my last update.  Here's what's been happening....

My 30-day Trial License for unRaid was ready to expire this week.  Even if it expires, and you keep the unRaid array up and running, it'll be fine until the next reboot or if the array is stopped.  I figured I'm happy with it, so I paid for the 12-storage-device license.  I have 9 devices now:  2 x parity + 4 x array storage + 2 x caching SSDs + 1 x unattached SSD.  The boot flash (thumb drive) isn't counted in the device limit.

That's above the lower-priced 6 device license and below the unlimited license.  Note:  "unlimited" is what the license covers, but unRAID can't handle more than 30 storage devices.  Maybe they'll design an upgrade that increases that limit, but I thought I'd make it clear since there's going to be that one guy who buys a pallet of old disk drives from an auction and then gets bent out of shape when he can't configure 298 drives in a single unRaid server!   (That a LOT of pron -- even by AH's standards!) :rofl:

I spent most of this time messing with Nginx web server and how it might perform reverse-proxying for my home-based servers.  I was able to easily set up Let's Encrypt following video directions, but it involved forwarding web traffic to another unRaid container, and only using an additional port -- not a separate IP address from the unRaid O/S.

What I wanted was to use the same Let's Encrypt docker container and configure it for all my reverse-proxy connections, for simplicity.

Most of the video and written tutorials involve installing Nginx on your Linux VM along with the Apache web server -- totally separate from your unRaid instance.  That might be okay for my needs, but it complicates things if I leave ports 80 & 443 (http & https) mapped to the Let's Encrypt docker from my main router.  You can only target one server at a time on the same port.  The alternative is to map different ports from the Internet to your servers, but:

1.  the incoming URL would have to specify the port if other than 80 or 443 (as in http://my.dns-registered-server.net:8181), which is messy, and
2.  I think running Nginx in a docker is more secure, lighter in terms of overhead, easier to manage, and is well suited to run on unRaid for not ony reverse-proxy duty, but also for managing and renewing SSL certificates.

A new detail I learned about Let's Encrypt -- it will look at your SSL certificates and attempt to renew any that are within 30 days of expiration.  They are issued with a 90 day expiration.  That provides the administrator (you) a month to notice any renewal errors so they can be corrected prior to the expiration date.  We all know what a pain it is when a website we frequent lets their SSL certificates lapse.   :geekdanc:

One 6-month-old video I watched said that Let's Encrypt has registered more than a BILLION SSL certificates ... and all for free.   :thumbsup:  Since it's now 90% automated, and it's FREE! (open source, but whatever), there's really no excuse for not having an SSL certificate issued by a Certificate Authority that's trusted.  You can still pay $60 - $600 for a certificate if you feel you have to have one issued by VeriSign, but I see no legit reason for that now. 

What have I accomplished besides spending $89?  I now have my official domains pointing to my leased Virtual Private Server (in Michigan), and my development domains pointing to my home router where Nginx decides which IP and port location gets the requests based on the URL sent by the browser.

I've created load-balancing gateways using Linux in the past for applications like IRC Chat.  I did what companies wanted $30 to do with 2 spare 486 PCs and a few spare network cards.  Result:  The entire Pacific Fleet could chat while load-balanced across my 3-6 servers all accessed using a single IP address.  I could take a server (or more than one) offline for updates or hardware changes with zero downtime for chat users.

This reverse-proxy setup is the same function, but it sits one more level removed from the kernel and OS.  I could have saved myself a lot of work if this had been mainstreamed in the early 2000s!   

Oh, well.  Such is the nature of computers.  It never stands still.

There are a few other dockers I've been messing around with.  (No, not the kind you buy at May's!).    One is a reverse-proxy manager.  It can create and manage your server connections in a graphical interface, making the process super easy.  I just need to see if it'll work on the Let's Encrypt implementation of Nginx without too much effort.

I'm a system integrator by trade.  Lately, I've had to learn to step back and see if there are already integration paths available without creating something new.  Otherwise, I not only waste my time, but upgrades and minor changes become a bear if I have to test, change and retest my own "system hooks".