Works and Days

So my blog has been rather silent for some time.  Basically, I was suddenly catapulted to a more prestigious and labor-intensive position at work, and I have been working to a degree that is unprecedented in my life, both professionally and at home.

I do want to report back with the templates and timelines I have followed to earn this success, as the originating and constant impetus for this blog is to share with fellow men of God those ways of success which I have discovered in this world, but that time will have to wait a bit more.

I will say this: it has done me extreme good to avoid the outrages of the day entirely.  Historically, I worried that it is my responsibility to tend to the news and affairs of politics lest I reduce myself to idiocy, but I will say to those who are interested: turn your focus to your life exclusively, and you will impress yourself with your achievements.  They will, one day, render a base of operations for you whose stability will allow a measured and effective return to societal matters.  But I expect that a man in his 30s owes his efforts more exclusively and directly to his work and his family.

This glossed passage from Hesiod has led me well:

“…listen to right and do not foster violence; for violence is bad for a poor man. Even the prosperous cannot easily bear its burden, but is weighed down under it when he has fallen into delusion. The better path is to go by on the other side towards Justice; for Justice beats Outrage when she comes at length to the end of the race. But only when he has suffered does the fool learn this…lay up these things within your heart and listen now to right, ceasing altogether to think of violence.”

“Badness can be got easily and in shoals; the road to her is smooth, and she lives very near us. But between us and Goodness the gods have placed the sweat of our brows; long and steep is the path that leads to her, and it is rough at the first; but when a man has reached the top, then is she easy to reach, though before that she was hard.”

“But do you at any rate, always remembering my charge, work…that Hunger may hate you, and venerable Demeter richly crowned may love you and fill your barn with food; for Hunger is altogether a meet comrade for the sluggard.”

“whatever be your lot, work is best for you, if you turn your misguided mind away from other men’s property to your work and attend to your livelihood as I bid you.”

“So do: and avoid the talk of men. For Talk is mischievous, light, and easily raised, but hard to bear and difficult to be rid of. Talk never wholly dies away when many people voice her: even Talk is in some ways divine.”

I recommend the practice wholeheartedly.  After six months of this practice, my home and professional life have never before been so well ordered. For a brief list to provide a general sense of the direction in which my efforts have moved:

  1. The Home Server
    1. Off-Site Backup for All Family Documents:  The process is working consistently and without need for my intervention.  I am using RaspBSD and a Raspberry Pi 3 with a Seagate 4TB Desktop USB disk at a remote site to receive differential snapshots from my OpenZFS zpool every night, as facilitated by a short and easy zsh script which I hope to make public soon.
    2. A Paperless Home:  I purchased an Epson ES-400 scanner and fully digitized all paperwork in the home, storing related documents in single PDF files (e.g. each vehicle, appliance, etc., has its purchase documents, title, and maintenance documents concatenated in chronological order in a single, easily searched file).  A nice little inbox is placed in a central location in the home from which documents are processed into our digital storage solution.
      1. Combined with item 1.1, these documents’ integrity is fully protected by OpenZFS and their availability is assured even in the event of a disaster which consumes my home.
    3. Home videos and pictures are well organized and tagged on the home server, allowing for engaging reviews of our family time which brings joy and a sense of order and progress to my children and myself.  The inspiration to further engage with one’s family which can be had with such documentation and reviews should not be underestimated, in my opinion.
      1. I have found that these pictures also provide the extreme benefit of allowing one to document and let go of possessions which only have sentimental value insofar as they were the sole documentation of important memories.  By documenting them with pictures and videos (say, of the children playing with particular toys or whatnot), it is that much easier to process those items out of the home when the time comes.  Not everything must go, of course, but many items are, as I said, without real value aside from that documentation which is better had in video form, anyhow.
  2. Professional IT Work
    1. CISSP Certification:  In my new position, it became critical to achieve this certification.  I intend to finalize and publish a personal reading list to give others an idea of the material which I covered, as I was able to pass this examination with only two weeks of dedicated study (my historical work made this possible, so I believe I could offer a very clear path to certification for others).
    2. Splunk:  I have personally constructed a mid-sized Splunk cluster (seven index nodes, two search heads, about 200 GB of data processed and indexed daily).  I have installed and configured proprietary Apps and Apps which I developed for data sources without such offerings as well as custom Data Models, Lookup tables, Dashboards, and Alerts for our organization.
    3. Other Projects have been undertaken as well, keeping me extremely busy, but it is rewarding work.
  3. Miscellaneous Home Logistics
    1. Processed out underutilized valuables in the home:  It is truly remarkable what can be accomplished with a persistent digital yard sale (eBay); I was able to sell (often within 24 hours of posting the item!) old IT components for which I have no present use (the Intel Core i3 and Core i5 processors which once drove the home server, now using a Core i7 processor) and other items (my middle school trombone!), converting senseless burdens into hundreds of dollars.
    2. Home storage has been optimized: we have reduced our attic storage from some 30 boxes worth of hastily-packed junk, keepsakes, and valuables into a mere 8 well-labeled and indexed boxes.  Everything else was either digitized (lots of papers crammed into boxes…), sold, donated, or thrown away.  It is a massive relief to know we do not have all of that stuff awaiting our attention should it become again necessary (but hopefully we won’t move from this house until far, far into the future).
    3. Clutter has been removed:  All of our daily items have been reduced to a near-optimal state through the process above (3.2).  There is little lying around which we do not actually use on a regular basis.

Without alleviating the pressures and wasted time incurred by the daily outrage factory that is our society as of late, I could not have done these things, and I would be in a dramatically inferior position when it comes to assisting with any genuine disaster whose occurrence does legitimately seem to be more plausible given our current political situation.

The fruit of this labor is, indeed, a constant sense of peace that has hitherto been irregularly periodic at best.

Posted in Governing the Body, Information Technology, Politics, Religious Studies, Scorpionfish | Leave a comment

Fedora, Firefox 52, and NPAPI Plug-Ins

Quick Fix Up Front:

  1. Open up Firefox 52, navigate to about:config.
  2. Right click and add a boolean configuration variable named “plugin.load_flash_only” and set it to False.
  3. Restart Firefox 52

A Brief Write-Up:

So, Firefox 52 was just pushed to Fedora 25 and it immediately broke my F5 extension which I use to access my place of employment’s VPN on my Fedora 25 systems.

Ever so nicely, the update to Firefox doesn’t come with any useful information informing you why you are now being continually prompted for reinstallations of your NPAPI extensions.  If you look in your “Add-Ons” menu in Firefox, it appears the extension is properly installed and in good shape.  However, any attempt to use a site which requires it will cause the site to prompt you for reinstallation.

As it turns out, this is because Firefox 52 is attempting to remove all support for NPAPI extensions.  This is theoretically a good thing due to the security risks of the NPAPI framework, but of course not every vendor out there will necessarily update their code accordingly (at least quickly enough to be helpful), and if you’re like me and relying on one of these extensions, you’ll need a work-around.

Very lamely, Firefox is still allowing the Adobe Flash NPAPI extension, but only that one (since they know removing support for it would kill their browser usage).  So, that explains the name of the hidden configuration parameter you have to manually add into your about:config page to get this to work.

The proper solution here is, of course, for Firefox to permit explicitly-designated NPAPI extensions to be used for a good while.  I understand that they’re trying to increase security with some tough love here, but it really just adversely impacts Linux-based end users rather than coerce organizations with little incentive to provide Linux distribution support in the first place.

I don’t know how this will pan out; hopefully F5 will up their game, but if they don’t, I guess I’ll have to have a back-versioned Firefox browser on my system just for my VPN..

Posted in Information Technology | Tagged , | Leave a comment

FreeBSD and OpenZFS Enable Scientific Discovery: How Long Do Seagate Hard Disks Live?

Ok, I don’t usually post little “yay for me” articles ’round here, but I’ve been buried in work to the point of not being able to post much at all so at the risk of taking a solid step in a bad direction for the blog, I’ll provide some evidence of the value in running an arrangement as described in the Engineering Walden Technical Reference Model:

It’s pretty incredible to think that I’m typing this post on a GNU/Linux/KVM/QEMU-driven (Fedora, of course) Windows 10 guest domain which is sharing hardware with the FreeBSD guest domain which is using IOMMU-based PCI-passthrough to access and manage four 3 TB Seagate Barracuda hard disks arranged in a OpenZFS zpool as a striped set of mirrored pairs.

Right now, I am comfortably typing a blog post and gaming (Elite Dangerous is amazing, btw) in a Windows 10 guest domain hosted on the same rig which is simultaneously rebuilding my home server’s principal storage pool and hosting a home media server which is streaming Futurama to my now-sleeping family.  Its uptime is relatively uninterrupted (I powered it down for the disk replacement just to be safe) thanks to the extreme awesomeness of OpenZFS on FreeBSD.

And today’s stalwart servant sent to that bitbucket in the sky, the great /dev/null from which all arise and to which all shall return, a Seagate Barracuda Desktop Edition who lived a long and steadfast life of service at 28,811 hours of uptime.

He will join his brother, who arrived before him after a staggering 38,000 hours of uptime.

So, anecdotally, these are some pretty good drives, I’d say.

More demonstrably, we have some amazing machines in these homes of ours, I’d say.

But most importantly: with OpenZFS, my data integrity is monitored by associated checksum values so I know the age of my disks isn’t causing me to lose any data.  Because OpenZFS is designed to operate without trusting the individual underlying block devices, I can get the most time possible out of my consumer-grade hardware without worrying that the files I store with them are being corrupted.

And tranquility ensues.  Engineering is a fine spiritual venture.

Posted in Information Technology | Tagged , , , , , | Leave a comment

Fedora, NFS Home Directories, and You: Proprietary NVidia Driver Edition

Well that was one hell of a circuitous problem solving tale of woe.  So here’s the rundown:

After some upgrades, I ran into the following slew of errors on reboot (in reverse chronological order):

Jan 31 23:31:48 gdm[1217]: GLib: g_hash_table_find: assertion ‘version == hash_table->version’ failed
Jan 31 23:31:48 login[2729]: pam_systemd(login:session): Failed to release session: Interrupted system call
Jan 31 23:31:48 gnome-session-binary[1417]: GLib-GObject-CRITICAL: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed
Jan 31 23:31:48 gdm-x-session[1658]: GLib-GObject: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed
Jan 31 23:31:48 gdm-x-session[1658]: GLib-GIO: g_task_return_boolean: assertion ‘task->result_set == FALSE’ failed

Jan 31 22:22:16 gnome-settings-daemon.desktop[1896]: lock: No locks available

Jan 31 22:10:29 pulseaudio[2540]: Daemon already running.
Jan 31 22:08:37 abrt-hook-ccpp[1899]: Process 1888 (gnome-session-failed) of user 1000 killed by SIGSEGV – dumping core
Jan 31 22:08:37 gnome-session-binary[1678]: Unrecoverable failure in required component org.gnome.Shell.desktop
Jan 31 22:08:23 pulseaudio[1865]: [pulseaudio] bluez5-util.c: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote appl
Jan 31 22:08:23 pulseaudio[1865]: [pulseaudio] core-util.c: lock: No locks available
Jan 31 22:07:33 pulseaudio[1865]: [pulseaudio] core-util.c: lock: No locks available

Looking a little closer, I found:

Jan 31 22:20:07 gnome-shell[1857]: cogl-sampler-cache.c:200: GL error (1280): Invalid enumeration value

Jan 31 22:20:06 gnome-shell[1857]: clutter_input_device_get_device_id: assertion ‘CLUTTER_IS_INPUT_DEVICE (device)’ failed
Jan 31 22:20:06 gnome-shell[1857]: STACK_OP_ADD: window 0x1200001 already in stack

Jan 31 22:08:37 kernel: gnome-session-f[1888]: segfault at 0 ip 00007fe59d2e1899 sp 00007ffc1269b3f0 error 4 in[7fe59d003000+6f6000]

Though I fixated on that segmentation fault like a narrow-minded fool, I also noted what I should have considered immediately as abnormal and important:

Jan 31 22:15:39 kernel: lockd: server OK
Jan 31 22:15:39 kernel: lockd: server not responding, still trying

And the reason that’s so abnormal and important, in particular, is not only that my lockd service is having trouble contacting my NFS server, but that I use NFSv4…which doesn’t use lockd.

After banging my head against these errors by reinstalling my NVidia drivers, attempting the use of KDE Plasma rather than GNOME, and the like, all to no avail.  Finding the issue to be common to GNOME and KDE Plasma (which doesn’t make use of the segfaulting libgtk, obviously), and running down a bug report on bugzilla that shows the error is actually a failure of GDM which likely occurs long after the real culprit, I actually took note of the fact that a number of services (gnome-settings-daemon, pulseaudio above) were citing the absence of locks as problems.

And that, my friends, was the issue.  I had been messing around with my NFS server, trying to get the Windows NFS client to make proper use of file locking, and in the course of doing so, I had inadvertently caused the NFSv4 service to stop, leaving only NFSv3 and a busted-up lockd arrangement for my poor Fedora system.  It automatically downgraded to NFSv3 when it found NFSv4 to fail, and being stuck without file locking, this actually caused the entire problem.

So, the moral of the story is:  When using NFS-mounted home directories, sometimes NFS problems can even look like graphics driver problems.

Now, I firmly believe the benefits of NFS-mounted home directories outweigh the added potential for issues such as the above, so I’ll be keeping mine, thanks, but just know you should check on that stuff at the first sign of trouble and make sure all is well.  The issues that can be caused by a file system problem with your home directory can look like just about anything.

Posted in Information Technology | Tagged , , , | Leave a comment

Potential Resolution for Remmina Error ERRCONNECT_CONNECT_TRANSPORT_FAILED [0x2000D]

If you attempt to connect to a Windows 7 workstation using the RDP protocol TLS security, you may encounter the following error information from Remmina:

Dec 30 21:25:00 hostname remmina.desktop[4836]: [21:25:00:443] [4836:5478] [ERROR][com.freerdp.core] – freerdp_set_last_error ERRCONNECT_CONNECT_TRANSPORT_FAILED [0x2000D]
Dec 30 21:25:00 hostname remmina.desktop[4836]: [21:25:00:443] [4836:5478] [ERROR][com.freerdp.core.transport] – BIO_read returned an error: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

To resolve the issue, you may simply need to edit the connection in Remmina, head to the Advanced tab, and choose TLS for Security (by default, it is “Negotiate”).  I don’t know why the security negotiation appears to be failing in my case, but that resolved the issue for me.


Posted in Information Technology | Tagged , , | Leave a comment

Bug Report: mutter.3.22.2-1.fc25 Breaks GDM and GNOME for Proprietary NVidia Drivers

MAN.  That took hours to figure out.  I simply ran a dnf upgrade on my Fedora 25 system (using an NVidia GeForce GTX 960 with the proprietary drivers) and, when it rebooted, I was greeted with a failure to start GDM.  The system would simply hang at the text list of services being started.

Switching into another terminal, I found error messages such as these repeated with every effort to start GDM:

Dec 01 15:07:04 hostname gdm[1181]: GLib: g_hash_table_find: assertion ‘version == hash_table->version’ failed
Dec 01 15:06:51 hostname abrt-hook-ccpp[1460]: Process 1455 (gnome-session-failed) of user 42 killed by SIGSEGV – dumping core

Being segmentation faults, I suspected I needed to reinstall my NVidia drivers (which seems to happen once in a while with kernel or mesa updates).  I tried this to no avail.  Many times.

Because I’m a moron and was filtering my journal for error-level messages only, I didn’t notice this helpful message was also being logged:

Dec 01 15:06:51 hostname kernel: gnome-session-f[9433]: segfault at 0 ip 00007f23d0001579 sp 00007ffd1a6248c0 error 4 in[7f23cfd23000+6f0000]

So I started checking around for that same segmentation fault in other users, and lo and behold I hit upon this over at the Arch Linux forums.  If I’m reading the tail end of the related bug report properly, it looks like a bug was fixed upstream, but given that Fedora 25 is still hosting mutter-3.22.2-1.fc25 for download, we may still be waiting on the package which includes the fix.  Or, it may be that the fix included in package 3.22.2-1 is incomplete (a user at the tail end of the forum link reports just a few days ago using the package which includes the apparent fix to no avail).

So anyway, if you, too, are experiencing GDM/GNOME session failures with the proprietary NVidia drivers on Fedora 25, try simply downgrading mutter:

[you@yourplace ~]$ sudo dnf downgrade mutter

That brought me back to mutter-3.22.1-8.fc25.x86_64 and all is well for me.

Now to get rid of lightdm and roll back all my insane troubleshooting installations…

Posted in Information Technology | Tagged , | Leave a comment

Adding a New Boot Disk in Fedora 23 with UEFI

Well THAT was a hell of a learning experience.  You know, I spent all this time learning about GRUB 2, the MBR, and bootloader operation in general, and just when I thought I really had the bootloader stuff down, I realized I had to learn about UEFI before I could consider myself to have mastered the area from a system administration perspective.

But that was a lot of junk to read, so I didn’t.

So then, I found I needed to swap out an SSD in my Fedora system so that I could replace it with a larger drive.  Of course, it had to be my boot volume.

THUS BEGAN MY QUEST.  I read quite a lot.  I can recommend this guy for a high-quality informal read in BIOS and UEFI technology.  But, I think I can make a quick and dirty SysAdmin overview for ya:

  1. As you know, Fedora boots using the GRand Unified Bootloader (GRUB) version 2.
  2. As you may know, GRUB 2 operates either with BIOS or UEFI firmware.
    1. If operating with BIOS firmware, GRUB 2 installs bootloader code at the beginning of a bootable disk, in what’s known as the Master Boot Record.  Even more particularly, it installs to s a very small space at the start of a disk within the MBR which exists prior to partition information (and then a second “Stage 1.5” is installed in a subsquent space).  Many people have experience with bootloaders wiping one another out (such as, say, Microsoft Windows and Fedora) during installation as they contend for this same extremely limited space.
    2. If operating with UEFI firmware, GRUB 2 installs bootloader code in a special EFI partition which must be formatted with the FAT 12, 16, or 32 file system (use mkfs.vfat in Fedora to create any of those file system types).  Technically, the UEFI specification describes a very specific implementation of the FAT file system, but Fedora’s mkfs.vfat command seems to produce file systems of UEFI’s liking.
      1. Once the UEFI bootloader code is in place, there is a final important step for system administrators, and that is to update the UEFI firmware’s boot manager to point to the new code.

That last step there was what had me hung up for about an hour.  Fedora, and GNU/Linux distributions in general, use a tool called efibootmgr to control the UEFI firmware’s boot manager from within the operating system.  That’s pretty sweet.  Amazingly enough, your motherboard is not likely to provide as much capability in managing the UEFI boot manager as this handy tool.  My motherboard doesn’t even seem to let me create boot entries within the UEFI interface, so I have to rely on efibootmgr.

If you check out the man page, you’ll see some pretty standard options.  Basically:

  • Use efibootmgr -v to list the boot manager entries in your UEFI firmware.
  • Use grep efibootmgr /var/log/anaconda/program.log to locate the command used by Fedora when your OS was installed.  It will look something like this:
    • efibootmgr -c -w -L Fedora -d /dev/sde -p 1 -l \EFI\fedora\shim.efi
      • The “-c” option creates a new boot entry
      • The “-w” option writes a signature to the MBR if necessary (which it is not, in a UEFI environment, so this can probably be dropped)
      • The “-L” option creates the name for the boot entry which you will see in your UEFI firmware
      • The “-d” option points to the disk device on which the EFI System Partition (your FAT file system) resides
      • The “-p” option indicates the partition number on the disk device on which the EFI Partition resides
      • The “l” option points to the code within the EFI partition which should be executed first by the UEFI firmware.

Now, armed with this super secret knowledge, you will be able to easily and handily create a new boot device on your Fedora system.  Really, all you have to do is:

  1. Create the necessary EFI and boot partitions on the new device
    1. sudo cfdisk /dev/sdb or whatever and make a 200MB EFI partition and a 500MB boot partition.
  2. Create the necessary file systems in the new partitions
    1. `sudo mkfs.vfat /dev/sdb1`
    2. sudo mkfs.ext4 /dev/sdb2
  3. Make some temporary locations and mount the partitions to them so you can modify their contents
    1. sudo mkdir /mnt/boot2 /mnt/efi2
    2. sudo mount /dev/sdb1 /mnt/efi2 && sudo mount /dev/sdb2 /mnt/boot2
  4. And then just rsync over your current boot and EFI partitions:
    1. sudo rsync -a /boot/ /mnt/boot2
    2. sudo rm -r /mnt/boot2/efi/EFI
    3. sudo rsync -a /boot/efi/ /mnt/efi2
  5. Now just fix your /etc/fstab so that the boot and EFI partitions point to the right new GUIDs
    1. Obtain the file system UUIDs from `cfdisk` (displayed at the bottom)
    2. Swap UUIDs in /etc/fstab
  6. And finally, use efibootmgr to create a new boot entry for your system which points to the proper boot device.
    1. You need only change the -d option in the command from your Anaconda program.log.
    2. You cannot rename UEFI boot manager entries with efibootmgr (sadly), so just delete the old entry after you prove that your system boots with the new entry.

And that is it!  Fantastico.

So whereas with older MBR/BIOS systems, you need to reinstall GRUB after installing Microsoft Windows (if you installed onto the same disk as your Fedora system) in order to overwrite the Windows bootloader in the MBR (and then chainload Windows with GRUB, making GRUB the sole true bootloader for the system), with EFI, you have more options.  You could create two separate EFI partitions for Windows and Fedora, or you could try to put all the bootloader code in a single EFI partition and use efibootmgr to create separate boot entries in your UEFI firmware to point to the same disk and partition, but separate bootloader code for each OS.

It’s actually a lot easier to manage, but it requires this additional understanding to get it right.  Once you know the sequence of events and the relationships between the components, managing issues becomes a lot easier.  If you are no longer able to boot Microsoft Windows on a dual-boot system from within your UEFI firmware, for example, you now know you simply need to boot into your GNU/Linux OS and use efibootmgr to create the appropriate entry.  If your Windows EFI partition was overwritten or the code was lost, you can attempt the use of Microsoft utilities (as described here) to repair that matter.

Posted in Information Technology | Tagged , , , | 1 Comment