Broken YUM on CentOS 7 (or how I learned to love DNF)

It's a simple thing to keep your system updated. A quick yum update every few days - just like brushing your teeth. Completely painless, quick, efficient.

Not today it wasnt.

---> Package screen.x86_64 0:4.1.0-0.26.20120314git3c2946.el7 will be updated
---> Package screen.x86_64 0:4.1.0-0.27.20120314git3c2946.el7_9 will be an update
---> Package skypeforlinux.x86_64 0:8.67.0.96-1 will be updated
---> Package skypeforlinux.x86_64 0:8.71.0.36-1 will be an update
Error: Invalid version flag: or

This was the error I received. Okay, so when yum update fails, there's always the tried and true command sequence that fixes it:

sudo yum clean all
sudo yum update --skip-broken

This time it didn"t work. I still received the same error. Time to google for more info... 

I found a kool command sequence on John S. De Stefano's blog that looked promising:

sudo yum check all                # tells you of any problems
sudo package-cleanup --problems   # lists all known package problems
sudo package-cleanup --dupes      # lists duplicate packages
sudo package-cleanup --cleandupes # actually cleans up duplicates
sudo yum check all                # run again to check for remaining problems
sudo yum-complete-transaction --cleanup-only

However, this command sequence failed to fix the issue too. Looks like I'm going to have to work out what's broken and why. No easy way out with this problem.

Rich Dependencies

Scrolling through bugzilla, I found the following entry:

There was a mistake made in the rpmlib() dep for rich deps. You need
at least rpm 4.13 for the base rich deps, and rpm 4.13.1 for the rest.

yum and related packages are no longer actively developed.
They are being replaced with dnf, dnf-utils, etc.

I'm closing this bug because it's most likely never going to be fixed.
If you still consider your bug report important, reopen it, please.

https://bugzilla.redhat.com/show_bug.cgi?id=1578942

This was actually a bug report for F28+. Since Fedora uses DNF primarily and YUM is deprecated, no one seemed particularly interested in fixing it in Fedora. RHEL/CentOS 8 both use DNF as well. Could this bug have percolated down into CentOS 7? Time to check rpm versions and make sure I have at least 4.13:

$ yum --showduplicate list rpm

Installed Packages
rpm.x86_64                          4.11.3-45.el7                          @base
Available Packages
rpm.x86_64                          4.11.3-45.el7                          base 

Well, that's just peachy!

I'm running an old version of rpm that doesn't support rich dependencies. It also appears that yum may not handle them well either - although the indications are the problem really lies with rpmlib().

I've never really spent a lot of time looking at the inner workings of package managers and their respective update managers. Now that ignorance is coming back to haunt me and it's time for some old fashioned studying the matter.

From rpm.org there's an excellent description of how the boolean operators work and how they help avoid dependency hell. This is what is being referenced in the error I received. The 'or' operator is unknown because my version of rpm is too old. Boolean operators enable what is termed "rich dependencies". Basically providing a logical sequence for resolving dependency issues across multiple versions. A good example is community-mysql and mariadb. Both packages do the same thing - provide a mysql style database. Without rich dependencies, if another package requires mysql, you have to choose which package is required. With rich dependencies you can state:

Package A: Requires: mysql
Package mariadb: Provides: mysql
Package community-mysql: Provides: mysql
Suggests: mariadb to Package A.

Which means that if community-mysql is already installed, that is used as the dependency, otherwise mariadb is installed.

This is really cool, but yum simply ignores it.

Put simply, the root cause of the problem is:

1. RPM supports "rich dependencies"
2. DNF supports resolving packages with "rich dependencies"
3. YUM does not support resolving packages with "rich dependencies"

The solution then is obvious: Since CentOS 7 supports DNF, it's time to switch. Fiddling with yum and manually resolving the dependencies will only delay the inevitable.

YUM vs DNF 

source

"Dandified YUM" or DNF, is the replacement package update manager for RHEL/CentOS. It's been part of Fedora for a long time now. I acknowledge it is clearly superior to YUM in most aspects, plus it doesn't suffer from some of the issues that Debian apt does. The major goal is to eliminate (where possible) dependency hell. But it also has other advantages.

It was also designed to be as drop-in replaceable to yum as possible. It comes very close, but some commands have no equivalent (some of these are deliberate actions). So, for me, if your used to using yum, dnf just represents yet another sequence of commands that must be memorised just to continue doing your job.

I'm not going to bore you with lists here of features, commands, comparisons etc. The links I've provided do that well enough. Plus, if you want a deep dive into dnf, you can go here. Suffice it to say that I decided that installing and using dnf was the simplest and most effect potential solution to the immediate and potentially long term problems.

Install DNF on CentOS7

Pretty simple really, however there are a number of dependencies since it leverages by Py2 and Py3. A total of 11 dependent packages were installed, however YMMV.

$sudo yum install dnf
...
Resolving Dependencies
--> Running transaction check
---> Package dnf.noarch 0:4.0.9.2-2.el7_9 will be installed
--> Processing Dependency: python2-dnf = 4.0.9.2-2.el7_9 for package: dnf-4.0.9.2-2.el7_9.noarch
--> Running transaction check
---> Package python2-dnf.noarch 0:4.0.9.2-2.el7_9 will be installed
--> Processing Dependency: dnf-data = 4.0.9.2-2.el7_9 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: python2-libdnf >= 0.22.5 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: python2-libcomps >= 0.1.8 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: python2-hawkey >= 0.22.5 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: libmodulemd >= 1.4.0 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: python2-libdnf for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Processing Dependency: python-enum34 for package: python2-dnf-4.0.9.2-2.el7_9.noarch
--> Running transaction check
---> Package dnf-data.noarch 0:4.0.9.2-2.el7_9 will be installed
---> Package libmodulemd.x86_64 0:1.6.3-1.el7 will be installed
---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed
---> Package python2-hawkey.x86_64 0:0.22.5-2.el7_9 will be installed
--> Processing Dependency: libdnf(x86-64) = 0.22.5-2.el7_9 for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: libsolvext.so.0(SOLV_1.0)(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: libsolv.so.0(SOLV_1.0)(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: libsolvext.so.0()(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: libsolv.so.0()(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: librepo.so.0()(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
--> Processing Dependency: libdnf.so.2()(64bit) for package: python2-hawkey-0.22.5-2.el7_9.x86_64
---> Package python2-libcomps.x86_64 0:0.1.8-14.el7 will be installed
--> Processing Dependency: libcomps(x86-64) = 0.1.8-14.el7 for package: python2-libcomps-0.1.8-14.el7.x86_64
--> Processing Dependency: libcomps.so.0.1.6()(64bit) for package: python2-libcomps-0.1.8-14.el7.x86_64
---> Package python2-libdnf.x86_64 0:0.22.5-2.el7_9 will be installed
--> Running transaction check
---> Package libcomps.x86_64 0:0.1.8-14.el7 will be installed
---> Package libdnf.x86_64 0:0.22.5-2.el7_9 will be installed
---> Package librepo.x86_64 0:1.8.1-8.el7_9 will be installed
---> Package libsolv.x86_64 0:0.6.34-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

Next I tried dnf update:

$ sudo dnf update
<snip>
Running transaction check
Error: transaction check vs depsolve:
(libatomic or libatomic1) is needed by skypeforlinux-8.71.0.36-1.x86_64
rpmlib(RichDependencies) <= 4.12.0-1 is needed by skypeforlinux-8.71.0.36-1.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

DNF is smart enough to know why the update failed. The package 'skypeforlinux' uses rich dependencies. These dependencies require a version of rpmlib that is greater than the CentOS 7 repositories can provide. Therefore, the dependencies cannot be resolved. There's a tip there to use rpm directly to reconcile the issues, but since I know that the version of rpm is also too low, that won't work.

I decide that I can live without skypeforlinux, so I remove it and both dnf and yum are happy.

At this point in time, since both work, I can use either to keep my system updated. However, having installed dnf and now that I'm comfortable working with it (having learned the syntax and equivalent commands) I think I'll use it from now on.

How SORBS ruined my life (old blog)

When this blog entry was written, Open Relay blockers like SORBS were considered the primary defence mechanism against spam email. They are still often used, but not as much as they used to be - for many of the reasons discussed in this article. SPF, DKIM & DMARC have largely taken their place.

 

 

I've had a helluva week so far, and I've got SORBS to thank for it.

SORBS (Spam and Open Relay Blocking System) is an email server reputation service. It (along with other reputation services) collect data on servers that are are either poorly configured (and open to abuse) or actively send spam. Reputation services are an excellent way of protecting against spam. They have very high hit rates and (generally) very low false positives. They also have a very low performance overhead compared with heuristic and bayesian filtering techniques. The mail servers I admister are configured to quarantine all email that is listed by either SORBS, SpamHaus or SpamCop (two other reputation services).

However, sometime last week (when I was off work sick), SORBS listed pretty much all of the hotmail, windows live, yahoo, bigpond, optusnet and myplace servers. There may have been others listed as well.

Responding to a few complaints that came in on Monday about email not being received, I began checking. Then began the sinking feeling that goes along with knowing that I'd have to change the email filtering rules, do some regression tests and then resubmit a week's work of spam through the email filters. SORBS is a very aggressive filter and I've been quite reliant on it for some time.

Less fun was trying to explain the problem to the CEO.

The first part of the quick fix was to move the SORBS check to the end of the anti-SPAM rules of the DMZ mail filter and set it to monitor only - not block. The DMZ mail server is exposed to the Internet and performs basic/quick checks only and does not look at the content of the email messages. It is highly robust and is meant as the front line of defence. It experiences an average of 90 security attacks per hour. It filters about 75% of the email traffic as inappropriate before passing the "possibly okay" messages to the second filter.

The second part was to add a check for zero day threats on the DMZ server and tag messages accordingly.

The third part was to tighten the screws a bit on the second mail filter. This meant decreasing the tolerance limits a little and added a few more checks which included looking for the tags on the email messages placed there by the DMZ mail filter. Usually, this server intercepts about 8% of messages pass to it. Messages quarantined by this server may result in a quarantine message and even a self-release option depending upon the spam score the email receives.

Now that these changes were made, I resubmitted 2143 spam quarantined messages. With the new rules in place, 1963 messages were blocked and 180 passed through to the second filter which blocked a further 68 messages leaving 112 messages sailing through to the mail server. I don't know how what the spam:ham ratio of the 112 messages was but the count seemed to be manageable so I released the remaining 14,000 quarantined messages and sent an email instructing people to forward any received spam to the spam submit mailbox for heuristic analysis. So far, I have 35 spam messages submitted by users which (if accurate) we now have a spam hit rate of 99.75% with (hopefully) a close to zero false positive rate. The hit rate has dropped to 59.4% for the DMZ server and risen to 12.5% for the second filter. Time and monitoring will determine how successful the changes have been.

Moving forward, I have written replacement rules that will no longer quarantine email based upon failed reputation. The new rules will look at SPF (Send Policy Framework - RFC4088) in addition to reputation. If the SPF check passes, the email will be accepted. If SPF fails, it will be dropped. If SPF is 'softfail' or 'none' then it will be subject to a reputation check. Any failure will drop then connection with a 46x error - Temporary Failure, with details as to why sending failed. The sending MTA will then notify the recipient that the email failed to send. The sending server then has the option of implementing SPF. I'd like at some point to add a DKIM check, but that's a reasonably difficult task.

BTW, I am heavily influenced by Ming Weng Wong's whitepaper on Messaging Anti-Abuse:

http://www.openspf.org/blobs/sender-authentication-whitepaper.pdf

 

 

The Dream (Creative Writing)

The world around me starts to dissolve into nothing. Two men are begging, pleading with a man in a suit "No! Please he'll forget us."

"He won't forget you. He'll see you again on TV. You will be a science fiction show and you.." he turns to the second man "..will be a soap opera"

"No!" they both shout as they are dragged away.

"Excuse me!" I say

The suited man looks surprised, turning both ways to see if there is anyone else I may be speaking to.

"Yes you" I demanded "I'm talking to you!"

"I'm sorry" he replies "but you shouldn't be able to perceive me, let alone speak to me. Something is obviously wrong with the system at the moment. Are you sick, ill or delirious?” he asks.

"I have a head cold at the moment, but that's beside the point."

"Ah! That must be it, you're delirious - seeing things."

"I'm fine, but who are you?"

"Well, since you've asked, I am your id."

"My what?"

"The anthropomorphic personification of your imagination. I am the one who keeps control of your dreams - tries to help you make sense of them." then his demeanor becomes more sinister "And then eliminates them before you awake."

At that moment I recognise my id as a dead ringer for James Spader.

"You’re serious?"

"Oh yes. Right now you're asleep - probably feverish actually, it's the only explanation for why you can perceive me. Usually you are blissfully unaware of my existence. But come now, dream number three is about to start."

I turn to ask him what he means, but as I do so, I realize I'm on a footpath, somewhere, who knows where. I'm dragging a some pieces of cardboard around and a few meager possessions rolled up in a blanket. I select a good piece of footpath, under cover and begin to lay out the cardboard on the ground. Soon four other men come near to do the same, but they see me there and look disappointed.

"Everything okay?" I ask.

"Sorry, it's just that we usually setup here. We'll find somewhere else."

"There's room for all of us." I say, and they look overjoyed.

They begin to setup and soon it looks like there's no place for me. I turn to walk away and one of them says, ‘Hey your spot is over here, in between all of us. We'll keep you safe tonight in return for letting us stay here.’

I thank them and put down my cardboard, rudimentary mattress and pillow, other possessions and lie down, wrapped in a minke blanket to keep me warm. As soon as I think 'I shouldn't be here' I'm instantly outside of myself, observing. Standing right next to my id. Standing next to him is another-me wrapped up tightly in white linen and bound with blinking Christmas tree lights and struggling to get out. It looks like he's going to succeed.

"What? I though I got rid of you?"

"Who is that?” I ask, pointing to homeless-me.

"Isn't it obvious? That's your fear of failure. It's quite a common theme. Frankly, I find it boring. The feckless self-pity, the meekness rewarded with gratitude. Seriously B grade. I wish he would get shanked once in a while just to add some reality."

"What? You want to see me shanked?"

"Oh no. Not you. Just that perception of yourself. I am you."

"I'm nothing like you!"

"Oh please! Spare me the self-loathing. I do enough of it for the both of us. You revel so much in your analytical capabilities and your critical thinking that every night I have to cleanse your soul of the monsters you unleash upon your psyche. If it wasn't for me you'd wake up a gibbering mess that needs to be spoon fed."

"May the shades of Jung and Freud haunt you forever."

"Plagiarist! You got that from Zelazny. Remember: everything you know, I know. But come, it's time for Christmas dinner." he finishes as the other-me successfully escapes from his bonds.

I turn around and realise I'm in a buffet of some kind. I offer my plate to the server and receive a slice of roast ham. Further down the line, there's turkey, cranberry sauce, duck and other foods typically found at Christmas. I realise I am moving slowly and my skin is wrinkled. I am old. I make my way to a large table full of people I know: my kids - all adults now - with their wives and children. I sit down at the table and a small girl climbs onto my lap and kisses my cheek and looks at me with a sparkle in her eye.

"Merry Christmas Grandpa! I love you."

A wave of bittersweet happiness washes over me as I look at the empty seat next to me. As I do so I think again 'I shouldn't be here' and once more I am an observer, standing next to my id.

"I can't seem to keep you in one place right now, can I? It's a pity, I like this one, however you need to do something about that empty chair."

"Why is it empty?" I ask "Did someone die."

"No. Someone never happened. Your future-me will always be like that as long as you keep feeding homeless-me. However, I no longer have any time to indulge you, I have work to do."

In an instant we are on top of a ten-story building. It looks like the outside portion of a luxurious penthouse apartment. I instantly know my id lives there. There are three representations of me standing on the ledge, all pleading for their lives. They notice me.

"Don't let him do it!"

"Who are they? What parts of me do they represent?"

"Destructive influences. You're better off without them."

"Who gets to decide that? You?"

"Why yes, as a matter of fact. But more to the point *you* did, by giving that power to me."

"I don't remember doing that!"

"Of course you don't. You didn't want the responsibility. However I tell you what, I'll let them decide."

My id pulls out a handgun.

"I'll make all of you a one-time deal. Trust me, have faith and jump. You have my word that you will be unharmed. Hesitate, and I will shoot you in the head. You have three seconds. One...two...."

All three of them jump. Seconds later there is a sickening thud. I rush to the side and peer over. There, lying on the ground are the bodies of my three doppelgängers.

"You said they'd be unharmed!"

"I lied. Time to go."

Instantly I find I'm sitting on a recliner watching TV with what feels like a heavy, soft warm blanket over me. I have comfort food at my disposal: cheeses, olives, salmon, chocolates that I am constantly eating. The room I'm in is a large one and there's a constant stream of people entering the room at the rear and muttering their disapproval of the situation. My youngest son is beside me also watching. I'm on TV, telling jokes - all ones I've told before. Comical-me has a TV audience, all laughing at every single joke. I start thinking this is simply a comedy show, but the jokes are intertwined around a complex melodramatic love triangle. I can tell this is going to end badly for comical-me so I change channels. I end up with an explicit porno playing. Since my son is watching beside me I try to change channels but I can't. This is when I notice the blanket on top of me is actually partly a naked woman and what I thought was the remote control was actually one of the blanket-woman's breasts that I'm squeezing.

"Don't stop!" the blanket-woman moans. Instantly I'm an observer again.

Blanket-woman looks at me and says "Great! A threesome!"

I find the remote and quickly change the channel. It's a televangelist channel and I am the preacher.

"Good choice!" says preacher-me to real-me. He then points an accusing finger at lustful-me "You need to repent!"

I change the channel again, this time it's a an episode of Star Trek. I'm one of the engineers. The engines are about to explode, but engineer-me grabs some tools and climbs into the bowels of the engine room saying "Don't worry. I can fix anything!"

I walk to the back of the room where my id is congregating with the disapproving crowd.

"Had enough yet?" he asks smugly.

"I'm still trying to make sense of it all."

"Give up. You'll go crazy." He leads me out of the room into a featureless wasteland. "And this, I'm afraid, is where we part company. You're about to wake up and within thirty minutes, you won't remember a thing. Subconscious memory doesn't translate to long-term memory - as you know."

"No, I don't think so. I've seen enough to know I don't want you in charge."
"That's preposterous! I AM you. Remember? I am simply a representation of your id. I'm the one who keeps the nightmares at bay. Without me, you wouldn't be functional from day to day."

"And yet, strolling through my dreams here, amongst the random collections of my Jungian archetypes, there's one thing I've noticed."

"And what's that? That they all look like you?" he sneers.

"Almost. They all look like me EXCEPT for you."

The color drains from his face as the featureless background starts to dissolve, slowly becoming the Christmas cafeteria scene from earlier.

"You know, I'm starting to think about this, about how it all fits together, and what your place in this is. More importantly, how did you get to be in charge here."

My id retorts uncertainly "I told you, *you* did.”

“Yes, I know. And yet something’s been bothering me. My id is supposed to be the primal, uncontrolled part of myself. That’s not you, that’s why you don’t look like me at all. You’re not my id, you’re my Super-Ego.”

As I say the words, my fake id deflates a little and the Christmas cafeteria fully appears. Seated at the table are all the representations of me. The empty chair beside mine is still empty. I wave my hand at the table.

“These are the various manifestations of my id...including you”

“I though you said I was your super-ego.” He sneered.

“That’s a part you play, as the ruler archetype. You’re both: my super-ego as I imagine you to be. An idealised version of myself that I can never achieve.”

“You’re making a mistake - can’t you see that?”

“The mistake I make is in suppressing my realities. I *need* my fear of failure - it keeps me from making stupid decisions.”

My super-ego is noticeably shorter now - and younger.

“I’m just happy to eat the crumbs from your table.” responds homeless-me. Lustful-me (complete with blanket-woman) is already gorging himself on food.

"Oh shut up!" retorts my super-ego.

"I'm the sum total of all my fears and dreams - positive and negative."

"Well, I know when I'm not wanted. I guess you want to go and shoot myself then!"

"That's where you're wrong. See the empty seat at the table? Well, that's yours. I do need to be reminded from time to time that I have responsibilities and obligations outside of myself. However, I also have an obligation to myself as well."

Almost with tears in his eyes my super-ego sits down. The next time I look at him I see myself - my twelve-year old child self. I realize I am being judged by the expectations that I made of myself before I became an adult. He looks up at me and says "You won't win. I told you, you'll forget all about this within thirty minutes."

I smile back and say "Not if I write this down as soon as I awake."

As the scene fades to wakefulness, I enjoy watching all of my archetypes arguing with each other over Christmas dinner.

"Time to write." I say to myself.

Why is Backup so hard?

Performing a backup never used to be hard, in fact, it used to be the easiest job a sysadmin could perform. So, the ever present question many sysadmins end up asking is: Why is something that used to be so easy, now so difficult? The next equally valid question is:

Why is it also so EXPENSIVE?

As with most things, we are dealing with a history lesson.

In the early days of mainframes, you dealt with a monolithic system. Everything (and I mean everything!) came from a single vendor - and that included a backup system.

How the backup system actually worked was never much of a question. It just worked OOTB (Out Of The Box). Despite this, there was a reasonable degree of busy work involved. In fact, the primary job of the sysadmin was backups.

Consider the situation: You have several rows of IBM 72x TBUs all purring away madly. Now the tape software may know what tape it is writing to, but these are reel-reel tapes. There are no barcode labels to be read and no link between the media header on the tape and what you write on the tape (and it's box). You have to be super organised. A bell (literally a bell!) will go off on the operator console, you read the message to remove tape xyz01 from TBU 23 and replace it with tape qwe12 etc. So you run around like a blue-assed fly to make sure the tapes are properly stored in sequence and you know where all the catalog tapes are just in case the storage decides to die - a not uncommon occurrence.

In fact, in days of very expensive storage, tape was considered the mainstay. Several units were often dedicated for "offline" and "standby" storage. The only difference between the two was that standby storage was on tapes that hadn't been removed yet. Often the TBUs would double up - standby during the day and backup operation at night. Tape was cheap, disk was expensive.

The popularity of UNIX destroyed proprietary backup systems. On a UNIX system, a TBU was just another device to be written to or read from. The 'tar' utility (Tape ARchiver) turned one an entire subdirectory structure into a single file for writing (or reading from) a tape device. For raw block level backups of a disk volume, "dump" and "restore" were popular from early days of UNIX. All of these commands were frequently used in script. Sysadmins developed their own scripts and cronjobs. Usually, very little effort was put into documenting these scripts. The sysadmin 'knew' what was going on - and that was all that mattered.

As the desktop computer began pushing into the business computer market, initially little thought was put towards backups. You made copies of floppies and if you were lucky enough to have a hard disk drive, you just manually copied files onto floppy disk. Some early backup systems began to appear like Fastback which offered full duplex support and compression.

As the size of desktop computer HDDs increased, the ability to 'back up to floppy' dropped off considerably. Once again, proprietary backup systems began to appear. One of these was the 4mm Colorado Jumbo, which was an inexpensive TBU that connected to the FDD interface - meaning a separate controller did not need to be purchased. Colorado bundled some backup software - but it wasn't particularly good.

Around this time, Novell Netware 2 and later Netware 3.12 was increasing in popularity. Vendors would sell Netware 5 user version and install it on a glorified workstation. It was common to install either a Colorado Jumbo TBU or a more expensive QIC TBU. The latter would require a SCSI card. 

However, Netware didn't have a native backup software.

About the only backup software that was available for Netware was ArcServe. 

It was horrible.

Sure, it wasn't too bad at backing up your files. It just a hard time restoring them. It also had a habit of crashing servers. On Netware this was called an ABEND. Pretty soon, people the idea of setting up a dedicated backup server, so if the backup server crashed, it wasn't too bad. People also found that particular SCSI cards and TBUs caused Arcserve to be more stable than others. Particular servers were also kinder. Unfortunately, this "stability" came at a price.

Arcserve wasn't too expensive - which was its only real saving grace. The other was the complete lack of any alternative. When Windows NT came out, Arcserve made a version for it - which managed to port all the same stability problems to the new platform.

Then came Backup Exec. It was a breath of fresh air! Simple. Stable. Affordable.

Okay, it was more expensive, but you didn't need a dedicated backup server. In fact, there was a cut down version that came with a SCSI card and a Colorado Travan TBU. It all worked out of the box for under $500! Overnight, pretty much everyone switched to Backup Exec.

Successive versions added more options: open file, DR, Groupwise, BTree, NDS, Windows, Unix, Linux. However the stability gradually fell away. It was still good, but the dedicated backup server was resurrected. The price increased too. It doubled, then tripled. Soon, the best thing you could really say about Backup Exec was that it was better than ArcServe - which by now had been purchased by Computer Associates and was called ArcserveIT.

Veritas (who owned Backup exec) spawned another backup product very similar to Backup Exec - Netbackup. It became the standard for heterogeneous backup. To celebrate this, they added a zero on the price tag.

Netbackup Management Console

Then came a sequence of enterprise backup products. They were all better than Backup Exec. More stable. Heterogeneous. Great support. Agents for everything you could imagine. Policy based. But by this time, they cost in the tens of thousands of dollars.

They were Syncsort, CommVault, HP Data Protector, Portlock. No sooner did one product come out, but another did that was better with extra features. 

Then the game changer of them all came out: Veeam.

It was like Backup Exec all over again. Relatively cheap. designed primarily for virtual machines, it could do what all the others struggled with: Restore a complete working server in minimal time. 

Pretty soon, Veeam became the dominant backup software. It was a little feature poor at first, but you could do cloud based backup - meaning you didn't need to buy expensive tapes or TBUs. Veeam charged very little for cloud storage.

Slowly, as successive versions of Veeam came out, features were added. However costs began to go up again and reliability and stability began to drop. Veeam also started increasing the cost of cloud restore operations - so whilst backing up to the cloud was cheap, restoring from it cost a fortune!

Now, organisations using Veeam are casting around for alternatives. There's  EMC Networker, Altaro, Nakivo plus there are appliance based systems like Datto that work on a different paradigm. They all have one thing in common:

They are very expensive.

The goal seems to be to create hassle free simple backups. Over time, software companies sem to forget that and overload their software with Netfeatures few people use.

It seems to me the solution is to create a two tier system: the simple software that always works and the heterogenous one that deals with all of the weird and wonderful situations. 

That sounds like a simple solution, unfortunately it's always the low cost product that makes the profits and drives innovation. And nobody wants to run two pieces of software. This means the larger business that pay the most for your product, don't use the cheaper versions. When Backup Exec came out, all the large organisations stuck with ArcServe. The same is true of Veeam. Netbackup continues to hold sway in larger organisations. A quick persual of the Netbackup support forums gives the distinct impression that those who administer NetBackup have only one job to do: Backup support. If that's your only job, you don't really care that much if it's difficult to administer and requires arcane knowledge and sophisticated scripting skills. It also doesn't matter that it's hyper-expensive. In fact, that expense is a good thing as it masks you salary in the TCO.

So, I fear history is doomed to continually repeat itself. Think about that the next time a new piece of backup software appears on the market that seems too good to be true.

Critical Thinking: Why facebook & twitter are bad places for asymmetric discussions

"The beauty of concision, you know, saying a couple sentences between two commercials, the beauty of that is you can only repeat conventional thoughts. Suppose I go on Nightline, whatever it is, two minutes, and I say Gaddafi is a terrorist, Khomeini is a murderer etcetera etcetera...I don't need any evidence, everyone just nods. On the other hand, suppose you're saying something that isn't just regurgitating conventional pieties, suppose you say something that's the least bit unexpected or controversial, people will quite reasonably expect to know what you mean. If you said that you'd better have a reason, better have some evidence. You can't give evidence if you're stuck with concision. That's the genius of this structural constraint."
    - Noam Chomsky 'Manufacturing Consent"

Quite recently I was reported on Facebook for "Hate Speech". Since it was my first ever report, there was no consequence, otherwise I would have been in Facebook gaol for a period of time and unable to post or make comment. Interestingly there was a 'dispute' button I could press. I did so, and was informed less than four hours later that my post and been reviewed and the decision upheld.

Justice is quick on Facebook, apparently.

Facebook has very clear standards on what it considers 'Hate Speech':

 

Facebook's Community Standards go into further detail. I'm not complaining about this. Facebook is a private company. They have been repeatedly determined to be a publisher and not a common carrier - making them liable to content. As such it is perfectly just and fair they be allowed to restrict what is and is not content on their site. 

 

The "Community Standards" are reasonable, however for some reason, they take a different position in their Hard Questions section tacitly questioning the 'boundaries' of free speech. In doing so, they contradict their initial position, replacing the initial fairly strict definition of "hate speech" and introduce a new concept of Dangerous Speech and equate hate speech with violence. This is a very disturbing extension. They justify this by claiming that free speech is under academic debate and there "is very important academic work in this area that we follow closely". This completely ignores the fact that the free speech debate dates back to the 5th and 6th centuries and was fully hammered out a couple of centuries ago by eminent English philosophers John Milton (1608–74), John Locke (1632–1704) and in France by Denis Diderot, Baron d'Holbach and Claude Adrien Helvétius. John Stuart Mill (1806–1873) seminal work On Liberty, published in 1859 became a classic defence of the right to freedom of expression. In her biography of Voltaire, Evelyn Beatrice Hall coined the following sentence to illustrate Voltaire's beliefs: "I disapprove of what you say, but I will defend to the death your right to say it."

 

The standard definition of free speech is perhaps most simply defined by Noam Chomsky: 

"If you're in favor of freedom of speech, that means you're in favor of freedom of speech precisely for views you despise."

So influential was the work of Locke that it was used as the inspiration for the United States constitution. 

 

The driving force behind the philosophers of the renaissance and their analysis of free speech and freedom of expression was the invention of the printing press. As the "menace" of printing spread, more governments attempted to centralize control. The French Crown repressed printing and the printer Etienne Dolet was burned at the stake in 1546. In 1557 the British Crown thought to stem the flow of seditious and heretical books by chartering the Stationers' Company. The right to print was limited to the members of that guild, and thirty years later the Star Chamber was chartered to curtail the "greate enormities and abuses" of "dyvers contentyous and disorderlye persons professinge the arte or mystere of pryntinge or selling of books." The right to print was restricted to two universities and to the 21 existing printers in London. Confrontation with authority made printers radical and rebellious, with 800 authors, printers and book dealers being incarcerated in the Bastille in Paris before it was stormed in 1789. (Wikipedia)

Indeed, whenever there is technological change that tends to extend the boundaries of communication, it is opposed by Governments and fascistic elements. The current situation is no different than any other. So called "new" arguments are trotted out relying on the generally ignorance of the previous debates and conclusions. Facebook's musings fit this category as does the Dangerous Speech Project - which is essentially an argument for censorship.

Another complicating factor is the existence of various defamation laws that exist in jurisdictions around the world. No two jurisdictions seem to be able to agree on what is or what isn't defamation, however the basic components involve the competition between a person's right to their reputation not being harmed and the public right to know. Nearly all countries regard defamation (and I'm including libel here) as a civil matter, however in Singapore and Queensland (yep!) it's part of the criminal code. In most cases, absolute truth is NOT a sufficient defence. 

However, Facebook can't have it both ways: It's not a common carrier, so it is subject to defamation and other claims. As a private company, it has both a right and a duty to protect itself from both civil and criminal prosecutions arising from hosted content. This gives it a right to take down content that may be damaging or even content it simply doesn't like. The same applies to all social media companies. But don't pretend this has anything to do with free speech. It doesn't. It never did. And also don't pretend you are trying to work out the "limits" of free speech, because there aren't any limits on freedom of expression. There's free speech, and there's censorship. You may like the censorship, but that doesn't make you a free speech advocate. 

Since the last Facebook infringement I've received two more. The first one was for 'hate speech' when I ironically quoted "Arbeit Macht Frei" in response to a discussion on limiting free speech. The phrase is German for "Work makes you free" and adorned the gates of Auschwitz. 

The warning gave me the option of accepting that I had written hate speech (without any penalty) or appealing. If the appeal failed, then I would be banned for one day.

Since even a casual reading of the context would reveal I hadn't written anything objectionable, I clicked the appeal button. The appeal was instantly denied and I received a one day Facebook ban! Obviously no one had time to actually look at the thread. No human being had actually been involved in the appeal process. It was a sham. The veneer of oversight.

Just recently, I received a third warning: this time for 'bullying and harassment'. No appeal available. No content was provided at all. I don't even know what it was for and received a three day ban. The only thing I can think it was for is that I asked for evidence for a claim that a particular type of nuclear reactor produced waste that lasts for 10,000 years (hint: it doesn't). So apparently, asking for someone to provide evidence for claims made is bullying - depending upon what side of the argument you are on.

And this is where it becomes asymmetric. As soon as you start to split hairs on an issue, you are subject to bias. When only one person is deciding what is and isn't "free speech", this is inevitable. History has shown this. Yet Facebook has chosen to repeat the mistakes of history by introducing its own "star chamber". 

Being in Facebook gaol is serious business now. You can't administer or moderate any pages or groups. Your business may suffer. Although Facebook is not legally a common carrier, it may as well be. For a business, being locked out of your business page can be as damaging as having your phone cut off.

What came out of the renaissance was that the best antidote for any 'dangerous' or 'hate' speech was sunlight: Allow those expressing their views to air them freely - and then debate them. Free speech allows you to say whatever you like, but it doesn't protect you from criticism and debate. The consensus was that the only ideas that needed protection were those ideas that could not withstand rigorous scrutiny. This is why free speech is the enemy of the rich and powerful. Not only does the emperor have no clothes, but he will throw you in gaol for stating it.

A growing number of Facebook users are complaining that Facebook and Twitter are exercising their power to block and ban to further a particular political agenda. Both of these scoial media companies seem to be using the concept of 'hate speech' to cover any ideological they find objectionable.

So, this leads us to one of two possible scenarios:

1) They are ignorant of the history surrounding the development of free speech and are blindly repeating past mistakes.

2) They are fully aware of what they are doing and are following fascistic tendencies to suppress ideas they despise.

Installing Apache on Windows Server 2016

Occasionally, someone will want something that runs best on Linux to be installed on a Windows server. There are various reasons for this, but usually it comes down to "I'm not comfortable with Linux, I want this to run on Windows". Regardless of the reason, if that's what the customer wants, that's what the customer gets.

Just to be clear this point, I'm referring to a WAMP server (Windows Apache MySQL PHP) as close as possible to a 'standard' LAMP server (Linux Apache MySQL PHP) that you'll get from the repo of your favourite distro.

That being said, the LAMP server you'll get from RHEL/CentOS will be significantly different from the one you get from Ubuntu.

When it comes to running Apache on Windows, for strange reasons, that simple decision makes the world a whole lot more complex. You see in Linux, when you decide to run Apache, pretty much all the decisions of 'how' have been made for you by the distribution you are using. And unless you decide to roll-your-own - which most people don't do AND increasingly this is a bad thing to do - they are the decisions you now deal with.

Under Windows, that's not the case. There are more ways to run Apache under Windows than you can poke a stick it at. (If you want to get pedantic, you can say there are an infinite number of ways, but don't go there). A lot of these methods are excluded by the "why" you are running Apache, we'll look at a few of these.

The default way is to download Apache from Apache.org and go from there. That's a lot of work and leads to a lot of stuff being left out.

However, if you want to run Apache as bog standard as possible, without having to do too much low level work, Apache Lounge is a good way to go and the one I'll be detailing here.

Procedure

Firstly download the binaries from here. Choose the appropriate version (generally 64 bit) and unzip the archive. While you're waiting for it to download, get the Windows c++ runtime redistributable that is appropriate. Apache lounge is written in c++ and requires this in order to run. The Apache Lounge readme and download page will show you which version is necessary - which is currently 2017. Later versions will (may) work, but it's best to get the version the application was written for. Earlier versions will not work.

Apache Lounge is easier to work with than the official Apache distribution. It also 'feels' more flexible. Try both and you'll see what I mean. What I really like is being uncoupled from the "program files" paradigm of Windows. I personally prefer to have both the binaries and the data separate from the C: drive, so that's the assumption we'll be using here.

Next extract the zip files and copy the Apache24 directory structure to D:\apache24. It may be a good idea to change the folder preferences for full visibility including hidden files and showing the filename extension.

As an aside here, it is also a good idea to prep your server to have the following utilities installed and available:

 - Chrome browser (or firefox, safari, opera etc)
 - putty
 - winscp
 - telnet client
 - tftp client

Edit the D:\apache24\conf\httpd.conf file to add/edit lines as follows (in most cases this is done by looking for the near identical line and either editing it or remove the '#':

Define SRVROOT "D:/Apache24"
Listen *:80

LoadModule rewrite_module modules/mod_rewrite.so

ServerAdmin <youremailaddress> 

ServerName <your servername or localhost>:80

DocumentRoot "D:/webroot" 

AllowOverride All  

<Directory "D:/webroot"> 

Check the config file carefully. Work here will pay off later.
Change to the bin directory and run the following:

httpd -t

If the configuration is fine, check that ports 80 and 443 are not in use by running:

netstat -ao | findstr ":80"
netstat -ao | findstr ":443"

Most likely you'll find port 80 in use. The last number is the process ID of the service tying up the port. Identify the service from the process ID using Task Manager and disable that service. If the PID is 4, then this is the system service http.sys and it becomes annoying and potentially a problem....

Under later versions of Windows Server, some processes communicate using http via port 80. The purpose of http.sys is to redirect that data to the appropriate service. You can test this by using the telnet client to telnet to 127.0.0.1 80 and pressing return twice. You should get something like the following as output:

What this means is that one or more native windows services are dependent upon the http.sys service to operate. You have to decide if you can live without these services. If not, then you need to select a different port for Apache to use for http traffic or use it for https only. If this is the case, edit the config file and change the listening port from 80 to something else (for now). We can remove http and make the server https only later on.

If you can live without the services, firstly stop the http.sys service using

net stop http

You will get something like the following:

Answer 'y' and stop the services. You can (and should) script this process by taking not of all the services that need to be stopped and explicitly stopping them. You will likely have to do this frequently.

MySQL/MariaDB

Download MySQL installer and MySQL workbench

(From https://seravo.fi/2015/10-reasons-to-migrate-to-mariadb-if-still-using-mysql) 

The original MySQL was created by a Finnish/Swedish company, MySQL AB, founded by David Axmark, Allan Larsson and Michael ”Monty” Widenius. The first version of MySQL appeared in 1995. It was initially created for personal usage but in a few years evolved into a enterprise grade database and it became the worlds most popular open source relational database software – and it still is. In January 2008, Sun Microsystems bought MySQL for $1 billion. Soon after, Oracle acquired all of Sun Microsystems after getting approval from the European Commission in late 2009, which initially stopped the transaction due to concerns that such a merger would harm the database markets as MySQL was the main competitor of Oracle’s database product.
 

Out of distrust in Oracle stewardship of MySQL, the original developers of MySQL forked it and created MariaDB in 2009. As time passed, MariaDB replaced MySQL in many places

 https://downloads.mariadb.org/

Current is 10.3.15 stable. Download x64 msi package.

Installation of MySQL

Download MySQL Workbench. Trust me: You need this!

 https://dev.mysql.com/downloads/workbench/
 

Current version 8.0.16 Requires

Microsoft .NET Framework 4.5
Visual C++ Redistributable for Visual Studio 2015

MySQL Community Server

https://dev.mysql.com/downloads/windows/

The full installer includes all windows packages and is a 373MB download. The installer is 32 bit but it includes both 32 and 64 bit binaries.

The installer does not permit upgrades between major and minor version numbers - these are installed as separate server instances - but it will perform upgrades within a release series eg: 5.7.18 to 5.7.19.

The installer also requires .NET Framework 4.5.2 or later.

Post installation testing of MySQL/MariaDB

The service should start immediately after installation. Check the MySQL service is running.

Check you can login from the command line using the root user and password generated before. 

Installing PHP

(From https://www.php.net/manual/en/install.windows.php)

 

PHP 5.5+ require at least Windows 2008/Vista, or 2008r2, 2012, 2012r2, 2016 or 7, 8, 8.1, 10. Either 32-Bit or 64-bit (aka X86 or X64. PHP does not run on Windows RT/WOA/ARM). As of PHP 7.2.0 Windows 2008 and Vista are no longer supported.

 

PHP requires the Visual C runtime(CRT). Many applications require that so it may already be installed. 

 

PHP 5.5 and 5.6 require VC CRT 11 (Visual Studio 2012). See: » https://www.microsoft.com/en-us/download/details.aspx?id=30679

 

PHP 7.0 and 7.1 require VC CRT 14 (Visual Studio 2015). PHP 7.2 and 7.3 require VC CRT 15 (Visual Studio 2017). PHP 7.4+ requires VC CRT 16 (Visual Studio 2019). 

 

The Microsoft Visual C++ Redistributable for Visual Studio 2019 is suitable for all these PHP versions, see » https://visualstudio.microsoft.com/downloads/. 

So we will be installing 7.3 from https://windows.php.net/download

Which version do I choose?

IIS

If you are using PHP as FastCGI with IIS you should use the Non-Thread Safe (NTS) versions of PHP.

Apache

Please use the Apache builds provided by Apache Lounge. They provide VC14 and VC15 builds of Apache for x86 and x64. We use their binaries to build the Apache SAPIs.
With Apache you have to use the Thread Safe (TS) versions of PHP.

VC14 & VC15

More recent versions of PHP are built with VC14 or VC15 (Visual Studio 2015 or 2017 compiler respectively) and include improvements in performance and stability.
- The VC14 builds require to have the Visual C++ Redistributable for Visual Studio 2015 x86 or x64 installed
- The VC15 builds require to have the Visual C++ Redistributable for Visual Studio 2017 x64 or x86 installed

TS and NTS

TS refers to multithread capable builds. NTS refers to single thread only builds. Use case for TS binaries involves interaction with a multithreaded SAPI and PHP loaded as a module into a web server. For NTS binaries the widespread use case is interaction with a web server through the FastCGI protocol, utilizing no multithreading (but also for example CLI).

What is PGO?

Profile Guided Optimization is an optimization feature available in Microsoft's Visual C++ compiler that allows you to optimize an output file based on profiling data collected during test runs of the application or module.

Links:

• PGO on MSDN
• Visual C++ Team Blog - PGO with PHP
• PHP PGO build for maximum performance (old)

amd64 (x86_64) Builds

PHP 7 provides full 64-bit support. The x64 builds of PHP 7 support native 64-bit integers, LFS, 64-bit memory_limit and much more.

  copy to D:\php7

  copy php.ini

  edit php.ini

Add C:\php to the path environment variable

To ensure Windows can find PHP, you need to change the path environment variable. Open Settings, type ‘environment variables’ into the search field and open the result. Select the “Advanced” tab, and click the “Environment Variables” button.
Scroll down the System variables list and click on “Path” followed by the “Edit” button. Click “Edit text” and add ;C:\php to the end of the Variable value line (remember the semicolon).

 edit httpd.conf to add index.php to the DirectoryIndex variable

At the bottom of the file, add the following lines (change the PHP file locations if necessary):

# PHP5 module
LoadModule php5_module "c:/php/php5apache2_2.dll"
AddType application/x-httpd-php .php
PHPIniDir "C:/php" 

Save the configuration file and test it from the command line (Start > Run > cmd):

cd Apache2bin
httpd -t

PECL extensions are pre-built for Windows and available from: » http://windows.php.net/downloads/pecl/releases/

 

Some extensions use features specific to some Unix systems and so are not available on Windows. Otherwise, all extensions are available for Windows. 

PEAR
Open a command prompt with administrator privileges. Use curl to download the pear install file into the php7 directory

curl -OL http://pear.php.net/go-pear.phar

If you use a browser to download the file, it renders as text but with binary characters embedded. This will stop the script from working, hence the need to use curl.

set PHP_PEAR_SYSCONF_DIR=D:\php7

php -d phar.require_hash=0 go-pear.phar

You will then be prompted with a few questions. If the defaults don't work, you probably haven't elevated the privileges of the command prompt. If option 12 lists the Windows path, you haven't set the PHP_PEAR_SYSCONF_DIR variable. Editing here doesn't work, which is why you need to set the variable. You will need to enter the path to cli.exe- in our case this is d:\php7.

Note: If any of the directories already exists, you may get a 'permission denied' error for that directory. In which case, delete the directory and run the script again.

The rest of the defaults should be okay, but check them anyway. Once finished, a reg file called PEAR_ENV.reg will be created. Execute that to add the necessary keys to the registry.

Finally, check that PEAR is working using pear version:

Conclusion

Now, hopefully you read all this through before attempting it and are utterly confused by the process. If you, that's great! No go and install Linux and deploy Apache that way. Trust me! It's much easier and safer. Plus when you get into trouble and need to troubleshooting, pretty much any article you'll find will assume you are using Linux.


The Sony Vaio VGN and Linux

For quite some time now, my preferred desktop OS has been Ubuntu. I made the switch from Fedora after a bad experience with graphics drivers between Core 5 & 6 and decided that Fedora was just a little too cutting edge for me. This was around when I had been playing around with Ubuntu for client desktops and warming to with each new version. For me, the transition to a Debian base system was uncomfortable and I avoided it for a while. The crunch came in early 2007 when I received a Sony Vaio VGN series "hip-top" computer. At the time it was pretty stunning and Sony were pushing them hard - which is probably why they were giving them away to guys like me: so that we would show them off to our clients. In the end, tablets killed off this technology avenue.

Sony Vaio VGN series

I was pretty impressed with the VGN, mainly because it was loaded with peripherals and came with a great docking station. Of course, the first task was to give Windows the flick and install Linux.

With Fedora, almost nothing worked, so I blew that away and installed SuSe. This was much better and most things "worked". The problem was the peculiar UI - which for a touch screen was difficult to work with. Changing the UI on SLED was a problem and it felt a little too "locked down" for my liking. It became painful, so it had to go as well.

Ubuntu seemed to be about halfway there. Some of the special buttons would not work, but I could live without them. It fitted with my purposes at the time, so I was off and running with Ubuntu. It didn't take too long before I preferred it to Fedora.

One of the things I used to dislike about Ubuntu was the default Gnome interface - I preferred KDE. Yes, I know it's also the default for Fedora, but switching interfaces with Fedora was simple. For the Vaio, I chose to install Kubuntu; which at the time was the specific distro for Ubuntu users with KDE. Later I decided to switch back to Gnome (when Gnome updated)and that proved to be a superior experience. The ever present issue was always screen resolution - which never really "fit" the screen. 

I persevered for a few years with the Vaio - mainly for it's 'wow' factor - until I bought a Asus Transformer T300 tablet with keyboard and after that I almost immediately stopped using the Vaio. Despite the tablet being far less powerful, in had a 10" screen and a real keyboard. To make the Vaio usable, I had to become "walking tech" by carrying around a folding keyboard, port replicator, external HDD and a wireless mouse - plus the charger. I ended up selling the Vaio to a colleague for $50.