There I was, minding my own business at the Google party after the third day of the 2007 Linux Conference Australia, sitting at a table by myself when I hear a voice say "Do you mind if we sit here?"
I look up and see Linus there with his wife Tove and reply "No worries, go right ahead."
What followed was some awkward silence, interspersed by the occasional exchange in Finnish between Linus and Tove. I valiantly attempted a conversation.
"Have you been doing much kernel maintenance lately?" (It sounded bad as soon as I uttered it)
"Oh no, not much. I try to leave that to others."
Try again "Will you be giving many lectures while you are here?"
His reply "Just my introduction to Andrew Tanenbaum's keynote address tomorrow. I really hate public speaking, I only do it when I have to."
I figured I may as well dive into the specific issues I had with Linux at the time.
"So, how do you feel Linux is going to handle enterprise deployment issues?"
Linus seemed genuinely surprised, as though he didn't understand the question - or simply the need for it. I tried to elaborate.
"The problem I have with convincing clients to use Linux is the lack of enterprise tools available - the ability to image workstations and servers, manage desktops, deploy applications and printers etc."
Linus scoffed at these requirements, but I pushed on "These things are the reasons why Linux has over 50% of the web server marker, but less than a 1% penetration into the corporate desktop."
Linus' answer was that if there was really a demand these tools, then there would be open source projects for them.
To me, that was cart before the horse stuff and I said so. Also the lack of a reliable network file sharing protocol (nfs is not up to snuff and neither is samba) to which he asked "Well, what do you use?"
"NCP - Netware Core Protocol" I replied.
"Netware!? That's a dead operating system. No one uses it anymore."
"That's not true, all of my clients use it. The Queensland Gov't uses it. In fact, the larger the organisation, the more likely they are to use it. It's strength is its scalability - which is far superior to Windows and currently lacking in Linux."
"When was the last time you installed a new Netware server?" He scoffed.
"This month: Two installations in fact. But anyway, NCP and eDirectory run on either Netware or SLES."
At that point he asked if I worked for Novell and when I said no, he still checked my conference badge. Once he confirmed I wasn't one of the Novell apparatchik he proceeded to rip on Netware and Novell in general. He had a couple of good points that I (and just about every other Netware engineer) agreed with - such as the increasing problem of driver support for closed source operating systems.
However, then it turned kinda personal - almost pityingly personal actually. In the world according to Linus: I was a dinosaur, a relic of a bygone era reminiscent of the last telegraph operator or a septuagenarian steam train driver.
This wasn't at all what I expected. I'd hoped for a robust discussion; some thoughtful insights and maybe the occasional "Hmmm, interesting point." Instead my treasured discussion points may as well have been heretically nailed to a church door in the presence of Tomas de Torquemada. I think I would have had a fairer hearing.
I excused myself and tried to enjoy the party that Google threw for us geeks before Linus could burn me any further. I found it difficult to enjoy the festivities and left early for my conference domicile and pondered the conversation before I went to sleep. This is always a bad idea for me as I tend have strange dreams - really strange dreams that would delight any Jungian psychotherapist. Tonight was not going to be an exception.
I found myself at the Battle of Zama as Hannibal commanding the Carthaginian Army against the Roman forces under Scipio Africanus. I know this battle well and deeply despaired. Historically, this was Hannibal's last battle and a major victory for Scipio. I looked at the Carthaginian Army and noticed they were all wearing red with the Novell logo. I rode forward to parlay with Scipio who was bearing the banner of a penguin. When Scipio removed his helm - there was Linus' face staring back at me with that same pitying smile he gave me at the party.
"We seek terms for surrender." I said
"There will be no surrender. After today Carthage will be only a memory." He replied without changing his facial expression. Linus replaced his helm and we both reformed the lines.
"Your orders?" asked my trusted aide. I thought for a while before replying.
"Take out your sword and cut off my head."
And then without any hesitation, he complied.
Wednesday 17 July 2013
Monday 15 July 2013
How to know if you are a bad sysadmin
Just about every sysadmin I have met have one thing in common: they all think they are awesome at their job. However, rarely (well IMHO anyway) is this correct. Most will be quite offended if I have something to say to them about their ineptness. So I have created a short self-evaluation questionnaire that you can use to find out just how bad (or good) you are at your job. If you are a "good" sysadmin, you should score very low on this questionnaire and at least recognise the issues that exist. If you don't understand the reasons for these questions: It's time to either take steps to remedy these issues or do the rest of us a favour and leave the industry forever.
1. Do I ever have to ask my users for their passwords?
No sysadmin should ever need to know a users password. You should have procedures in place and methodologies and/or technologies to ensure this is not necessary - ever.
A corollary to this is that password resets should be immediately be followed by a forced password expiration. All end-user passwords should be rotated at regular intervals with duplicates not allowed.
2. Do I ever use the enterprise Administrator password?
The administrator or root password should never been used except on standalone systems. All administrators should have their own administrator password separate from their usual login. An extension to this is that all administrative activity should be logged.
3. Do I physically have to go to a users workstation?
For anything other than doing physical work, this should be unnecessary. You should actually have more capability through remote access than sitting at their desk.
4. Do I never conduct trial restores from Backup?
Just because your backup software says "successful backup" it does not necessarily follow you will be able to restore data from it. Check regularly, so you become familiar with the process. At least once every six months do a complete trial disaster recovery for one of your servers. Time yourself, try to beat that time.
5. Do I have to manually setup workstations for new users?
A new user should need only login with their password to get:
- All their software
- Their drive mappings
- Their printers
There is simply no need for a sysadmin to get involved in this process. It should all be automated.
6. Do I use statically mapped drives?
This should never be required. Scripting should take care of all contingencies.
7. Do I use user-based file system rights/permissions?
These are close to impossible to administer. If you have user-based permissions for anything beyond services and home directories, chances are your file system security is non-existent.
8. Do I allow direct access to the Internet?
This is a serious security issue. Access to the internet for ANY protocol should be via the DMZ using proxied or relayed access. No exceptions.
9. Do I use 'Same as xxx user' when creating new user accounts?
This is not only lazy it is insecure and leads to non-repeatable actions. The new accounts will usually have way too many permissions - many of which you will be unable to explain the reason for.
10. Am I unable to name any new technology that I have trialled in the last six months?
Good sysadmins spend significant time on system development. This includes trialling new technologies as they are released to determine the relevance in your environment.
1. Do I ever have to ask my users for their passwords?
No sysadmin should ever need to know a users password. You should have procedures in place and methodologies and/or technologies to ensure this is not necessary - ever.
A corollary to this is that password resets should be immediately be followed by a forced password expiration. All end-user passwords should be rotated at regular intervals with duplicates not allowed.
2. Do I ever use the enterprise Administrator password?
The administrator or root password should never been used except on standalone systems. All administrators should have their own administrator password separate from their usual login. An extension to this is that all administrative activity should be logged.
3. Do I physically have to go to a users workstation?
For anything other than doing physical work, this should be unnecessary. You should actually have more capability through remote access than sitting at their desk.
4. Do I never conduct trial restores from Backup?
Just because your backup software says "successful backup" it does not necessarily follow you will be able to restore data from it. Check regularly, so you become familiar with the process. At least once every six months do a complete trial disaster recovery for one of your servers. Time yourself, try to beat that time.
5. Do I have to manually setup workstations for new users?
A new user should need only login with their password to get:
- All their software
- Their drive mappings
- Their printers
There is simply no need for a sysadmin to get involved in this process. It should all be automated.
6. Do I use statically mapped drives?
This should never be required. Scripting should take care of all contingencies.
7. Do I use user-based file system rights/permissions?
These are close to impossible to administer. If you have user-based permissions for anything beyond services and home directories, chances are your file system security is non-existent.
8. Do I allow direct access to the Internet?
This is a serious security issue. Access to the internet for ANY protocol should be via the DMZ using proxied or relayed access. No exceptions.
9. Do I use 'Same as xxx user' when creating new user accounts?
This is not only lazy it is insecure and leads to non-repeatable actions. The new accounts will usually have way too many permissions - many of which you will be unable to explain the reason for.
10. Am I unable to name any new technology that I have trialled in the last six months?
Good sysadmins spend significant time on system development. This includes trialling new technologies as they are released to determine the relevance in your environment.
Enterprise File System Security
I'd have to say that very few (read 'two') organisations I've encountered implement good file system security. This is a pity because file system security is one of the most basic and easy things to get right. Many sysadmins, however, implement security for their systems on a user basis - which makes administration a function of the number of users you have.
Some time ago I established a reasonably bullet-proof filesystem security template that I have applied across NFS, CIFS/SMB and NCP file systems. The required exceptions are rare and (usually) easily managed. It does require management buy-in so you need to establish the business case very well (greater security, better administration etc). You will also need to lock-down the new user and modify user processes tightly which will require cooperation with HR. Coordination with departmental heads to ensure they have the ability to do their work unhindered by the file system security is essential.
General Notes
Never use masks or revoke permissions. The number of cases where this needs to be done is so rare it is almost non-existent. If permissions need to be revoked, simply apply zero permissions to the appropriate user or group. Trying to track down complex permissions with revocation is a nightmare - so don't do it.
With the exception of home directories, user specific directories, administrative accounts or services: Never, ever apply permissions to users. Create a group - even if that group has only one member - and apply rights/permissions to the group.
For Windows servers, apply all security via NTFS permissions and not via the shared permissions. You'll thank me for this one someday...
Also for Windows, be careful of the ACLs. There is no true inheritance in NTFS, so if you copy or move a directory structure, make sure you re-apply the permissions.
Get a handle on how the FS security works for your system. Know it backwards.
Home Directories
This is fairly simple. Only users have rights to their home directory with one possible exception (see submit). Make sure the users don't have the ability to add others to their directory or sub-directory. For Windows sysadmins, this means giving users 'Modify' instead of 'Full' permissions.
Departmental Directories
In general, you will want all members of the department to have R/O access to this tree. A group will need to be created for those that have write access. Beyond this, apply extra permissions at the lowest level only and create groups for those with extra access. Use a naming convention that allows you to know exactly by looking at the group what it is used for.
Shared Directory
These will generally a single tree with subdirectories for each department. Members of that department will have R/W access and others will have R/O access.
Temp Directory
This is to facilitate the temporary transfer of files. All users have R/W access and the directory has size restrictions on it. The entire structure is deleted every weekend after backup.
Software Directory
A R/O structure containing installable software and other R/O files. This structure is rarely backed up.
Submit Directory (Optional)
This requires some clever scripting. Essentially, there is a subdirectory for every user. Everyone has W/O access to these directories. They can copy files there, but not see or read the contents. Once copied, a script runs which moves the submitted files to the users Home Directory/submit and an email is sent notifying the user of the file, its location and who copied it there.
Additional FS Security
For anything not covered, create a group or groups with the necessary permissions. Make sure your naming conventions makes this fairly self-explanatory. Ensure that these group memberships are part of the user account creation process, which should NEVER have the statement 'Same as xxx' - this is simply bad administration.
Administration
Once you have settled upon the appropriate rights/permissions, tested them etc. Dump them to a text file and check them carefully. With the exception of home directories, write a script which will re-apply these permissions, deleting any others and have it run every night. When administering, modify the text file (or create a new one). By doing this, if someone with administrator access fools around and adds permissions (or removes them), all will be put back to rights overnight. As an extension, dump the rights/permissions before modification and compare them using diff with the original. Send an email alert out if there is a difference.
Followed properly, all you should need to do to alter a users permissions is to change their group membership. The number of times you will actually need to add more permissions will be small and now you can make them subject to a change request rather than a service request.
Enforcement
Executables should not exist in home directories or departmental directories. It is a simple matter to write a script to quarantine executable files (or make them non-executable) on a daily basis. If executables need to be shared, a change request should be submitted to the IT Department.
This is just a small but crucial part of best practice administration. It is simply better to have a total handle on your file system security.
Some time ago I established a reasonably bullet-proof filesystem security template that I have applied across NFS, CIFS/SMB and NCP file systems. The required exceptions are rare and (usually) easily managed. It does require management buy-in so you need to establish the business case very well (greater security, better administration etc). You will also need to lock-down the new user and modify user processes tightly which will require cooperation with HR. Coordination with departmental heads to ensure they have the ability to do their work unhindered by the file system security is essential.
General Notes
Never use masks or revoke permissions. The number of cases where this needs to be done is so rare it is almost non-existent. If permissions need to be revoked, simply apply zero permissions to the appropriate user or group. Trying to track down complex permissions with revocation is a nightmare - so don't do it.
With the exception of home directories, user specific directories, administrative accounts or services: Never, ever apply permissions to users. Create a group - even if that group has only one member - and apply rights/permissions to the group.
For Windows servers, apply all security via NTFS permissions and not via the shared permissions. You'll thank me for this one someday...
Also for Windows, be careful of the ACLs. There is no true inheritance in NTFS, so if you copy or move a directory structure, make sure you re-apply the permissions.
Get a handle on how the FS security works for your system. Know it backwards.
Home Directories
This is fairly simple. Only users have rights to their home directory with one possible exception (see submit). Make sure the users don't have the ability to add others to their directory or sub-directory. For Windows sysadmins, this means giving users 'Modify' instead of 'Full' permissions.
Departmental Directories
In general, you will want all members of the department to have R/O access to this tree. A group will need to be created for those that have write access. Beyond this, apply extra permissions at the lowest level only and create groups for those with extra access. Use a naming convention that allows you to know exactly by looking at the group what it is used for.
Shared Directory
These will generally a single tree with subdirectories for each department. Members of that department will have R/W access and others will have R/O access.
Temp Directory
This is to facilitate the temporary transfer of files. All users have R/W access and the directory has size restrictions on it. The entire structure is deleted every weekend after backup.
Software Directory
A R/O structure containing installable software and other R/O files. This structure is rarely backed up.
Submit Directory (Optional)
This requires some clever scripting. Essentially, there is a subdirectory for every user. Everyone has W/O access to these directories. They can copy files there, but not see or read the contents. Once copied, a script runs which moves the submitted files to the users Home Directory/submit and an email is sent notifying the user of the file, its location and who copied it there.
Additional FS Security
For anything not covered, create a group or groups with the necessary permissions. Make sure your naming conventions makes this fairly self-explanatory. Ensure that these group memberships are part of the user account creation process, which should NEVER have the statement 'Same as xxx' - this is simply bad administration.
Administration
Once you have settled upon the appropriate rights/permissions, tested them etc. Dump them to a text file and check them carefully. With the exception of home directories, write a script which will re-apply these permissions, deleting any others and have it run every night. When administering, modify the text file (or create a new one). By doing this, if someone with administrator access fools around and adds permissions (or removes them), all will be put back to rights overnight. As an extension, dump the rights/permissions before modification and compare them using diff with the original. Send an email alert out if there is a difference.
Followed properly, all you should need to do to alter a users permissions is to change their group membership. The number of times you will actually need to add more permissions will be small and now you can make them subject to a change request rather than a service request.
Enforcement
Executables should not exist in home directories or departmental directories. It is a simple matter to write a script to quarantine executable files (or make them non-executable) on a daily basis. If executables need to be shared, a change request should be submitted to the IT Department.
This is just a small but crucial part of best practice administration. It is simply better to have a total handle on your file system security.
Subscribe to:
Posts (Atom)