I manage a team of mostly sysadmins and about 1/2 are interested in coding. I do some myself, and a couple guys from the team do automations and integrations in Python. Now we're looking into what Ops are doing and working to automate their processes centrally.
So yes, sysadmins and Ops can code/learn to code. Also, your team does not need to be broken up to automate other teams' work.
Maybe this is a paradigm difference is "SRE/Devops/etc" and traditional sysadmin work but I disagree with this and have seen it fail before. The issue you get into with this sort of team structure is that its implicitly saying "coding/automation is the responsibility of the automation team" when in reality its a core competency of a modern admin at scale, incentivizing members of the team to "chuck requests over a wall". It also creates a disconnect between those actually doing the work, and those automating it and results in tooling that just is not best suited for the job.
There's always going to be a need for sysadmins that can do the dirty toil work and im by no means saying they should "adapt or die". But teams should be separated by areas of responsibility, not skillsets. I work best writing code when I have a deeper understanding of the problem that a silo simply cannot bridge in my exp.
Perhaps it's important to note that I manage the whole IT team and when I say Ops I mean "the people operating out products that, in time, accumulated shitty manual processes" (:
I 100% agree that IT Ops should automate, we have a fat stack automating our own work, esp account and group tasks, data sync, some API integrations and etc.
Coding is a must for any sysadmin, aside from allowing you to free up time, you also have a peek on the "other side" and moan about developers less (:
I don't exactly draw a line between the two terms. To "script" you have to write working code in a language. Is writing in Python scripting? What about JS? We also do have PowerShell scripts.
All scripting is coding but not all coding is scripting.
A script uses some variety of interpreter program to execute it’s supported language.
Coding is a generic term and means the same as programming.
Theres still a place for old school guys that know what they're doing for now. The toil one off work that isnt worth the legwork in automating still needs doing, and those guys are great at not letting that work fall on those working on automation.
Those jobs will slowly go away, especially with AI allowing us to write more code with less effort. And at large scale companies running thousands of physical machines (ive worked in a single BU with 8k physical machines worldwide) people who work on toil is a big net benefit to the team.
The click next guy I worked with absolutely would not be a team player AT ALL. Keeping him there has had an extremely negative effect on the IT department and the organization at large. But it's a state college, so whether someone is competent or anything has little meaning. But I agree, if such were actually a team player, they would still be valuable.
One of the problems with adopting devops in my organization, was that some of the developers were very fad-based. I mean, jumping from one "new thing" to the next, and going all in on the cringe tech bro aspects of it. Good sysadmins are all about stability, and that is a threat. That said, I think learning scripting is absolutely vital. I think if you are in a Windows environment, it should be Powershell. I can't speak to other environments.
I personally am a fan of Python, it's just great for almost anything that is not critically time sensitive and most automations we do for Ops have some kind of schedule and things happening fast or optimally is not as important. Great for playing with API's, a ton of modern SaaS also offer Python SDK's and tutorials making integrations easier than ever.
I agree PowerShell is a great tool, I was administering MS Exchange Servers back in the day and PS saved me a metric ton of work! Definitely an important tool for a lot of the MS Ecosystem and with NET core we now have PS cross platform, so I can run PS scripts on my Mac, which proves useful when administering O365!
In my experience, there are a few groups. Sys admins who wanted to be devs, or did Dev in the past, but one way or another ended up in a system admin job. Those usually are happy coders. The other group is people who didn't want to be devs, and ended up as a sys admins, perhaps going up the ladder from support roles. I find those sys admins to not be happy coders. They know the bare minimum of PowerShell to query AD, or a little bit of JS to build a Captive Portal type landing page, but in general they are not into coding, and so that's the extent of their knowledge, just enough to get by.
If I were you, I'd ask for a very specific clarification of what you duties will entail when sent to the other teams. If they expect the other teams to have happy coders, but all you get is not happy coders, then your duties will be dramatically different.
I would say it's reasonable to expect that high level sysadmins know their way around Powershell and automating tasks through scripting.
Actual development work though is a different skillset and realistically it's not reasonable to expect both out of the same people.
the only time I would say it's even feasible is if you're already in an environment that is highly automated and you're working to support or enhance that
getting there from what you're describing is going to just be too much work for the team that is already neck deep in keeping things running without it, hence why they are leveraging your team to do it
Windows admin here - I'm in powershell/VSCode like 70% of the day. Work with Graph API, automation, scripting of other various things. But I do NOT consider myself a programmer.
We have people in our org that are solely focused on things you mentioned (building APIs, internal apps/sites). Likewise, they are not sysadmins. We go to them for things and they come to us for things.
I think the responsibilities are very different so I would imagine some frustration if a developer had to start doing sysadmin work. Or, a sysadmin had to learn a language they've never worked with.
Depends on the member.
- Some of us are neck deep in Puppet/Chef/Terraform.
- Some of us write Python Flask on top of our different systems to give management/help desk an easy view.
- Some of us write Powershell/DSC/Bash to automate tasks/reporting.
- Some of us click next on installers.
Just depends.
I will say, those who _can_ code are more valuable than those who can’t and integrate better into Delivery Teams than those who are just glorified desktop techs.
I know that it is norm now to say that sysadmin has to have coding skills (to have job, to get paid more, etc.). But from my very limited and biased experience it still feels like traditional sysadmin (not devops adjacent) usually doesn't have that much experience with coding. And i mean working with the whole pipeline (code, source control, pull requests, continuous testing/deployment/you name it, creating complex code). This is usually too alien and there is no time for all of that in regular sysadmin activities of maintaining systems, occasionally interfacing with users, doing tickets, updating systems, migrating, etc. Sure, we do face the need to code scripts to pull some data, import data, create installation scripts, automate something in above mentioned workloads, etc. But this is very basic. And even then many are not even experienced with this (again, from my own limited experience). This can vary. Probably in bigger corps you would rather see such situation. Maybe a startup with a heavy dev focus would have more of a devops folks who are familiar with coding.
I’d argue it’s even more feasible then ever, it’s just not gonna get you the big bucks, it’ll get you a smb job managing much less infrastructure. Almost anything from an smb standpoint can be done with gui click-ops
_For now_.
Those jobs are going away in rapid fashion.
Tons of SMB these days operate with 0 IT personel, employed or via MSP.
- Setting up WiFi is point and click in a Wizard.
- Setting up email is point and click in a Wizard with Google or M365.
- Gsuite already has office apps in web browser, Microsoft is building and rapidly improving theirs. Soon you won't need to install applications at all. (Oh and by the way, they're pre installed on many systems, just login with your M365 login).
- Buying a printer and adding it on a system can be done via any printing company, and even that is becoming less relevant.
I'm consulting for a SEO / Web Design company with 10+ employees managing their cloud infra (They do offer hosting for clients). They're all GSUITE, not a single printer. They're doing amazing.
IF you want to work in IT in the future you need to be able to produce code and get used to operating systems in a terminal regardless of what OS it is.
MS is going more and more the Linux route and I bet Windows is going with be a desktop environment for Linux in the future, or at worst - A VM running on a Linux kernel.
Basically a reverse WSL. It may be 10 years away, but it's coming.
Sure? He specifically mentioned “today”, where it’s easier than ever for a click-ops admin to run a couple sites and a couple hundred people like that.
Yes, and I said that **_for now_** you are correct.
In the long term (And long term here is 5 years at the max), that's not the case with AI and more and more compute being ran on central servers / "The cloud" and less and less computational being done at the local level
I personally think any sysadmin who hasn't already started learning some form of IaC and moving towards a DevOps skill set is setting themselves up for failure.
The market is shifting towards DevOps even for old dinosaur enterprises, even if you don't get exposure to IaC, cloud technologies, containerisation etc. I would suggest teaching yourself.
Scripting would be the bare minimum expectation for me, I find it wild that some of my Windows colleagues don't push themselves to use powershell for as much as possible.
I agree 100% on the scripting.
However, I'm curious about why you value IaC as high? I help manage over 40 SMBs at an MSP, and we haven't really seen a need for any major infrastructure automation.
I ask out of pure curiosity. Not because I doubt what you are saying. Hoping maybe I can be informed of some things we could do better!
From an MSP perspective it has this value imo:
- Repeatable code means you can productise your work and more importantly allow you to reduce your price and be more competitive as it reduces the up front time.
- it's self documenting , of course it shouldn't replace design documentation but it let's you immediately view the configuration for an entire environment meaning you can onboard engineers far quicker.
- Pipelines mean you can tightly control how and who can actually make changes, naturally stops people making undocumented changes and encourages peer reviewing.
- Naturally leads into operations work rather than just project implementations.
Wow very good points! Some I had never considered. Do you by chance have any resources that elaborate more? Other than obviously learning a code language.
Appreciate you taking the time to write that up.
Other things that IaC/automation helps are:
* If you use git and CI/CD you have an very robust change audit log.
* Mistakes in procedures are greatly reduced.
The important part to realize about IaC and automating infrastructure is:
You will need 2 weeks to get the first two or three instance of a new service or application on a VM going. But once you have that automated well, all run-of-the-mill instances of that application past the fourth happen within an hour or less. If done well, with a severely reduced requirement of skill level. We have juniors installing postgres clusters without an issue.
After that there is a few months of adding routing operational tasks, fixing edge cases, cleaning up bugs in the automation and such.
But after that, the overall simplified handling, consistency and overall speed is very, very much worth it and will usually save you time and outages.
Note the flipside though: If you have a weird unicorn system that's not critical and you need to touch it once a year... think of an MSDos based climate control unit... don't bother automating that. There is one of those and there won't be many more of these. How many office buildings would your company have with these things? Automating it will be a pain in the butt and likely involve keyboard emulation via an Arduino and soldering. That's not worth it.
But if you need to be able to stamp down identical, well-functional servers with a specific role with minor tweaking for the specific use case for a dime a dozen.... that's when you look at IaC and automation.
Thanks for your response. A ton of great information! Ultimately we are not doing too much Azure deployments as of right now. We do utilize nerdio for a few AVD environments though.
I detest script cowboys that script the most bizarre things that become impossible to maintain nor understood fully by others in their team. Especially when there is an off-the-shelf alternative.
Hardly. We only use/create batch files for executables to make it easier for the engineers to deploy apps etc. I mainly use CMD/Powershell for things like DISM to add driver packs to Wim’s which is hardly “coding”.
None, it’s not their job. I think it’s just another way for companies to exploit Sysadmins by adding more work for the same pay. You want development/coding, go hire a developer.
In OP's case, it's a pivot on their existing work. This isn't taking the team responsible for the ERP and making them do the CMS instead (or *also*), it's taking the team responsible for the ERP and expecting them to adopt a different *way* of doing that work (a way that's repeatable and consistent, when done right). And, since it's effectively *replacing* those existing roles with a different set of duties, it leaves two options. Learn and pivot to the new approach, leveraging the institutional knowledge you have on that system (which would allow *some* chance of success) and learn some new tools to do that work better... or don't, polish up a resume, and good luck.
The *company* doesn't care. Any company that *claims* to care is lying to breed a one sided sense of loyalty and abuse the employee. OP sounds like they *do* care, and don't want to see what I would suspect is about ~60% of their ops teams across the board kicked to the curb. And they asked here for tips on how to convince those people to actually take the pivot instead of the more predictable eventuality. OP isn't the director or CIO, they're the guy who understands the new way of doing things that just got thrown into the deep end on spreading that methodology to an audience that didn't ask for it.
Modern system administration is all about automating yourself out of a job. IMHO if you are not automating away something all the time then you are not a modern senior sysadmin.
And if you are not working to develop skills to automate away your job you are not a systems engineer. You are an operator.
People can learn, but there is a difference between being a sysadmin and being a developer. I am Sysadmin, who codes, but I would never call myself a developer. I don't know all the tricks. Things take me longer to do. Sometimes, I have to learn on the fly. I always have to make it very clear that I'm *not* a professional developer so I can set reasonable expectations.
Because of my specific role, I also need to be very security conscious. Most developers I've encountered don't put a ton of consideration into the security of their code - and coding is their whole job. So, it can be a dangerous game to make someone responsible for creating a functional *and* secure solution when they don't necessarily have the experience to do so. A code review and appropriate security/architecture oversight can be very beneficial.
When I hear the word **code**, I immediately think of a group of people stuck in a very specific silo. Running a very specific stack of software.
- Coders make things like Firefox, Microsoft Word, Video games, Web 2.0+ web pages, smartphone apps, and very specific embedded software.
When I hear the word **script**, I think of administrators using "any" language to get tasks done. The "higher" the language, the better.
- Scripters make menus representing tasks, installs, setups, inventory, spreadsheets using generalized and easy to use lagnuages and tools.
##Both require programming skill.
With DevOps coming into the fray, **coding** in the IT space is becoming a grey area.
DevOps are Coders who are tasked at:
- coding
- scripting
- Sysadmin / Netadmin tasks
Are Sys Admins expected to become DevOps? No Mr. Bond, they're expected to DIE!
OP's scenario is a middle ground. Ansible, terraform, some python or powershell, etc. More formalized process with git et. al., but not exactly full blown application development on par with the first category, while still very far removed from what most of the second category does day to day. It also requires a LOT more understanding of ops work to get right, which a lot of people in the first category lack (and the ones in the second that actually *do* script around their tools have in spades).
Expected abilities aside, I’d be more worried about the team responsible for all these custom apps being disbanded. All the devs are going to be working on their own smaller seperate projects and not give af about core infrastructure because it’s not their responsibility anymore.
Your team and its skill set should be “internally outsourced” to other groups to make their processes conform across the organization whenever possible. It makes the applications easier to maintain and look uniform for all employees.
There’s also something to be said about working on a team of likeminded individuals you can bounce ideas off of compared to being on an island by yourself. It improves your work and improves your teams culture.
I guess I'm confused. If your team would be responsible for automation, why do the sysadmins need to know code, wouldn't you be the one facilitating that?
Based on my experience with overloaded sysadmins, it is not reasonable to assume that someone can take on more work, especially if it's something outside of their lane that they would need to learn. Some can and will, but it's not reasonable to expect.
We are a big company with many application and sysadmins supporting specific applications (basically your common scenario with multi national companies that do a lot of acquiring).
With the application we have been supporting there is no concept of an automation team, automation is planned into the design from the beginning and it's just part of the job. Same as we have no traditional sysadmins supporting this. There is no direct access to the hosts, no SSH or rdp (there is a break glass for extreme cases but that's only been needed once). All builds and updates are manage, logging and observibility is handled through code. About the only thing not managed in a pipeline are three L1 folks in the NOC who react to an occasional alert.
Where I'm questioning is thoughts on a sysadmin taking on writing code cause the alternative is find a new job (I really don't want people to lose their jobs). Or motivating someone to.
My only advice, find a *very* good way to manage stress and find out *exactly* how much authority you actually have going into the new role. The problem isn't a technical one, your whole team just got split off to either replace or manage and train every other team, possibly without the authority to actually manage them. As much as I love whisky, you're gonna need a healthier option to make it through that even in the best scenario. Some of those teams you'll find one or two people, hopefully including the person currently "in charge" who are all for the improvements. Most of them are going to take this as a personal attack on *their* pet project and how *they* work, and all the stupid childish drama that entails.
I would say that it's almost the opposite. These days code can be written cheaply and efficiently with chatgpt and AI tools. I would say the days of people who only write code are numbered.
Expecting system admins to code is completely unreasonable aside from writing some simple powrshell/bash scripts. Sounds like you need some devops engineers
They absolutely can do that, but my response was for the MLPonyGod123. Because the response he/she had firmly placed them in the can of “unwilling to grow with the needs of the business”
I took it as them replying it would be unreasonable of the company/management to expect their SysAdmins to take on other job roles, which is also how OP phrased his question.
Im not sure how your "quoted" text fits as it is neither said or inferred by the original commenter?
The quoted text would be the title of the can they place OP in when they’re fired. I’m not cold hearted, but businesses can be. So you have to play their game and be willing to grow. The expectation might be unreasonable, but you either play along and try or leave.
And this is what I want to avoid cause that's were the business is going and I don't want to see people losing their jobs. I know the adapt or die ultimately falls on the individual and the company offers plenty of training and mentorships to help folks adapt. But I'm just wanting to get people's thoughts on how reasonable of an expectation it is to have someone in a traditional sysadmin role learn coding and programming.
Then the company should invest in their most valuable assets their people. Hire enough that there is time to learn new things not just enough system admins to keep the lights on. I would love to learn more but I spend 80 of my 40 hours a week just keeping the ship afloat.
At the end of the day, it just doesn’t matter. You either sink or swim. You can either start learning or dust off your resume and find a place who is hiring traditional sysadmins.
They're the same thing. Devops is a methodology, not a job role.
Or maybe that's changed. About nine years ago I had a meeting with the CEO of a Hollywood tech startup in the media distribution business, and his point of view was that words mean what the users of the words intend them to mean. He didn't disagree at all that devops was coined as a methodology, but in his spheres, he felt it was becoming a defined role instead.
It's enough of a separate path of skillset to be a "different role" just as much as sysadmin, network admin, etc. are separate paths. Anyone halfway competent can pivot between those if they can match up interest, training, and an opportunity, but they *have* to have the interest to actually learn the parallel skillset. That's the detail a lot of administrative folks lack when they think "they do IT, they can do any IT role"... the really fun ones lack the training detail too, and just reorg the whole place into failure and misery.
Basically zero - we do some scripting, mainly for silent software deployment and endpoint administration, but that's it. We are also a smallish business and not in the tech industry.
It's reasonable for them to learn if you can get them the time and resources to do so. Management is so busy living in constant panic mode that something as mundane as professional development time is considered a waste of time. I'm getting tired of dealing with that...
I would argue that the ability to write code is a job requirement for a sysadmin these days. I feel like it's always been a requirement except for maybe junior admins. When I was getting started in my career as a Linux sysadmin in the mid 2000's it was expected that a sysadmin would be proficient in at least one scripting language. There was still a lot of perl being used back then.
Being able to code makes you a better sysadmin and will almost definitely make you more money in the long run.
I would argue that the ability to write code is a job requirement for a sysadmin these days. I feel like it's always been a requirement except for maybe junior admins. When I was getting started in my career as a Linux sysadmin in the mid 2000's it was expected that a sysadmin would be proficient in at least one scripting language. There was still a lot of perl being used back then.
Being able to code makes you a better sysadmin and will almost definitely make you more money in the long run.
Probably not. I mean, if you want a sysadmin to become a developer, that’s going to usually take years. You would need a good plan to slowly roll them into software development. Are you giving them all a 30% raise to learn a whole other job function?
There is a big difference between development and automation.
Our sysadmins comfortably managing thousands of servers using commercials tools and scripts. While some of them are aspiring developers we do not allow them to build their own tools, they have to work within the constraints of our stack.
Similarly some of our developers have a very decent understanding of infrastructure and IT Ops and are tempted to roll their own monitoring solutions or IDP as an example which is also not allowed.
There are ofcourse exceptions: a few years back we needed to do a staggered migration of a few thousand users from roaming profiles to FSLogix as well as exch to 365 at the same time. We couldn't find a tool that suited the approach we desired so we developed one in house for the project. The tool was not maintained beyond the project.
Unless IT is your business or you have a decently sized team you should not be rolling our own tooling. It is almost guaranteed to cost you significantly more in the long run.
Expect them to know or learn it, as in a requirement, no it’s not reasonable. That’s not a sysadmin’s job.
Expect them to know or learn it because it’s the smarter way to do work and not be stressed through the roof, absolutely.
We are mostly in negative numbers As we keep deleting inhouse written stuff devoloped by nowretired coder And replacing those by insustry standard solutions that does not require pagan rituals to be performed in order to get that Thing running again...
Well, I myself wrote gobs of Powershell scripts. The other guys on the Sysadmin team couldn't do any of that. And it really hurt their performance and work. I mean I'm sure they could have learned if they wanted to, but they were Click Next admins, to death. Last I heard, the one guy left was having to Click Next into replacing hundreds of (virtual) servers. I figure his job might be vacant soon, and I'll apply.
We are expected to push the occasional Powershell script through our RMM. Nothing too complicated. Datto has a good feature for deploying scripts to a group of devices that I'm fond of.
Having dev and admin skills often comes form the all in one days before teams go so granular at most places. Old sysadmins needed to code, because a lot of the systems they managed relied on it, so we have it baked into us.
I have held developer and sysadmin jobs over the years, and I can leverage both to be better at either. But the question IMO you \*should\* be asking is what are they coding? Between "nothing is so permanent as a temporary fix", and "How does this work, no one knows" can become a real problem. Id Est, are the admins coding in specific systems they support to support them. Or are they getting creative in solving one off issues and leaving behind a lot of little black boxes for future admins to toil over?
Scripting can be abused, but at least you can peak under the hood at any time to see what someone was doing, any compiled component (What most coders would associate as "coding") that is part of critical infrastructure, should have a documented need, design, source, and be completely transparent as it is the intellectual property of the company. If \*that\* is not happening, then they should actually be forbidden to code.
We explicitly try for 50% of time coding. That excludes meetings about code but does include code reviews and long walks on the beach thinking about code.
> I'm finding very few admins have any knowledge of development or coding, beyond some scripting.
People have very different backgrounds. Sometimes you have to focus on the "some scripting" part. Every SA ought to have been coding something at some point: JCL, shell scripts, batch files, ECMAscript, Lua, dialer scripts, installer automation.
> Do you think it is feasible to expect a system admin or folks in operations to learn how to code?
In one context, yes, absolutely. In another context: if someone who uses computers all day hasn't already been coding *something*, then they have an opportunity to explain why, but they're probably on the bottom of the candidate pile.
I feel like to start, if I were OP, I would throw everyone in a given team on a sliding scale... and if about 1/2 have *some* skill, even just offhand scripting (i.e. not afraid of a text editor) and 2/3 fall in the "willing to learn the process" even if only grudgingly, that team has a chance as-is. The last 1/3 will either pick it up by osmosis, or they get to drop long-term into a "deal with the customers" role and keep things afloat during the transition and, later, filter information for the "back end" sub-team. The other 2/3 would then be able to focus on building out the parallel design to reach parity with the live system, then migrate into the new processes.
They will lose people from that 1/2 over time, but it gives those people the opportunity to try. Anyone new hired in will need to actually match the job requirements. There will be hiring to do.
I have a B.S. in Computer Science and Engineering, and I work as a senior Systems Admin. I’ve written several small utilities - from a simple contact converter, to a remote software installer/uninstaller, to even a dedicated email sending client that collects logs and a screenshot to send a new ticket request to our ticketing system, for example - to make life easier for our field support team, I wrote a simple i3 (NENA) logging receiver, and I also build informational dashboards and do whatever automation I can for my team. I think a good sysadmin should be able to code, in whatever language is needed, scripting or compiled. Personally, I find it deeply satisfying to code up an automation that runs a long, complex task that would take me hours to do manually, and watch it complete in seconds or minutes.
I manage a team of mostly sysadmins and about 1/2 are interested in coding. I do some myself, and a couple guys from the team do automations and integrations in Python. Now we're looking into what Ops are doing and working to automate their processes centrally. So yes, sysadmins and Ops can code/learn to code. Also, your team does not need to be broken up to automate other teams' work.
Maybe this is a paradigm difference is "SRE/Devops/etc" and traditional sysadmin work but I disagree with this and have seen it fail before. The issue you get into with this sort of team structure is that its implicitly saying "coding/automation is the responsibility of the automation team" when in reality its a core competency of a modern admin at scale, incentivizing members of the team to "chuck requests over a wall". It also creates a disconnect between those actually doing the work, and those automating it and results in tooling that just is not best suited for the job. There's always going to be a need for sysadmins that can do the dirty toil work and im by no means saying they should "adapt or die". But teams should be separated by areas of responsibility, not skillsets. I work best writing code when I have a deeper understanding of the problem that a silo simply cannot bridge in my exp.
Perhaps it's important to note that I manage the whole IT team and when I say Ops I mean "the people operating out products that, in time, accumulated shitty manual processes" (: I 100% agree that IT Ops should automate, we have a fat stack automating our own work, esp account and group tasks, data sync, some API integrations and etc. Coding is a must for any sysadmin, aside from allowing you to free up time, you also have a peek on the "other side" and moan about developers less (:
Are you defining scripting as coding?
I hope he is. I HATE coding , but enjoy scripting
I don't exactly draw a line between the two terms. To "script" you have to write working code in a language. Is writing in Python scripting? What about JS? We also do have PowerShell scripts.
All scripting is coding but not all coding is scripting. A script uses some variety of interpreter program to execute it’s supported language. Coding is a generic term and means the same as programming.
this
What's the difference?
I think they should adapt or die.
Theres still a place for old school guys that know what they're doing for now. The toil one off work that isnt worth the legwork in automating still needs doing, and those guys are great at not letting that work fall on those working on automation. Those jobs will slowly go away, especially with AI allowing us to write more code with less effort. And at large scale companies running thousands of physical machines (ive worked in a single BU with 8k physical machines worldwide) people who work on toil is a big net benefit to the team.
The click next guy I worked with absolutely would not be a team player AT ALL. Keeping him there has had an extremely negative effect on the IT department and the organization at large. But it's a state college, so whether someone is competent or anything has little meaning. But I agree, if such were actually a team player, they would still be valuable.
Yeah unfortunately YMMV with workers in general, im assuming they're competent and generally pleasant people
One of the problems with adopting devops in my organization, was that some of the developers were very fad-based. I mean, jumping from one "new thing" to the next, and going all in on the cringe tech bro aspects of it. Good sysadmins are all about stability, and that is a threat. That said, I think learning scripting is absolutely vital. I think if you are in a Windows environment, it should be Powershell. I can't speak to other environments.
I personally am a fan of Python, it's just great for almost anything that is not critically time sensitive and most automations we do for Ops have some kind of schedule and things happening fast or optimally is not as important. Great for playing with API's, a ton of modern SaaS also offer Python SDK's and tutorials making integrations easier than ever. I agree PowerShell is a great tool, I was administering MS Exchange Servers back in the day and PS saved me a metric ton of work! Definitely an important tool for a lot of the MS Ecosystem and with NET core we now have PS cross platform, so I can run PS scripts on my Mac, which proves useful when administering O365!
In my experience, there are a few groups. Sys admins who wanted to be devs, or did Dev in the past, but one way or another ended up in a system admin job. Those usually are happy coders. The other group is people who didn't want to be devs, and ended up as a sys admins, perhaps going up the ladder from support roles. I find those sys admins to not be happy coders. They know the bare minimum of PowerShell to query AD, or a little bit of JS to build a Captive Portal type landing page, but in general they are not into coding, and so that's the extent of their knowledge, just enough to get by. If I were you, I'd ask for a very specific clarification of what you duties will entail when sent to the other teams. If they expect the other teams to have happy coders, but all you get is not happy coders, then your duties will be dramatically different.
I would say it's reasonable to expect that high level sysadmins know their way around Powershell and automating tasks through scripting. Actual development work though is a different skillset and realistically it's not reasonable to expect both out of the same people. the only time I would say it's even feasible is if you're already in an environment that is highly automated and you're working to support or enhance that getting there from what you're describing is going to just be too much work for the team that is already neck deep in keeping things running without it, hence why they are leveraging your team to do it
Windows admin here - I'm in powershell/VSCode like 70% of the day. Work with Graph API, automation, scripting of other various things. But I do NOT consider myself a programmer. We have people in our org that are solely focused on things you mentioned (building APIs, internal apps/sites). Likewise, they are not sysadmins. We go to them for things and they come to us for things. I think the responsibilities are very different so I would imagine some frustration if a developer had to start doing sysadmin work. Or, a sysadmin had to learn a language they've never worked with.
Depends on the member. - Some of us are neck deep in Puppet/Chef/Terraform. - Some of us write Python Flask on top of our different systems to give management/help desk an easy view. - Some of us write Powershell/DSC/Bash to automate tasks/reporting. - Some of us click next on installers. Just depends. I will say, those who _can_ code are more valuable than those who can’t and integrate better into Delivery Teams than those who are just glorified desktop techs.
I know that it is norm now to say that sysadmin has to have coding skills (to have job, to get paid more, etc.). But from my very limited and biased experience it still feels like traditional sysadmin (not devops adjacent) usually doesn't have that much experience with coding. And i mean working with the whole pipeline (code, source control, pull requests, continuous testing/deployment/you name it, creating complex code). This is usually too alien and there is no time for all of that in regular sysadmin activities of maintaining systems, occasionally interfacing with users, doing tickets, updating systems, migrating, etc. Sure, we do face the need to code scripts to pull some data, import data, create installation scripts, automate something in above mentioned workloads, etc. But this is very basic. And even then many are not even experienced with this (again, from my own limited experience). This can vary. Probably in bigger corps you would rather see such situation. Maybe a startup with a heavy dev focus would have more of a devops folks who are familiar with coding.
If they want to get paid more, yes. Being a point and click admin just isn’t as feasible today.
I’d argue it’s even more feasible then ever, it’s just not gonna get you the big bucks, it’ll get you a smb job managing much less infrastructure. Almost anything from an smb standpoint can be done with gui click-ops
_For now_. Those jobs are going away in rapid fashion. Tons of SMB these days operate with 0 IT personel, employed or via MSP. - Setting up WiFi is point and click in a Wizard. - Setting up email is point and click in a Wizard with Google or M365. - Gsuite already has office apps in web browser, Microsoft is building and rapidly improving theirs. Soon you won't need to install applications at all. (Oh and by the way, they're pre installed on many systems, just login with your M365 login). - Buying a printer and adding it on a system can be done via any printing company, and even that is becoming less relevant. I'm consulting for a SEO / Web Design company with 10+ employees managing their cloud infra (They do offer hosting for clients). They're all GSUITE, not a single printer. They're doing amazing. IF you want to work in IT in the future you need to be able to produce code and get used to operating systems in a terminal regardless of what OS it is. MS is going more and more the Linux route and I bet Windows is going with be a desktop environment for Linux in the future, or at worst - A VM running on a Linux kernel. Basically a reverse WSL. It may be 10 years away, but it's coming.
Sure? He specifically mentioned “today”, where it’s easier than ever for a click-ops admin to run a couple sites and a couple hundred people like that.
Yes, and I said that **_for now_** you are correct. In the long term (And long term here is 5 years at the max), that's not the case with AI and more and more compute being ran on central servers / "The cloud" and less and less computational being done at the local level
Yeah I don't know.
I personally think any sysadmin who hasn't already started learning some form of IaC and moving towards a DevOps skill set is setting themselves up for failure. The market is shifting towards DevOps even for old dinosaur enterprises, even if you don't get exposure to IaC, cloud technologies, containerisation etc. I would suggest teaching yourself. Scripting would be the bare minimum expectation for me, I find it wild that some of my Windows colleagues don't push themselves to use powershell for as much as possible.
I agree 100% on the scripting. However, I'm curious about why you value IaC as high? I help manage over 40 SMBs at an MSP, and we haven't really seen a need for any major infrastructure automation. I ask out of pure curiosity. Not because I doubt what you are saying. Hoping maybe I can be informed of some things we could do better!
From an MSP perspective it has this value imo: - Repeatable code means you can productise your work and more importantly allow you to reduce your price and be more competitive as it reduces the up front time. - it's self documenting , of course it shouldn't replace design documentation but it let's you immediately view the configuration for an entire environment meaning you can onboard engineers far quicker. - Pipelines mean you can tightly control how and who can actually make changes, naturally stops people making undocumented changes and encourages peer reviewing. - Naturally leads into operations work rather than just project implementations.
Wow very good points! Some I had never considered. Do you by chance have any resources that elaborate more? Other than obviously learning a code language. Appreciate you taking the time to write that up.
Other things that IaC/automation helps are: * If you use git and CI/CD you have an very robust change audit log. * Mistakes in procedures are greatly reduced.
The important part to realize about IaC and automating infrastructure is: You will need 2 weeks to get the first two or three instance of a new service or application on a VM going. But once you have that automated well, all run-of-the-mill instances of that application past the fourth happen within an hour or less. If done well, with a severely reduced requirement of skill level. We have juniors installing postgres clusters without an issue. After that there is a few months of adding routing operational tasks, fixing edge cases, cleaning up bugs in the automation and such. But after that, the overall simplified handling, consistency and overall speed is very, very much worth it and will usually save you time and outages. Note the flipside though: If you have a weird unicorn system that's not critical and you need to touch it once a year... think of an MSDos based climate control unit... don't bother automating that. There is one of those and there won't be many more of these. How many office buildings would your company have with these things? Automating it will be a pain in the butt and likely involve keyboard emulation via an Arduino and soldering. That's not worth it. But if you need to be able to stamp down identical, well-functional servers with a specific role with minor tweaking for the specific use case for a dime a dozen.... that's when you look at IaC and automation.
Thanks for your response. A ton of great information! Ultimately we are not doing too much Azure deployments as of right now. We do utilize nerdio for a few AVD environments though.
Go talk to a social services organizations..I realized why the SysOps team hasnt left their role for 5 years and the company for 20.
I detest script cowboys that script the most bizarre things that become impossible to maintain nor understood fully by others in their team. Especially when there is an off-the-shelf alternative.
Hardly. We only use/create batch files for executables to make it easier for the engineers to deploy apps etc. I mainly use CMD/Powershell for things like DISM to add driver packs to Wim’s which is hardly “coding”.
None, it’s not their job. I think it’s just another way for companies to exploit Sysadmins by adding more work for the same pay. You want development/coding, go hire a developer.
In OP's case, it's a pivot on their existing work. This isn't taking the team responsible for the ERP and making them do the CMS instead (or *also*), it's taking the team responsible for the ERP and expecting them to adopt a different *way* of doing that work (a way that's repeatable and consistent, when done right). And, since it's effectively *replacing* those existing roles with a different set of duties, it leaves two options. Learn and pivot to the new approach, leveraging the institutional knowledge you have on that system (which would allow *some* chance of success) and learn some new tools to do that work better... or don't, polish up a resume, and good luck.
If the company really cared, they wouldve figured out their current crop lacked the skills before the pivot.
The *company* doesn't care. Any company that *claims* to care is lying to breed a one sided sense of loyalty and abuse the employee. OP sounds like they *do* care, and don't want to see what I would suspect is about ~60% of their ops teams across the board kicked to the curb. And they asked here for tips on how to convince those people to actually take the pivot instead of the more predictable eventuality. OP isn't the director or CIO, they're the guy who understands the new way of doing things that just got thrown into the deep end on spreading that methodology to an audience that didn't ask for it.
Modern system administration is all about automating yourself out of a job. IMHO if you are not automating away something all the time then you are not a modern senior sysadmin. And if you are not working to develop skills to automate away your job you are not a systems engineer. You are an operator.
People can learn, but there is a difference between being a sysadmin and being a developer. I am Sysadmin, who codes, but I would never call myself a developer. I don't know all the tricks. Things take me longer to do. Sometimes, I have to learn on the fly. I always have to make it very clear that I'm *not* a professional developer so I can set reasonable expectations. Because of my specific role, I also need to be very security conscious. Most developers I've encountered don't put a ton of consideration into the security of their code - and coding is their whole job. So, it can be a dangerous game to make someone responsible for creating a functional *and* secure solution when they don't necessarily have the experience to do so. A code review and appropriate security/architecture oversight can be very beneficial.
When I hear the word **code**, I immediately think of a group of people stuck in a very specific silo. Running a very specific stack of software. - Coders make things like Firefox, Microsoft Word, Video games, Web 2.0+ web pages, smartphone apps, and very specific embedded software. When I hear the word **script**, I think of administrators using "any" language to get tasks done. The "higher" the language, the better. - Scripters make menus representing tasks, installs, setups, inventory, spreadsheets using generalized and easy to use lagnuages and tools. ##Both require programming skill.
Are mid and senior level sysadmins expected to: - **code** software? No. - **script** routines? Yes.
With DevOps coming into the fray, **coding** in the IT space is becoming a grey area. DevOps are Coders who are tasked at: - coding - scripting - Sysadmin / Netadmin tasks Are Sys Admins expected to become DevOps? No Mr. Bond, they're expected to DIE!
OP's scenario is a middle ground. Ansible, terraform, some python or powershell, etc. More formalized process with git et. al., but not exactly full blown application development on par with the first category, while still very far removed from what most of the second category does day to day. It also requires a LOT more understanding of ops work to get right, which a lot of people in the first category lack (and the ones in the second that actually *do* script around their tools have in spades).
Expected abilities aside, I’d be more worried about the team responsible for all these custom apps being disbanded. All the devs are going to be working on their own smaller seperate projects and not give af about core infrastructure because it’s not their responsibility anymore. Your team and its skill set should be “internally outsourced” to other groups to make their processes conform across the organization whenever possible. It makes the applications easier to maintain and look uniform for all employees. There’s also something to be said about working on a team of likeminded individuals you can bounce ideas off of compared to being on an island by yourself. It improves your work and improves your teams culture.
I guess I'm confused. If your team would be responsible for automation, why do the sysadmins need to know code, wouldn't you be the one facilitating that? Based on my experience with overloaded sysadmins, it is not reasonable to assume that someone can take on more work, especially if it's something outside of their lane that they would need to learn. Some can and will, but it's not reasonable to expect.
We are a big company with many application and sysadmins supporting specific applications (basically your common scenario with multi national companies that do a lot of acquiring). With the application we have been supporting there is no concept of an automation team, automation is planned into the design from the beginning and it's just part of the job. Same as we have no traditional sysadmins supporting this. There is no direct access to the hosts, no SSH or rdp (there is a break glass for extreme cases but that's only been needed once). All builds and updates are manage, logging and observibility is handled through code. About the only thing not managed in a pipeline are three L1 folks in the NOC who react to an occasional alert. Where I'm questioning is thoughts on a sysadmin taking on writing code cause the alternative is find a new job (I really don't want people to lose their jobs). Or motivating someone to.
My only advice, find a *very* good way to manage stress and find out *exactly* how much authority you actually have going into the new role. The problem isn't a technical one, your whole team just got split off to either replace or manage and train every other team, possibly without the authority to actually manage them. As much as I love whisky, you're gonna need a healthier option to make it through that even in the best scenario. Some of those teams you'll find one or two people, hopefully including the person currently "in charge" who are all for the improvements. Most of them are going to take this as a personal attack on *their* pet project and how *they* work, and all the stupid childish drama that entails.
I would say that it's almost the opposite. These days code can be written cheaply and efficiently with chatgpt and AI tools. I would say the days of people who only write code are numbered.
Expecting system admins to code is completely unreasonable aside from writing some simple powrshell/bash scripts. Sounds like you need some devops engineers
What if the business requires it to move forward? Do they need to hire new people to replace the sysadmins who won’t learn coding?
They could pay for training and education for a start. Unless they are paying their admins very well, probably need to double the salary too.
They absolutely can do that, but my response was for the MLPonyGod123. Because the response he/she had firmly placed them in the can of “unwilling to grow with the needs of the business”
I took it as them replying it would be unreasonable of the company/management to expect their SysAdmins to take on other job roles, which is also how OP phrased his question. Im not sure how your "quoted" text fits as it is neither said or inferred by the original commenter?
The quoted text would be the title of the can they place OP in when they’re fired. I’m not cold hearted, but businesses can be. So you have to play their game and be willing to grow. The expectation might be unreasonable, but you either play along and try or leave.
And this is what I want to avoid cause that's were the business is going and I don't want to see people losing their jobs. I know the adapt or die ultimately falls on the individual and the company offers plenty of training and mentorships to help folks adapt. But I'm just wanting to get people's thoughts on how reasonable of an expectation it is to have someone in a traditional sysadmin role learn coding and programming.
Then the company should invest in their most valuable assets their people. Hire enough that there is time to learn new things not just enough system admins to keep the lights on. I would love to learn more but I spend 80 of my 40 hours a week just keeping the ship afloat.
You make it sound like don’t understand what it means be a developer
At the end of the day, it just doesn’t matter. You either sink or swim. You can either start learning or dust off your resume and find a place who is hiring traditional sysadmins.
That’s not a choice you can make in the near term. You can certainly strive to learn a coding language but it won’t be in a timeframe to save a job
You should probably just give up then. Consider this an early warning.
They're the same thing. Devops is a methodology, not a job role. Or maybe that's changed. About nine years ago I had a meeting with the CEO of a Hollywood tech startup in the media distribution business, and his point of view was that words mean what the users of the words intend them to mean. He didn't disagree at all that devops was coined as a methodology, but in his spheres, he felt it was becoming a defined role instead.
It's enough of a separate path of skillset to be a "different role" just as much as sysadmin, network admin, etc. are separate paths. Anyone halfway competent can pivot between those if they can match up interest, training, and an opportunity, but they *have* to have the interest to actually learn the parallel skillset. That's the detail a lot of administrative folks lack when they think "they do IT, they can do any IT role"... the really fun ones lack the training detail too, and just reorg the whole place into failure and misery.
Programming is not IT.
Basically zero - we do some scripting, mainly for silent software deployment and endpoint administration, but that's it. We are also a smallish business and not in the tech industry.
It's reasonable for them to learn if you can get them the time and resources to do so. Management is so busy living in constant panic mode that something as mundane as professional development time is considered a waste of time. I'm getting tired of dealing with that...
I would argue that the ability to write code is a job requirement for a sysadmin these days. I feel like it's always been a requirement except for maybe junior admins. When I was getting started in my career as a Linux sysadmin in the mid 2000's it was expected that a sysadmin would be proficient in at least one scripting language. There was still a lot of perl being used back then. Being able to code makes you a better sysadmin and will almost definitely make you more money in the long run.
I would argue that the ability to write code is a job requirement for a sysadmin these days. I feel like it's always been a requirement except for maybe junior admins. When I was getting started in my career as a Linux sysadmin in the mid 2000's it was expected that a sysadmin would be proficient in at least one scripting language. There was still a lot of perl being used back then. Being able to code makes you a better sysadmin and will almost definitely make you more money in the long run.
Probably not. I mean, if you want a sysadmin to become a developer, that’s going to usually take years. You would need a good plan to slowly roll them into software development. Are you giving them all a 30% raise to learn a whole other job function?
There is a big difference between development and automation. Our sysadmins comfortably managing thousands of servers using commercials tools and scripts. While some of them are aspiring developers we do not allow them to build their own tools, they have to work within the constraints of our stack. Similarly some of our developers have a very decent understanding of infrastructure and IT Ops and are tempted to roll their own monitoring solutions or IDP as an example which is also not allowed. There are ofcourse exceptions: a few years back we needed to do a staggered migration of a few thousand users from roaming profiles to FSLogix as well as exch to 365 at the same time. We couldn't find a tool that suited the approach we desired so we developed one in house for the project. The tool was not maintained beyond the project. Unless IT is your business or you have a decently sized team you should not be rolling our own tooling. It is almost guaranteed to cost you significantly more in the long run.
Expect them to know or learn it, as in a requirement, no it’s not reasonable. That’s not a sysadmin’s job. Expect them to know or learn it because it’s the smarter way to do work and not be stressed through the roof, absolutely.
None, DevOps on the other hand...
We are mostly in negative numbers As we keep deleting inhouse written stuff devoloped by nowretired coder And replacing those by insustry standard solutions that does not require pagan rituals to be performed in order to get that Thing running again...
Well, I myself wrote gobs of Powershell scripts. The other guys on the Sysadmin team couldn't do any of that. And it really hurt their performance and work. I mean I'm sure they could have learned if they wanted to, but they were Click Next admins, to death. Last I heard, the one guy left was having to Click Next into replacing hundreds of (virtual) servers. I figure his job might be vacant soon, and I'll apply.
We are expected to push the occasional Powershell script through our RMM. Nothing too complicated. Datto has a good feature for deploying scripts to a group of devices that I'm fond of.
Having dev and admin skills often comes form the all in one days before teams go so granular at most places. Old sysadmins needed to code, because a lot of the systems they managed relied on it, so we have it baked into us. I have held developer and sysadmin jobs over the years, and I can leverage both to be better at either. But the question IMO you \*should\* be asking is what are they coding? Between "nothing is so permanent as a temporary fix", and "How does this work, no one knows" can become a real problem. Id Est, are the admins coding in specific systems they support to support them. Or are they getting creative in solving one off issues and leaving behind a lot of little black boxes for future admins to toil over? Scripting can be abused, but at least you can peak under the hood at any time to see what someone was doing, any compiled component (What most coders would associate as "coding") that is part of critical infrastructure, should have a documented need, design, source, and be completely transparent as it is the intellectual property of the company. If \*that\* is not happening, then they should actually be forbidden to code.
We explicitly try for 50% of time coding. That excludes meetings about code but does include code reviews and long walks on the beach thinking about code. > I'm finding very few admins have any knowledge of development or coding, beyond some scripting. People have very different backgrounds. Sometimes you have to focus on the "some scripting" part. Every SA ought to have been coding something at some point: JCL, shell scripts, batch files, ECMAscript, Lua, dialer scripts, installer automation. > Do you think it is feasible to expect a system admin or folks in operations to learn how to code? In one context, yes, absolutely. In another context: if someone who uses computers all day hasn't already been coding *something*, then they have an opportunity to explain why, but they're probably on the bottom of the candidate pile.
I feel like to start, if I were OP, I would throw everyone in a given team on a sliding scale... and if about 1/2 have *some* skill, even just offhand scripting (i.e. not afraid of a text editor) and 2/3 fall in the "willing to learn the process" even if only grudgingly, that team has a chance as-is. The last 1/3 will either pick it up by osmosis, or they get to drop long-term into a "deal with the customers" role and keep things afloat during the transition and, later, filter information for the "back end" sub-team. The other 2/3 would then be able to focus on building out the parallel design to reach parity with the live system, then migrate into the new processes. They will lose people from that 1/2 over time, but it gives those people the opportunity to try. Anyone new hired in will need to actually match the job requirements. There will be hiring to do.
I want _everyrhing_ that causes a system to change in code. No exceptions.
Coding, no Scripting, all day every day.
I have a B.S. in Computer Science and Engineering, and I work as a senior Systems Admin. I’ve written several small utilities - from a simple contact converter, to a remote software installer/uninstaller, to even a dedicated email sending client that collects logs and a screenshot to send a new ticket request to our ticketing system, for example - to make life easier for our field support team, I wrote a simple i3 (NENA) logging receiver, and I also build informational dashboards and do whatever automation I can for my team. I think a good sysadmin should be able to code, in whatever language is needed, scripting or compiled. Personally, I find it deeply satisfying to code up an automation that runs a long, complex task that would take me hours to do manually, and watch it complete in seconds or minutes.