|
At my previous site, the system administrator (server side) was just that.. the administrator. And the network people could do there job.
Following a 'merger' and shift of responsibilities to corporate, neither can do there job.. the system administrator does not have access rights to the nodes to make sufficient changes - he is a glorified part swapper, and the network? That is contracted out to a firm - you now just hold the door for them and escort them throughout the mill.
Bigger is not always better; let people do there job.
|
|
|
|
|
Welcome to the world of PCI compliance where even sysadmins don't have any privileges. Where some random QSA (Qualified Security Auditor) who is basically unfamiliar with your business or business rules gets to convince management that removing IT access to all servers and data, except the glorified few, somehow automagically makes everything "better".
And where getting anything deployed to production requires an act of congress because everyone knows that developers, who have worked for the same company for 30 years, can't be trusted.
And if something bad should happen to the only two people who are actually allowed to approve deployments to production, then no one is deploying anything until the one network person who actually knows where all the skeletons are buried can worm his way into the deployment system and approve someone else.
Because apparently, this is how everyone else does it.
"...JavaScript could teach Dyson how to suck." -- Nagy Vilmos
|
|
|
|
|
Sander Rossel wrote: Computers are my job, I kind of know how they work...
Because every time I've heard that it was aaaaalwaaays true! 
(Not implying that it doesn't actually apply to you!  )
I'm pretty sure I would not like to live in a world in which I would never be offended.
I am absolutely certain I don't want to live in a world in which you would never be offended.
Freedom doesn't mean the absence of things you don't like.
Dave
|
|
|
|
|
I am so in the same space right now. Had some guy from "IT" respond to a ticket I raised with some cr@p that started "I've talked to Microsoft and...". :sniff:...anyone else smell that? Then a business critical month end process crashes and when I ask for a registry entry to be changed he demands "a business case" before IT can "authorise the use of that Access database"
As if it's not soul - destroying enough to be working as a "VBA Developer" I have to watch a bunch of cowboys pass themselves off as an "IT" department providing zero service to the business. Jeez.
|
|
|
|
|
My job title, and the AD groups I am in, says I am a DBA and System & Server Admin.
We (my coworker, in the same job) have been trying to install an MS SQL cluster on a server since August.
I think I'll just leave it at that. Use your imagination: think about permissions and HBSS and DNS and AD and VMs and about how each teeny tiny aspect of an enterprise system is individually controlled by different people, in different parts of the country, and how each request has to go through a ticketing system that averages about three days from submission to assignment.
|
|
|
|
|
Wow, good luck! 
|
|
|
|
|
Okay so reading this made me cringe and here is why: the world will not improve if you do not do your part in improving it.
Of course you could have done everything on your own, no body is doubting that you are good with computers. But time is not on your side; there are deadlines, there are priorities, there are projects to be delivered, bills to be paid, etc. Besides, evolution taught us that humans excelled the most when they worked together, when ideas clashed, etc, it's not always good to do everything on you own.
You work in a company where you chose to be a developer presumably because this is what you like/enjoy the most, and this is what brings the best of you. So focusing on this role should be your highest priority.
I'm just stating these in order to establish a context.
Now having these in mind, here is how the scenario could have been handled:
0 Encounter with a sysadmin, understood that he is way beyond his capabilities
1 ask for his supervisor/senior/manager etc, solve the problem in one hour maybe two.
2 transfer your knowledge to this new sysadmin
2a this would have helped him a lot in his career
2b this would have given him the interest to dig deeper and enjoy his job
2c this would have saved a lot of time the next time you come to him
2d this would have saved a lot of time for the next developer that comes to him
2e this would have given you a privilege over all other developers because you became a special person for this sysadmin
2f this would have made him like developers and helping them because he would have felt that there is always a lesson to learn
... and so many more benefits.
Instead, by being frustrated at him, it made him feel bad about his job, made him feel that he will never grow in this position because developers can always do his job in a fraction of the time and he is just there because of some company policy, made him hate developers, etc...
and this would for sure help creating a toxic work environment
|
|
|
|
|
It seems you missed my point.
I explicitly stated that I'm not complaining about him not knowing his job and I don't think that's what I made him feel like.
It's just that this has been a process that took me six weeks to get right on various environments.
The sysadmins are in no way involved with the entire project.
And now that it needs to go to production I need to transfer six weeks of knowledge, gained from struggling with third party code, services, and certificates, in just a couple of minutes, to someone who'll further remain uninvolved.
Mostly it's not a problem, except when I need to disturb him in his work to OPEN A TEXT FILE because I don't have access to it.
Of course I can open a text file, or click an exe, my grandma could do that!
So here I am, disturbing him, being slowed down myself, to open a text file...
Basically, that's what his job is now, showing me the content of some text files.
Tell me, how is that focusing on my role, how is that helping or improving him?
At least you are right about the toxic work environment 
I could really use him now by the way, I need to run a program for five seconds and view the logs, but there's no sysadmin anywhere in sight... 
|
|
|
|
|
Separation of responsibilities is an important thing; I've had the joy of being on an assessment team that turned into an incident response team overnight because of it.
"There are three kinds of lies: lies, damned lies and statistics."
- Benjamin Disraeli
|
|
|
|
|
I feel your pain. You feel my pain. (fortunately, I got a guy who knows a guy ... )
|
|
|
|
|
Sounds like a process issue to me. Probably just need a layer of management in-between to approve the changes to be deployed.
Because, of course, they'd understand the changes and process, be easily accessible and be anything more than a rubber stamp. 
What if, rather than add layers of bureaucracy, we hired smart people, and trusted them to do the right things (which includes trusting them to know when they're out-of-scope and get help).
|
|
|
|
|
The one time I had to work in that kind of environment, we complained until we got the permissions we needed to do our job.
Otherwise, I've been responsible for everything, including debugging or working around Microsoft or other large company bugs, because the only support we could get was like that situation you just described, where we knew what the problem was, but couldn't get past the support person to the next level, assuming we could even understand that person, given their heavy accent.
So, I've asked for a clay tablet and stylus for a Christmas present this year.
|
|
|
|
|
1/2 day? ... How about 6 months waiting for a DBA to "approve" your design.
What ever happened to IBM and SQL Server and Oracle "DBA"'s? Did they ascend into heaven?
Oh, right; desktop database engines.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Sander Rossel wrote: but this took an entire afternoon while it should've taken about an hour.
...and so if the production machine goes down or gets hacked who is the person that is on the hook for that?
Myself it has been a long time since I would even accept credentials on a production machine when there were actual ops personnel around. And I discourage handing out such access to others. If there is a problem then figuring out the source is easier when there are far few fingers in the mix.
|
|
|
|
|
But actually getting anything done becomes pretty much impossible.
|
|
|
|
|
Sander Rossel wrote: But actually getting anything done becomes pretty much impossible.
That would then be a management problem.
But I haven't experienced that situation myself. Operations that fails to meet requests do not last long. Just as any other position that doesn't do their job doesn't last.
|
|
|
|
|
Sounds like you need some sort of UAT environment between DEV and PROD.
On our team, when you make a change to UAT, you document it with enough detail that it can be applied to PROD, but someone else has to make the PROD changes based on your documentation alone.
You fix the documentation until the PROD deployment succeeds.
What happens if you need to stand up the same app for a different site, department, client, or Disaster Site?
If you want to test an application upgrade path starting with an identically built machine...
If someone wants to upgrade the OS under your app?
3 years from now it will be a lot harder to remember all of the little details that got it working. It is also likely that various things were tried on the DEV machine that did not quite work correctly... Throw DEV away after a bit and clone the UAT environment for the next round of DEV.
|
|
|
|
|
That COULD work if all software I deployed to PROD worked.
More specifically, getting the software to run on PROD wasn't the issue, getting it to run PROPERLY was.
We do have a DTAP street, but production is always a little different.
For example, it uses SSL (I don't know why we don't use that for everything, even internal services), it should sent out actual emails to actual people, it uses different certificates (because we can't connect to production services from a non-production environment), the firewall is configured differently, etc.
Sometimes you just need to do some testing on production, especially when third party services are involved (which is the case here).
But when you need someone to open a text file for you that process becomes very difficult.
I do agree that we SHOULD document the deployment process for every bit of software though (but we don't).
|
|
|
|
|
I've been on both sides of this issue. The problem is too many companies have been burned by developers releasing code that hasn't been properly tested and confirmed to work in the target environment. Just because it works on my [development] system and server doesn't mean it will work on the CEO's.
|
|
|
|
|
Well, I guess not giving developers permissions to their target platform solves that issue as no software gets released anymore!
But seriously, if untested software makes it to production then your issue is with testing, not releasing.
If anything, the chances that software doesn't work as expected is only bigger if people who know nothing about the software do the releasing.
|
|
|
|
|
Not speaking of you directly, but generally, developers in many companies do not have have the security or legal compliance mindset. This has been the case at most every company I have worked at, and unfortunately is the case at my current company.
A few months back, the new Senior Manager that is over both my team and the applications (development) team came up to me and informed me that he was making me the "gatekeeper" over all security aspects of our internal AS/400 system. I am not an AS/400 person. I have very limited experience with it as a user. I basically know how to unlock an account, reset a password, and restart a stuck print queue. Any of the AS/400 administration has been handled by the applications team for decades.
So, why am I now the security "gatekeeper"? He explained that the developers would just create a new service account with full sysadmin access for every application and every robot and every agent. Whenever anyone with a Supervisor or above title asked for access to something, they granted that access without questions asked. My Sr. Manager had discovered that one of our warehouse workers had access to just about every system, including Accounts Receivable and Payable (because his supervisor wanted him to be able to check to see if some customers were current on their invoices before shipping out).
I had to audit user access across the board. I took away access that was not properly approved and documented. Because people could not do their jobs the (wrong) way they were used to doing it, production ground to a screeching halt for almost a full week while access was straitened out, and people were re-trained to do their jobs the correct way. It cost the company thousands upon thousands of dollars. Granted, if something serious had happened and we had been found to be out of legal compliance, it could have cost millions.
I still do not have the know-how to administer the AS/400 (although I am slowly learning). So, how do we make this work? The developers must submit security requests to me for review. I properly log the requests, ensure that all the required parties have reviewed the request and approved it, then I create a work order and send it back to the developer/admins to actually execute the security change. If they are found to have made security changes without following the procedure, they are subject to disciplinary action. (It only had to happen once.)
- - - - - -
I'll go out on a limb and say that most (I did not say all) developers are focused on "Making it work" and are not focused on security and legal compliance. That is why there needs to be a separation of development from administration.
It is indeed sad that the separation is implemented so poorly at your company. It is regrettable that an appropriate separation is not yet achieved here at my company. The unfortunate reality is that giving production administration to development teams is not the way to go.
In your case, you need a procedure in place to have a copy of your production system and config pulled to a dev/test system, allow you to make the changes needed to deploy/repair whatever is needed, then pass the specific changes (properly documented) back to the admins to review and implement as a whole, not in piecemeal. That would not only make implementing the changes easier, but would also make the sysadmin's manager's job easier in determining what training or additional personnel are needed for them to give you better response time and better collaboration.
Money makes the world go round ... but documentation moves the money.
|
|
|
|
|
It's not so much about doing everything yourself, it's just that it would be really nice and no-risk if I just had access to a log file.
I don't know much about security, IIS, reverse proxies, and user management.
But I do know how to open a log file and run a query on a database.
I consider the latter to be a part of my job, and those are the things I cannot currently do.
The former can be handled by someone else.
|
|
|
|
|
I agree with you. If they had it set up properly, you would have at lease read access to the logs, unless there is something else in those logs that would be considered confidential.
In that case, they should be parsing the logs, and giving you access to the appropriate portions.
Money makes the world go round ... but documentation moves the money.
|
|
|
|
|
Thank you for your post willichan, it was enjoyable
|
|
|
|
|
Of course, if your application had been deployed in UAT or some other environment with representative security - then you could have worked through the issues on systems you had access to. Instead of futzing around in production. Then your install/deployment would have catered for all of the issues that arose.
I would also ask about whether all of the custom things you did made it back into the systems documentation - so that in five years time when someone else needs to re-install that they know to repeat the things you did to make it work.
Not suggesting that this is representative of your situation - just that there are two sides to the coin. Separation of duties is painful, but sometimes the pain is in forcing the documentation and awareness into the process. Does not sound like they way it worked for you though.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
