|
Update the resume and start discreetly looking?
You know who will get the blame when it all falls apart ... 
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
|
Run Forrest, run!
"In testa che avete, Signor di Ceprano?"
-- Rigoletto
|
|
|
|
|
My "issues" can usually be traced to lack of visibility; putting off that dashboard or "log" I know the app needs. Like driving with an oil light only. (And the bulb may be burned out.)
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
|
|
|
|
|
Like when you get that "engine" light on the dashboard, but just ignore it....
|
|
|
|
|
That sounds so much like the situation where I am working... And no one wants to dig into this archeological dig of a mess to find the underlying causes...
modified 21-Feb-24 21:52pm.
|
|
|
|
|
Long ago I rather enjoyed fixing bugs in other people's code. Send the code to me. Maybe I'm still good at it.
|
|
|
|
|
This is also called the project death march. You need an enemy to make hard decisions. No one wants to own up to the mess.
My very first contract job was with a DMV processing company. I got settled in, did my work, poked around and came across a couple of boxes of Project Zeus. Going through it, it was high level OO design out the wazoo to replace a functioning system. After reading a bit, I asked the VP who ran my group about Project Zeus. He got a very concerned look on his face....
The story goes on from there.
This project went through a full year of design. Mind you design. Not at one time did the whiz kids ask the grunts who knew the working system how things worked. Slip after slip, finally the head VP said, when am I going to have a working system? The week before that date, the entire team quit.
Five years later, I come along. This place did have one nice feature - it had a cafeteria and cooks that could cook. Excellent blueberry pancakes. So, I'm sitting there eating my breakfast and the VP comes up, sits down just to chat.
He says, "you know, we're thinking about shutting this product down...."
me: "Why?"
Him: "Well it's not new tech."
me <thinking this="" system="" sits="" in="" the="" corner="" and="" makes="" 25="" million="" a="" year...="">me: "I'll buy it."
Him: "what?"
me: "you heard me. What's your price?"
Him: "haha, you're joking.."
me: "nope, point me in the right direction."
last I heard it's still running in the corner making millions.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
I totally agree with @OriginalGriff, time to quietly move on.
But, just for laughs, this situation always reminds me of this: Abbott & Costello - YouTube[^]
Graeme
"I fear not the man who has practiced ten thousand kicks one time, but I fear the man that has practiced one kick ten thousand times!" - Bruce Lee
|
|
|
|
|
"It's nearly done" has been the mantra of most programming teams since I have been in the business (1994).
What happens is that teams always do the easy stuff first. Naturally and with management's glowing approval and encouragement.
So of course the "almost done" parts are the parts that noone has yet figured out how to do.
This scenario will not change until programming managers wise up, and insist that teams COMPLETE THE HARD PARTS FIRST.
And LOL, good luck with that...
|
|
|
|
|
Member 9625785 wrote: COMPLETE THE HARD PARTS FIRST
All such broadly sweeping statements are problematic.
|
|
|
|
|
Including broad statements about broad statements I presume?
|
|
|
|
|
Yes, those are the broadest statements of all. It's like broad squared.
|
|
|
|
|
The 840 kW Solar system, along with the 2.5 MW generator system that supports it, went teats up Friday. I spent 12 hours Saturday, and 16 hours yesterday, trying to get it back running again, powering a million visitors per year resort. The first notice we had was a message that the generators had lost all communication with the solar controller that rules them. The way we're configured is a fiber link from the solar (plus battery) yard to the generators about 2 miles away. The message actually originated from the generators - I have no such alert function in the solar/battery installation.
Within the solar + battery installation I have an unmanaged switch that allows all of the components of the installation to work together and keep the lights on. During this event I asked the Battery supplier (Tesla) for a tool to allow me to view network traffic or log files for commands sent to the generators during this event. They told me that no such tool exists.
So I'm asking for a tool that let's me monitor and record network activity on a particular group of IP addresses. I'm not asking for any fancy tools, just a simple tool to allow creating a log of messages from one IP address to another.
Thanks, in advance, if you have anything to offer.
Will Rogers never met me.
|
|
|
|
|
The usual recommendation would be to use Wireshark
Problem one is that it can only analyze the traffic on the computer it sits on.
Problem 2 is that a switch is passing the traffic only between the nodes that are talking, if you would plug a PC with Wireshark installed to the switch it would only see broadcast messages on the network.
Solution one, use a hub instead of a switch. Sorry, going back to the stone age is hardly a solution.
So, the solutions are either to get a switch where you can monitor traffic directly, or program it to mirror a port to create a data tap.
What switch is it you have? Sometimes "unmanaged" switches aren't totally stupid, or rather sitting somewhere in between smart and stupid switches.
|
|
|
|
|
Thanks, Jorgan... I rather thought that would be the answer, but I was hopeful that there might be a way around that switch limitation. This switch is quite limited, I'm afraid. It's Hirschmann, and meant for industrial solutions, and therefor dumb but reliable. Sigh...
Will Rogers never met me.
|
|
|
|
|
Maybe find a router to replace the switch. Many routers will have some sort of configurable logging built into them so you'd just need one that was robust in that respect.
Alternatively, it's inelegant, but what is probably possible would be to use a basic proxy server through which the traffic routes bidirectionally between these things. It would require a different configuration of things though (to talk to the proxy server and let it handle the routing). Basically, use port forwarding to forward not just to a different internal port but also to a different internal IP. Wiresharking the proxy then should be able to give you what you want.
|
|
|
|
|
I'm not sure how you're going to monitor for an event that has already happened; and for which there is no log. All you will get up to a point, is "normal" traffic; in anticipation of an event that may never happen again under the same circumstances. One should be focusing on future "recovery" plan(s); instead of diagnostics; at this point (IMO).
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
|
|
|
|
|
A large part of diagnosing a problem is having visibility into what has recently happened - this system (by Tesla design) completely lacks that visibility. The situation cannot be improved until I have some means of seeing inside the black box and reading its history. If this happens again tonight, we are back to square one with no insight as to what should be done, or even where to investigate.
Will Rogers never met me.
|
|
|
|
|
I have to believe Tesla "has the tools"; but they're "company tools". I would try leaning on them. Talk about all your "connections". (On the other hand, some hardware vendors have used my tools to debug the new product my tool will use).
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
|
|
|
|
|
If you know something should do something every N-minutes/seconds or whatever, the lack of evidence is evidence of something lacking.
|
|
|
|
|
From the post, it is a "black box". There was an "event". And there is no "log". Where is the "evidence"?
Psycho-babble about "no evidence" being evidence, is nonsense.
I expect a "chance" of evidence before I go looking for it. Regardless of "orders".
This is like exception handling without "try blocks" ... or messages.
Try and post this in Q&A under: "It's working now, but it stopped working and I don't know why" ... and see what you get.
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
|
|
|
|
|
|
Interesting problem.
Ignoring cost I figured it should be easy to find a solution. But I didn't find anything at all.
Most suggestions point out that the device has to be configured to generate logs. So in your case the 'switch'.
That of course depends on the exact vendor, product, version, etc.
Presuming that there is a way to do that you are also going to need somewhere to store it. And quite possible there is going to be a lot of data.
You probably also want to set up a round robin. So it only stores, for example, last 7 days before reusing the storage space. If the device doesn't do that you will need to put a server in place to proxy that.
Then you will need to figure out how to analyze the data that you do get.
|
|
|
|
|
Surprisingly, I actually did find a tool for this. Port sniffers have been around in software for years, bit I discovered a number of physical traffic monitors out the on the interwebs. Since the Site Controller runs through a simple switch, then through a media converter to interface with the fiber running to the generators, I think I can install one in the path between the media converter and the switch. It won't help with the recent event, but it might be a huge help if this kind of failure ever occurs again. As you mentioned, too, I only need a few days' of recording depth. If nobody visits in that time, the Grand Canyon West resort has been nuked, or nothing bad happened.
Will Rogers never met me.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
