Network Operations Center

[cod3poet]

So I use linux every 2 seconds of every day, Unless there is a major internet outage or I slip into a Coma this will be a fact of life for me for the rest of my days.

Just wondering where a good place to share my work life with you folks would be? Is here good?

Also Anyone else using linux @ work?

[Jason]

Just wondering where a good place to share my work life with you folks would be? Is here good?

Yep. Do you do development for a living?

[cod3poet]

Yes and No.

Yes I develop systems methods queries and management scripts. And I write code to tie all the systems together. No I do not write full web applications or software that get installed onto a computer. I create wrappers and automation every day though.

My Stack, CentOS linux because the latest Red Hat charges way too much for licensing vs Red Hat 6. I like RHEL however there are barriers to how much I can scale out when we get bigger. From a few nodes to a hundred or so.

Right now my test stuff is hosted out of https://www.ovh.com/ca/en/ which shows it's french bleeding through all it's admin panels but the hardware is inexpensive and works really well.

I am running zabbix http://www.zabbix.com/whats_new that aggregates all our montiored information and stores the metrics, then it runs threshold analysis and if it falls outside of the acceptible value it can alert or run an automated script.

IF value > 300 - Send alert and Wake Brian.
IF servicename status != running - Send alert and Wake Brian.
IF Host == Down 2+ hours - Send alert Fire Brian.

We also have metrics visualization http://grafana.org/ that updates every 5 seconds and can go back months so we can perform trend analysis to see Oh, it spikes every tuesday must be something automated. Yup it's a database backup.

All of this is running on PHP, Mysql, Javascript, and GO, running on Apache and Nodejs. Once I start scaling out I will be using https://saltstack.com/ to manage all my linux hosts so that I get continuity of configuration across the board.


The real kicker to all of this is that this entire system is designed to monitor and alert for pure windows networks. Because we needed higher stability than the networks we are monitoring we decided to go with linux.

I spend 100% of my time in the commandline working with this system, and the only time I spend in the web front end is making changes and administrative items so my team can do their jobs.

To look back and where I started in tech here in Peterborough, to where I am today I am very grateful to have access to a positive tech forward group of friends and colleagues and enough people saying "Wow that's really cool" rather than the glazed eyes response that it has kept me motivated to do all this and build this out.

So thanks to you for letting me share this.

P.s. Word of advice from production. Do not set your logging to verbose, and log rotation policy to none. You will end up with 138GB of log files in a week. I blame Martin. Tracking down disk usage in a more human way I use:

https://dev.yorhel.nl/ncdu

Because my world starts at 80x24.

[Jason]

That's pretty cool, Brian. It's interesting that Linux has the back of windows networks, that is, it helps keep track with your various scripts and monitoring software of Windows machines. It's kind of like Windows instability keeps you busy

Can you say what company you work for (or run?).

[cod3poet]

It's a work from home company that provides helpdesk services to managed service providers. They decided to dive into NOC so I get to fix all the blinky lights from my own gear at home. I also take care of our internal systems and have inherited a growing company's systems.

All in all it's been challenging but very rewarding and a definite flex of skills between windows / linux and networking.