Contributed by deanna on from the dept.
Over the weekend, one of our OpenBSD servers, an internet router running OpenBGPD, had a fan die. Thanks to the sensors framework, and the Nagios plugin I wrote, I found out it was broken, and I could also tell that the rest of the fans in the server were doing a fine job keeping it cool. That means I was able to replace the fan at my convenience. Without the monitoring, I would probably not have noticed the fan being out until more fans died and the server overheated and failed.
After this, I got excited about the sensors again and updated the check with the ability to check the sensors that report their status. Since many sensors support this, it can make the size of your sensorsd.conf much smaller. For example, check_hw_sensors will automatically check these two sensors:
hw.sensors.76=esm0, Fan 4, OK, fanrpm, 3629 RPM hw.sensors.77=esm0, Fan 5, CRITICAL, fanrpm, 0 RPMIt will report the status listed to Nagios. For 76, it would report OK, for 77 it would report CRITICAL. You don't need to put anything in a config file to support those.
I have this check running on 10 servers with a variety of different hardware, checking a total of 273 sensors. It sure makes me sleep better knowing that if something breaks, I will get a text message on my cell phone letting me know.
The variety of hardware includes:
- 2 ISP1100
- 5 Dell PowerEdge 2450
- 1 Dell PowerEdge 4300
- 1 Dell PowerEdge 6450
- 1 Whitebox Celeron 300
Nagios is available at http://nagios.org, and Andrew's bio and sensors plugins are at http://openbsd.somedomain.net/nagios/
(Comments are closed)
By linc (199.18.139.77) on
Comments
By andrew fresh (66.185.224.30) andrew@mad-techies.org on http://openbsd.somedomain.net
> Are you handling everything over gige and handing off to switches or Ciscos?
> Just curious how you have this set up.
hah, gige, I wish we could get that sort of thing here in the boonies. We are still waiting to get a T3, but there is always "no facility" when they get close to installing. Currently the people trying to get us the bandwidth are in a lawsuit about it. Plus, up to about 20Mb it is cheaper to get it over T1 here.
The current setup (which is actually in the list archives) is like this. 4 T1s from one provider terminating into two Sangoma A102u cards in a PowerEdge 2450. Plus, 2 T1s from a second provider and 2 more from a third provider into more A102u cards in another 2450. Both routers running BGP, one session with each provider, one to each other and one to my core router. Those all terminate 100 Mb into my DMZ where I have some DNS servers, mail servers and web servers. Stuff that is more Internet facing than anything internal use.
Then, I have a core router that has 20 dc interfaces (currently several of them not in use, but waiting on colo customers), a couple of sks and an fxp in a PowerEdge 6450. The core router runs OpenBGP and OpenOSPF and has sessions with each of the Internet routers. It also terminates into the DMZ.
Eventually I want to get trunk set up on some of the ethernet interfaces on these routers, and set up a second core router that is carped with the first for redundancy, but I have not had time or hardware so far.
l8rZ,
Comments
By linc (199.18.139.77) on
> > Are you handling everything over gige and handing off to switches or Ciscos?
> > Just curious how you have this set up.
>
> hah, gige, I wish we could get that sort of thing here in the boonies. We are still waiting to get a T3, but there is always "no facility" when they get close to installing. Currently the people trying to get us the bandwidth are in a lawsuit about it. Plus, up to about 20Mb it is cheaper to get it over T1 here.
>
> The current setup (which is actually in the list archives) is like this. 4 T1s from one provider terminating into two Sangoma A102u cards in a PowerEdge 2450. Plus, 2 T1s from a second provider and 2 more from a third provider into more A102u cards in another 2450. Both routers running BGP, one session with each provider, one to each other and one to my core router. Those all terminate 100 Mb into my DMZ where I have some DNS servers, mail servers and web servers. Stuff that is more Internet facing than anything internal use.
>
> Then, I have a core router that has 20 dc interfaces (currently several of them not in use, but waiting on colo customers), a couple of sks and an fxp in a PowerEdge 6450. The core router runs OpenBGP and OpenOSPF and has sessions with each of the Internet routers. It also terminates into the DMZ.
>
> Eventually I want to get trunk set up on some of the ethernet interfaces on these routers, and set up a second core router that is carped with the first for redundancy, but I have not had time or hardware so far.
>
> l8rZ,
Very interesting. I was wondering how folks were implementing OpenBSD boxes as routers.
I work for a small telecom, and sometimes I wonder if something like that could handle our backbone. We currently use Junipers to do everything on the backbone and then just trunk everything out from there on switches.
By sthen (85.158.44.146) on
If you mostly use ethernet-type connections it's usually cheaper and easier to get lots of ports that way, and in fewer U... shifting a link from one router to another can be done without hands onsite too (manually or CARP). And, a bit of a silly reason, but it saves installing snmpd on the routers ...
I currently have a mixture of tagged vlans and direct ports but I'm seriously thinking of moving the remaining few direct ports onto switches (currently using trunk(4) for switch resilience which is quicker, but probably changing over to STP bridge(4) since it picks up a few more failure modes - comments on this subject invited :-)
Comments
By andrew fresh (66.185.224.30) andrew@mad-techies.org on http://openbsd.somedomain.net
> easier to get lots of ports that way, and in fewer U... shifting a link
> from one router to another can be done without hands onsite too
> (manually or CARP). And, a bit of a silly reason, but it saves
> installing snmpd on the routers ...
I wish we could do that for our Internet connections. I have requested that they terminate our new bandwidth as ethernet, but we shall see.
When we were last ordering bandwidth, the sales rep said "oh, we have this new thing, 10 megs over ethernet", it sounded kewl and way cheaper to get into the router, so we ordered that. A month or so later when we called to find out the status, "oh, you can't get that in your area". So, we ended up with 4 T1s instead, and Sangoma A102u cards are a LOT more than an ethernet card so that really sucked.
l8rZ,
Comments
By Anonymous Coward (199.18.139.77) on
> > easier to get lots of ports that way, and in fewer U... shifting a link
> > from one router to another can be done without hands onsite too
> > (manually or CARP). And, a bit of a silly reason, but it saves
> > installing snmpd on the routers ...
>
> I wish we could do that for our Internet connections. I have requested that they terminate our new bandwidth as ethernet, but we shall see.
>
> When we were last ordering bandwidth, the sales rep said "oh, we have this new thing, 10 megs over ethernet", it sounded kewl and way cheaper to get into the router, so we ordered that. A month or so later when we called to find out the status, "oh, you can't get that in your area". So, we ended up with 4 T1s instead, and Sangoma A102u cards are a LOT more than an ethernet card so that really sucked.
>
> l8rZ,
Yeah, but you are at least doing your peering on your OpenBSD box, so you can tweak and play. Now I'm wondering if you could use a couple of old 2600s or 1700s with T1 cards, but still do the peering on the PC?
Damnit, like I don't have enough pointless projects :)
Linc
Comments
By sthen (85.158.44.146) on
Yes, i-BGP there's no problem anyway, e-BGP you'll need configured as multihop (both your side and at the peer's side).
By sthen (85.158.44.146) on
Being on a relatively small island (the UK :-) there are some things that are easier to get done (and some things much more difficult of course!). ISPs have used telco "lan extension" (LES) circuits for quite some time to handle multiple customers in a city, and there are some telcos (3 or 4 come to mind) doing 'pseudo-wire' circuits, usually ethernet-over-mpls over their own backbone with various means for last-mile (including LES, microwave and SHDSL) which generally end up as ethernet - and the more sensible ones have direct presence at a lot of large carrier-neutral sites.
Fault diagnosis can be complicated in some cases - ethernet link status isn't usually carried across the network (only to the next box you're connected to) and where the cheaper last-mile circuits are involved there can be a couple of companies to contact to get things fixed. But hey, we have things like OSPF and STP to handle things until that's done :-)
It might make sense to connect to IP providers by ethernet at carrier-neutral colo sites and run your own WAN circuits from there rather than using provider's circuits - you can get boxes that you run at both sides which bridge 4x or 8x T1/E1 (and larger circuits) into ethernet (google for words like 'ethernet over t1', 't1 ethernet imux', 't1 ethernet inverse multiplexer'). I'm sure these bring their own problems to learn and work-around (for starters you might want to keep a bit closer eye than usual on mbuf use, altq may help) but can be a useful way to bridge the gap between T1/T3...
By DS (68.104.220.48) on
Would you consider posting these to the Nagios Exchange?
http://www.nagiosexchange.org/Would be easy to find in the future and available to more people.
DS
By David Gwynne (203.173.42.48) loki@animata.net on