OpenBSD Journal

Routers for the ISP and WISP

Contributed by marco on from the dept.

This is a direct quote from isp-planet.

In a discussion thread on the ISP-Wireless list concerning high end routers for WISPs, DF posted his opinions on the major players:

OpenBGPD/OpenBSD: Free OpenBSD-based EGP/IGP routing platform. Solid, secure, free, and very scalable.
Again, you're operating without vendor support. Non-standard of BGP functionality (modeled after PF).
Awesome integration with CARP and PF, makes for great firewalls, routers and route servers.
If you are system administrator and appreciate Unix, you will fall in love with OpenBGPD.
If you are a Linux admin, you will be surprised at the lack of learning curve involved. Community support is actually pretty good.

(Comments are closed)


Comments
  1. By nuintari (64.246.119.65) on

    We are currently using Mikrotiks running routerOS, which I personally hate. The OSPF implmentation in them is bogus, everyone of them seems to think it has to be in area 0, so you can forget about having any Miktotiks entirely inside a non 0 area. Yuck.

    I love the OpenOSPF system, I use it at home for testing, and I really want to deploy it on our network for a lot of the tower sites we have. However, we have a major issue with using OpenBSD for our tower based router: PPPoE.

    So, for a little FYI for those of you who don't run a WISP. DHCP is a BAD idea for a WISP. We use Canopy, with all the good encryption stuff turned on, so it is not like a random punk with a wireless laptop can hop on our network, but we still want to use a username/password auth system. Also, DHCP can be a nightmare if a customer wires up his brand new linksys the wrong way, and all of a sudden half the customers on the same segment are getting private IP's (we have a competitor who has this problem all the time, we decided to go the alternate route).

    Like I said, we use Canopy, so we _could_ vlan every single customer and dhcp assign a /30, but that is wasteful of space, and a royal pain to setup. Not even sure how I would automate that in the billing system. PPPoE uses radius, which was automated from when we were primarily a dialup provider. Super easy jump to add in support for the wireless customers.

    Where am I going with this? All that said, the PPPoE server in OpenBSD _sucks_. Throughput is just god awful. So, my choices are, working PPPoE service, and working, albeit, weird OSPF implmentation. Or, wonderful OSPF imp, and crappy PPP support. Sorry, I'll stick with the mikrotiks, much as I hate them.

    When OpenBSD has a better PPPoE server, I'll be the first to jump on board. But as it stands, I can only recommend it for a WISP if you are a) giving every customer a static address, or b) using a shitload of vlans.

    Sorry to ramble all over the place, I just got a new coffee maker, and I am a little wired.

    Comments
    1. By weka bidii (196.201.255.58) on

      why not just set it up so that you put run everything but PPPoE on the box and then just forward traffic to another box that has an implementation of PPPoE you like?


      Comments
      1. By nuintari (64.246.119.65) on

        Easy answer, several of them in fact.

        Power consumption on the towers is a big issue, Every device we add draws more off the batteries, the more power we consume, the more wires, and batteries, and chargers, we need in a little box at the site. Which leads me to my next point.

        Space is at a premium for WISP's. We don't have a huge datacenter at each site, we have a little box that has to stay watertight. Here, this is a typical box, has enough power in this configuration for 1 router, 6 radios, and 1 GPS sync unit. Like I said, space and power are at a premium.

        Access Point

        Comments
        1. By weka bidii (196.201.255.58) on

          ok, then how about virtualisation? Work is progressing on porting OpenBSD to Xen. Not sure how well it works atm, but certainly an option. You then can run something else to run the PPPoE. All on one box with minimal extra demands on RAM, CPU and disk space. Just a thought.

          Comments
          1. By nuintari (64.246.119.65) on

            Dunno if you'd get it to work on something small enough to fit in our enclosures. But it is something to contemplate.

    2. By Ely (72.193.175.51) on

      Maybe you could hire a developer to improve the PPPoE server until it works well enough to implement? Might make business sense for you...

      Comments
      1. By nuintari (64.246.119.65) on

        > Maybe you could hire a developer to improve the PPPoE server until it works well enough to implement? Might make business sense for you...

        That would be me, if there was enough incentive, my boss would say, "Nui, do it." But I am swamped enough as is, and the value of a better OSPF implementation is not high enough to the chief to rationalize me going nutty over an extensive software project.

    3. By ea1x (212.27.60.48) on

      > We are currently using Mikrotiks running routerOS, which I personally hate. The OSPF implmentation in them is bogus, everyone of them seems to think it has to be in area 0, so you can forget about having any Miktotiks entirely inside a non 0 area. Yuck.


      and for wireless link it sucks ?

      Comments
      1. By Anonymous Coward (64.246.119.65) on

        > > We are currently using Mikrotiks running routerOS, which I personally hate. The OSPF implmentation in them is bogus, everyone of them seems to think it has to be in area 0, so you can forget about having any Miktotiks entirely inside a non 0 area. Yuck.
        >
        >
        > and for wireless link it sucks ?

        Not really, just that every tower has to be in area 0, which is not a huge problem right now, we have 14 tower sites. But if we keep growing at the rate we are, we will eventually want to use more than 1 area, to keep the LSA's down, and the size of the routing table in each router. We can't do that with the current routerOS builds.

    4. By claudio@ (62.48.30.1) on

      > I love the OpenOSPF system, I use it at home for testing, and I really
      > want to deploy it on our network for a lot of the tower sites we have.
      > However, we have a major issue with using OpenBSD for our tower based
      > router: PPPoE.
      >
      > Where am I going with this? All that said, the PPPoE server in OpenBSD
      > _sucks_. Throughput is just god awful. So, my choices are, working PPPoE
      > service, and working, albeit, weird OSPF implmentation. Or, wonderful
      > OSPF imp, and crappy PPP support. Sorry, I'll stick with the mikrotiks,
      > much as I hate them.
      >

      We know that our pppoe solution sucks for ISP. It was written for end
      customers and not for ISPs. The problem is that pppoe(8) runs in userland
      and uses bpf(4) to capture the packets and tun(4) via ppp(8) to write the
      packets back. So in the end every packet is copied multiple times between
      kernel and userland which results in bad performance.
      On the otherhand pppoe(4) runs in the kernel but does not support server
      mode and all the needed goodies like radius aaa. This comes from the fact
      that sppp(4) has the control plan in the kernel -- which was a bad idea.

      In the end somebody needs to sit down and rewrite/merge/split the
      different ppp implementations into one scalable, fast thing. This is not
      something you do in an afternoon. So if you really want to switch you
      should not only whine about it but change it. Either by doing it yourself
      or by paining someone to do it for you.

      --
      :wq Claudio

      Comments
      1. By Anonymous Coward (24.37.236.100) on

        > > I love the OpenOSPF system, I use it at home for testing, and I really
        > > want to deploy it on our network for a lot of the tower sites we have.
        > > However, we have a major issue with using OpenBSD for our tower based
        > > router: PPPoE.
        > >
        > > Where am I going with this? All that said, the PPPoE server in OpenBSD
        > > _sucks_. Throughput is just god awful. So, my choices are, working PPPoE
        > > service, and working, albeit, weird OSPF implmentation. Or, wonderful
        > > OSPF imp, and crappy PPP support. Sorry, I'll stick with the mikrotiks,
        > > much as I hate them.
        > >
        >
        > We know that our pppoe solution sucks for ISP. It was written for end
        > customers and not for ISPs. The problem is that pppoe(8) runs in userland
        > and uses bpf(4) to capture the packets and tun(4) via ppp(8) to write the
        > packets back. So in the end every packet is copied multiple times between
        > kernel and userland which results in bad performance.
        > On the otherhand pppoe(4) runs in the kernel but does not support server
        > mode and all the needed goodies like radius aaa. This comes from the fact
        > that sppp(4) has the control plan in the kernel -- which was a bad idea.
        >
        > In the end somebody needs to sit down and rewrite/merge/split the
        > different ppp implementations into one scalable, fast thing. This is not
        > something you do in an afternoon. So if you really want to switch you
        > should not only whine about it but change it. Either by doing it yourself
        > or by paining someone to do it for you.
        >
        > --
        > :wq Claudio

        Do I smell OpenPPPoE in the near future? >=)

      2. By nuintari (64.246.119.65) on

        > We know that our pppoe solution sucks for ISP. It was written for end
        > customers and not for ISPs. The problem is that pppoe(8) runs in userland
        > and uses bpf(4) to capture the packets and tun(4) via ppp(8) to write the
        > packets back. So in the end every packet is copied multiple times between
        > kernel and userland which results in bad performance.
        > On the otherhand pppoe(4) runs in the kernel but does not support server
        > mode and all the needed goodies like radius aaa. This comes from the fact
        > that sppp(4) has the control plan in the kernel -- which was a bad idea.

        Yes, I have read all the pertinent info on it, know exactly what is wrong with it and all that jazz. I was just saying, "here is my wishlist", however....
        >
        > In the end somebody needs to sit down and rewrite/merge/split the
        > different ppp implementations into one scalable, fast thing. This is not
        > something you do in an afternoon. So if you really want to switch you
        > should not only whine about it but change it. Either by doing it yourself
        > or by paining someone to do it for you.

        Who said anything about whining? I would love to take care of it, problem is, I work about 60 hours a week as is, I am still overloaded with projects piling up faster than I can knock em down, and like I said, my current system "basically works," I just happen to hate it.

    5. By Anonymous Coward (217.116.110.188) on

      > We are currently using Mikrotiks running routerOS, which I personally hate. The OSPF implmentation in them is bogus, everyone of them seems to think it has to be in area 0, so you can forget about having any Miktotiks entirely inside a non 0 area. Yuck.
      >
      > I love the OpenOSPF system, I use it at home for testing, and I really want to deploy it on our network for a lot of the tower sites we have. However, we have a major issue with using OpenBSD for our tower based router: PPPoE.

      If you already use Mikrotik, which is Linux-based, you might want to take a look at Lintrack [1], which is free and open source, has Quagga and RP's PPPoE server included.

      [1] http://www.lintrack.org/

  2. By Anonymous Coward (81.216.200.10) on

    "Community support is actually pretty good."

    Hahahaha, good one. Official OpenBSD support goes like this:
    Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
    A: Fuck off and die! Go back to Linux and get hacked, you retard!

    Also: Lack of learning curve? The curve might be flat, but lack of one? That's just stupid.

    Comments
    1. By Anonymous Coward (80.57.127.8) on

      > "Community support is actually pretty good."
      >
      > Hahahaha, good one. Official OpenBSD support goes like this:
      > Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
      > A: Fuck off and die! Go back to Linux and get hacked, you retard!
      >
      > Also: Lack of learning curve? The curve might be flat, but lack of one? That's just stupid.

      I've had answers to all my questions! Thanks guys :)

    2. By Anonymous Coward (24.37.236.100) on

      > "Community support is actually pretty good."
      >
      > Hahahaha, good one. Official OpenBSD support goes like this:
      > Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
      > A: Fuck off and die! Go back to Linux and get hacked, you retard!
      >
      > Also: Lack of learning curve? The curve might be flat, but lack of one? That's just stupid.

      Where do you get "Official OpenBSD Support" that's live like that, with responses??? I think you're mixing IRC (sounds like efnet) or something else with "Official Support" because AFAIK, there is no "Official Support" other than the FAQ & man pages - the rest is unofficial, which means you can get responses from a lot of assholes and nicer responses from others.

      Comments
      1. By nuintari (64.246.119.65) on

        > > "Community support is actually pretty good."
        > >
        > > Hahahaha, good one. Official OpenBSD support goes like this:
        > > Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
        > > A: Fuck off and die! Go back to Linux and get hacked, you retard!
        > >
        > > Also: Lack of learning curve? The curve might be flat, but lack of one? That's just stupid.
        >
        > Where do you get "Official OpenBSD Support" that's live like that, with responses??? I think you're mixing IRC (sounds like efnet) or something else with "Official Support" because AFAIK, there is no "Official Support" other than the FAQ & man pages - the rest is unofficial, which means you can get responses from a lot of assholes and nicer responses from others.

        I deal with a pile of technical, and non-technical people everyday, and I must say I totally understand why some of the guys default to assuming all questions come from a moron who hasn't even tried to read the docs.

        I do read the docs, and have yet to need to send any questions to tech@ or misc@, but I do read them, and I think the attitude issue is a) overrated, and b) well deserved when it does happen.

        I'd contribute more, but I have enough stress in my life without all the Linux newbs bugging me to tell em how to install RPM's in obsd.

      2. By Anonymous Coward (81.216.200.10) on

        > > "Community support is actually pretty good."
        > >
        > > Hahahaha, good one. Official OpenBSD support goes like this:
        > > Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
        > > A: Fuck off and die! Go back to Linux and get hacked, you retard!
        > >
        > > Also: Lack of learning curve? The curve might be flat, but lack of one? That's just stupid.
        >
        > Where do you get "Official OpenBSD Support" that's live like that, with responses??? I think you're mixing IRC (sounds like efnet) or something else with "Official Support" because AFAIK, there is no "Official Support" other than the FAQ & man pages - the rest is unofficial, which means you can get responses from a lot of assholes and nicer responses from others.


        The example question above is valid and I couldn't find it documented, and it's the kind of question that gets that sort of reply on email lists from head people of OpenBSD. And god help you if you ask about something OpenBSD can't to. They'll send hitmen after you nagging you that YOU DON'T REALLY NEED THAT, WE KNOW BEST ABOUT YOUR PARTICULAR SITUATION!

        Comments
        1. By jb (jb) on

          > > > "Community support is actually pretty good."

          It really is, if you bother to look something up first.

          > The example question above is valid and I couldn't find it documented, and it's the kind of question that gets that sort of reply on email lists from head people of OpenBSD. And god help you if you ask about something OpenBSD can't to. They'll send hitmen after you nagging you that YOU DON'T REALLY NEED THAT, WE KNOW BEST ABOUT YOUR PARTICULAR SITUATION!

          Perhaps it's not documented because it's not a usable feature? I've found that handling inbound and outbound traffic on the same interface is better than doing them in the same line. Do you always have UDP and TCP traffic in the same lines?

          Secondly, you could always just do a "pass on $INTERFACE from $HOST to ($INTERFACE) port $SERVICE"

          It's pretty explicitly stated in the <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=pf.conf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html'>man page</a> that it will.

          From pf.conf(5):
          <pre>
          in or out
          This rule applies to incoming or outgoing packets. If neither in
          nor out are specified, the rule will match packets in both direc-
          tions.
          </pre>

          So, your example is a void one, since anyone who's bothered to read the pf FAQ and the man page should have seen that explained.

          Comments
          1. By Anonymous Coward (81.216.200.10) on

            > Secondly, you could always just do a "pass on $INTERFACE from $HOST to ($INTERFACE) port $SERVICE"

            No, I want to do something like:

            pass on fxp0 to (fxp1:network)

            except :network should include multihops as well, that is "is to be routed out fxp1". (fxp1) would just be short for whatever IP(s) I have assigned to fxp1, right?

            iptables -I FORWARD -i eth0 -o eth1 -j ACCEPT

            Also, I want to combine rdr and nat to bounce on my server, but last I checked, you can't rdr out the same interface, so no dice.

            Also, I want to traffic-shape on incoming traffic. Stop it, don't say it. Yes I do. YOU may not want to, but lots of people do.

            Comments
            1. By Anonymous Coward (82.40.182.26) on

              > Also, I want to traffic-shape on incoming traffic. Stop it, don't say
              > it. Yes I do. YOU may not want to, but lots of people do.
              >

              I'm sure lots of people want to, however a little thought will reveal to you why this is not possible. You can only meaningfully assign outbound traffic to queues. This is not a limitation of PF it is fundamental to the network, you can't limit how fast people send you packets. This is why DDoS works.

              Comments
              1. By Anonymous Coward (81.216.200.10) on

                > I'm sure lots of people want to, however a little thought will reveal to you why this is not possible.

                This, and the rest of the comment is exactly what I'm talking about.
                1) You are wrong
                2) Other systems can do it

                YES! For gods sake YES, we are aware this will not stop a DoS, that's NOT what this is about. It has nothing to do with keeping your line from being filled by a malicious user.

                This is about maybe limiting how fast someone can scp TO your server, without having control of an external firewall.

                And in the spirit of OpenBSD I must add: you stupid retard!

                Comments
                1. By CODOR (CODOR) on

                  > This is about maybe limiting how fast someone can scp TO your server, without having control of an external firewall.

                  Won't setting up queueing on the internal interface do this?

                  Comments
                  1. By Anonymous Coward (81.216.200.10) on

                    > > This is about maybe limiting how fast someone can scp TO your server, without having control of an external firewall.
                    > Won't setting up queueing on the internal interface do this?

                    Judging from what's gotten through the cursing and "no you don't want to do that"; no, that's not possible with openbsd.

                    People have been telling openbsd-developers for years that "yes, we do".

                  2. By Anonymous Coward (81.216.200.10) on

                    > > This is about maybe limiting how fast someone can scp TO your server, without having control of an external firewall.
                    >
                    > Won't setting up queueing on the internal interface do this?

                    Oh, and what internal interface? lo0? This is not a firewall.

                  3. By Anonymous Coward (81.216.200.10) on

                    > > This is about maybe limiting how fast someone can scp TO your server, without having control of an external firewall.
                    >
                    > Won't setting up queueing on the internal interface do this?

                    FYI:

                    pass in on fxp0 proto tcp from any to any port 22 flags S/SA \
                    keep state queue ssh

                    PF will record the queue in the state table entry so that packets traveling back out fxp0 that match the stateful connection will end up in the ssh queue. Note that even though the queue keyword is being used on a rule filtering incoming traffic, the goal is to specify a queue for the corresponding outgoing traffic; the above rule does ****NOT**** queue incoming packets.


                    ------
                    emphasis added by me

                    Comments
                    1. By Anonymous Coward (57.79.167.12) on


                      > FYI:
                      >
                      > pass in on fxp0 proto tcp from any to any port 22 flags S/SA \
                      > keep state queue ssh
                      >
                      > PF will record the queue in the state table entry so that packets traveling back out fxp0 that match the stateful connection will end up in the ssh queue. Note that even though the queue keyword is being used on a rule filtering incoming traffic, the goal is to specify a queue for the corresponding outgoing traffic; the above rule does ****NOT**** queue incoming packets.
                      >
                      >
                      > ------
                      > emphasis added by me

                      I forgot to write that this was from the FAQ

            2. By sthen (85.158.44.149) on

              > > Secondly, you could always just do a "pass on $INTERFACE from $HOST to ($INTERFACE) port $SERVICE"
              >
              > No, I want to do something like:
              >
              > pass on fxp0 to (fxp1:network)
              >
              > except :network should include multihops as well, that is "is to be routed out fxp1". (fxp1) would just be short for whatever IP(s) I have assigned to fxp1, right?
              >
              > iptables -I FORWARD -i eth0 -o eth1 -j ACCEPT

              you can do something similar with

              pass in on fxp1 tag From_fxp1
              pass out on fxp0 tagged From_fxp1

              > Also, I want to combine rdr and nat to bounce on my server, but last I checked, you can't rdr out the same interface, so no dice.

              Nothing stopping you. But you do need the nat as well as the rdr.

              Comments
              1. By Anonymous Coward (81.216.200.10) on

                > pass in on fxp1 tag From_fxp1
                > pass out on fxp0 tagged From_fxp1

                You did not read the simple question. I clearly said, in the same rule. Scales better that way for readability. And it's an example.

                > > Also, I want to combine rdr and nat to bounce on my server, but last I checked, you can't rdr out the same interface, so no dice.
                >
                > Nothing stopping you. But you do need the nat as well as the rdr.

                Really? Since when? Last I checked, you are NOT allowed to go out the same interface you went in.

                Comments
                1. By sthen (85.158.44.148) on

                  > > pass in on fxp1 tag From_fxp1
                  > > pass out on fxp0 tagged From_fxp1
                  >
                  > You did not read the simple question. I clearly said, in the same rule. Scales better that way for readability. And it's an example.

                  You didn't say anything about that in the direct question you asked. You said «No, I want to do something like: pass on fxp0 to (fxp1:network) except :network should include multihops as well, that is "is to be routed out fxp1"» Why should I tramp through the other posts to find some other anonymous post and work out that it's the same person when I'm trying to help out?

                  If you must have it in a single rule, write a rule generator of some kind, or use iptables, but if you choose iptables, don't go talking about readability...

                  > > > Also, I want to combine rdr and nat to bounce on my server, but last I checked, you can't rdr out the same interface, so no dice.
                  > >
                  > > Nothing stopping you. But you do need the nat as well as the rdr.
                  >
                  > Really? Since when? Last I checked, you are NOT allowed to go out the same interface you went in.

                  Since always. www/faq/pf/rdr.html explains all about this (even the first revision of the file from nearly 4 years ago). You can't have that in a single rule either, though.

                  Comments
                  1. By Anonymous Coward (57.79.167.12) on

                    > > You did not read the simple question. I clearly said, in the same rule. Scales better that way for readability. And it's an example.
                    > You didn't say anything about that in the direct question you asked.

                    Quote: "How do I filter in pf on both incoming and outgoing interface in the same rule?"

                    > but if you choose iptables, don't go talking about readability...

                    That's a matter of taste and what you're used to. I think it's clearer than pf. I didn't bring it up since it's a matter of taste.

                    Easier to first get started? Sure, pf wins there. Complex firewall rules? Gak! pf degenerates into a mess.

                    > Since always. www/faq/pf/rdr.html explains all about this (even the first revision of the file from nearly 4 years ago). You can't have that in a single rule either, though.

                    Have you *tried* it? Admittedly I hadn't seen that (discouraged) solution, and haven't tried it. It's second-hand info, which everyone so far has confirmed.

                    Comments
                    1. By sthen (85.158.44.148) on

                      > > > You did not read the simple question. I clearly said, in the same rule. Scales better that way for readability. And it's an example.
                      > > You didn't say anything about that in the direct question you asked.
                      >
                      > Quote: "How do I filter in pf on both incoming and outgoing interface in the same rule?"

                      If you'd referred to the other post when you talked about 'like :net but multihop' (paraphrasing) I might have found it. It might be considered polite to make it clear you're just making an example rather than asking for actual assistance too...

                      > > Since always. www/faq/pf/rdr.html explains all about this (even the first revision of the file from nearly 4 years ago). You can't have that in a single rule either, though.
                      >
                      > Have you *tried* it?

                      Yes, it works fine. The addresses don't even have to be local.

            3. By Daniel Hartmeier (dhartmei) on

              It's a good thing to know precisely what you want.

              Now all you have to figure out is how to achieve those goals.

              Comments
              1. By Anonymous Coward (81.216.200.10) on

                > Now all you have to figure out is how to achieve those goals.

                Yes, and with OpenBSD it's often "not with openbsd you can't".

                Well, actually the reply is "YOU STUPID RETARD, YOU DON'T WANT THAT AT ALL. WE KNOW WHAT YOU WANT. SIT DOWN AND SHUT UP".

                Comments
                1. By Daniel Hartmeier (dhartmei) on janitor

                  > > Now all you have to figure out is how to achieve those goals.
                  >
                  > Yes, and with OpenBSD it's often "not with openbsd you can't".

                  Your strategy of achieving a goal is to tell someone about your needs and repeat yourself until your needs are being met?

                  Does that work well for you in other contexts where you're not a paying customer?

                  Comments
                  1. By Anonymous Coward (81.216.200.10) on

                    > Your strategy of achieving a goal is to tell someone about your needs and repeat yourself until your needs are being met?

                    No, it's usually three steps:
                    1) I ask how I do something
                    2) people say, through "no you don't want that at all" and cursing, that openbsd can't do that
                    3) I solve the problem without openbsd

                    I repeat myself here because you don't seem to understand, but that's outside the three-step process outlined above.

                    > Does that work well for you in other contexts where you're not a paying customer?

                    While comments such as this does not help or hurt you really, since I'm not (and probably won't ever be) a customer of yours, it does discourage people from using or developing for openbsd. If the openbsd project didn't care about that, why does it have an advocacy-list?

                    And even if someone won't be YOUR customer, bashing them for no reason except for your elitism will only hurt in the long run.

                    Compare this to the commercials where the message is "Buy a new car -- The automobil industry" where there isn't even a name of a manufacurer, simply "Buy a new car from SOMEONE" does help everyone in the industry.

            4. By Lars Hansson (203.65.246.12) lars@unet.net.ph on

              > Also, I want to traffic-shape on incoming traffic. Stop it, don't say it. Yes I do. YOU may not want to, but lots of people do.

              We all want the impossible and I'm sure there are companies out there ready to sell you a solution that supposedly does that.
              Meanwhile, in the real world, you can only shape incoming traffic indirectly since, by law of nature, by the time you get the packets they have already gone over the wire. Indirect shaping is not that hard to do and it sort of works well enough. It has also been covered a billion times on the mailinglists.

              Comments
              1. By Anonymous Coward (81.216.200.10) on

                > > Also, I want to traffic-shape on incoming traffic. Stop it, don't say it. Yes I do. YOU may not want to, but lots of people do.
                >
                > We all want the impossible and I'm sure there are companies out there ready to sell you a solution that supposedly does that.

                Yes, just like W^X was impossible on x86 (what? the competition already does it? what? itlb cache and dtlbcache? I thought there was only one!)

                > Meanwhile, in the real world, you can only shape incoming traffic indirectly since, by law of nature, by the time you get the packets they have already gone over the wire. Indirect shaping is not that hard to do and it sort of works well enough. It has also been covered a billion times on the mailinglists.

                I'm gonna have to call bullshit on that. Answers to this question is always "no, you don't want to", "it can't be done even in theory" (both of these are basically "I'm too stupid to see what you mean") or "can't be done with openbsd".


    3. By Toby (68.148.1.194) weingart@tepid.org on

      > "Community support is actually pretty good."
      >
      > Hahahaha, good one. Official OpenBSD support goes like this:
      > Q: How do I filter in pf on both incoming and outgoing interface in the same rule?
      > A: Fuck off and die! Go back to Linux and get hacked, you retard!

      I'm somewhat surprised. Community Support != Official Support. If you
      paid some company for official support, and that was the response you
      got, I'd say "move on". If you need, I'd be more than happy to point
      you at some of the good companies giving OpenBSD support.

      --Toby.

  3. By daver (83.13.175.18) on

    Juniper: Great gear, expensive, very high performance. An M7i with next day 24/7 support, 2 port DS-3, redundant DC power, 850 Mhz route engine, 1.5 GB RAM, 256M B Flash, and 20 GB HDD is about $25,000. I believe the J-series runs Linux.
    ^^^^^
    Mistake.

    It's *BSD http://en.wikipedia.org/wiki/Berkeley_Software_Distribution


    daver

    Comments
    1. By Amir S Mesry (208.52.133.98) on

      > Juniper: Great gear, expensive, very high performance. An M7i with next day 24/7 support, 2 port DS-3, redundant DC power, 850 Mhz route engine, 1.5 GB RAM, 256M B Flash, and 20 GB HDD is about $25,000. I believe the J-series runs Linux.
      > ^^^^^
      > Mistake.
      >
      > It's *BSD http://en.wikipedia.org/wiki/Berkeley_Software_Distribution
      >
      >
      > daver
      >

      Yep, JunOS is BSD based, they know what they are doing. Thats one reason they are in a lot of network cores.

  4. By Dan (80.178.87.106) on

    I don't think OpenBSD can be a high end router.

    As long the routing and filtering is done in software, it will not be a match to most L3 switches.

    Even this 1400$ switch will outperform OpenBSD: http://tinyurl.com/yqngug

    Don't get me wrong, I love OpenBSD. And it got great features, which I know will work and are well documented.

    Comments
    1. By Nate (Nate) on

      Okay then, um-- I'd like to go ahead and, yeah, well, go ahead and disagree with you there, Dan, the hardware in a, uh, Cisco machine is generally nothing to get overly excited about. It's sort of, a, tiny machine with a specialized operating system on it, that, well, runs effeciently in it's place.

      With the same exact hardware, sure, OpenBSD would, you know, get out performed, but with a box custom selected for the job, the price/performance ratio wouldn't be too different and the, uh, systems administrator guy, would gain the increased tools and, uh, managability available to OpenBSD.

      With Cisco IOS you have little you can do, at least with OpenBSD you have the ability to, you know, change things, look at things, ask questions about how things ought to be, maybe see if there are ways to make things run a little smoother. Other than just, uh, getting more parts from Cisco. Even most of the, uh, hardware upgrades are cheaper on a random, uh, PC box. Yeah.

      Oh, and remember, next Friday is Hawaiian shirt day. So, you know, if you want to, go ahead and, uh, wear a Hawaiian shirt and jeans.

      Comments
      1. By nuintari (64.246.109.26) on

        > Okay then, um-- I'd like to go ahead and, yeah, well, go ahead and disagree with you there, Dan, the hardware in a, uh, Cisco machine is generally nothing to get overly excited about. It's sort of, a, tiny machine with a specialized operating system on it, that, well, runs effeciently in it's place.
        >
        > With the same exact hardware, sure, OpenBSD would, you know, get out performed, but with a box custom selected for the job, the price/performance ratio wouldn't be too different and the, uh, systems administrator guy, would gain the increased tools and, uh, managability available to OpenBSD.
        >
        > With Cisco IOS you have little you can do, at least with OpenBSD you have the ability to, you know, change things, look at things, ask questions about how things ought to be, maybe see if there are ways to make things run a little smoother. Other than just, uh, getting more parts from Cisco. Even most of the, uh, hardware upgrades are cheaper on a random, uh, PC box. Yeah.
        >
        > Oh, and remember, next Friday is Hawaiian shirt day. So, you know, if you want to, go ahead and, uh, wear a Hawaiian shirt and jeans.

        Okay, so here is why none of that matters. I have explained this to about a million people, so I will make this brief. *update, I failed to make it brief, sorry*

        Cisco and other expensive router vendors have specialized hardware that is far more efficient at routing packets than a conventional PC. And it is NOT about having a faster CPU, because that isn't where the bottle neck is. The individual line cards have a lot of control over where packets go, and can speak directly to each other. The CPU is used to manage the routing and flow tables, and route the few packets that it actually needs to think about. 99% of all the packets going through a cisco or juniper are "express forwarded", meaning the packet never hits the cpu, it just goes from the incoming line card, over the bus, to the outbound line card.

        Even an OpenBSD router with a massive CPU and tons of ram doesn't do this, as PC hardware wasn't designed with it in mind. Nor can it keep pace with weaker hardware designed specifically for routing. Everything is done in software, which means, it is far easier to overload.

        This is also why the route processor in your average cisco can crash, and the thing will still pass traffic while IOS reloads. Just don't expect it to make any updates to the routing table for a bit. Might drop a lot of ICMP traffic for a while too.

        Want to see what happens when you tell a cisco router to CPU route everything? sign into one, and type this:

        enable
        <password>
        conf t
        no ip cef

        watch it groan to a crawl

        This isn't noticeable on a pc router with a pair of nics in it, but try a bigger setup with a bunch of different interfaces. You can really bog down the CPU, and more importantly, the bus. PC's were not designed to handle traffic coming and going on 6 different gigE line cards all at once at high capacity. It is nothing against OpenBSD, I use it for routers myself. But one should know the real reason why ciscos are still useful, even if you plan on deploying OpenBSD as a major router platform in your infrastructure.

        Comments
        1. By Dan (80.178.87.106) on

          This is not how Cisco's routers works.

          Generally, there are two types of routers; software and hardware routers:

          Software: (18XX,28XX,38XX and older routers)
          _All_ packets goes through the CPU. CEF (Cisco express forwarding) is what makes a CPU running Cisco forward packet faster then OpenBSD on the same CPU. Here is the trick: Don't go through ip_input(). For every network interrupt it will use a cache to try and put the packet directly on the output interface Queue during the interrupt context. This of course got to be very very quick, thats what special about CEF. _All_ the routing table is cached.
          IIRC in OpenBSD the interrupt is putting the packet in mbuf and queue it for ip_input() processing.


          Hardware: (L3 switches, some special 7XXX engines)
          The CEF table is populated to ASICs, also the access-lists are populated to special memory/asic structures called TCAM.
          Each packet going through does not reach the CPU, and it routed using hardware at _wire speed_. You can buy on eBay a used Cisco L3 switch costing 1500$ to route millions packets per second. No OpenBSD hardware can match it.

          But then again, I would trust OpenBSD router much MUCH _MUCH_ more then any cisco device.

          Dan

          Comments
          1. By nuintari (64.246.109.26) on

            > This is not how Cisco's routers works.
            well, I was a bit off on some finer points of my collected knowledge, but my point was valid. You can't rheow lots of hardware at an OpenBSD router and claim it'll beat one from Cisco or Juniper.
            >
            > Generally, there are two types of routers; software and hardware routers:
            >
            > Software: (18XX,28XX,38XX and older routers)
            > _All_ packets goes through the CPU. CEF (Cisco express forwarding) is what makes a CPU running Cisco forward packet faster then OpenBSD on the same CPU. Here is the trick: Don't go through ip_input(). For every network interrupt it will use a cache to try and put the packet directly on the output interface Queue during the interrupt context. This of course got to be very very quick, thats what special about CEF. _All_ the routing table is cached.
            Oh, I thought cef cached the routing table on the line cards, this explains a lot about the behavior of our cisco 26xx at the office here....
            > IIRC in OpenBSD the interrupt is putting the packet in mbuf and queue it for ip_input() processing.
            Like I said, this explains a lot about our 26xx at the office. Like why it groans to a halt whenever you make a change to it. It has to recalc all of that cef data, and death occurs for about 45 seconds. fortunately, we never change stuff over here.
            >
            >
            > Hardware: (L3 switches, some special 7XXX engines)
            > The CEF table is populated to ASICs, also the access-lists are populated to special memory/asic structures called TCAM.
            > Each packet going through does not reach the CPU, and it routed using hardware at _wire speed_. You can buy on eBay a used Cisco L3 switch costing 1500$ to route millions packets per second. No OpenBSD hardware can match it.
            Yeah, this is how our 7507's work to my understanding, but you certainly explained it better than I did. Thanks!
            >
            > But then again, I would trust OpenBSD router much MUCH _MUCH_ more then any cisco device.
            Considering at one point our choices for new IOS software were "local crash exploit", "remote security exploit", and "really new and untested and probably full of exploits." The local crash exploit replaced a previously used, buggy as sin version of IOS we needed to get away from.

            Thanks for your details. I didn't know those differences.

            Comments
            1. By nuintari (64.246.109.26) on

              Damnit, I meant to click Preview!

            2. By Dan (87.69.56.199) on

              Actually the 7500 does not got any ASICs.

              The trick on the 7500 is caleed DCEF. Or distributed CEF.
              Each slot is using VIP, which is a normal CPU on that slot. Each VIP got its own CEF table and will do the routing on behalf the main CPU on the RSP.

          2. By Anonymous Coward (57.79.167.12) on

            > But then again, I would trust OpenBSD router much MUCH _MUCH_ more then any cisco device.

            To do what? Not crash? Even IF openbsd was perfectly stable (and it's most assuredly not) I wouldn't trust PC hardware to be as stable as cisco hw.

            Also, while a proper backup scheme will make replacing any system easy, a PC (or similar)-based HW will never beat the "put another 2600 in place, push conf" that dedicated firewall has.

        2. By Dan Farrell (66.252.236.93) danno@appliedi.net on http://danno.appliedi.net/drupal/

          > This isn't noticeable on a pc router with a pair of nics in it, but try a bigger setup with a bunch of different interfaces. You can really bog down the CPU, and more importantly, the bus. PC's were not designed to handle traffic coming and going on 6 different gigE line cards all at once at high capacity. It is nothing against OpenBSD, I use it for routers myself. But one should know the real reason why ciscos are still useful, even if you plan on deploying OpenBSD as a major router platform in your infrastructure.

          Useful for who and at what level? I think the post here is a bit deceptive, considering what level of traffic and service offering you have to be at before the Cisco-type devices outstrip today's (not yesterday's) commodity-based PC hardware.

          To keep this post brief I won't ponder every flavor of PCI-X, but sufficed to say PCI-Express X4 lanes can deliver 8 Gbps of traffic, and it only gets better from there. With 64-bit dual-core and quad core CPU's, and 1033 Mhz FSB's, you have to push quite a bit of bandwidth to push the limits of PC commodity hardware.

          So how much bandwidth and what level of service offering do you have to reach before your statement becomes accurate? 6 Gbps of traffic, 12?

          And then you have to ask yourself, how many providers push that much traffic through a SINGLE router?

          Your argument becomes accurate when comparing a Cisco 1200-series router being stacked against a PIII-800. Sure, if you are running AT+T's pop in Herndon, you might want something other than your workstation in place.

          Remember, Cisco uses the hardware they use not because it is 'better' but because it's cheaper. They are trying to make a buck... and your argument is what they use to convince the meek and ignorant that their products are superior to an Open Source commodity-based implementation. Basically, as one put it, you just drank the marketing kool-aid.

    2. By Lars Hansson (203.65.246.12) lars@unet.net.ph on

      > I don't think OpenBSD can be a high end router.

      Depends on your definition of "high end". Multi-gigabit flows? Probably not. A couple of hundred megabits? Sure.

      > Even this 1400$ switch will outperform OpenBSD:

      It's a switch, not a router.

      Comments
      1. By Dan (80.178.87.106) on

        Its a Layer3 switch, which means it does routing at _wire speed_, millions packets per second, and it runs OSPF,BGP,RIP...

        Comments
        1. By sthen (85.158.44.149) on

          > Its a Layer3 switch, which means it does routing at _wire speed_, millions packets per second, and it runs OSPF,BGP,RIP...

          I'm not sure what it costs to transfer licenses on a 3550 (and with ebay it's usually safe to say that EMIs haven't been purchased through legitimate channels), but buying new to avoid license-transfers and using a similar platform but with more than 6 months before software maintenance ceases, you're looking at more like $4000 for 3560-24TS-E and SFPs (because without faster uplinks there's not a huge advantage).

          And you still need a route-server (and networks happy to run multihop sessions) if you want to take full routing tables over it ...

          If you're pushing up to around 3-400Mb per router at reasonable packet sizes, using OpenBSD routers is pretty attractive and not too hard to achieve, there's a range above that needing more careful hardware selections but still do-able now, then you're into specialized-hardware territory (though I'm wondering if the I/OAT stuff will help there someday).

        2. By Anonymous Coward (74.115.21.120) on

          > Its a Layer3 switch, which means it does routing at _wire speed_, millions packets per second, and it runs OSPF,BGP,RIP...

          No, it means cisco says that, but they are lying. It switches at switch speed, and routes horribly slow. And if you use any ACLs at all, you can just forget about moving packets at all. Ddos attacks routinely overwhelmed our 6500, we had to turn it into just an expensive switch, and use a $1000 1u "server" to do the routing using openbsd.

          Comments
          1. By Anonymous Coward (66.252.236.93) on

            > > Its a Layer3 switch, which means it does routing at _wire speed_, millions packets per second, and it runs OSPF,BGP,RIP...
            >
            > No, it means cisco says that, but they are lying. It switches at switch speed, and routes horribly slow. And if you use any ACLs at all, you can just forget about moving packets at all. Ddos attacks routinely overwhelmed our 6500, we had to turn it into just an expensive switch, and use a $1000 1u "server" to do the routing using openbsd.

            Let's tear that apart even more--

            The following operations take place in the CPU, not the line cards:
            OSPF
            BGP
            RIP
            SNMP
            ISIS
            ACL filtering

            So the moment you want to actually use the L3 switch as anything other than an hyped-up L2 switch, you will hurt it.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]