OpenBSD Journal

Introducing ports.su - ports-readmes mirror and ports search

Contributed by pitrh on from the In Soviet Russia, system packages you! dept.

Constantine A. Murenin writes in about his new ports database site, ports.su:

Taking on espie's dated call on doing something cool with databases/sqlports port, and with his own initial work through databases/ports-readmes port, I've decided to fork ports-readmes, and create some very simple web-site to mirror the content generated.

Introducing http://ports.su/ .

The web-site is completely static, and all the pages get regenerated daily by downloading a fresh copy of the sqlports package from the snapshots, and running the forked ports-readmes.

The site has been first announced on ports@ on 2013-02-28, and as of April, most of the 8k or so pages are already indexed by Google, and are thus searchable.

Why another ports site?

I like openports.se. However, one thing that never works there is their search. Try searching for anything interesting there, and there's a good chance that you won't find it. How do we address the issue? Google to the rescue!

This site, in its current form, is by no means an openports.se (formerly ports.openbsd.nu) replacement; however, I believe it could exist to complement openports.se, and I've included links to openports.se in all the pages, so you could always go where you want without any extra hassle. Also included are links to CVSweb on OpenBSD.org and allbsd.org, so, sources can quickly be checked.
Scripts that generate the whole site are available in github and bitbucket, under /cnst/ports-readmes.

Why would I use ports.su instead of openports.se?

Due to ports.su being based on sqlports / ports-readmes:

 * any given port can have more than one parent category

 * category pages will include all the ports that list said category, 
   and not just those for which the category is primary

 * library, build and run dependencies are listed for each port

 * based on official sqlports -- no parsing bugs ^_^

 * full-text search with Google

Basically, not trying to reinvent a square wheel.

Plus, a shorter and simpler ports.su address is better if you are too tired to type, or want to look through the ports tree on a PDA. :-P

Comments, questions, forks and pull requests are welcome. Powered by OpenBSD and nginx, available via IPv4 and IPv6.

(Comments are closed)


Comments
  1. By cnst (cnst) on http://bxr.su/s?q=%22Constantine%20A.%20Murenin%22%20OR%20cnst

    BTW, you can search like this:

    http://ports.su/search?q=git
    Since the nginx.conf for ports.su has this:
    	set	$goog	http://www.google.com/search?q=site:ports.su+$arg_q;
    	location = /search {
    		return	307	$goog;
    	}
    

    :-)

  2. By sneaker (sneaker) sneaker@noahpugsley.net on

    This is great. So much better then grepping INDEX. :-)

    Seriously though, nice work. I think I'll get a lot of use out of it.

  3. By Marc Espie (213.41.185.88) espie@openbsd.org on

    I hope it's going to move to the dynamic version, which can do lots more things (for instance, it's tied into pkglocatedb as well, and features an actual search box)

    If you want to try it out, install ports-readmes-dancer on your local hosts, and start it, then point your browser at localhost:3001

    That's a perl dancer application, tying that up to any kind of webserver is reasonably easy, see the Dancer::Deployment manpage.

    Comments
    1. By sthen (2001:8b0:648e:cc01:f2de:f1ff:fef9:a752) on

      > I hope it's going to move to the dynamic version, which can do lots more things (for instance, it's tied into pkglocatedb as well, and features an actual search box)
      >
      > If you want to try it out, install ports-readmes-dancer on your local hosts, and start it, then point your browser at localhost:3001
      >
      > That's a perl dancer application, tying that up to any kind of webserver is reasonably easy, see the Dancer::Deployment manpage.

      A "reverse search" i.e. "show me what depends on X package" would be lovely, I tried adding it but haven't managed to make it work yet..

      Comments
      1. By Marc Espie (espie) on


        > A "reverse search" i.e. "show me what depends on X package" would be lovely, I tried adding it but haven't managed to make it work yet..

        This now works (in the perl dancer port).

        I really hope Constantine will manage to deploy the dynamic version at
        some point. The static version makes ways less sense...

        Comments
        1. By Constantine A. Murenin (cnst) on

          >
          > > A "reverse search" i.e. "show me what depends on X package" would be lovely, I tried adding it but haven't managed to make it work yet..
          >
          > This now works (in the perl dancer port).

          It would be nice if you had it in a git repo, and with some kind of web interface. ;-)

          A reverse search could probably be implemented to be shown in the static pages of a non-dancer ports-readmes, too.

          > I really hope Constantine will manage to deploy the dynamic version at
          > some point. The static version makes ways less sense...

          I will take a closer look at a homebrew search at some point indeed. However, the whole point of ports.su is search through regular search-engines like Google (unfortunately, Bing, Yandex and Baidu decided not to crawl the site yet, but, hopefully, that'll change later on).

          Just as you don't like openports.se reinventing a square wheel, I think that searching through a regular general-purpose search-engine might as well be better than going about a non-fulltext non-typo-correcting homebrew search of individual fields. Port developers would probably want more, and it sounds like ports-readmes-dancer is a great tool for that, but I would guesstimate that Google search is well sufficient for the average ports or packages user who simply needs to quickly reference or find some port.

          And, Google integration works surprisingly well for ports.su, I must say. It's almost to the point of being scary that Googlebot can index 8k pages of an unknown and newly-launched web-site so quickly and easily.

          Comments
          1. By Marc Espie (espie) on

            > >
            > > > A "reverse search" i.e. "show me what depends on X package" would be lovely, I tried adding it but haven't managed to make it work yet..
            > >
            > > This now works (in the perl dancer port).
            >
            > It would be nice if you had it in a git repo, and with some kind of web interface. ;-)

            It's not such a big project that setting it up in git makes a lot of sense.

            > A reverse search could probably be implemented to be shown in the static pages of a non-dancer ports-readmes, too.

            Well, the more I look at it, the less the non-dancer port makes sense.
            I mean, you could actually just run the dancer version and crawl it (with wget for instance) to generate a static version. ;p

            Actually, the only reason the static version is still around is because you are using it...

            > Just as you don't like openports.se reinventing a square wheel, I think that searching through a regular general-purpose search-engine might as well be better than going about a non-fulltext non-typo-correcting homebrew search of individual fields. Port developers would probably want more, and it sounds like ports-readmes-dancer is a great tool for that, but I would guesstimate that Google search is well sufficient for the average ports or packages user who simply needs to quickly reference or find some port.

            Well, I think that both search methods are justified. Sometimes, you want full text, other times, you want specific search parameters.

          2. By sthen (2001:8b0:648e:cc01:f2de:f1ff:fef9:a752) on

            > And, Google integration works surprisingly well for ports.su, I must say. It's almost to the point of being scary that Googlebot can index 8k pages of an unknown and newly-launched web-site so quickly and easily.

            They crawl dynamic sites too, these aren't the days when their servers were made of Lego..

            Comments
            1. By Marc Espie (espie) on

              > > And, Google integration works surprisingly well for ports.su, I must say. It's almost to the point of being scary that Googlebot can index 8k pages of an unknown and newly-launched web-site so quickly and easily.
              >
              > They crawl dynamic sites too, these aren't the days when their servers were made of Lego..
              >

              Actually, they won't even notice that the dancer site is dynamic, as it
              serves clean urls.

  4. By Anonymous Coward (213.16.232.143) on

    Someone also improve the search in undeadly.org, please!

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]