Contributed by jose on from the oldland-security dept.
"We've had this discussion over and over again and the camp splits quite evenly into:This is not uncommon to find situations where upgrading quite yet is an option, and it gets put off almost indefinitely. While the newest software always does have the known fixes in it, it may also introduce more complications than are acceptable.This doesn't just hold for OpenBSD 3.0 but all the previous versions.
- Your problem: upgrade,
- Just keep going and patch as things come out.
I've started following two separate techniques: the first, and obvious, is to install over OpenBSD native versions whenever something "necessary" appears (for example: apache, openssl, sendmail...) but the new method I have been following is to apply OpenBSD 3.1 patches where possible.
I thought this was never going to work but, for example, the recent LPD patch works just fine because the code is from 1999. So, with a grain of salt you can happily patch your OpenBSD 3.0 source tree with OpenBSD 3.1 patches. Another patch which worked was the openssl patch - I had tried compiling the latest OpenSSL from source and was unable to reproduce the exact setup (including dynamic libraries) of the OpenBSD source tree.
Would it make sense to start having an archive of patches which apply clean to previous versions of OpenBSD and fix vulnerabilities? This need not be an "official" page but it would definitely help."
Its sometimes not possible to just apply the new code to an old tree, as APIs change (as has happened in OpenSSL, for example), breaking software. However, with some effort, new patches can be rolled for the older versions which preserve the compatability while fixing known issues. The real question is are any of you doing this?
(Comments are closed)
By Anonymous Coward () on
Comments
By Anonymous Coward () on
Comments
By Anonymous Coward () on
new custom kernel takes just another reboot and some broken features...
always nice to have few boxes for important purpose
Comments
By pravus () on
if you can't even have down-time for maintenance, you really should re-think your strategy for resource allocation.
By Anonymous Coward () on
And if you think one can just replicate a configuration that is spread on 50+ computers to make it work as it should, then you may have unlimited funds or something.
By gwolf () gwolf@gwolf.cx on http://www.gwolf.cx
I have upgraded many machines from 2.2 to 3.0, 2.2 to unstable, 3.0 to unstable, etc. - Never even had to reboot for this. And if Linux is not your stuff, you can install Debian/NetBSD, although it is still quite experimental, and help the project :)
Comments
By Gerardo Santana Gómez Garrido () santana at openbsd.org.mx on http://www.openbsd.org.mx/~santana/
stick with Debian and go away.
By tedu () on
Comments
By Anonymous Coward () on
By Anonymous Coward () on
You can't upgrade the kernel without rebooting.
Apparently he is running an ancient and probably highly vulnerable Linux kernel.
By djm () on
Unless you run Debian Unstable, you are likely using 2+ year old software (no nice 6-monthly release cycle). The usual response of Debian advocates to this is "use unstable, it is really stable", but so is OpenBSD -current for me.
Their security integration isn't all it is cracked up to be either: http://lwn.net/Articles/22624/
You can also say goodbye to all the other wonderful things that OpenBSD's release model brings you: whole-of-tree audits, patches via CVS, the ability to easily rebuild the entire OS from source, etc.
By elmore () on www.screamingelectron.org
So what I typically do is track stable, 1 release behind on the production box and track stable for the current release on the backup computer. When end of life happens for the production box I merely turn it off and the other picks up for it. No downtime, no one ever even knows.
Obviously in your situation people would know but, you could cut downtime however to an absolute minimum, no more than a minute or two this way.
The only problem I see with this is if data is stored on the OBSD box itself I.E. peoples home's. In my environment homes and all data are mounted via NFS I don't really have a problem with that. Anyways just an idea. Maybe it'll help you out.
Comments
By Matt () on
I would think you need to do at least a few things like syncing user accounts, change ip, change hostname... what else would be required to make a drop-in replacement?
Comments
By Bas () beetjebrak78@hotmail.com on mailto:beetjebrak78@hotmail.com
By elmore () on
By cycloon () cycoon at is-root dot org on mailto:cycoon at is-root dot org
After i read about the new altq-merges and pf I additionally thought about updating to -current. The first thing i tried was cvs, but i got a lot of problems there (most standing in the update-mini-faq).
So last week i decided to upgrade with those binary snapshots. I did this like described in the faq. First, simply unpacking the base33.tgz, comp33.tgz and so on. Afer that I rebooted and of course got some config problems (since i didnt upgrade /etc). So i updated those things by hand (mainly merging nat.conf in pf.conf and updating things like ppp.linkup). Another reboot and everything worked, with some errors, but it worked.
So now I decided to update /etc. Like described I unpacked everything to ~/new_etc and manually looked over the changes. most configs weren't changed by me so i could simply copy them over to /etc. Ater that another reboot and everything was fine.
Great:
OpenBSD 3.3 (GENERIC) #19: Thu Mar 6 18:43:26 MST 2003
Conclusion: dont worry about updates at all.
Comments
By Anonymous Coward () on
By Anonymous Coward () on
Comments
By Ben Johnson () on
I had two 2.9 fiewalls/database servers with the OpenSSH vunerability - and being lazy and stupid, instead of patching OpenSSH and actually figguring things out, I just ploped in a new 3.2 firewall infront of the old firewall.
Just a thought.
Comments
By RC () on
The only thing people are compalining about is that they are forced to reboot (eg. 2 seconds of down-time) to upgrade to a new release, and hence, a new kernel.
By Anonymous Coward () on
By click46 () click46@genmay.net on mailto:click46@genmay.net
Comments
By Noob () on
I would rather invest money in the purchase of the latest release CDROM and help future developments.
The upgrade proceedures which already exist work for me just fine. I'm actively using OpenBSD 3.2-stable on production systems and running OpenBSD 3.3-current on my test systems.
I have upgraded in the past without problems or any critical downtime.
By Anonymous Coward () on
By Anonymous Coward () on
.../OpenBSD/snapshots/
For our major architectures, we tend to build mini releases of unknown stability and quality about every month or so. This is where we place those test releases.
Comments
By cycloon () on
And as i wrote, i wanted to test some features i will use at work in future.
By tedu () on
you don't have to upgrade every week, of course. but if there's some feature you want in current, don't be afraid to try it out.
By click46 () click46@genmay.net on www.genmay.net
Comments
By pravus () on
Comments
By click46 () click46@genmay.net on mailto:click46@genmay.net
By grey () on
Until that situation changes, we can't expect developers to maintain stable branches of trees older than already outlined (and again, I realize that Arrigo Triulzi wasn't asking for that - so I hope some folks have some other suggestions for him). That said, there are methods by which to minimize the impact that upgrades have in production environments.
Separation of duties is a good first - break off your httpd, smtpd, DNS server, user files (this is aimed at the guy with the 50+k shell users) et al into separate -dedicated- machines, that way upgrades can be done incrementally. If your resources allow, hopefully at least one rollover machine can be purchased (this is less of a challenge if you're dealing with x86 hardware) such that you can make a preparation machine - bring it up to the level (3.3-current/whatever) you need, get the service you need (say, named) all working hunky dorey, and then have a cutover. The machine that is removed from service can then be the next rollover machine [that's if you can only afford to have one extra box].
The ability to have planned maintenance windows really should not be sacrificed - and with proper resources, they can be pretty transparent to the user (even shell servers - look at the old netcomxx.netcom.com shell situation as an example, where a group would be cycled out periodically for upgrades, allowing users to round robin to still up machines all in turn [of course, this meant that their user home directories were handled differently than just a pure local fs, think about it]).
If maintenance windows are still a real challenge, maybe some of the newer load balancing features of pf can assist in redirecting traffic to machines during transition, but that is going to be pretty case specific.
If the challenges in upgrading are due to breaking other packages (whether in house or external) well, it's hard to say (especially since admins often inherit systems) but if possible, try to use alternatives that are maintained themselves - that or find a way to hide the old (possibly vulnerable) machine from the rest of the world by putting something in between it and everything else.
These are all primarily administrative methods of approach to reducing downtime while still tracking patches/new features. Asking OpenBSD developers to devote their resources to dealing with something which is primarily a user methodology issue isn't all that wise.
Feel free to donate more money to OpenBSD in hopes that they can maintain older trees - but I don't really hear a groundswell of support for that [binary patches would yield much bigger support IMO]. And ask yourself, do you really _want_ support for older releases and the tree system that tends to entail (just look at the horror of the FreeBSD tree system). Even superficially it looks complicated - I can only imagine the pains of insuring that patches really fix the right problems across several different supported areas.
Comments
By Anonymous Coward () on
I agree, and anyone willing to prove otherwise, within reason, is welcome.
By Stef () on http://www.userfriendly.org
By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net
Make sure nobody is logged in and there is no crap in
/tmp.
I generally start by doing this in /bin/sh:
# cd /tmp
# ftp ftp.openbsd.org
ftp> cd /pub/OpenBSD/snapshots/i386
ftp> prompt
prompt off
ftp> mget bsd *gz
ftp> quit
# tar xpzf etc33.tgz
# rm etc33.tgz
# cd /
# rm -r /usr/include /usr/share /usr/libexec
# for i in /tmp/*gz; do
> tar xzpf $i
> done
# mv /bsd /obsd
# mv /tmp/bsd /bsd
start like this:
# cat /tmp/etc/group
# vi /etc/group
# vipw
# cd /tmp/etc
# cp rc login.conf netstart daily weekly monthly security fbtab moduli /etc
# cat /etc/rc.conf
# cp /tmp/etc/rc.conf /etc
# vi /etc/rc.conf
# cat /etc/rc.securelevel
# cp /tmp/etc/rc.securelevel /etc
# vi /etc/rc.securelevel
# cat /etc/rc.shutdown
# cp /tmp/etc/rc.shutdown /etc
# vi /etc/rc.shutdown
# cat /tmp/var/cron/tabs/root
# crontab -e root
# cd /dev
# rm -r altq
# ./MAKEDEV all
Of course, you may be upgrading from an ealier version
than 2.8. Also, you may be upgrading a system like sparc64 where the binary format changed. In this
case, you may want to preserve gzip and tar for
the extraction before you do it. So, do this in place
of the 'for' loop above:
# cp /usr/bin/tar /tar
# cp /bin/gzip /gzip
# mv /bsd /obsd
# mv /tmp/bsd /bsd
# for i in /tmp/*gz; do
> /gzip -dc $i | /tar xvpf -
> done
Of course, you have to be a lot more careful when
you are upgrading like this. You will have to reboot after the for loop and before you do anything else! Because you are most likely going to have to reinstall EVERY package and anything else that you compiled, this is obviously a more complicated method. For people not familiar with how the system is organized, you may just be better off installing from scratch if you are starting from a very old version of OpenBSD.
There are always little things that will linger after the upgrade that you have to take care of. You will want to recompile major programs/ports that you depend on, like database servers, MTAs or apache modules. You may need to wipe out your whole site_perl directory and recompile the modules if you are moving from non-propolice to a propolice version of Perl.
On my home machines I always delete all the ports and recompile everything since they can be down for a little bit and KDE is always more reliable when every library and everything else is current.
Finally, some people may have custom kernels and/or other things to pay attention to. Don't forget to use config to update the new kernel before you reboot.
If you are using bind4, bind9 is going to require that you re-do your config in the new format. And, the config goes in /var/named/etc now.
Having just upgraded about 20 machines in the past week, I can say that this only takes about 10-30 minutes per machine, depending on how customized /etc is and if the kernel is custom and so on. That's why I say it's EASY. I do it remotely all the time. Once you've practiced a bit, it is all downhill from there. The only sticky points are when you have a lot of apps that depend on ports and such. Recompiling ports, mysql, etc can take quite a while.
FYI, I normally download the release/snapshot that i want to upgrade with to a local FTP server and then install all the other machines from that FTP server rather than use ftp.openbsd.org every time.
Upgrading OpenBSD involves quite a bit of work if you are going to recompile the software/packages, but you can always get away with running the same old crappy stuff if both releases are binary compatible. So, if you are upgrading i386 from 2.8 ro 3.3, no biggie. Upgrade the packages when you have time. Perl and Apache modules are the only problem points that I've had with this lazy method. Of course, anything compiled under 3.3 will get ProPolice protection, and you will probably be getting a newer, (possibly) more secure version of the software/port/package that you are installing since this is 2.5 years after the obsd 2.8 release!!!
Expect any platform switch to ELF to be more complicated, of course. i386 will do this eventually. You
will have no choice but to upgrade all your Perl and Apache modules, and probably all your ports at this point. For people who maintain lots of machines, remote machines, and/or machines that have lots of software/data that you don't want to fuck with, this method is vastly superior to installing from scratch and copying things back over.
Another method is to format only your / and /var and /usr
partitions but keep your data partitions when you do the reinstall. Obviously, this cannot be done remotely and requires a bit more reconfiguration.
The first method outlined here is pretty close to what you'd have to do using the (U)pgrade option on the boot floppy, only you are unpacking the distribution and making devices yourself.
Hope this helps people, like I said I have machines that have been upgraded this way since 2.1. If you aren't comfortable with the system then you may get lost upgrading like this because too many things will bite you in the ass. Reaing the upgrade-minifaq can give you hints about other things that have changed between releases that I don't talk about here.
Have fun
Comments
By Anonymous Coward () on
By Anonymous Coward () on
-# mv /bsd /obsd
+# cp /bsd /obsd
# mv /tmp/bsd /bsd
much better!
Comments
By tedu () on
By jose () on http://monkey.org/~jose/
1.uninstall all packages
2. download bsd.rd and boot that (ramdisk)
3. delete all system files outside of /etc, /var, /dev/, and /home ... ie everything being upgraded. no stale libs, binaries, etc ...
4. unpack tgz's as chris does (tar -zxvpf), install new kernel
5. if needed (ie ELF upgrade on sparc) install new boot blocks
6. reboot new kernel.
7. mergemaster on /etc files
8. new packages get installed (i should make a metaport of my stuff to smooth the process).
hope that helps. its definitely a very hackish way to do it.
By jose () on http://monkey.org/~jose/
1.uninstall all packages
2. download bsd.rd and boot that (ramdisk)
3. delete all system files outside of /etc, /var, /dev/, and /home ... ie everything being upgraded. no stale libs, binaries, etc ...
4. unpack tgz's as chris does (tar -zxvpf), install new kernel
5. if needed (ie ELF upgrade on sparc) install new boot blocks
6. reboot new kernel.
7. mergemaster on /etc files
8. new packages get installed (i should make a metaport of my stuff to smooth the process).
hope that helps. its definitely a very hackish way to do it.
Comments
By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net
But, you do want to at least rm /usr/lib/*c_r* to get rid of the old threaded crap.
By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net
I have been upgrading my production boxes since 2.1.
Make sure nobody is logged in and there is no crap in
/tmp.
I generally start by doing this in /bin/sh:
=download openbsd 3.3 into /tmp=
# cd /tmp
# ftp ftp.openbsd.org
ftp> cd /pub/OpenBSD/snapshots/i386
ftp> prompt
prompt off
ftp> mget bsd *gz
ftp> quit
=at this point you should still be in /tmp=
# tar xpzf etc33.tgz
# rm etc33.tgz
# cd /
=get rid of directories which will have old crap that can interfere with your compiles/ports/stuff=
# rm -r /usr/include /usr/share /usr/libexec
=here is a for loop that actually does the unpacking of the new system. the P flag is very important, don't forget it=
# for i in /tmp/*gz; do
> tar xzpf $i
> done
=preserve your old kernel in case the new one is bogus on your hardware=
# mv /bsd /obsd
=copy over the new kernel=
# mv /tmp/bsd /bsd
=Now you must edit /etc/group and add in all the
groups that start with _underscore from /tmp/etc/group=
start like this:
# cat /tmp/etc/group =Take notes=
# vi /etc/group =edit the group file=
=Now you must edit /etc/master.passwd and add in all
the users that start with _underscore from /tmp/etc/master.passwd=
# vipw
=Now you should copy over all the standard scripts from
/tmp/etc to /etc=
# cd /tmp/etc
# cp rc login.conf netstart daily weekly monthly security fbtab moduli /etc
=Take note of your /etc/rc.conf=
# cat /etc/rc.conf
=Copy over the 3.3 rc.conf in place of your old one=
# cp /tmp/etc/rc.conf /etc
=Add your old changes to the stock 3.3 rc.conf=
# vi /etc/rc.conf
=Take note of your /etc/rc.securelevel=
# cat /etc/rc.securelevel
=Copy over the 3.3 rc.securelevel in place of your old one=
# cp /tmp/etc/rc.securelevel /etc
=Add your old changes to the stock 3.3 rc.securelevel=
# vi /etc/rc.securelevel
=Take note of your /etc/rc.shutdown=
# cat /etc/rc.shutdown
=Copy over the stock 3.3 rc.shutdown in place of your old one=
# cp /tmp/etc/rc.shutdown /etc
=Add your old changes to the stock 3.3 rc.shutdown=
# vi /etc/rc.shutdown
=Now you must edit root's crontab and bring it up to sync
with the new stuff in /tmp/var/cron/tabs/root=
# cat /tmp/var/cron/tabs/root
=Add sendmail and whatever else is new from there to your old root crontab=
# crontab -e root
=make sure you have new devices, not old ones=
# cd /dev
=altq is gone now, get rid of the directory=
# rm -r altq
=make youre new devices=
# ./MAKEDEV all
Of course, you may be upgrading from an ealier version
than 2.8. Also, you may be upgrading a system like sparc64 where the binary format changed. In this
case, you may want to preserve gzip and tar for
the extraction before you do it. So, do this in place
of the 'for' loop above:
=preserve the old tar and gzip so you can keep running them over and over in the loop=
# cp /usr/bin/tar /tar
# cp /bin/gzip /gzip
=copy over the new kernel ahead of time=
# mv /bsd /obsd
# mv /tmp/bsd /bsd
=here is the loop=
# for i in /tmp/*gz; do
> /gzip -dc $i | /tar xvpf -
> done
Of course, you have to be a lot more careful when
you are upgrading like this. You will have to reboot after the for loop and before you do anything else! Because you are most likely going to have to reinstall EVERY package and anything else that you compiled, this is obviously a more complicated method. For people not familiar with how the system is organized, you may just be better off installing from scratch if you are starting from a very old version of OpenBSD.
There are always little things that will linger after the upgrade that you have to take care of. You will want to recompile major programs/ports that you depend on, like database servers, MTAs or apache modules. You may need to wipe out your whole site_perl directory and recompile the modules if you are moving from non-propolice to a propolice version of Perl.
On my home machines I always delete all the ports and recompile everything since they can be down for a little bit and KDE is always more reliable when every library and everything else is current.
Finally, some people may have custom kernels and/or other things to pay attention to. Don't forget to use config to update the new kernel before you reboot.
If you are using bind4, bind9 is going to require that you re-do your config in the new format. And, the config goes in /var/named/etc now.
Having just upgraded about 20 machines in the past week, I can say that this only takes about 10-30 minutes per machine, depending on how customized /etc is and if the kernel is custom and so on. That's why I say it's EASY. I do it remotely all the time. Once you've practiced a bit, it is all downhill from there. The only sticky points are when you have a lot of apps that depend on ports and such. Recompiling ports, mysql, etc can take quite a while.
FYI, I normally download the release/snapshot that i want to upgrade with to a local FTP server and then install all the other machines from that FTP server rather than use ftp.openbsd.org every time.
Upgrading OpenBSD involves quite a bit of work if you are going to recompile the software/packages, but you can always get away with running the same old crappy stuff if both releases are binary compatible. So, if you are upgrading i386 from 2.8 ro 3.3, no biggie. Upgrade the packages when you have time. Perl and Apache modules are the only problem points that I've had with this lazy method. Of course, anything compiled under 3.3 will get ProPolice protection, and you will probably be getting a newer, (possibly) more secure version of the software/port/package that you are installing since this is 2.5 years after the obsd 2.8 release!!!
Expect any platform switch to ELF to be more complicated, of course. i386 will do this eventually. You
will have no choice but to upgrade all your Perl and Apache modules, and probably all your ports at this point. For people who maintain lots of machines, remote machines, and/or machines that have lots of software/data that you don't want to fuck with, this method is vastly superior to installing from scratch and copying things back over.
Another method is to format only your / and /var and /usr
partitions but keep your data partitions when you do the reinstall. Obviously, this cannot be done remotely and requires a bit more reconfiguration.
The first method outlined here is pretty close to what you'd have to do using the (U)pgrade option on the boot floppy, only you are unpacking the distribution and making devices yourself.
Hope this helps people, like I said I have machines that have been upgraded this way since 2.1. If you aren't comfortable with the system then you may get lost upgrading like this because too many things will bite you in the ass. Reaing the upgrade-minifaq can give you hints about other things that have changed between releases that I don't talk about here.
Have fun
Comments
By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo
Thanks for posting it!
Comments
By jose () on http://monkey.org/~jose/
By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo
The key points for those who took this posting as a troll:
1) it is clear, accepted (at least by me) and eminently understandable that the OpenBSD project cannot support all the releases,
2) there are some situations in which upgrading is not possible: production systems in the middle of nowhere come to mind.
3) "remote production systems" is my problem, and it is exacerbated by the fact that PCs are hopeless in headless configurations. I have done zillions of installs and upgrades over serial terminal concentrators on systems which support this. The PC was simply not designed for this.
4) I am aware of a number of pretty upgrade options, some better than others. This is *not* being disputed.
5) The point I am trying to make is that, if necessary, it is possible to keep older systems alive without excessively compromising on security.
6) "without excessively compromising on security" means making an educated decision as to whether the limitations of the current system (e.g. no propolice) are acceptable or not.
Other than that if anyone knows what happened to those people who sold PC cards which gave you a serial interface into the BIOS (ie. you could intercept output from BIOS startup onwards, a real serial console setup) I would love to know.
Arrigo
P.S. Just for reference: it is trivial to upgrade a remote OpenBSD Sun box or indeed Alpha via the serial console in comparison to a PC...
Comments
By Anonymous Coward () on
Comments
By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo
Do you have any experience with them? Do they really work?
Comments
By grey () on
http://www.cyclades.com/products/index.php?id=4
(approximately $300)
Or, for something on the cheap (main downside being no sshd), where you have more rackspace:
http://unixsurplus.zoovy.com/product/TERM_TANDEM_8PORT
The DIY cyclades would be doable with something like a soekris board.
The realweasel appears to be pretty useful in general, though keep in mind you can obviate the need for one if you don't care about BIOS options, by simply editing your boot.conf to redirect to a serial port on your x86.
Ideally, if you have the rack space and the finances, you could do it all DIY with each unit (serial console device & actual server) having two serial ports, and cross connect them to each other, so that you can use one to upgrade the other if necessary. A little crazy, but it would be pretty flexible.
And well, realweasel could also be paired well with just a modem in the event that things are really horked (e.g. whole network down, so you couldn't access a cyclades if you wanted). Some folks combine these features into their remote serial console devices (I believe even cyclades does, though it'll probably cost a bit more).
ymmv
By Anonymous Coward () on
"I love my Weasel." -- Paul Vixie
By Piotr Kapczuk () Piotr.Kapczuk@hoop.pl on mailto:Piotr.Kapczuk@hoop.pl
There are also servers with integrated remote management http://h18004.www1.hp.com/products/servers/management/ilo/index.html
By jose () on http://monkey.org/~jose/
uprading hardware may not be an option tho, either ... but something to keep in mind for future stuff.
Comments
By Marco Peereboom () slash@peereboom.us on www.peereboom.us
By Anonymous Coward () on
openbsd has too many core changes to become a "stable trusted environment". It excells at network security and little else. It's too volatile to trust in a server environment (sorry if that pisses anyone off but I appear to be the only one being realistic).
Comments
By Anonymous Coward () on
No, it just appears that your definition of "server environment" is different from mine.
For instance, I don't mind wiping out everything on every server once every year or two, reinstalling new version and restoring user's data - it only takes one night to do it and couple more nights to tweak and finalize. May be I am just lucky that our servers need to be operational only for 14-16 hours a day...
By Anonymous Coward () on
By Anonymous Coward () on
servers to debian, great solution, very
realistic.
why are you on deadly.org anyway? please
return to debian-trolls posthaste.
Comments
By Stefan () - on mailto:-
If you want no downtime at all, you would need a standby/failover-solution - so you can reboot one system while the other acts as a "standby".
However, if you want to use some kind of "shared storage" and you want to upgrade
the cluster-filesystem, then there's no way you can get away w/o downtime
or maybe at least a three-or-more-system-cluster?
So far, I haven't seen a "free" (as in coffee) solution for this ...
Maybe someone has an idea?
cheers
stefan