release and patch/errata info in (easily) machine readable format?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

release and patch/errata info in (easily) machine readable format?

openbsd-misc
I mostly follow -stable, and have scripts/tools that enable me to (re)build
stable from source with minimal human intervention.

To further automate this process, it would be helpful to have the current
release number and (at least) the most current patch number.

Obviously this information is clearly documented in various web pages, and if
absolutely necessary,
I could extend my toolset to scrape this info from the website and/or the www
directory in CVS,
but I am wondering if this information is already available somewhere as
data?

I've found that www/build/Makefile contains:

        STABLE_VERSION= 5.8

So that is one place I could look, although I am not excited about having to
parse a Makefile either, but I haven’t yet
found anyplace where the patch numbers are available as non-html data.

One approach would be to scrape http://www.openbsd.org/errata.html
<http://www.openbsd.org/errata.html>, and figure out the release numbers, and
then scrape the errata page of a particular release to obtain the patch
numbers.

Is this information available somewhere in the tree in some easily parseable
format (YAML, JSON, etc) ?

If not, I’ll proceed to scrape this info.

It seems to me that the errata.html and errata<release>.html files could be
generated from the kind of data source
I’m describing, and that both the resulting html files AND the data source
file could then be statically served from the website.
If this isn’t the way these files are generated today, and if there were
interest in migrating to this approach, I would
be willing to develop and contribute the code to implement that…

Reply | Threaded
Open this post in threaded view
|

Re: release and patch/errata info in (easily) machine readable format?

trondd-2
On Sat, December 5, 2015 2:20 pm, [hidden email]
wrote:
> I mostly follow -stable, and have scripts/tools that enable me to
> (re)build
> stable from source with minimal human intervention.
>
> To further automate this process, it would be helpful to have the current
> release number and (at least) the most current patch number.

What is your build process?  The machine doing the build is running the
same version it's building, right?  Does 'uname -r' not work for you?

As for the patch number, someone can correct me if I am wrong, but I don't
believe it is recorded anywhere else.  I used to parse the errata page but
to be kinder to the server, I started parsing my local mirror which I
actually found to be easier to get the info from.

I maintain a "patchlevel" file on each system to keep track of what patch
I have applied and I check it against the patches on my mirror in
daily.local so I keep getting notified of out of date systems.  I also add
it to the motd so I see it when I log in, as well.

I prefer this slightly manual intervention because I like to know what is
changing on my systems.  I'm already patching manually, so also
maintaining the patchlevel file is minor.

Tim.

Reply | Threaded
Open this post in threaded view
|

Re: release and patch/errata info in (easily) machine readable format?

openbsd-misc
> On Dec 5, 2015, at 11:51 AM, trondd <[hidden email]> wrote:
>
> On Sat, December 5, 2015 2:20 pm, [hidden email]
> wrote:
>> I mostly follow -stable, and have scripts/tools that enable me to
>> (re)build
>> stable from source with minimal human intervention.
>>
>> To further automate this process, it would be helpful to have the current
>> release number and (at least) the most current patch number.
>
> What is your build process?  The machine doing the build is running the
> same version it's building, right?  Does 'uname -r' not work for you?

My build process begins outside of OpenBSD itself, so if I do not have a
machine running the current release version,
a machine running that release needs to be created.  There are several ways to
make that happen, and currently
I spin up a virtual machine.  At the moment, this is not an automated part of
my process, but I would like to make it so...

> As for the patch number, someone can correct me if I am wrong, but I don't
> believe it is recorded anywhere else.  I used to parse the errata page but
> to be kinder to the server, I started parsing my local mirror which I
> actually found to be easier to get the info from.

Yes, if I end up writing a scraper, I will very likely obtain the html pages
from the www directory of my local CVS mirror, rather than making http
requests
of the OpenBSD website.  In addition to reducing bandwidth demands of the
website, getting the information from my local mirror might lower the risk
that the website is more recent that my local mirror….

Another nice piece of data to have about a patch level would be the revision
number in CVS for that patch.
At present, the only place I see that information is inside the patch.sig
file, e.g.

        http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/004_smtpd.patch.sig

If I had that, i could ensure that the release I am about to build actually
contains the changes indicated by the patch level.
I am not looking forward to parsing these .sig files either :-(

> I maintain a "patchlevel" file on each system to keep track of what patch
> I have applied and I check it against the patches on my mirror in
> daily.local so I keep getting notified of out of date systems.  I also add
> it to the motd so I see it when I log in, as well.
>
> I prefer this slightly manual intervention because I like to know what is
> changing on my systems.  I'm already patching manually, so also
> maintaining the patchlevel file is minor.

My approach is to build an entire new release for the current patch level.
I understand this is way overkill, but given that is is a (mostly) automated
process, I prefer this
approach to manually applying and rebuilding….

I don’t apply patches to running systems, I re-install them from scratch,
and automated
configuration management restores the system to where it should be.

I do not now, nor envision, that the re-imaging of a machine would happen
automatically.

I can imagine that at some point I can have my build system send me a
notification that a new patch is available, and a bit later,
that a new release has been built and is available for installation, if/when I
so choose.

Your idea of a patch level file and adding that to motd is great,  I will add
that to my configuration management, just to make it obvious when shelling
into a server.
A follow-on addition to that idea is to add a “patch level fact” to ones
configuration management tool of choice, so that the patch level is
reported….

Reply | Threaded
Open this post in threaded view
|

Re: release and patch/errata info in (easily) machine readable format?

trondd-2
On Sat, December 5, 2015 4:08 pm, [hidden email]
wrote:

> Yes, if I end up writing a scraper, I will very likely obtain the html
> pages
> from the www directory of my local CVS mirror, rather than making http
> requests
> of the OpenBSD website.
>
> Another nice piece of data to have about a patch level would be the
> revision
> number in CVS for that patch.
> At present, the only place I see that information is inside the patch.sig
> file, e.g.
>
> http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/004_smtpd.patch.sig
>

I meant, I parse the files in patches/*/common, not the html in cvs.  I
found it easier.  I get a nice little report of the errata number, name
and synopsis.  It's mostly wasted as I see the cvs checkin first and am
already rebuilding. :)

> My approach is to build an entire new release for the current patch level.
> I understand this is way overkill, but given that is is a (mostly)
> automated
> process, I prefer this
> approach to manually applying and rebuildingâ*¦.

This is also what I do.  I rebuild the whole release and then apply the
kernel or base tarball to update the systems.  This way I have an
up-to-date release available for new system deployments as well.  I don't
have the luxury of redundency or down time to fully redeploy a system to
update it.

I do maintain configuration changes in siteXX.tgz files in case I do have
to redeploy.  Unfortunatly, they are mostly untested which is bad, but it
should get me 95% of the way.  A dev/staging environment is in the works.

> I can imagine that at some point I can have my build system send me a
> notification that a new patch is available, and a bit later,
> that a new release has been built and is available for installation,
> if/when I
> so choose.
>

My build machine monitors cvs for -stable updates in the code, as well as
changes to the patches/$version/ folder.  I see the cvs changes, and if
errata newer than the build machine's patchlevel have been created.  I
mirror the patches directory so all the systems compare their patchlevel
to my internal mirror.  Currently, I manually kick off a new release build
and then initiate an update from each system.

The patchlevel handles errata, but I can't yet be sure a system has the
latest stable changes.  The theory is that it's the errata that's
important, so I haven't solved that problem yet.

As it stands I have way better insite into the changes going onto my OBSD
machines as compared to the linux ones with rolling updates of hundreds of
packages.

Reply | Threaded
Open this post in threaded view
|

Re: release and patch/errata info in (easily) machine readable format?

trondd-2
In reply to this post by openbsd-misc
On Sat, December 5, 2015 4:08 pm, [hidden email]
wrote:

> Yes, if I end up writing a scraper, I will very likely obtain the html
> pages
> from the www directory of my local CVS mirror, rather than making http
> requests
> of the OpenBSD website.
>
> Another nice piece of data to have about a patch level would be the
> revision
> number in CVS for that patch.
> At present, the only place I see that information is inside the patch.sig
> file, e.g.
>
> http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/004_smtpd.patch.sig
>

I meant, I parse the files in patches/*/common, not the html in cvs.  I
found it easier.  I get a nice little report of the errata number, name
and synopsis.  It's mostly wasted as I see the cvs checkin first and am
already rebuilding. :)

> My approach is to build an entire new release for the current patch level.
> I understand this is way overkill, but given that is is a (mostly)
> automated
> process, I prefer this
> approach to manually applying and rebuildingâ*¦.

This is also what I do.  I rebuild the whole release and then apply the
kernel or base tarball to update the systems.  This way I have an
up-to-date release available for new system deployments as well.  I don't
have the luxury of redundency or down time to fully redeploy a system to
update it.

I do maintain configuration changes in siteXX.tgz files in case I do have
to redeploy.  Unfortunatly, they are mostly untested which is bad, but it
should get me 95% of the way.  A dev/staging environment is in the works.

> I can imagine that at some point I can have my build system send me a
> notification that a new patch is available, and a bit later,
> that a new release has been built and is available for installation,
> if/when I
> so choose.
>

My build machine monitors cvs for -stable updates in the code, as well as
changes to the patches/$version/ folder.  I see the cvs changes, and if
errata newer than the build machine's patchlevel have been created.  I
mirror the patches directory so all the systems compare their patchlevel
to my internal mirror.  Currently, I manually kick off a new release build
and then initiate an update from each system.

The patchlevel handles errata, but I can't yet be sure a system has the
latest stable changes.  The theory is that it's the errata that's
important, so I haven't solved that problem yet.

As it stands I have way better insite into the changes going onto my OBSD
machines as compared to the linux ones with rolling updates of hundreds of
packages of who-knows-what.

Reply | Threaded
Open this post in threaded view
|

Re: release and patch/errata info in (easily) machine readable format?

Todd Mortimer
In reply to this post by openbsd-misc
On Sat, Dec 05, 2015 at 11:20:41AM -0800, [hidden email] wrote:
> One approach would be to scrape http://www.openbsd.org/errata.html
> <http://www.openbsd.org/errata.html>, and figure out the release numbers, and
> then scrape the errata page of a particular release to obtain the patch
> numbers.

If you're just looking for a raw number that will increase for every
patch added, you shouldn't need to scrape the website. There is a
tarball containing all patches for a given release, eg:

http://ftp.openbsd.org/pub/OpenBSD/patches/5.8.tar.gz

It is updated once a day. If you retrieve this, you can get a crude
'patch level' by just counting the number of entries in this tarball:

$ tar -tzf 5.8.tar.gz | wc -l                    
      10

If you want to remove the whitespace, you can pipe through tr:

$ tar -tzf 5.8.tar.gz | wc -l | tr -d '[:space:]'
10$

But it sounds like you're using CVS, so I don't think you even need to
bother with the errata page. If you are updating CVS via cron, then
add the -q flag to your cvs up command. If anything in the stable tree
is updated, then you'll get an email from cron listing the updated files
and then you can go build a new release using your usual method.
Everything that ends up on the errata page is also in CVS, so you won't
miss anything.

If you do something like this, then the 'patch level' can be reduced to
just the day you did the build, and you can include this in your build
by creating a siteXX file:

http://www.openbsd.org/faq/faq4.html#site

Where your siteXX.tgz contains a file with the date you did the build
(say, /etc/builddate). Then you can easily see if your machines are
running the most recent build by comparing the contents of that file to
the date you did your most recent build.

Anyway, I know this isn't quite what you were after (machine readable
version / patch level), but if you're just looking to know when to do a
new release build, then you should be able to get there by just tracking
CVS. If you're just looking to track errata, then the tarball can help
you without having to scrape the website, and if you'd prefer to not hit
the website unnecessarily, then tracking the www CVS repo can tell you
when the errata pages have changed, and then you can retrieve the patch
tarball without having to scrape out info about the actual patch
numbers.

Todd