ldapd(8) binary incompatibility, 5.4 -> 5.5

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

ldapd(8) binary incompatibility, 5.4 -> 5.5

Matthew Weigel
I finally upgraded my last machine - that runs ldapd(8) for user
logins, mail aliases, and a few other odds and ends - from 5.4 to 5.5.

I'm left wondering if I'm the only one who actually uses the stock
ldapd(8), because it is not called out at all in upgrade55.html as
having problems with the Year 2038 fixes that went into 5.5.

I ended up having to create a 5.4 VM (I stuck with the same amd64 arch
as my actual server, and have not investigated or tested under what
constraints this might work across architectures) to load the ldapd(8)
database files, use third party LDAP tools to create a text dump in
LDIF format, and then load the LDIF into an empty database of 5.5
ldapd(8).

It looks like particularly the btree_stat and btree_meta structs used
in the ldapd(8) btree implementation are the culprits, as it looks like
they are the only time_t bits actually stored on disk.  Since it
appears my problems are now solved, I'm mostly sending this message as
a heads up in case there is anyone still getting ready to upgrade to
5.5 that uses ldapd(8).

Something probably deserves to be in update55.html as well, but I don't
have a repeatable, documented procedure for what I did.
--
  Matthew Weigel
  hacker
  unique & idempot . ent

Reply | Threaded
Open this post in threaded view
|

Re: ldapd(8) binary incompatibility, 5.4 -> 5.5

Olivier Mehani-5
On 2014-07-22, Matthew Weigel <[hidden email]> wrote:
> I finally upgraded my last machine - that runs ldapd(8) for user
> logins, mail aliases, and a few other odds and ends - from 5.4 to 5.5.

Haha! I just did the same two days ago.

> I'm left wondering if I'm the only one who actually uses the stock
> ldapd(8), because it is not called out at all in upgrade55.html as
> having problems with the Year 2038 fixes that went into 5.5.

No, I'm here.

> I ended up having to create a 5.4 VM (I stuck with the same amd64 arch
> as my actual server, and have not investigated or tested under what
> constraints this might work across architectures) to load the ldapd(8)
> database files, use third party LDAP tools to create a text dump in
> LDIF format, and then load the LDIF into an empty database of 5.5
> ldapd(8).

I'm currently trying to cobble together a binary importer which reads
5.4 dbs, and writes them as 5.5 dbs. It's a bit ugly, based on
frankensteined code from ldapd and ldapctl. I haven't found a straight
way to write back into a file, so I'm trying go down the compacting way,
which appears to be rewriting an entirely new database.

Hopefully, it should work in the end.

I thought about the VM/dump option, but all I could find was for slapd
(using slapcat). Could you give more details on the tools you use?

> It looks like particularly the btree_stat and btree_meta structs used
> in the ldapd(8) btree implementation are the culprits, as it looks like
> they are the only time_t bits actually stored on disk.  Since it
> appears my problems are now solved, I'm mostly sending this message as
> a heads up in case there is anyone still getting ready to upgrade to
> 5.5 that uses ldapd(8).

I think only the btree_meta is relevant, as I don't see the btree_stat
being written on disk.

--
Olivier Mehani <[hidden email]>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655
Confidentiality cannot be guaranteed on emails sent or received unencrypted.

Reply | Threaded
Open this post in threaded view
|

Re: ldapd(8) binary incompatibility, 5.4 -> 5.5

Matthew Weigel
On 7/22/14, 9:03 PM, Olivier Mehani wrote:

>> I ended up having to create a 5.4 VM (I stuck with the same amd64 arch
>> as my actual server, and have not investigated or tested under what
>> constraints this might work across architectures) to load the ldapd(8)
>> database files, use third party LDAP tools to create a text dump in
>> LDIF format, and then load the LDIF into an empty database of 5.5
>> ldapd(8).
>
> I'm currently trying to cobble together a binary importer which reads
> 5.4 dbs, and writes them as 5.5 dbs. It's a bit ugly, based on
> frankensteined code from ldapd and ldapctl. I haven't found a straight
> way to write back into a file, so I'm trying go down the compacting way,
> which appears to be rewriting an entirely new database.
>
> Hopefully, it should work in the end.
>
> I thought about the VM/dump option, but all I could find was for slapd
> (using slapcat). Could you give more details on the tools you use?

Someone else asked about this off the list.  After setting up the VM
and copying /etc/ldapd.conf, /etc/ldap/*.schema, and /var/db/ldap/*.db
into it, I started up ldapd(8) and connected to it with ldapvi(1) from
ports.  I wrote out the contents of that buffer to a separate file, and
that was my not-exactly-LDIF text backup.

To add those entries to the 5.5 server, I replaced the numeric
identifier ldapvi(1) uses for existing entries with the special key
'add' like so:

0 dc=example,dc=net
objectClass: dcObject
objectClass: organization
objectClass: top
dc: example
o: example.net
description: Account and Group LDAP Identity Database

was changed to

add dc=example,dc=net
objectClass: dcObject
objectClass: organization
objectClass: top
dc: example
o: example.net
description: Account and Group LDAP Identity Database

I had to use ldapadd(1) from the openldap-client package to populate
the root object before ldapvi(1) would work, however.  I think I also
had to add the structure of the LDAP tree first, and do a second round
of edits to populate leaf nodes.

>> It looks like particularly the btree_stat and btree_meta structs used
>> in the ldapd(8) btree implementation are the culprits, as it looks like
>> they are the only time_t bits actually stored on disk.  Since it
>> appears my problems are now solved, I'm mostly sending this message as
>> a heads up in case there is anyone still getting ready to upgrade to
>> 5.5 that uses ldapd(8).
>
> I think only the btree_meta is relevant, as I don't see the btree_stat
> being written on disk.

Maybe, I didn't dig too deep into it once I solved my problems.
--
  Matthew Weigel
  hacker
  unique & idempot . ent

Reply | Threaded
Open this post in threaded view
|

Re: ldapd(8) binary incompatibility, 5.4 -> 5.5

Matthew Weigel
On 7/22/14, 9:37 PM, Matthew Weigel wrote:

> into it, I started up ldapd(8) and connected to it with ldapvi(1) from
> ports.  I wrote out the contents of that buffer to a separate file, and

Actually I didn't notice it this weekend but ldapvi(1) has --in and
--out arguments that do exactly the right thing - just read and write
straight LDIF files.
--
  Matthew Weigel
  hacker
  unique & idempot . ent

Reply | Threaded
Open this post in threaded view
|

Re: ldapd(8) binary incompatibility, 5.4 -> 5.5

Olivier Mehani-5
Hey Matthew,

On 2014-07-23, Matthew Weigel <[hidden email]> wrote:
>> into it, I started up ldapd(8) and connected to it with ldapvi(1) from
>> ports.  I wrote out the contents of that buffer to a separate file, and
> Actually I didn't notice it this weekend but ldapvi(1) has --in and
> --out arguments that do exactly the right thing - just read and write
> straight LDIF files.

Yup, this did the trick nicely (once I remembered that the DB LDAPD
looks for is in /var/db/ldap):

  ldapvi --out --host localhost -D cn=root,dc=example,dc=net > dump.ldiff
 
I have the openldap-client package installed, so ldapadd did the trick
to reimport all that into a fresh and empty DB

  ldapadd  -H ldapi://%2fvar%2frun%2fldapi -D cn=root,dc=example,dc=net -W < dump.ldiff

I'll give up on my binary importer (:

Thanks for the pointers!

--
Olivier Mehani <[hidden email]>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655
Confidentiality cannot be guaranteed on emails sent or received unencrypted.