Question about lang/ghc module (trying to port git-annex)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about lang/ghc module (trying to port git-annex)

James Cook-2
Hi ports@,

I'm trying to write a port for git-annex, which is built using
Haskell's Cabal. I'm new to OpenBSD porting and don't completely
understand the lang/ghc module.

Basic question: do all of the Haskell dependencies (from git-annex.cabal) need
to already have their own ports (like devel/hs-async, devel/hs-network, etc)?
(Or will the missing ones be magically built like cabal-install normally does?)
If they do need to all be ported, how isn't there e.g. a devel/hs-containers
port --- doesn't basically every Haskell project depend on the containers
package?

git-annex has at least 50 dependencies I couldn't find in ports, so I
want to make sure I understand this before I start porting them one at
a time (or just give up)...

Also, I'm using devel/darcs/Makefile as my model but don't understand
the following comment. Should I also be listing git-annex's
dependencies in BUILD_DEPENDS instead of LIB_DEPENDS?

    # Yes, build dependencies, because GHC libs are still static and we
    # don't want to pull in all of ghc.
    BUILD_DEPENDS = archivers/hs-zip-archive>=0.2.3,<0.4 \
    archivers/hs-zlib>=0.6.1.2,<0.7.0.0 \
    ...

Another question: what is MODGHC_PACKAGE_KEY? ghc.port.mk describes it
as the "package key" of the library, but I don't know what that means.

Here is what I have so far. I only included the dependencies that are
already in ports. Currently make build fails with a compiler error
"Could not find module ‘System.FilePath.ByteString’", indicating that
it is indeed a problem that there's no filepath-bytestring port.

diff --git a/devel/git-annex/Makefile b/devel/git-annex/Makefile
new file mode 100644
index 00000000000..57ae7fe1079
--- /dev/null
+++ b/devel/git-annex/Makefile
@@ -0,0 +1,54 @@
+# $OpenBSD$
+#
+# TODO:
+# Use /usr/ports/infrastructure/bin/portcheck
+
+COMMENT = manage files with git without checking in file contents
+
+DISTNAME =      git-annex-8.20210127
+CATEGORIES = devel
+HOMEPAGE = https://git-annex.branchable.com/
+
+MODULES =       lang/ghc
+MODGHC_BUILD =  cabal hackage nort
+
+# Many licenses listed in COPYRIGHT. All permit redistribution.
+PERMIT_PACKAGE = Yes
+
+# TODO
+# "make port-lib-depends-check" can help
+#WANTLIB = ???
+
+# TODO
+# Dependencies
+#BUILD_DEPENDS = ???
+RUN_DEPENDS = net/rsync devel/git-lfs
+LIB_DEPENDS = devel/hs-network-uri>=2.6 \
+ devel/hs-exceptions>=0.6 \
+ devel/hs-data-default \
+ devel/hs-random \
+ devel/hs-dlist \
+ devel/hs-unix-compat \
+ devel/hs-async \
+ devel/hs-hslogger \
+ devel/hs-utf8-string \
+ devel/hs-sandi \
+ devel/hs-monad-control \
+ devel/hs-edit-distance \
+ devel/hs-resourcet \
+ devel/hs-conduit \
+ devel/hs-old-locale \
+ devel/hs-unliftio-core \
+ devel/hs-vector \
+ devel/hs-unordered-containers \
+ devel/hs-regex-tdfa \
+ devel/hs-byteable \
+ security/hs-crypto-api \
+ devel/hs-split \
+ textproc/hs-attoparsec \
+ devel/hs-QuickCheck
+#version doesn't match
+#devel/hs-network>=3.0.0.0
+TEST_DEPENDS = net/rsync
+
+.include <bsd.port.mk>
diff --git a/devel/git-annex/distinfo b/devel/git-annex/distinfo
new file mode 100644
index 00000000000..9d9fa1c5788
--- /dev/null
+++ b/devel/git-annex/distinfo
@@ -0,0 +1,2 @@
+SHA256 (ghc/git-annex-8.20210127.tar.gz) = Y29DlCDyipKoJQufi0IlZ+Q5MV8/LSPLC+o7Cg5XVcM=
+SIZE (ghc/git-annex-8.20210127.tar.gz) = 1361993

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
> Here is what I have so far.

P.S. to anyone trying to build git-annex, you'll need the following
patch, which I didn't yet get around to adding to my WIP port:

https://git-annex.branchable.com/bugs/__91__PATCH__93___OpenBSD__58___fix_Utility.DirWatcher.Kqueue/?updated#comment-0b3828545e6cf6ec417c0f82645888cb

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
In reply to this post by James Cook-2
Hi James,

James Cook <[hidden email]> writes:

> I'm trying to write a port for git-annex, which is built using
> Haskell's Cabal. I'm new to OpenBSD porting and don't completely
> understand the lang/ghc module.
>
> Basic question: do all of the Haskell dependencies (from git-annex.cabal) need
> to already have their own ports (like devel/hs-async, devel/hs-network, etc)?
> (Or will the missing ones be magically built like cabal-install normally does?)
> If they do need to all be ported, how isn't there e.g. a devel/hs-containers
> port --- doesn't basically every Haskell project depend on the containers
> package?

Yes, that's the way it currently is in the ports tree.

> git-annex has at least 50 dependencies I couldn't find in ports, so I
> want to make sure I understand this before I start porting them one at
> a time (or just give up)...

This is exactly the problem I want to solve with cabal.port.mk. You
can try to look at
https://marc.info/?l=openbsd-ports&m=160858285410366&w=2
A quick search in the archives will show the justification and the
history of the effort.

The current state of the work is I'm waiting on cabal 3.4 official
release gated by ghc-9.0. I'm also looking for people needing this work
and your attempt to add git-annex gives me more motivation to finish.

BTW, trying to build with cabal also needs devel/libmagic port, or else:

cabal: Missing dependency on a foreign library:
* Missing (or bad) C library: magic

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
Greg Steuck <[hidden email]> writes:

>> git-annex has at least 50 dependencies I couldn't find in ports, so I
>> want to make sure I understand this before I start porting them one at
>> a time (or just give up)...
>
> This is exactly the problem I want to solve with cabal.port.mk. You
> can try to look at
> https://marc.info/?l=openbsd-ports&m=160858285410366&w=2
> A quick search in the archives will show the justification and the
> history of the effort.
>
> The current state of the work is I'm waiting on cabal 3.4 official
> release gated by ghc-9.0. I'm also looking for people needing this work
> and your attempt to add git-annex gives me more motivation to finish.

FWIW, a version of git-annex port that can display its help message:
https://github.com/openbsd/ports/commit/0d970c085441b82f0080afee13e2034e47f475c0

I didn't bother to pare down the dependency list to exclude the test
stuff, so MODCABAL_MANIFEST is twice the length it should be.

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
On Sun, Jan 31, 2021 at 11:58:40PM -0800, Greg Steuck wrote:

> Greg Steuck <[hidden email]> writes:
>
> >> git-annex has at least 50 dependencies I couldn't find in ports, so I
> >> want to make sure I understand this before I start porting them one at
> >> a time (or just give up)...
> >
> > This is exactly the problem I want to solve with cabal.port.mk. You
> > can try to look at
> > https://marc.info/?l=openbsd-ports&m=160858285410366&w=2
> > A quick search in the archives will show the justification and the
> > history of the effort.
> >
> > The current state of the work is I'm waiting on cabal 3.4 official
> > release gated by ghc-9.0. I'm also looking for people needing this work
> > and your attempt to add git-annex gives me more motivation to finish.
>
> FWIW, a version of git-annex port that can display its help message:
> https://github.com/openbsd/ports/commit/0d970c085441b82f0080afee13e2034e47f475c0

Thanks. I'm currently building it and will try it with my repos.

In devel/cabal/Makefile would it make sense to fix the ghc version,
i.e:

  BUILD_DEPENDS += lang/ghc=8.10.3

I quickly discovered cabal won't build with 8.6.4, which is what I have
right now, because the port expects ghc-prim-0.6.1 which doesn't match.
With the above change, "make build" kicks off a build for ghc-8.10.3
(which I'm now waiting for).

> I didn't bother to pare down the dependency list to exclude the test
> stuff, so MODCABAL_MANIFEST is twice the length it should be.

Hm, I thought git-annex doesn't have test stuff. cabal v2-test doesn't
work; the way to test it is to build the binary and run git annex test.

Assuming it works and that "git annex test" doesn't show any problems,
is there anything else I can do to help with git-annex? I use darcs a
fair bit; would it help if I built it on your branch and tested it out?

> Thanks
> Greg

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
James Cook <[hidden email]> writes:

> On Sun, Jan 31, 2021 at 11:58:40PM -0800, Greg Steuck wrote:
> Thanks. I'm currently building it and will try it with my repos.
>
> In devel/cabal/Makefile would it make sense to fix the ghc version,
> i.e:
>
>   BUILD_DEPENDS += lang/ghc=8.10.3

Good call. I was updating everything in lock-step so I never noticed.

> I quickly discovered cabal won't build with 8.6.4, which is what I have
> right now, because the port expects ghc-prim-0.6.1 which doesn't match.
> With the above change, "make build" kicks off a build for ghc-8.10.3
> (which I'm now waiting for).

It's not fast. I build with MAKE_JOBS=N to help a bit.

>> I didn't bother to pare down the dependency list to exclude the test
>> stuff, so MODCABAL_MANIFEST is twice the length it should be.
>
> Hm, I thought git-annex doesn't have test stuff. cabal v2-test doesn't
> work; the way to test it is to build the binary and run git annex
> test.

Right, that was a mistake on my part. I later realized the list can't be
shorter. I was just taken aback with the sheer mass of dependencies.

> Assuming it works and that "git annex test" doesn't show any problems,
> is there anything else I can do to help with git-annex?

The idea to link the test code into the main binary is quite
unorthodox. I can imagine some arguments for it, but I'll have to
overcome a lot of prior beliefs to accept this as a best practice.

$ touch /home/greg/.config/git-annex/program
$ git-annex test
...
    get (ssh remote):                                     FAIL (1.20s)
      ./Test/Framework.hs:57:
      get of file failed (transcript follows)
      get foo (from origin...) /bin/sh: git-annex-shell: not foundrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]rsync error: error in rsync protocol data stream (code 12) at io.c(228) [Receiver=3.2.3]rsync exited 12  rsync failed -- run git annex again to resume file transfer  transfer failed  Unable to access these remotes: origin  Try making some of these remotes available:          a45e9dcd-c958-4026-96e4-a0d9a6fc322b -- test repo [origin]failedgit-annex: get: 1 failed
...
6 out of 975 tests failed (1002.59s)
  (Failures above could be due to a bug in git-annex, or an incompatibility
   with utilities, such as git, installed on this system.)

The failing tests are due to unconfigured rsync stuff.

> I use darcs a fair bit; would it help if I built it on your branch and
> tested it out?

Please do. You are the first person I know who still uses darcs on
openbsd :)

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
> > Assuming it works and that "git annex test" doesn't show any problems,
> > is there anything else I can do to help with git-annex?
>
> The idea to link the test code into the main binary is quite
> unorthodox. I can imagine some arguments for it, but I'll have to
> overcome a lot of prior beliefs to accept this as a best practice.

This is the only place I've seen it. Maybe the author did it this way
so that users can reassure themselves that their storage backends are
working (using the more specialized "git annex testremote" command).

> $ touch /home/greg/.config/git-annex/program
> $ git-annex test
> ...
>     get (ssh remote):                                     FAIL (1.20s)
>       ./Test/Framework.hs:57:
>       get of file failed (transcript follows)
>       get foo (from origin...) /bin/sh: git-annex-shell: not foundrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]rsync error: error in rsync protocol data stream (code 12) at io.c(228) [Receiver=3.2.3]rsync exited 12  rsync failed -- run git annex again to resume file transfer  transfer failed  Unable to access these remotes: origin  Try making some of these remotes available:          a45e9dcd-c958-4026-96e4-a0d9a6fc322b -- test repo [origin]failedgit-annex: get: 1 failed
> ...
> 6 out of 975 tests failed (1002.59s)
>   (Failures above could be due to a bug in git-annex, or an incompatibility
>    with utilities, such as git, installed on this system.)
>
> The failing tests are due to unconfigured rsync stuff.

I don't think that's an rsync problem. Unfortunately the
git-annex-shell binary is missing, and that's pretty critical (needed
to access the repository from other machines; sort of like git
push/pull). Incidentally, the man pages also seem to be missing.

(I think git-annex tries to connect a local rsync process to a remote
git-annex-shell process, which would explain why we also see an rsync
error message.)

I guess this could be solved with custom logic in
devel/git-annex/Makefile. I can try to write some custom install
commands, maybe tomorrow or Friday. (If you point me to the appropriate
variable for overriding/augmenting the install script, that would help;
I'm still pretty new to ports.)

Alternatively, I think the cabal (v2-)install commands installs what's
needed (at least, it gets git-annex-shell, which is the really
important thing). Is there some way to take advantage of that in your
cabal infrastrsucture?

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
James Cook <[hidden email]> writes:

> I don't think that's an rsync problem. Unfortunately the
> git-annex-shell binary is missing, and that's pretty critical (needed
> to access the repository from other machines; sort of like git
> push/pull).

This part is easy, patch attached.

> Incidentally, the man pages also seem to be missing.

Would you like to try your hand in extending post-install target with
some man formatting magic like we have in other ports?

> Alternatively, I think the cabal (v2-)install commands installs what's
> needed (at least, it gets git-annex-shell, which is the really
> important thing). Is there some way to take advantage of that in your
> cabal infrastrsucture?

From what I gathered the v2-install target is largely unusable for
installing packages outside of .cabal tree. At least neither I nor
FreeBSD maintainer found a way to leverage that. Hence the manual
install flow.

From f190225986ed8e355ecd0622812539f82e35ac36 Mon Sep 17 00:00:00 2001
From: Greg Steuck <[hidden email]>
Subject: [PATCH] fixup: git-annex post-install

All 975 tests passed (977.23s)
---
 devel/git-annex/Makefile  | 5 +++++
 devel/git-annex/pkg/PLIST | 2 ++
 2 files changed, 7 insertions(+)

diff --git devel/git-annex/Makefile devel/git-annex/Makefile
index 63996c8f9c3..134fe5a030f 100644
--- devel/git-annex/Makefile
+++ devel/git-annex/Makefile
@@ -27,6 +27,11 @@ MAKE_ENV = LC_ALL=en_US.UTF-8
 MODCABAL_STEM = git-annex
 MODCABAL_VERSION = 8.20210127
 
+# Manual reimplementation of postCopy hook of Setup.hs
+post-install:
+ @ln -s git-annex ${PREFIX}/bin/git-annex-shell
+ @ln -s git-annex ${PREFIX}/bin/git-remote-tor-annex
+
 # Produced by:
 # $ cabal-bundler --openbsd git-annex-8.20210127
 # Then manually remove: git-annex 8.20210127 0
diff --git devel/git-annex/pkg/PLIST devel/git-annex/pkg/PLIST
index 2a5b445b27d..d52ede016f8 100644
--- devel/git-annex/pkg/PLIST
+++ devel/git-annex/pkg/PLIST
@@ -1,2 +1,4 @@
 @comment $OpenBSD: PLIST,v$
 @bin bin/${MODCABAL_STEM}
+bin/${MODCABAL_STEM}-shell
+bin/git-remote-tor-annex
--
2.30.0

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
On Thu, Feb 04, 2021 at 11:13:43AM -0800, Greg Steuck wrote:
> James Cook <[hidden email]> writes:
>
> > I don't think that's an rsync problem. Unfortunately the
> > git-annex-shell binary is missing, and that's pretty critical (needed
> > to access the repository from other machines; sort of like git
> > push/pull).
>
> This part is easy, patch attached.

Thanks, I can now access my git-annex repositories remotely without an
awful kludge.

> > Incidentally, the man pages also seem to be missing.
>
> Would you like to try your hand in extending post-install target with
> some man formatting magic like we have in other ports?

Done (commit a1c5aec8) in my "git-annex" branch:

https://github.com/falsifian/ports/commits/git-annex

It's not pretty. I replicated some logic from Build/Man.hs, which is
called from Setup.hs. Normally cabal v2-install would do that work for
us (putting the result in .cabal/store/ghc-XXX/git-annex-XXX) but I
don't know if it's worth getting that working.

Caveat: I took a shortcut when testing this: instead of re-running
"make build" I just made the post-build target after adding this in.
I'll try building from scratch later if you don't beat me to it.

On that branch I also removed the runtime dep on devel/git-lfs (it's
just one of a large number of optional backends).

> > Alternatively, I think the cabal (v2-)install commands installs what's
> > needed (at least, it gets git-annex-shell, which is the really
> > important thing). Is there some way to take advantage of that in your
> > cabal infrastrsucture?
>
> From what I gathered the v2-install target is largely unusable for
> installing packages outside of .cabal tree. At least neither I nor
> FreeBSD maintainer found a way to leverage that. Hence the manual
> install flow.

I probably don't know all the subtleties, but when I run
"cabal v2-install", I get a nice set of files under
~/.cabal/ghc-XXX/git-annex-XXX, including the man pages and all three
needed binaries. I would guess that just copying those files to the
destination would do the trick for most cabal executable packages, but
I haven't actually tried it. Maybe it's better to be cautious.

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
> > From what I gathered the v2-install target is largely unusable for
> > installing packages outside of .cabal tree. At least neither I nor
> > FreeBSD maintainer found a way to leverage that. Hence the manual
> > install flow.
>
> I probably don't know all the subtleties, but when I run
> "cabal v2-install", I get a nice set of files under
> ~/.cabal/ghc-XXX/git-annex-XXX, including the man pages and all three
> needed binaries. I would guess that just copying those files to the
> destination would do the trick for most cabal executable packages, but
> I haven't actually tried it. Maybe it's better to be cautious.

I mean ~/.cabal/store/ghc-XXX/git-annex-XXX.

Also forgot to mention the following git-annex patch (already reported
upstream) is needed if anyone does want to experiment with using cabal
v2-install to get git-annex man pages.

diff --git a/Build/Mans.hs b/Build/Mans.hs
index 9fb29d4a3..672dcd71c 100644
--- a/Build/Mans.hs
+++ b/Build/Mans.hs
@@ -38,7 +38,8 @@ buildMans = do
  if (Just srcm > destm)
  then do
  r <- system $ unwords
- [ "./Build/mdwn2man"
+ [ "perl"
+                                       , "Build/mdwn2man"
  , progName src
  , "1"
  , src

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
In reply to this post by James Cook-2
James Cook <[hidden email]> writes:

>> Would you like to try your hand in extending post-install target with
>> some man formatting magic like we have in other ports?
>
> Done (commit a1c5aec8) in my "git-annex" branch:
>
> https://github.com/falsifian/ports/commits/git-annex

Very nice!

> It's not pretty. I replicated some logic from Build/Man.hs, which is
> called from Setup.hs.

Some quick observations. Makefile allows line-continuation with \ and
this is how people write long command lines. The for loop would be more
readable if it were split like that. I also don't see a lot of places in
ports make files with quotes around variables. I suspect we just expect
people to not go nuts with spaces in directory names when building
ports.

> Normally cabal v2-install would do that work for
> us (putting the result in .cabal/store/ghc-XXX/git-annex-XXX) but I
> don't know if it's worth getting that working.
>
> Caveat: I took a shortcut when testing this: instead of re-running
> "make build" I just made the post-build target after adding this in.
> I'll try building from scratch later if you don't beat me to it.

I believe `make rebuild` should reuse the cabal built pieces. post-build
seems like a reasonable place for this kind of work. Alternatively
sticking those man-formatting commands into post-install also seems
acceptable to me.

> On that branch I also removed the runtime dep on devel/git-lfs (it's
> just one of a large number of optional backends).

Good call.

>> From what I gathered the v2-install target is largely unusable for
>> installing packages outside of .cabal tree. At least neither I nor
>> FreeBSD maintainer found a way to leverage that. Hence the manual
>> install flow.
>
> I probably don't know all the subtleties, but when I run
> "cabal v2-install", I get a nice set of files under
> ~/.cabal/ghc-XXX/git-annex-XXX, including the man pages and all three
> needed binaries. I would guess that just copying those files to the
> destination would do the trick for most cabal executable packages, but
> I haven't actually tried it. Maybe it's better to be cautious.

I'll dig into this a bit. I simply didn't need to worry about this up
until now as all the other ports don't do much custom Setup.hs work.

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
On Fri, Feb 05, 2021 at 09:57:48PM -0800, Greg Steuck wrote:

> James Cook <[hidden email]> writes:
>
> >> Would you like to try your hand in extending post-install target with
> >> some man formatting magic like we have in other ports?
> >
> > Done (commit a1c5aec8) in my "git-annex" branch:
> >
> > https://github.com/falsifian/ports/commits/git-annex
>
> Very nice!
>
> > It's not pretty. I replicated some logic from Build/Man.hs, which is
> > called from Setup.hs.
>
> Some quick observations. Makefile allows line-continuation with \ and
> this is how people write long command lines. The for loop would be more
> readable if it were split like that. I also don't see a lot of places in
> ports make files with quotes around variables. I suspect we just expect
> people to not go nuts with spaces in directory names when building
> ports.

Thanks. I split up the long command and removed quotes:

https://github.com/falsifian/ports/commit/e24278e3c1d7fca1b3c4961a0e81a5bb2ca5bcf0

Two notes:

* Currently "make post-build" fails if you run it twice. Adding -p in
  "mkdir ${MAN1_STAGING_DIR}" would fix that. I don't know if it's
  better to fail or not if "make post-build" gets run more than once.

* I see you put "@" before the ln -s lines. I wasn't sure whether to
  add them to the other commands. The output is not that interesting.

> > Normally cabal v2-install would do that work for
> > us (putting the result in .cabal/store/ghc-XXX/git-annex-XXX) but I
> > don't know if it's worth getting that working.
> >
> > Caveat: I took a shortcut when testing this: instead of re-running
> > "make build" I just made the post-build target after adding this in.
> > I'll try building from scratch later if you don't beat me to it.
>
> I believe `make rebuild` should reuse the cabal built pieces. post-build
> seems like a reasonable place for this kind of work. Alternatively
> sticking those man-formatting commands into post-install also seems
> acceptable to me.

I confirmed the port works by installing on a different machine.

One hitch: I needed to manually uninstall the current cabal-install
port because it conflicts with the new cabal port. Is there a way to
tell ports that cabal is a successor to cabal-install?

(I considered putting the man-building commands in post-install, but I
wasn't sure where to put the staging directory if I did that. I wanted
the staging directory because I assumed I should use the ${INSTALL_MAN}
macro if possible to put the files in their final place.)

> > On that branch I also removed the runtime dep on devel/git-lfs (it's
> > just one of a large number of optional backends).
>
> Good call.
>
> >> From what I gathered the v2-install target is largely unusable for
> >> installing packages outside of .cabal tree. At least neither I nor
> >> FreeBSD maintainer found a way to leverage that. Hence the manual
> >> install flow.
> >
> > I probably don't know all the subtleties, but when I run
> > "cabal v2-install", I get a nice set of files under
> > ~/.cabal/ghc-XXX/git-annex-XXX, including the man pages and all three
> > needed binaries. I would guess that just copying those files to the
> > destination would do the trick for most cabal executable packages, but
> > I haven't actually tried it. Maybe it's better to be cautious.
>
> I'll dig into this a bit. I simply didn't need to worry about this up
> until now as all the other ports don't do much custom Setup.hs work.

I don't know if there will be any cases other than man pages and
git-annex's command aliases. Not sure how much effort it's worth.

> Thanks
> Greg

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
James Cook <[hidden email]> writes:
> Thanks. I split up the long command and removed quotes:
>
> https://github.com/falsifian/ports/commit/e24278e3c1d7fca1b3c4961a0e81a5bb2ca5bcf0

I pushed my update:
https://github.com/blackgnezdo/ports/commit/83db945f59633c848a9ae28627cd8e378e316a09

> Two notes:
>
> * Currently "make post-build" fails if you run it twice. Adding -p in
>   "mkdir ${MAN1_STAGING_DIR}" would fix that. I don't know if it's
>   better to fail or not if "make post-build" gets run more than once.

It seems unusual for `make rebuild` to fail. Your newly added action is
idempotent.

> * I see you put "@" before the ln -s lines. I wasn't sure whether to
>   add them to the other commands. The output is not that interesting.

I added them to your commands as I agree the output is not interesting.

> I confirmed the port works by installing on a different machine.

Thank you for testing this!

> One hitch: I needed to manually uninstall the current cabal-install
> port because it conflicts with the new cabal port. Is there a way to
> tell ports that cabal is a successor to cabal-install?

Yes, we'll need to update the quirks package which provides transitions
for this kind of cases.

> (I considered putting the man-building commands in post-install, but I
> wasn't sure where to put the staging directory if I did that. I wanted
> the staging directory because I assumed I should use the ${INSTALL_MAN}
> macro if possible to put the files in their final place.)

I think what we have now is fine.

>> I'll dig into this a bit. I simply didn't need to worry about this up
>> until now as all the other ports don't do much custom Setup.hs work.
>
> I don't know if there will be any cases other than man pages and
> git-annex's command aliases. Not sure how much effort it's worth.

The default action of doing nothing is valid :)

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
Hi Greg,

> >> I'll dig into this a bit. I simply didn't need to worry about this up
> >> until now as all the other ports don't do much custom Setup.hs work.
> >
> > I don't know if there will be any cases other than man pages and
> > git-annex's command aliases. Not sure how much effort it's worth.
>
> The default action of doing nothing is valid :)

Incidentally, I asked the maintainer how man pages are supposed to be
installed, and they said it's best to use the Makefile if you want a
complete installation. Probably not worth changing our port, but
something to keep in mind in case this turns out to be a pattern.
https://git-annex.branchable.com/bugs/__91__Patch__93___fix___34__mdwn2man__58___cannot_execute_-_...__34__/


A few notes / questions:

* I fixed a couple of typos in
  https://github.com/falsifian/ports/commit/98111e702634842865a3d27675b6848e3fb2b7ca

* I added example usage for cabal-bundler. I hope I got it right:
  https://github.com/falsifian/ports/commit/f21cb097f215831b1d08443f9074073895a13731

* If I find more time to work on this, should I polish my git-annex
  port, or is there something more useful I could be doing with your
  Cabal infrastructure? (I installed darcs and it seems to work.)

* I tried to enable tests, partly because Darcs has a comprehensive
  test suite. I ran into some trouble. My WIP is here:
  https://github.com/falsifian/ports/commit/99cb0e33e2263cf15979a19bf068d3ffff45630c
  One problem is that cabal-bundler doesn't seem to be outputting test
  dependencies. (I tried manually adding some, not included in my
  commit.) Another is that, even if I add test dependencies manually, I
  end up with strange results like this:

    falsifian moth darcs $ make test
    ===>  Regression tests for darcs-2.16.2
    cd /usr/local/ports/pobj/darcs-2.16.2/darcs-2.16.2  && /usr/bin/env -i PORTSDIR="/usr/ports" LIBTOOL="/usr/bin/libtool"  PATH='/usr/local/ports/pobj/darcs-2.16.2/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11R6/bin' PREFIX='/usr/local'  LOCALBASE='/usr/local' X11BASE='/usr/X11R6'  CFLAGS='-O2 -pipe'  TRUEPREFIX='/usr/local' DESTDIR=''  HOME='/darcs-2.16.2_writes_to_HOME' PICFLAG="-fpic"  BINGRP=bin BINOWN=root BINMODE=755 NONBINMODE=644  DIRMODE=755  INSTALL_COPY=-c INSTALL_STRIP=-s  MANGRP=bin MANOWN=root MANMODE=644 BSD_INSTALL_PROGRAM="/usr/local/ports/pobj/darcs-2.16.2/bin/install -c -s -m 755"  BSD_INSTALL_SCRIPT="/usr/local/ports/pobj/darcs-2.16.2/bin/install -c -m 755"  BSD_INSTALL_DATA="/usr/local/ports/pobj/darcs-2.16.2/bin/install -c -m 644"  BSD_INSTALL_MAN="/usr/local/ports/pobj/darcs-2.16.2/bin/install -c -m 644"  BSD_INSTALL_PROGRAM_DIR="/usr/local/ports/pobj/darcs-2.16.2/bin/install -d -m 755"  BSD_INSTALL_SCRIPT_DIR="/usr/local/ports/pobj/darcs-2.16.2/bin/install -d -m 755"  BSD_INSTALL_DATA_DIR="/usr/local/ports/pobj/darcs-2.16.2/bin/install -d -m 755"  BSD_INSTALL_MAN_DIR="/usr/local/ports/pobj/darcs-2.16.2/bin/install -d -m 755" HOME=/usr/local/ports/pobj/darcs-2.16.2 /usr/local/bin/cabal v2-test --offline --disable-benchmarks  -w /usr/local/bin/ghc  -j1  --flags="curl -library"
    Warning: No remote package servers have been specified. Usually you would have
    one specified in the config file.
    Resolving dependencies...
    cabal: Could not resolve dependencies:
    [__0] trying: random-1.2.0 (user goal)
    [__1] trying: splitmix-0.1.0.3 (user goal)
    [__2] trying: splitmix:!bench
    [__3] rejecting: splitmix:*test (cyclic dependencies; conflict set: random,
    splitmix)
    [__1] fail (backjumping, conflict set: random, splitmix)
    After searching the rest of the dependency tree exhaustively, these were the
    goals I've had most trouble fulfilling: random, splitmix

  splitmix's test target depends on random and random depends on
  splitmix. Normally this doesn't cause any problem; I don't know why
  it's a problem here.

* It's a pity that your approach doesn't allow Cabal-based ports to
  share dependencies. Some thoughts on mitigating or solving this, in
  decreasing order of plausability:

  * Is it possible to hand-pick some particularly slow ones, make
    packages for them, and then tell Cabal to use those packages? E.g.
    aeson takes a while to build.

  * nixpkgs (the only package system I'm really familiar with) deals
    with Cabal by automatically generating package descriptions for
    everything on hackage. I wonder if something along those lines
    would be do-able.

    I guess it would be ugly to fill the ports tree with hundreds(?) of
    directories each with an auto-generated package. I'm not sure if
    there's a way to smoosh it all into one big file. E.g. I see
    there's something called "multi-packages", but I'm guessing that's
    not designed to handle hundreds of subpackages.

  * Is it possible to cheat, by having all the Cabal-based ports share
    their Cabal directory? So if I build darcs then git-annex, Cabal
    has access to all the build products from darcs when it builds
    git-annex. I guess one problem is this might make the build output
    depend on the order in which you build things, if Cabal
    opportunistically uses whatever versions are there.

* Are you planning to commit these changes all at once? Or is there
  some way to have an intermediate state where both old-style and
  new-style packages work?

> Thanks
> Greg

--
James

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck-5
Hi James,

James Cook <[hidden email]> writes:

> Incidentally, I asked the maintainer how man pages are supposed to be
> installed, and they said it's best to use the Makefile if you want a
> complete installation. Probably not worth changing our port, but
> something to keep in mind in case this turns out to be a pattern.
> https://git-annex.branchable.com/bugs/__91__Patch__93___fix___34__mdwn2man__58___cannot_execute_-_...__34__/

Thanks. We can leave this as is unless it proves more trouble than
adjusting the recipe going forward.

> A few notes / questions:
>
> * I fixed a couple of typos in
>   https://github.com/falsifian/ports/commit/98111e702634842865a3d27675b6848e3fb2b7ca
>
> * I added example usage for cabal-bundler. I hope I got it right:
>   https://github.com/falsifian/ports/commit/f21cb097f215831b1d08443f9074073895a13731

Neat. I'll pick these up.

> * If I find more time to work on this, should I polish my git-annex
>   port, or is there something more useful I could be doing with your
>   Cabal infrastructure? (I installed darcs and it seems to work.)

Great! I don't think there's much else to do besides waiting for
cabal-install 3.4.0.0 to be pushed to Hackage. Depending on how this
aligns with 6.9 release we may or may not ship it then.

You could try to build your own port of a Haskell binary you use that we
don't have yet. Your experience will likely tell us something about the
level of maturity of this infra.

> * I tried to enable tests, partly because Darcs has a comprehensive
>   test suite. I ran into some trouble. My WIP is here:
>   https://github.com/falsifian/ports/commit/99cb0e33e2263cf15979a19bf068d3ffff45630c
>   One problem is that cabal-bundler doesn't seem to be outputting test
>   dependencies.

I keep track of this issue in https://github.com/blackgnezdo/ports/issues/13.
See if you agree with my assessment there.

> * It's a pity that your approach doesn't allow Cabal-based ports to
>   share dependencies.

I chose to trade-off compute to gain simlpicity and flexibility:
https://github.com/blackgnezdo/ports/issues/3#issuecomment-650823377

Until this choice turns to be expensive enough to cause port builders to
complain I'm inclined to keep this approach.

> Some thoughts on mitigating or solving this, in decreasing order of plausability:
>   * Is it possible to hand-pick some particularly slow ones, make
>     packages for them, and then tell Cabal to use those packages? E.g.
>     aeson takes a while to build.

Sadly this will move towards tree-wide configuration management which
was a very non-fun thing to do as you experienced first-hand trying to
deal with a ton of ports when you tried to add git-annex.

>   * nixpkgs (the only package system I'm really familiar with) deals
>     with Cabal by automatically generating package descriptions for
>     everything on hackage. I wonder if something along those lines
>     would be do-able.

cabal-bundler --openbsd is a play in this area. I just limited it
to the absolute minimum I could get away with.

>     I guess it would be ugly to fill the ports tree with hundreds(?) of
>     directories each with an auto-generated package. I'm not sure if
>     there's a way to smoosh it all into one big file. E.g. I see
>     there's something called "multi-packages", but I'm guessing that's
>     not designed to handle hundreds of subpackages.

I've seen how these things play out in much bigger code bases with many
people over time. We don't have enough people working on ports@ to
manage this.

>   * Is it possible to cheat, by having all the Cabal-based ports share
>     their Cabal directory? So if I build darcs then git-annex, Cabal
>     has access to all the build products from darcs when it builds
>     git-annex. I guess one problem is this might make the build output
>     depend on the order in which you build things, if Cabal
>     opportunistically uses whatever versions are there.

The root of the problem is GHC compilation speed. My preferred and
universally useful optimizations would be:

  * Use/build something like ccache for ghc
  * Speed up ghc itself

> * Are you planning to commit these changes all at once? Or is there
>   some way to have an intermediate state where both old-style and
>   new-style packages work?

I don't see how it would be possible to have the old and new coexist.
ghc-8.10 is incompatible with the haskell ports in the tree. My plan is
to land this as a bundle. I wrote it up in an email to ports@ before.

Thanks
Greg

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Ashton Fagg
In reply to this post by Greg Steuck-5
Greg Steuck <[hidden email]> writes:

> This is exactly the problem I want to solve with cabal.port.mk. You
> can try to look at
> https://marc.info/?l=openbsd-ports&m=160858285410366&w=2
> A quick search in the archives will show the justification and the
> history of the effort.
>
> The current state of the work is I'm waiting on cabal 3.4 official
> release gated by ghc-9.0. I'm also looking for people needing this work
> and your attempt to add git-annex gives me more motivation to finish.

Hi Greg,

I've been considering trying to port pandoc, but was put off by the
sheer amount of dependencies that would need to be ported as well.

I'll give your work a try and see if it works well for that also, if
that's of interest to you?

Cheers,

Ash

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Greg Steuck
On Fri, Feb 19, 2021 at 5:19 AM Ashton Fagg <[hidden email]> wrote

> Hi Greg,
>
> I've been considering trying to port pandoc, but was put off by the
> sheer amount of dependencies that would need to be ported as well.
>
> I'll give your work a try and see if it works well for that also, if
> that's of interest to you?
>

You could test pandoc as it is done:
https://github.com/blackgnezdo/ports/commit/609197ce3f84a2e4a147fcabcc5eb71d3cf10ca2

The full list is https://github.com/blackgnezdo/ports/commits/ghc-8.10

Thanks
Greg
--
nest.cx is Gmail hosted, use PGP:
https://pgp.key-server.io/0x0B1542BD8DF5A1B0
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0
Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

Ashton Fagg
Greg Steuck <[hidden email]> writes:

> You could test pandoc as it is done:
> https://github.com/blackgnezdo/ports/commit/609197ce3f84a2e4a147fcabcc5eb71d3cf10ca2
>
> The full list is https://github.com/blackgnezdo/ports/commits/ghc-8.10

Splendid! Thank you for this.

Reply | Threaded
Open this post in threaded view
|

Re: Question about lang/ghc module (trying to port git-annex)

James Cook-2
In reply to this post by Greg Steuck-5
> > A few notes / questions:
> >
> > * I fixed a couple of typos in
> >   https://github.com/falsifian/ports/commit/98111e702634842865a3d27675b6848e3fb2b7ca
> >
> > * I added example usage for cabal-bundler. I hope I got it right:
> >   https://github.com/falsifian/ports/commit/f21cb097f215831b1d08443f9074073895a13731
>
> Neat. I'll pick these up.
>
> > * If I find more time to work on this, should I polish my git-annex
> >   port, or is there something more useful I could be doing with your
> >   Cabal infrastructure? (I installed darcs and it seems to work.)
>
> Great! I don't think there's much else to do besides waiting for
> cabal-install 3.4.0.0 to be pushed to Hackage. Depending on how this
> aligns with 6.9 release we may or may not ship it then.
>
> You could try to build your own port of a Haskell binary you use that we
> don't have yet. Your experience will likely tell us something about the
> level of maturity of this infra.

The only unported Haskell binaries I use are personal projects that are
not on Hackage. I could try locally porting one of those as an
experiment.

> > * I tried to enable tests, partly because Darcs has a comprehensive
> >   test suite. I ran into some trouble. My WIP is here:
> >   https://github.com/falsifian/ports/commit/99cb0e33e2263cf15979a19bf068d3ffff45630c
> >   One problem is that cabal-bundler doesn't seem to be outputting test
> >   dependencies.
>
> I keep track of this issue in https://github.com/blackgnezdo/ports/issues/13.
> See if you agree with my assessment there.

Thanks, I commented there.

> > * It's a pity that your approach doesn't allow Cabal-based ports to
> >   share dependencies.
>
> I chose to trade-off compute to gain simlpicity and flexibility:
> https://github.com/blackgnezdo/ports/issues/3#issuecomment-650823377
>
> Until this choice turns to be expensive enough to cause port builders to
> complain I'm inclined to keep this approach.

Okay, that's reasonable. My only concern: later on if people do
complain and it's decided it's a real problem, is there a path forward
that builds on what you've already got? Or would you have to start from
scratch to address it?

(I could complain already, but for me at least your solution is a
definite improvement over the status quo. The build times aren't the
end of the world.)

> > Some thoughts on mitigating or solving this, in decreasing order of plausability:
> >   * Is it possible to hand-pick some particularly slow ones, make
> >     packages for them, and then tell Cabal to use those packages? E.g.
> >     aeson takes a while to build.
>
> Sadly this will move towards tree-wide configuration management which
> was a very non-fun thing to do as you experienced first-hand trying to
> deal with a ton of ports when you tried to add git-annex.

Maybe there could be a rule of thumb, like adding a reusable port is
only worth it if it reduces the total time to build all ports by at
least X hours?

I don't know if there are any libraries that meet that definition for
an appropriate value of X.

> >     I guess it would be ugly to fill the ports tree with hundreds(?) of
> >     directories each with an auto-generated package. I'm not sure if
> >     there's a way to smoosh it all into one big file. E.g. I see
> >     there's something called "multi-packages", but I'm guessing that's
> >     not designed to handle hundreds of subpackages.
>
> I've seen how these things play out in much bigger code bases with many
> people over time. We don't have enough people working on ports@ to
> manage this.

That makes sense. I imagine your approach will keep things easier for
maintainers: e.g. I'm not going to accidentally break darcs by updating
the dependencies of git-annex.

> >   * Is it possible to cheat, by having all the Cabal-based ports share
> >     their Cabal directory? So if I build darcs then git-annex, Cabal
> >     has access to all the build products from darcs when it builds
> >     git-annex. I guess one problem is this might make the build output
> >     depend on the order in which you build things, if Cabal
> >     opportunistically uses whatever versions are there.
>
> The root of the problem is GHC compilation speed. My preferred and
> universally useful optimizations would be:
>
>   * Use/build something like ccache for ghc
>   * Speed up ghc itself

Those would be cool!

> > * Are you planning to commit these changes all at once? Or is there
> >   some way to have an intermediate state where both old-style and
> >   new-style packages work?
>
> I don't see how it would be possible to have the old and new coexist.
> ghc-8.10 is incompatible with the haskell ports in the tree. My plan is
> to land this as a bundle. I wrote it up in an email to ports@ before.

I didn't know about the incompatibility. Maybe there could be two
separate ghc versions for a while, but I guess that's more trouble than
it's worth.

> Thanks
> Greg

--
James