#quicklisp

[earl-29]

周知の事実なんでしょうが、trivial-ssh はワナなので使ってはダメです。 quicklisp に登録されてるのでついつい使ってしまいそうになりますが、download-file が動かないのでファイル転送できないし、2019-11 以降メンテもされてない。

ssh とか scp/sftp したいなら、cl-libssh2 を直接使えば OK 。

(ql:quickload :libssh2)

[commonlisp-it]

r/Common_Lisp - JZON hits 1.0 and is at last on the latest QL release: a correct and safe JSON parser, packed with features, and also FASTER than the latest JSON library advertised here.

24 votes and 0 comments so far on Reddit

[commonlisp-it]

New projects: asdf-dependency-graph — A minimal wrapper around `dot` available at least on Linux systems to generate dependency-graphs....

[zblisp]

UIOP 3.3.0 problems

Ok, here's something that is causing problems when I build this month's Quicklisp dist.

UIOP 3.3.0 was recently released, and it's causing some stuff to apparently compile over and over again. Here's a real simple thing to try:

$ cd ~/quicklisp/local-projects/ $ curl -O https://common-lisp.net/project/asdf/archives/uiop.tar.gz $ tar xzvf uiop.tar.gz

Then:

CL-USER> (ql:quickload "circular-streams")

On my setup, I see cffi and babel stuff compiled twice:

To load "circular-streams": Load 1 ASDF system: circular-streams ; Loading "circular-streams" [package uiop/package]............................ .................................................. .................................................. [package alexandria.0.dev]........................ .................................................. [package babel-encodings]......................... [package babel]................................... .................................................. [package cffi-sys]................................ [package cffi].................................... .................................................. [package cffi-features]........................... [package alexandria.0.dev]........................ .................................................. [package impl-specific-gray]...................... [package trivial-gray-streams].................... [package uiop/package]............................ .................................................. .................................................. [package babel-encodings]......................... [package babel]................................... .................................................. [package cffi-sys]................................ [package cffi].................................... .................................................. [package cffi-features]........................... [package static-vectors].......................... [package fast-io]................................. [package circular-streams]..

If I remove uiop-3.3.0/ from local-projects, the output looks like this:

To load "circular-streams": Load 1 ASDF system: circular-streams ; Loading "circular-streams" [package alexandria.0.dev]........................ [package impl-specific-gray]...................... [package trivial-gray-streams].................... [package uiop/package]............................ .................................................. .................................................. [package babel-encodings]......................... [package babel]................................... .................................................. [package cffi-sys]................................ [package cffi].................................... .................................................. [package cffi-features]........................... [package static-vectors].......................... [package fast-io]................................. [package circular-streams]..

Any ideas?

update Commit 4ed76c32050753c8a4450c342a1592881e11d63d seems to reference this behavior, with the "fast-io" system given as an example. And indeed, when I try this with fast-io, I see similar recompilation.

[zblisp]

Quicklisp: beyond beta

I gave a talk at ELS 2015 in London in April about the past, present, and future of Quicklisp. The “future” part detailed several things I’d like to accomplish before removing the beta label from Quicklisp.

The slides and script of the talk are available on github. But since there are over a hundred slides and about twenty pages of script, I thought I’d summarize things in this post.

First, what’s Quicklisp for, anyway? The slogan I put in the talk is: Make it easy to confidently build on the work of others. (Also, work on as many Common Lisps on as many platforms as possible.) 

Quicklisp achieves part of that already. It runs well on almost all Common Lisp implementations on all platforms. It’s easy to install, use, and update. Things are tested together so everything usually works. But if something breaks on update, you can revert to a previous working version. And with those features you can build on hundreds of libraries already.

If it can do all that, why not drop the “beta” already? There are still a number of things that I want Quicklisp to do before I’m ready to say “Here, this is what I had in mind from the start.”

First, I’d like to improve the confidence in the code you download and run. By adding HTTPS, signature checking, and checksum validation, you can be sure that there is nobody intercepting and modifying the software provided by Quicklisp. The signature and archive integrity checks must be made complete and automatic to have the best results.

Second, I’d like to add comprehensive user and developer documentation. For users, that means being able to learn each command, feature, and behavior of Quicklisp, to be able to use it to its fullest. For developers, that means being able to build your own solutions on a Quicklisp foundation without starting from scratch.

Third, I’d like to make it easy to find the project that does what you need, evaluate its quality and popularity, and find out if its license is compatible with your goals. If you want to make changes to a project, I want it to be easy to get the original source of a project and send fixes or improvements to the upstream maintainer.

Fourth, I’d like to make it easy to hook into the archive-fetching component of Quicklisp in a way that makes it easy to support additional integrity checks, support local development policies, and add local mirrors or caches for Quicklisp software.

These changes and improvements will take time. When they’re done, I’ll be happy to drop “beta” from Quicklisp.

[zblisp]

The unknown dependency tree

After posting about the Quicklisp verbosity conundrum, a few people emailed me with variations on this theme: "Since Quicklisp knows what the dependencies of a system are, can't you just load those quietly first and then load your project verbosely?"

The problem is that the premise is not true. Quicklisp has an idea about the dependencies of Quicklisp-provided systems, but not of any other systems available through ASDF. 

And it's actually pretty difficult to answer the question, for a given system, "What systems must be loaded first?" It's not as simple as loading the system definition and then looking at it. The act of loading the system definition may trigger the loading of other systems, which then load other systems, which then load other systems. System definition files are not simply data files. They're Lisp programs that can do arbitrary computation and manipulation of the environment. 

Quicklisp knows about its system dependency structures because, for every system in Quicklisp, I load it, and record what got loaded to support it. That dependency structure is then saved to a file, and that file is fetched by the Quicklisp client as part of a Quicklisp dist. This data is computed and saved once, on my dist-constructing computer, not each time, on the Quicklisp client computer. The data is evident whenever you see something like "To load foo, installing 5 Quicklisp releases: ..."

But that "installing 5 Quicklisp releases" only works when foo itself is provided by Quicklisp. No dependency info is printed otherwise.

Quicklisp then loads foo by calling asdf:load-system. If some system that foo requires isn't present, ASDF signals an asdf:missing-dependency error, which Quicklisp handles. If Quicklisp knows how to fetch the missing dependency, it does so, then retries loading foo. Otherwise, the missing dependency error is fatal.

Ultimately, though, only the top-level asdf:load-system can be wrapped with the verbosity-controlling settings. The fetching-on-demand error handling only happens the first time a system is installed, so it's not a predictable point of intercession. After that first time, the system is found via asdf:find-system and no error handling takes place.

Writing this up has given me some twisted ideas, so maybe a fix is possible. I'll keep you posted.

[zblisp]

clbuild and Quicklisp

Listen, friends, to the story of clbuild and how it influenced the design and implementation of Quicklisp.

I can't tell a full story of clbuild, since I didn't use it very much, but here's what I remember.

clbuild was trivial to install. Download a single file, a shell script, to get started. From there, clbuild could fetch the source code of dozens of interesting projects and set up an environment where it was easy to use that code to support your own projects. It was also trivial to hack on most of the projects, since in most cases you were getting a source control checkout. It was nice to be able to hack directly on a darcs or git checkout of a useful library and then send patches or pull requests upstream.

Luke Gorrie created it, and, like many of his projects, quickly encouraged a community of contributors and hackers that kept evolving and improving clbuild.

clbuild was fantastic in many ways. So why didn't I use it? Why create Quicklisp, which lacks some of the best features of clbuild?

My biggest initial issue was the firewall at work.

Since clbuild checked out from various version control systems, some of them used ports outside of the range allowed by a typical corporate firewall. I was limited almost exclusively to HTTP or HTTPS service.

A subsequent problem was obtaining all the prerequisite version control tools. Although git and github are dominant today, in 2007, cvs, darcs, svn, and several other version control systems were more frequently used than today. It took a series of errors about missing commands before I could finally get things rolling.

In 2007, for 20 different projects, there might be 20 different computers hosting them. In 2014, it's more likely that 18 are hosted on github. Because of the diversity of hosting back then, it wasn't all that uncommon for a particular source code host to be unavailable. When that happened to a critical project's host, it could mean that bootstrapping your project from clbuild was dead in the water, waiting for the host to come back.

Even if everything was available, there was no particular guarantee that everything actually worked together. If the package structure of a particular project changed, it could break everything that depended on it, until everything was updated to work together again.

Pulling from source control also meant that the software you got depended heavily on the time you got it. If you had separate clbuild setups on separate computers, things could get out of sync unless you made an effort to sync them.

One final, minor issue was that clbuild was Unix-only. If you wanted to use it on Windows, you had to set up a Unix-like environment alongside your Lisp environment so you could run shell scripts and run cvs, darcs, svn, etc. as though they were Unix command-line programs. This didn't affect me personally, since I mostly used Linux and Mac OS X. But it did limit the audience of clbuild to a subset of CL users.

Elements of Quicklisp's design are in reaction to these issues.

Quicklisp's software structure shifts the task of fetching from source control, building, and distribution of software from the end user to a central server. Rather than continuously updating all sources all the time, the updates happen periodically, typically once per month.

This is based on the observation that although there are intermittent problems with software incompatibility and build-breaking bugs, most of the time things work out ok. So the Quicklisp process is meant to slow down the pace of updates and "freeze" a configuration of the Common Lisp project universe at a working, tested, known-good point in time. 

In Quicklisp terms, that universe is called a dist, and a dist version represents its frozen state at a particular point in time. The software is checked out of every source control system, archived into a .tar.gz file, built and tested, and then finally frozen into a set of HTTP-accessible archive files with a few metadata and index files. Fetching libraries is then a matter of connecting to the central server via HTTP to get the metadata and archives. There are no source-control programs to install or firewall ports to open. The build testing means there is a reduced risk of one project's updates being fatally out-of-sync with the rest of the project universe.

By default, a Quicklisp installation uses the latest version of the standard dist, but it's a short, easy command to get a specific version instead either at installation time or at some later time. So even if you install Quicklisp multiple times on multiple computers, you can make sure each has the same software "universe" available for development. The uncertainty introduced by the time of installation or update can be completely managed.

This works even for the oldest dist versions; if you started a project in October, 2010, you can still go back to that state of the Common Lisp library world and continue work. That's because no archive file is ever deleted; it's made permanently available for just this purpose.

In the mid-2000s, it would have been hard to make a design like that very reliable for a reasonable cost. Amazon web services have made it cheap and easy. I have had only a few minutes of HTTP availability issues with Amazon in the past four years. I've never lost a file. 

Quicklisp mitigates the Unix-only issue by using Common Lisp for the installation script and Common Lisp as the library management program. It fetches via HTTP, decompresses, and untars archives with code that has been adapted to work on each supported Common Lisp on each platform. No Unix shell or command-line tools are required. 

There are still some bugs and issues with Quicklisp on Windows, because it doesn't receive as much testing and use as non-Windows platforms, but it's just as easy to get started on Windows as it is anywhere else.

Despite fixing some of my personal issues with clbuild, Quicklisp is missing a big, key feature. When using clbuild, it's easy to get to the forefront of development for the universe of CL software. You can work with the bleeding-edge sources easily and submit bug fixes and features. With Quicklisp, it's harder to find out where a particular library came from, and it's harder to get a source-control copy of it suitable for hacking and tweaking. It's harder to be a contributor, rather than just a consumer, of projects that aren't your own.

I'd like to improve the situation in Quicklisp, but some of the old obstacles remain. It would require a bunch of Unix-only or Unix-centric command-line tools to be installed and properly configured. Maybe that's not such a big deal, but it's loomed large in my mind and blocked progress. Maybe someone will take a look at the Quicklisp project metadata and write a nice program that makes it easy to combine the best of clbuild and Quicklisp. If you do, please send me a link.

PS. clbuild lives on in clbuild2. It looks like it's still active, with commits from just a few months ago. Maybe that's the right thing to use when the hacking urge strikes? I'll have to give it a try.

[earl-29]

Quicklisp news: 2013-06