I have been blogging in English for some time, but this is my first time participating in an indie web conversation. So I think I should introduce myself first.
I’m Eltrac. I’m mainly a web developer but I like all kinds of thing about programming. I code in Go, Clojure and Lua, and I explore new tech all the time. I usually blog more than three times a week, but most of my posts are written in Chinese. I recently became an active member of
Codeberg e.V.
and have been working on my own FOSS project
weepinbell
for a while. It’s a new Webmention receiver written in Clojure.
In this post, I will talk about why I developed weepinbell, how I designed its architecture, what I think of the IndieWeb movement and many things.
Why weepinbell?
I wrote to
Brennan
recently and he was really nice to mention weepinbell
on his blog
. In that email, I talked about how I set a new goal for weepinbell (i.e. an alternative to webmention.io) because of Aaron Parecki rewriting webmention.io with Claude. Later, in
Aaron’s response
to
Brennan’s new post
, he mentioned weepinbell as “a direct response to my rewrite”.
I would like to make things clear. I didn’t start developing weepinbell because of Aaron’s rewrite. I started early in this May to learn Clojure. I’ve been blogging (in Chinese) for years and of course I love the concept of IndieWeb. Naturally, my practice project for learning a new language involves IndieWeb technologies.
The motive is that I want to get my hands dirty with the Webmention standard. I read through the
W3C specification document
and started putting things in parentheses. You can still find snippets from that document in my code as comments. I bascially code against the specification. It was a fun experience for me.
(if-let [error-msg (check-request source target allowlist)]
;; respond if request verification failed (bad-request {:status400:message error-msg})
;; proceed if request is ok (do;; The receiver SHOULD check that target is a valid;; resource for which it can accept Webmentions.;; This check SHOULD happen synchronously to reject;; invalid Webmentions before more in-depth verification begins.;;;; Webmention verification SHOULD be handled asynchronously;; to prevent DoS (Denial of Service) attacks. (put! wmc new-wm)
;; respond with 201 when Webmention passed checks and;; was sent into queue, include a Location header (let [wm-id (digest (str source target))
wm-url (str (:base-url config) "/api/mention/" wm-id)
response {:status201:message"Webmention is accepted and pending for verification.":url wm-url}]
(created wm-url response))))))
After the initial quick and rough implementation, I stopped working on weepinbell for a few months. And it had been receiving Webmentions for my blog during that time. Everything worked and I was content. It was bascially a personal tool.
Then the webmention.io rewrite came to my attention.
Why does it bother me? I don’t even use webmention.io any more. I see some of my friends seeking alternatives, and… there aren’t many to choose from! There is
webmentiond
by Horst Gutmann and some other implementations here and there. My friend
Chlorine
developed
CircleAt
in Rust. And I’m sure there are many more. But frankly, none of them truly count as “alternative to webmention.io”.
Don’t get me wrong. I think many of them are great for their purpose, i.e. small service one can self-host. However, they can be quite difficult for the non-tech indie web folks and those who simply do not own a server. They need a public instance.
Refactoring for scalability
I decided to continue development of weepinbell and make it “the next webmention.io”, at least that’s what I’m going for.
So I took a closer look at my code. They’re not that messy, well partitioned (for the most part) and maintainable. I can swap some of the parts easily because I used hook functions and vtables.1 The architecture is flexible enough, and the Lisp power also makes modifying software really easy.
The following part will be somewhat technical. Feel free to jump to the next section if you don’t want to read about system design. The bottom line is: weepinbell would be lightweight enough for people to self-host; it would also be scalable enough to support large traffic.
I’m a Go programmer by the living.2 And Clojure’s
core.async
library greatly resembles Go’s concurrey programming style, which means my skills of building scalable applications in Go translates.
Webmention is not big data. Storage is definitely not the bottleneck. In fact, up until
a few commits ago
, weepinbell does not even support any type of database. It unloaded everything in a single file and then loaded it back when needed. And it just… worked, for months.
Of course this is neither performant nor safe. If I want to build something reliable, I need a database. If you have taken a look at that commit I linked, you might notice that the business logic is untouched. All that’s added is a new storage implementation, a new config option, and a few lines in the main component (core.clj).
I’ll probably elaborate on architecture flexibility in another post. That’s something I’m really proud of. The gist is that using a new type of database and ORM is easy, and will be just as easy if I ever decided to redo the storage layer again.
;; all I need to make sure is that the new storage implementation;; returns a valid storage vtable, like this:(defn use-store [db-file]
(let [ds (make-datasource db-file)]
(create-tables ds)
{:append (partial insert-webmention ds)
:update (partial update-webmention ds)
:delete (partial delete-webmention ds)
:get (partial get-webmention ds)
:get-by-domain (partial get-webmentions-by-domain ds)}))
Beside a new SQLite implementation, I added a storage worker that handles I/O asynchronously. So when processing a new webmention, the processor won’t have to wait for DB transaction (or appending to a file; it does not care about what’s going on underneath) to end. It simply delegates to the storage worker and keeps processing the next webmention.
The processor delegates by putting an operation object into a channel. It’s similar to what I already have for receiver. When a new webmention request is sent to the web server, receiver does some basic checks (e.g. does source and target look like real URLs?) and then put the request into the webmention queue. A processor will pick it up and make sure source has truly mentioned target, parse microformats, wrap data into a valid record, and hand it over to storage worker.
Reading from a channel is thread-safe. And since I/O is handled internally by storage worker, it can also be thread-safe, without the need for a mutex lock. So we can add as many processors as we want to handle webmentions concurrently.
The overall architecture looks like this:
Architecture of the receiver part of weepinbell, drawn using Excalidraw
That’s horizontal scaling. For self-hosters, simply set pipeline-concurrency to 1, and there will be only one processor handling webmentions, using a reasonable amount of RAM and CPU.
For large groups or someone who wants to host a public instance, change a number in config and you’re good to go.
It’s just regular HTML with standardized class names like h-card and p-name. It’s easy to add a few classes to your HTML. However, it’s still hard to parse structural data from HTML with just a few more classes.
Good thing there are already some parsers available… right?
Production ready parsers are available for: Go, Node, PHP, Python, Ruby.
Right… I’m using Clojure…
Clojure can use Java code, since it runs on top of it. I found
mf2j
, an “In-development Java parser for microformats2”. It hasn’t been updated since… 10 years ago.
There is a
Clojure implementation
. It’s archived, with its last update committed 11 years ago.
Do I implement a new one myself? Do I pick up where they left off? How do I make sure I don’t burn out and abandon it as well? After all, Clojure is a niche language and IndieWeb is also kind of niche. I can’t imagine there being another Clojurist building something that relies on microformats. At least, I don’t know anyone like that.
I briefly considered running a micro-service that wraps the Go parser, and weepinbell can talk to this service using RESTful API or gRPC. But it feels heavy and everything just won’t fit in a single .jar.
So how can Clojure run programs written in a non-JVM language? What else runs on JVM? Scala, Groovy… Wait, will WebAssembly work?
Yes! WebAssembly runs on JVM. There is a native JVM WebAssembly runtime called
chicory
.
I spent an afternoon studying
Extism
and chirocy. Then I wrote a tiny little Go wrapper of Will Norris’
parser library
.
package main
import (
"bytes""github.com/extism/go-pdk""willnorris.com/go/microformats")
//go:wasmexport parsefuncparse() int32 {
input := pdk.Input()
r := bytes.NewReader(input)
data := microformats.Parse(r, nil)
pdk.OutputJSON(data)
return0}
Then I compiled it to .wasm using
tinygo
compiler, bundled the .wasm inside .jar, and called the Extism plugin from Clojure using chicory. Voilà! Here’s
mf2clj
, a Clojure wrapper of WebAssembly compiled from a Go wrapper of an existing microformats parser.
By the way, I made a little logo just for fun.
You use it just like a normal library in Clojure. And it’s surprisingly fast.
More surprisingly, bundling a .wasm binary and the WebAssembly runtime inside the .jar is not as heavy as I thougt it would be. The .jar only grew 2 MB in size. Am I really running Java?
Nevertheless, the nightmare isn’t over yet. In the begining, I settled for the incomplete mf2j library and used it for a while. At that time, I called the JSON output “data structure from Hell”. I still do —— Everything is an array! Everything is deep inside something’s properties or children! Nothing is deterministic! Every website’s format is different!
In the end, I had to write a smaller parser to parse output a the original parser. I called this feature microformats reduction. I remember webmention.io having something similar.
If something does not exists, it’ll just be null. The format is not final yet, I will make some adjustments. This can be turned off by ?microformats=raw when querying webmentions.
Also, I think more than half of the bugs I found in weepinbell have something to do with microformats-related features. Maybe we should switch to Open Graph and JSON-LD, folks?
Anyway, I’m just glad it’s finally working.
Slowing down
At this point, I’ve almost finished everything I have planned for the v1.0.0 release: a proper database, horizontal scalability, up-to-date microformats parser. As for hosting a public instance, that’ll involve authenticating users and making a dashboard for them to manage webmentions. That would be a separate project.
I plan to develop a service that sits in front of weepinbell, handling logging in and stuffs. I mentioned the idea in the project’s
PHILOSOPHY.md
.
I want to keep the core of weepinbell minimal. Those who want to be their own sysadmins don’t have to care about authentication. They don’t even need a frontend. They only need to self-host one .jar and use API to query webmentions, use RSS to receive notifications and use webhook to automate stuff. These are the built-ins.
A platform like webmention.io needs more than that. If I want to keep weepinbell “for both individuals and large groups”, I have to separate things. Also, this makes everything modular and swappable, a pretty good architecture decision.
My writing to Brennan was an attempt to get more Clojure developers on the team. Well, that didn’t go well. For the new platform, which will be named Ringabell4, I want to use more mainstream stack, probably Go, so that it’s easier for people to get involved.
For now, I’m still working on weepinbell’s test suite. The current test coverage is more than 83%, but there are still some edge cases that haven’t been covered by tests. I had to test a lot of paths by hand. Also it’ll take some time for the new SQLite implementation to be stable.
I don’t think we need to or should move fast when it comes to developing reliable software, with or without LLMs.
Independence, and now decentralization
In reply to Brennan’s recent question,
“What do we do now?”
, I think it’s finanlly time to get more decentralized.
Decentralization is never about moving away from the corporate web (that’s independence). It’s about not relying on a
single point
, no matter what the point is and who it involves.
I agree with Brennan on not conflating IndieWeb and the lower-case indie web. The
IndieWeb camp
is a nice community and a great introduction to the indie web. But I think we should form more indie web communities other than the IndieWeb camp.
omg.lol
is an example. It’s a fun web community by
Neatnik
. They provide infrastructure like a simple profile page, Fediverse instance and email forwarding. Brennan is also building
folk.zone
, which hosts a fork of webmention.io and many more.
Those two may not be exactly alternatives to the IndieWeb camp, but they can be. And there can be more for people to choose from.
Tech folks can self-host their own infrastructure if they want. For the non-tech folks, there should be more alternatives to what the IndieWeb camp has to offer. Poeple who are capable can host public services and take a small community.
After all, communities are what brings the indie web together. We can and should build more communities (this is probably more like polycentricity rather than decentralization). We can have the LispyWeb for lisp programmers, ArtsyWeb for artists, and Indie Indie Literature Club!5 for book lovers. Those are just some random names I came up with.
Perhaps the upcoming Ringabell might be the next cozy small community. Who knows? To everyone who’s nice enough to finish reading this, I hope we can all find our people.
A vtable in Clojure is basically a hashmap whose values are functions. It’s a technique mentioned in one of Uncle Bob’s books, Functional Design. ↩︎
Actually I’m at the very begining of my career. Don’t think of me as an expert of anything. XD ↩︎
Clojure runs on JVM, so it compiles to JVM bytecode, just like Java. ↩︎
标照 Rich 本人对简单的定义,标准库里的 map、filter、reduce 等操作序列的函数,都不够简单,因为他们不仅涉及做什么操作(what to do),还涉及了怎么做(how to do it)——预设了惰性,预设了数据操作的顺序(序列都是从前往后线性遍历的),预设了数据的表示方式(返回的是序列)。
如果 map、filter、mapcat 等函数,能够与数据类型(序列)解耦呢?也就是说,如果 map 只定义做什么操作(what to do) ,把怎么做(how to do it)交给其他组件呢?
上一篇
我提及了如何用
Ring
开发 HTTP 服务器,也提到 Ring 提供了同步 API 和异步 API。那么,什么是异步编程?为什么需要异步?在什么情况下需要让一个程序同时做很多事情?
同步、异步、并发和并行
同步(synchronization)的中文名字可能会让人误以为是多个实体同时进行操作,实际上同步的意思是指系统中所有事件都依照相同的顺序进行协调。可以用工厂流水线举例,假如流水线中一个工人需要等上一个工人交付一部分工作之后才能干活,整个流水线都要所有人一起干活才能转动起来,这个流水线就是同步的。反之,假如某些工作可以分开做,比如一部分人负责生产 A 零件,另一部分人负责生产 B 零件,两者互不干扰,而最终把 A 和 B 组装在一起的又是另一拨人,这个系统就是异步的(asynchronous)。
;; A function that simulates a long-running process by calling Thread/sleep:(defn long-running-job [n]
(Thread/sleep3000) ; wait for 3 seconds (+ n 10))
;; Use `doall` to eagerly evaluate `map`, which evaluates lazily by default.;; With `map`, the total elapsed time is just under 4 * 3 seconds:user=> (time (doall (map long-running-job (range 4))))
"Elapsed time: 11999.235098 msecs"(10111213)
;; With `pmap`, the total elapsed time is just over 3 seconds:user=> (time (doall (pmap long-running-job (range 4))))
"Elapsed time: 3200.001117 msecs"(10111213)
core.async 几乎和 Go 语言异步编程的要素一模一样,尤其是 go block——把语句放在 go block 里面,语句就会异步执行。把消息队列 chan 当作一等公民的做法,也如出一辙。根据
Clojure 官方的博客文章
,这种异步编程风格其实有一段历史了,Go 也是借鉴并现代化了这种做法。
The roots of this style go back at least as far as Hoare’s Communicating Sequential Processes (CSP), followed by realizations and extensions in e.g. occam, Java CSP and the Go programming language.
无论如何,让我们来看看怎么用 go 和 chan 实现异步编程吧。如果你已经会写 Go,那么接下来的代码会很熟悉。
user=> (def c (chan1))
#'user/cuser=> (take! c
(fn [x]
(println "Clojure callback value " x)))
nil
user=> (put! c "XYZ")
Clojure callback value XYZ
true
user=> (put! c "XYZ")
true
最近在关注
Golda
,从 ch0ccyra1n 写的 Issues 来看,她近期并不打算推进太多功能开发,更多是在做各个平台的发包和新 Fork 落地的工作,还有清理 Hugo 的某些遗留问题(比如 Hugo 的传递依赖太多导致不太容易打包到 Guix 的
问题
)。我在帮忙移除了 hugo deploy 之后也找不到什么能帮忙的事情了。
不过我在尝试把博客从 Hugo 迁移到 Golda 的时候,发现我依赖的某些较新的 API 没有被 Golda 实现。我去翻了 Hugo 的更新日志和相关的 commit,发现这些功能有不少是用 Claude 写的,不能拿来用。Hugo 也废弃了某些 API。这样下去,Golda 的 API 会和 Hugo 的变得越来越不容易兼容,但一直盯着 Hugo 更新,等人重新实现他们用大模型生成的功能,也不现实。关于此事,我提了
Issue
,与维护者基本达成一致。不过,这也意味着我要为了消除 LLM 写的代码,耗费好多精力修改我原来的代码。
你说你能理解那些人。你理解有的人想要脱离职场的苦海,求助于这生来就不洁的技术,抱着投机的心态创业,指望做出爆火的产品解救自己的生活。你说你看见了他们的挣扎,却还是在读到他们用 Codex 和 Claude Code 时心头一震。你能要求他们去看看背后发生了什么吗?你说你理解有的人带着一种愚昧无知的上进心,理解他们关注“时代发展方向”,使用这项技术不过是为了自己的职业生涯考虑,是在适应变化。你自己也在做这样的事情,你深知你不在简历上写自己的 Agent 使用经验和上下文管理技巧就会被市场淘汰,你也在适应变化,你也是车轮下的蚂蚁。你与他们唯一的不同是,你非常清楚自己在车轮之下,而不是在风浪之上,你非常清楚自己随时会被轧死,而这不会让你变得更容易在车轮下生存,只会让你更容易陷入绝望。
职场前辈对你随口说的某句话你难以忘却。那天上午你没有工作,给 Golda 提交了代码之后,午餐时聊起此事,但避开了与大模型有关的词句。你说你给开源项目提 PR,前辈说挺好,“可以丰富自己的简历”。你总觉得自己没有背叛那个每天下午坐在电脑面前发现一个个奇迹的孩子,可你总归还是背叛了。那个孩子从来不会想到,追随自己的好奇心和他逐渐锻炼起来的品味,最终导向的仅仅是简历上的一行字,甚至写不进简历,因为没有人在乎你用 SSG 做网站、用 Lisp 写代码、用终端还是 IDE 做开发(神奇的是,他们很在乎你用不用大模型写代码,有没有跟上趋势)。
你说你清楚,你知道,人们只在乎成果。你也认可,你也不在乎某个数学家在证明某个定理时是否感到心潮澎湃。你知道你只需要交付可靠的软件(有很多时候,他们只要求你快速交付能符合短期要求的软件),没人在乎你用的是不是你喜欢的语言,没人在乎你用了什么设计模式,没人在乎你的代码整不整洁(“以前古法编程确实这有这个说法,现在都用 AI 写了啊”)。如果可以,他们真想完全除掉代码,而且他们已经这么干了。如果可以,你也想除掉数学、除掉昆虫、除掉令人大脑萎缩的短视频,而你也已经这么干了。你致力除掉你讨厌的东西,他们致力除掉他们讨厌的东西;你朝你的目标前进,他们朝他们的目标前进。
(defn make-route
"Returns a function that will only call the handler if the method and path
match the request." [method path handler]
(-> handler
(wrap-response)
(wrap-route-middleware)
(wrap-route-info [(or method :any) (str path)])
(wrap-route-matches method path)))
对谈中,Berkowitz 提及了真理的霸权(Tyranny of Truth)这个概念。他说他之前读到了西塞罗(Cicero)说过的一句话,那句话让他读过之后的几年内都感到内心不安:
I prefer, before heaven, to go astray with Plato, your reverence for whom I know, and admiration for whom I learn from your lips, rather than hold true views with his opponents.
第 95 期周刊
中我提到《独树不成林》主播仲树洗稿抄袭一事。有人发现她发布的好几期播客节目从内容、结构和例证上都与
Old School with Shilo Brooks
等播客十分相似,而舆论发酵后,仲树本人并没有回应,连着发布了三期播客之后,就陷入了一个多月的沉默期。上一节提到的播客是她在那之后发布的第一期播客。
新播客是一起访谈,仲树只是提问者,偶尔提出自己的见解,内容基本都是 Roger Berkowitz 输出的,所以不太可能构成洗稿。不过,先前的洗稿基本上坐实了,她自己也没有为自己开脱。我不能为其他听众说话1,我自己的感想是,这个有洗稿前科的人推出了新的内容,质量很不错,我听完也写了不少感想(见上),但这些内容出自一个“罪人”,所以我现在的心情很复杂。
Do you have an iPhone? A Mac or iPad?
你有 iPhone 吗?一台 Mac 或者 iPad?
Do you order from Amazon, buy groceries from Whole Foods, or use any of their other properties (Prime Video, Twitch, Audible, etc.)? Do you use AWS or any websites that rely on AWS?
你从亚马逊买东西,从 Whole Foods 买杂货,或者用过他们的资产(Prime Video、Twitch、Audible 等)吗?你用 AWS 或者用依赖 AWS 的网站吗?
Do you use GitHub or Office or Windows or Azure or Typescript? How about DuckDuckGo, which uses Bing?
你用 GitHub 或者 Office 或者 Windows 或者 Azure 或者 Typescript 吗?那 DuckDuckGo 呢,它用的是 Bing?
Do you run Linux, which now includes AI-assisted code? Do you use Claude or ChatGPT or any of those?
你运行 Linux 吗?现在它包含 AI 辅助生成的代码了。你用 Claude 或者 ChatGPT 或者其他同类软件吗?
Do you use any Meta social networks (Facebook, Threads, Instagram)?
你用 Meta 的社交网络吗(Facebook、Threads、Instagram)?
Do you write Ruby code? (Which features DHH prominently on its home page.)
你写 Ruby 代码吗?(他们的官网上显眼地放上了 DHH 的大名)
Do you have an Android phone or use Google search or Gmail or write code in Go?
你用 Android 手机或者用 Google 搜索或者 Gmail 或者用 Go 写代码吗?
没有人的技术选择是纯洁的,许多技术的诞生或许就不纯洁。我用 iPhone、Mac 和 iPad;我虽然远离 GitHub,但大半个软件世界的基础设施都依靠 GitHub;我用 Linux,不可能不用;我写 Go 代码,我也真的非常喜欢 Go。我是法西斯主义者吗?我当然不是。
那,那些 LLM 用户就要么无知要么邪恶吗?当然不是。
These days we need all the righteous fury we can rally — and we also need the clarity to know where to focus it.
It’s clear that if you are somebody who already has an effective throughput, then the specifics of the underlying mechanics are uniquely personal and you should continue doing whatever works for you.
LWM(Linux Weekly News)报道了 Scott Jenson 在 KDE 的年度会谈上的讲话。Jenson 在 Apple 和 Google 都有做 UI/UX 设计师的经验。如果读者也接触过 Linux 桌面端或者别的开源桌面软件,会发现自由和开源的桌面端软件中很难找到 UI/UX 设计精良的。Jenson 想要「现代化」开源桌面端。
Jenson 表示开源桌面端的一大问题是,他们在过去一直跟随 Windows 和 macOS 的设计,但事到如今,这些操作系统背后的公司其实也有些摆烂了,他们不再稳定产出值得参考的 UX 设计。我仔细回忆了一下最近 Microsoft 和 Apple 在桌面端做了些什么,我只能想到:
Windows 现在才在快照版本的 File Explorer 里支持 ~
,他们在 Windows 11 才支持 sudo(并且我自己体验下来觉得很难用),还有刚过去不久的 macOS Tahoe 的丑陋 UI 设计和噩梦般的性能消耗(Golden Gate 要好很多的,但依旧没有好的创新)。
I have to talk about AI, and it pisses me off because there’s so many cool things we could be talking about, and AI just sucks up all the oxygen, and it’s all you can talk about.
我不得不提到 AI,我很生气,因为我们明明有很多很酷的事情可以讨论,但 AI 占据了所有的氧气,你能谈的就只有 AI 了。
Jenson 认为开源用户也是在意 UI/UX 体验的。他记得之前向 KDE 提过一个功能,在 macOS 上,在窗口之间拖拽元素时,在鼠标释放之前,被拖拽元素所在的窗口都不会升起来(raised),也就是不会突然挡住要拖过去的那个窗口。窗口是在鼠标释放之后才升起来的。之前 KDE 不是这样,他提出之后,这个功能很快就被一个贡献者实现了。
Subtasks available:
template A meta-template for 'lein new' templates.
plugin A leiningen plugin project template.
default A general project template for libraries.
app An application project template.
todo 是 filter 和 map 返回的惰性序列,如果没有被访问就不会执行 process-data。之后通过 seq 检查 todo 是否非空,如果是的话就 doall,这个函数可以执行惰性序列中所有的元素。外层的 do 用来把多个表达式放在一个形式(form)里,不然会被 if 当作多个参数处理。