Auto saved by Logseq

2026-08-06 14:36:18 +00:00 · 2022-04-09 19:11:17 +02:00
parent 7e0373c45a
commit 610b5115ed
2 changed files with 57 additions and 0 deletions
@@ -0,0 +1,14 @@
+- The Agora started very fast (it was pleasant) and then proceeded to get very slow (with features, but mostly really with the number of nodes+edges that it's handling, see stats in [[nodes]] if interested).
+- It could be fast again -- we just need to fix some issues :)
+- As of right now, very often the Agora takes almost 10s (!) to load some pages, particularly when under load. There's a variety of reasons this happens -- I could write about them here but right now I'd rather just go ahead and try to fix them ;)
+	- Doing pomodoros today, [[2022-03-20]], to try to address this issue.
+	- When starting benchmarking, dev.anagora.org/do is regularly taking 3.7s to load (measured with [[time curl]]). In production this is often longer, but this seems like a good test node (because of the amount of pushes it has, which I think is related to the slowdown).
+		- The *first* load after restarting the dev server is often 10s, which is closer to what we'd see in prod (because it needs to load+process the whole Agora graph).
+	- Same on [[2022-04-09]].
+		- Increased organic traffic + crawler bot activity.
+		- Changed number of workers (it seems more is not necessarily better, trying running with 10.)
+		- Changed cache expiry to a random range to prevent thundering herd by workers hitting cache TTL in unison.
+		- Cached calls to node().
+		- I wonder if there's low hanging fruit remaining; dev.anagora.org just tends to feel so fast in comparison with prod, perhaps I'm missing where's the bottleneck. Will add debugging data.
+		- HA.
+		- The Agora workers were restarting *very often*, way more often than CACHE_TTL, because the bots that are causing most of the increase in load were also hitting URLs that produce 500s -- a nice favour really, as
@@ -0,0 +1,43 @@
+- [[flancia meet]]
+- [[agora]]
+	- LATER could probably improve the welcome message in [[index]] -- it's overly long, a bit confusing for first time users?
+	- [[agora server]]
+		- [[spaces to tabs conversion]]
+			- I didn't like the result so I took the liberty to revert. Some rationale in the node above :)
+			- Reach out if you think it's the wrong call! But I'd rather go with the Python community standard by default and not impose an extra burden on contributors (to override their editor defaults).
+		- LATER fix links in /journals
+		- LATER perhaps tell crawling bots to take it easy while I work
+		- LATER [[sqlite]] experiment
+		- LATER I should update to Python 3.8 so I can use f-strings with = at the end to print variable name and value in one swoop.
+			- Surely there are also other improvements :)
+		- NOW agora load balancing
+		  :LOGBOOK:
+		  CLOCK: [2022-04-09 Sat 17:36:23]
+		  :END:
+			- it is time.
+			- once the next item is ready, this unlocks arbitrary [[horizontal scaling]].
+			- I think I'll start with the simplest setup possible, solving balancing first and reliability later -- that is, add balancing to the [[nginx]] running in [[thecla]].
+		- NOW experiment with [[podman]] as [[docker]] replacement
+		  :LOGBOOK:
+		  CLOCK: [2022-04-09 Sat 15:45:51]
+		  :END:
+			- continue going through [[please contain yourself]] with [[podman]]?
+			- nice, got it running just fine; I said I was not going to default to [[rootless containers]] *but* they seem to work fine for simple examples, and the
+		- crawlers/bots are hammering anagora.org quite a bit, might need to actually write a [[robots.txt]] file to tell them to take it easy while we work on better performance :)
+- I've been thinking of [[time]] from the point of view of [[lady burup]] (as I imagine it) and I've found it [[interesting]].
+	- When it's light, it's the next day for her; she might not understand that [[dawn]] shifts from season to season?
+- [[twitter dall-e]]
+- [[doing laundry]]
+- #push [[agora speed]]
+	- [[scaling]] must happen
+		- NOW hmm, but there is [[low hanging fruit]]: the per-worker cache should not all expire in unison (!)
+		  :LOGBOOK:
+		  CLOCK: [2022-04-09 Sat 17:52:37]
+		  :END:
+			- also I wasn't caching calls to G.node() (?).
+- [[daniel dennett]]
+	- ethical [[almanac]]
+	- #push [[agora server]]
+		- [[almanac mode]]
+- [[fin de semana]]
+-