runxiyu's comments

On: Just Use Curl

[runxiyu]

Using -X POST is often wrong as it "changes the actual method string in the HTTP request [... and] does not change behavior accordingly" (Stenberg, 2015).

Although, it is correct for the article's mention of "Send POST requests"... just that typically people don't send POST requests out of the blue with no data.

[kaoD → runxiyu]

I think you misinterpret the text you're quoting (but it's hard to tell since you didn't include a link).

-X POST is not wrong, it's just superfluous when using other flags like -d where the method can be inferred.

POST requests are often sent with no data (anything that is not idempotent should, unless there's another verb that could fit better).

[jmholla → kaoD]

Here's the article in question. [0] I think runxiyu is correct.

The author delves a bit more into the issue.

> One of most obvious problems is that if you also tell curl to follow HTTP redirects (using -L or --location), the -X option will also be used on the redirected-to requests which may not at all be what the server asks for and the user expected. Dropping the -X will make curl adhere to what the server asks for. And if you want to alter what method to use in a redirect, curl already have dedicated options for that named --post301, --post302 and --post303!

Per the man page (`man 1 curl`),

> The method string you set with -X, --request will be used for all requests, which if you for example use -L, --location may cause unintended side-effects when curl does not change request method according to the HTTP 30x response codes - and similar.

`-d` and `--data` will appropriately change the headers of their requests. Funnily, `--post301` and `--post302` which have a similar effect as `-X POST` are RFC 7231 compliant, browsers just don't do that. [2][3] This is so ubiquitous that the error codes 307 and 308 were added to support the original behavior of repeating the request verbatim at the target address. Compare the following:

    > nc -l -p 8080 -q 1 <<< $'HTTP/1.1 301 Moved Permanently\nLocation: http://localhost:8081\n\n' & nc -l -p 8081 -q 1 <<< $'HTTP/1.1 200 OK\nContent-Length: 0\n\n' & curl -L --data test localhost:8080; wait
    > nc -l -p 8080 -q 1 <<< $'HTTP/1.1 301 Moved Permanently\nLocation: http://localhost:8081\n\n' & nc -l -p 8081 -q 1 <<< $'HTTP/1.1 200 OK\nContent-Length: 0\n\n' & curl -X POST -L --data test localhost:8080; wait
    > nc -l -p 8080 -q 1 <<< $'HTTP/1.1 308 Permanent Redirect\nLocation: http://localhost:8081\n\n' & nc -l -p 8081 -q 1 <<< $'HTTP/1.1 200 OK\nContent-Length: 0\n\n' & curl -L --data test localhost:8080; wait

What happens here:

1. In the 301 case with just `--data`, the request turns into a GET request when sent to the redirect.

2. In the 301 case with `-X POST`, the request stays a `POST` request, but doesn't send any data to the redirect.

3. Finally, in the case where the server returns a 308, we see the POST request is kept and the data is resent.

To further expand slightly on a different thing that might surprise some people, the data options will automatically set the content type by adding the header, `Content-Type: application/x-www-form-urlencoded`, as if sending form data from a browser. This behvaior can be overridden with a manual `-H`, `--header` argument (e.g., `-H 'Content-Type: application/json`).

Edit: cube00 pointed out that newer versions of curl than mine have `--json` which will do that automatically. [4]

[0]: https://daniel.haxx.se/blog/2015/09/11/unnecessary-use-of-cu...

[1]: https://www.rfc-editor.org/rfc/rfc7231

[2]: https://evertpot.com/http/301-moved-permanently

[3]: https://evertpot.com/http/302-found

[4]: https://news.ycombinator.com/item?id=45655409

On: Ask HN: What are you working on? (April 2025)

[runxiyu]

A git forge

On: 9front "clause 15 common elements of maus and star type" released

[runxiyu]

That cover image looks familiar.

On: Anubis Works

[runxiyu]

Probably not

On: Anubis Works

[runxiyu]

Anubis et al. are also looking into alternative algorithms. There seems to be consensus that SHA-256 PoW is not appropriate

[genewitch → runxiyu]

There's lots of other ones but you want hashes that use lots of RAM, stuff like scrypt used to be the go-to but I am sure there are better, now.

On: Anubis Works

[runxiyu]

Do you have a link to your own solution?

[JsonCameron → runxiyu]

I have a pretty similar one. (Works off of the same concept) https://github.com/JasonLovesDoggo/caddy-defender if you're curious. Keep in mind this will not protect you against residential IP scraping.

[prologic → runxiyu]

Not yet unfortunately. But if you're interested, please reach out! I currently run it in a 3-region GeoDNS setup with my self-hosted infra.

On: Powxy: Anti-Scraper Reverse Proxy

[runxiyu]

The main reason for the non-JS solution is because a lot of people in communities around me, me included, generally block JS in their browsers, and I thought it'd be cool to have a non-JS PoW solution.

To be honest I would prefer if this could be computed with a simple shell script, but calling sha256sum in a loop is ridiculously slow. I might consider other methods of doing proof of work in the future (IIRC it's possible with argon2).

[zzo38computer → runxiyu]

I agree. I also usually disable JavaScripts in the browser, and I think it is a good idea what you did with this. (Some users also may be using a browser without JavaScripts, so it is helpful with that too.)

I also agree that a simple shell script would probably be better, although nevertheless it is explained and someone could implement their own if they want to do, so it is good enough for now.