thanks for clarifying.
with a long enough command and pipes, you could probably put it into one command (the hash check and everything), but it would be very long. This simplifys things and adds the opportunity later for encrypted signature checking (like how PGP email works).
So it exactly enables bad behavior?
curl | sh is not significantly more difficult for anyone familiar with the command line than
git clone xyz
cd xyz
make whatever
or
wget .sh
< hash check command > .sh
< text editor > .sh
chmod +x .sh
./.sh
and yet much worse.
The problem is people recommending their users do it, not the lack of a secure layer on top of it. If they're not willing to do either of the above alternatives, what makes you think yours will be any different?
Atm people are just doing curl | sh because its easy and others are recommending it. Its a nice one-line soluction so they are more likely to do it than doing the
wget .sh
< hash check command > .sh
< text editor > .sh
chmod +x .sh
./.shThis does that later version atm in one line so its equivalent in difficulty to curl | sh but one step better in security.
Going forward, I would like the checks part to also check the signature of hte script against ones the user has added to their trusted public keys.
If your problem is with using wget instead of curl to download the script, we can update the client to use curl or git or whatever.
He'd have to fork it most likely, curl pipe isn't (primarily) intended for running remote install scripts blindly, and I doubt anyone involved with curl development wants it to be.
I doubt anyone involved with curl development wants it to be
agreed
People are lemmings and do silly **** every day. I really think everyone should be using PGP emailing systems to ensure their emails are encrypted and signed so the recipeint can prove who they came from who they said they came from. Yet this hasn't really taken off because its too difficult for people to understand and takes a few extra steps. If you provide them with something that is as-easy or simpler with better security it will get greater adoption and perhaps lead them to realize there is a more-secure solution. Something is better than nothing which is what I want to achieve. Its not fool-proof, just better than curl | sh. Add automated hash checking and in the future signature checking.
I'm of two minds about this. On the one hand, taking a step back and looking at it from 30,000', I think it would be useful to have a thread along the lines of, "Hey, I solved problem A, with script X and here is how the code works." It would be educational and it would likely plant a seed for the reader, about how he/she may approach a problem of their own.
On the other hand, I'm not really interested in seeing examples of curl in action. It's just too dangerous in the hands of a novice. But like I said, I'm conflicted. Precisely because it is dangerous, perhaps we should bring curl out in the open and shine some light on it? Not so much to promote its use, but to explain why it is dangerous.
TL:DR
We could share examples of interesting and/or creative solutions as well as examples of epic fails and/or unexpected results, due to the careless use of powerful commands. Since the primary focus should be to educate, the comments should be cranked up to 11.
BarkingMad did you read the first post? I think perhaps the thread title and the latter comments have made this thread look like something it's not (e.g. a thread for posting/sharing bash scripts etc)
That's my point.
let's say, for the sake of argument, that the people posting curl pipe instructions want to use your software. It isn't rolled into any distro by default (like wget, curl and git are,) so they have to point people to a ppa or github page in addition to providing a command.
They're using curl | because it's easy. Adopting a new tool, telling everyone to install something that doesn't come standard on most systems, just to use a different one line command, is not appealing.
Arguably much less appealing than just changing to one of the four line commands I listed off the top of my head.
I agree, which is why one of the first things to do is get prog-exec (or whatever new name) into packages for the various popular distributions. Before that we can just provide downloadable packages and a PPA. If its as easy as
sudo apt-get install prog-exec -y
prog-exec 31/3 "someHash"thats even easier than it is to add a ppa. I think its just a matter of adoption/popularity. If its as easy as above and other people say just use prog-exec to run the script like they do with curl, people will install/run it.
Fair enough. Agree to disagree I guess.
I'm not just trying to shoot you down, I think it's useful to have a critical perspective with any endeavor. I hope I've provided some semblance of that in the points I brought up.
Well said sir, whilst I think we both agree that packages is the real solution. I could do with making a tutorial on turning a BASH script/tool into a package in a PPA, but I got to the point of having created a deb and failed to get further from there. Perhaps I'm stupid but I found it bloody hard, probably because it uses a completely different toolset. I have never had to use a makefile or bazaar etc.
Your perspective is useful as that is exactly what the project will need to address to persuade users to adopt it. There is no point "preaching to the choir" as its said.
I just want to chime in here and say that yes this is a plausible solution to an increasing problem, it is also a case of trying to solve human behavior with a technical solution.
The problem being that the problematic practice in question curl | bash is still possible(allowed) and easier to do than:
- Finding out that there is a safer way
- finding prog-exec
- installing prog-exec
- using prog-exec
- having everyone update their guides to use prog-exec + new hashed scripts
Usually the people affected by this are new linux users and not the knowledgeable types who know better.
New linux users will anyway follow any random guides online not understanding what the code they copy-paste does. (Hence curl | bash) being a lesser concern in comparison.
I could upload a piece of malware with a valid hash and all that says exactly what it is and people would still not care to check what the script does and execute it.
The problem thus far really being around the problem of uninformed 'guide following', whereby new users blindly follow instruction along the line of ("if you know nothing you have to believe everything").
Causes of this problem thus being:
- Lack of official documentation and best practices leads to users using third party guides and scripts
- New Users are easily misguided and will blindly follow instruction not knowing better
- Copy pasting instructions without knowing better is bad. - But using safe checked packages from apt/pacman/dnf etc is good.
-
curl | bashis dumb.
If we solved the case of curl | bash from the bottom up we would still be left with the other three problems that negate the efforts of the 4th.
Ideally any package a user might want needs to be easily installable via a given distro's package manager and not using third party methods. So at best prog-exec is a bottom up approach patch for bad human behavior that simply adds a pipeline that less-knowledgeable users (who are most affected by this) will easily ignore by simply pressing y to continue since they do not understand the content anyway. This is similar to the Confirm to continue dialog with bad SSL certificates, uninformed users simply got good at ignoring sound advice to get what they want.
To take this another layer deeper, the problem also isn't really with the issue that a script on a legitimate website might get modified to be malicious. Because as a bad actor if I had deep enough access to change your website's html content (either direct or mitm) then I have enough access to change the script url + hash to one of my choosing too.
Now that is not to say that prog-exec is a bad idea, it's great for example for checking the integrity of script, but it is by no means a security solution.
TLDR: Hashes are not Signatures. If I can mod your script I can recalc the hash and post my own link to it. However If you can use PGP to sign a script we can prevent a malicious user from creating a valid signed version of the script at the same URL since said bad actor likely doesn't have access to the private keys used to sign the script. This however still only solves the case of a script being hosted separately to the website with the link to the script. If Both the website with link and script site are compromised or the script and website are on the same site that are compromised a malicious actor can still easily replace the entire instructions to use an entirely different install source without signature checking.
So TLSDR: We really need a way of easily distributing signed & checked scripts. The problem being how to establish a web of trust for script signatures. Which is very hard to implement without significant effort and active human 'checking' since machine automation of checking scripts is not easily done.
Something like Arch's PKGBUILD /AUR combined with a web of trust for certain keys could be a good approach. Currently AUR still requires users to manually add trusted keys.
1 Like
Great post, I just wish to address some points.
I could upload a piece of malware with a valid hash and all that says exactly what it is and people would still not care to check what the script does and execute it.
This will always be a problem, even with a PPA or a manually downloaded deb.
To take this another layer deeper, the problem also isn't really with the issue that a script on a legitimate website might get modified to be malicious. Because as a bad actor if I had deep enough access to change your website's html content (either direct or mitm) then I have enough access to change the script url + hash to one of my choosing too.
Let's say the script server did get hacked, and the malicious user edited the script and thus generated a new hash, this does not change the fact that the hash would not match the hash that was on the author's site such as digitalocean. Don't forget the author can always take the sha256 of their script and check it matches up with what the script server says the hash is and raise the alarm if the script server provides an incorrect prog-exec command hash. To overcome this, the malicous user would have to have access to both the scriptserver and the post where the author posts the command, such as digitalocean.
TLDR: Hashes are not Signatures.
Agreed, atm there is only hash checking, in future there will be signature checking, which would require authors to provide the signed version of their script, and ideally the prog-exec client pulls the public key from key-servers when adding to their trusted list. Private keys are not kept on the script server, that would make it pointless.
As mentioned before, there is no getting around malicious users uploading malicious scripts, but users can easily say they trust webupd8team, programster, and ubuntu and not execute scripts from any other authors.
So TLSDR: We really need a way of easily distributing signed & checked scripts. The problem being how to establish a web of trust for script signatures. Which is very hard to implement without significant effort and active human 'checking' since machine automation of checking scripts is not easily done.
Something like Arch's PKGBUILD /AUR combined with a web of trust for certain keys could be a good approach. Currently AUR still requires users to manually add trusted keys.
I take it AUR is arch specific? What you just described sounds exactly like what I'm trying to create here except that there are no automatically trusted keys and it is still up to users to add the keys they trust. This just isn't distro specific.
This is pretty much the ideal scenario for this at the moment, if larger trusted sites can provide trusted keys for scripts much of the threat can already be reduced.
One then only has to push to have author awareness to encourage widespread use the new technique and to widely discourage the older curl | bash technique since many will still use it out of simplicity.
So ultimately the goal has to be to make it easier to use prog-exec or whatever similar thing (maybe because it comes pre installed like you alluded to) than to use curl.
Yes AUR (Arch User Repository) is Arch and arch derivative specific. They really just need to follow the PKGBUILD way and some arbitrary dependencies. But aside from that the same code could work the same on any distro really.
1 Like
Yes i think the first step is just to get prog-exec into packages for the repositories so people can do yum install prog-exec or sudo apt-get install prog-exec, but we should wait until it has been updated to add signature checking.
Having widely trusted sites co-sign scripts would be great, and something people could perhaps pay for. E.g. i users trust canonical, so i pay canonical to review and sign my script and they have to check it before signing it otherwiser risk losing their reputation. That way users are likely to trust and run the script I authored as they don't trust me.
1 Like
Actually something like ast/scriptman install could be a good name.
AST = Advanced Script Tool (After APT Advanced Package Tool)
scriptman = script manager (After arch pacman (Package Manager))
1 Like
Well put; I did the whole devil's advocate thing because it's more engaging.
I'm confused, are you saying there is an existing tool called ast/scriptman that does what I'm talking about? because if it does I'll be happy to just look into and adopt that.