> We cannot issue an IPv4 address to each machine without blowing out the cost of the subscription. We cannot use IPv6-only as that means some of the internet cannot reach the VM over the web. That means we have to share IPv4 addresses between VMs.
Give a user a option for use IPv6 only, and if the user need legacy IP add it as a additional cost and move on.
Trying to keep v4 at the same cost level as v6 is not a thing we can solve. If it was we wouldn't need v6.
IPv6 does not work on the only ISP in my neighborhood that provides gigabit links. I will not build a product I cannot use.
Even when IPv6 is rolled out, it is only tested for consumer links by Happy Eyeballs. Links between DCs are entirely IPv4 even when dual stacked. We just discovered 20 of our machines in an LAX DC have broken IPv6 (because we tried to use Tailscale to move data to them, which defaults to happy eyeballs). Apparently the upstream switch configuration has been broken for months for hundreds of machines and we are the first to notice.
I am a big believer in: first make it work. On the internet today, you first make it work with IPv4. Then you have the luxury of playing with IPv6.
Whenever I see a comment that says "if you don't do the thing in the most efficient way possible, someone else will steal your lunch", I think that people vastly overestimate the likelihood that this will actually happen.
It's similar to "open source is the most secure because it has the most eyeballs on it", but in reality security bugs will exist for years with no one noticing because people vastly overestimate how any developers will actually spend their time analyzing any given open source software.
Sure, bugs are more likely to be caught in open source and it's more likely someone will take your market share with a more efficient and competitively priced product, but you're overblowing the likelihood of both by a large margin.
> "if you don't do the thing in the most efficient way possible, someone else will steal your lunch"
Well you’re leaving behind both a serious pain point for your users AND you’re leaving in the open a clearly more compute- and money-efficient way to achieve the objective on the table.
It’s literally giving your eventual competitors (because there will be competitors, eventually) a competitive advantage.
Then sure, the market is very wide but… just look at stackoverflow vs chatgpt. As soon as a better alternative came on the market, stackoverflow died to irrelevance within months.
Have you looked at each service running through a cloudflare tunnel or (HE offers something similar too)?
(PS: I use exe.dev quite a lot whenever I want to have a project and basic scripting doesn't work and I want to have a full environment, really thanks for having this product I really appreciate it as someone who has been using it since day one and have recommended/talked about your service in well regards to people :>)
You can get this effect today by installing Tailscale on your exe.dev VM. :)
The reason we put so much effort into exposing these publicly is for sharing with a heterogeneous team without imposing a client agent requirement. The web interface should be easy to make public, easy to share with friends with a Google Docs-style link, and ssh should be easy to share with teammates.
That said, nothing wrong with installing tunneling software on the VM, I do it!
This is great if you have IPv6 support from your ISP. Not so great if you don't.
Before someone mentions tunnels: Last time I tried to set up a tunnel Happy Eyeballs didn't work for me at all; almost everything went through the tunnel anyway and I had to deal with non-residential IP space issues and way too much traffic.
ISPs won't bother with IPv6 until they've either run out of IPv4 space or the internet starts to use IPv6's advantages.
Discussions about IPv6 quickly end with "we have enough v4 space and there are no services that require v6 anyway". As long as the extra cruft for v4 support remains free or even supported, large ISPs won't care. We're at the point where people need to deal with things like peer to peer connectivity with two sides behind CGNAT which require dedicated effort to even work.
I know it sucks if none of the ISPs in your area support IPv6 and you're left with suboptimal solutions like tunnels from HE, but I think it's only reasonable all this extra cost or effort becomes visible at some point. Half the world is on v6, legacy v4-only connections are becoming the minority now.
Are there really ISPs that don't support IPv6? I've had IPv6 from various ISPs since around 2010, and even my phone gets an IPv6 address from the cellular network.
I complained as a yearly tradition for couple of years to get v6 enabled in my ISP. They had the core network enabled on World IPv6 Launch in 2012, but not deployed to end customers.
One simple way to check if your ISP have some kind of IPv6 netowork is to see if CDN domains given by YouTube and Facebook have AAAA records.
We shouldn't have to ask for ISPs to add IPv6 support but here we are.
They could have done that in addition (and maybe they do), but for some of their customers it then may not work, for reasons hard to understand as a customer. Especially when changing locations frequently it may sometimes work and sometimes not ... not good for keeping customers
SSH waits for the server key before it presents the client keys, right? Does this mean that different VMs from different users have the same key? (Or rather, all VMs have the same key? A quick look shows s00{1,2,3}.exe.xyz all having the same key.) So this is full MitM?
2. Server side: use ForceCommand, either from within sshd_config or .ssh/authorized_keys, based on username or group, and forward the connection that way. I wrote a blogpost about this back in 2012 and I assume this still mostly works, but it probably has some escaping issues that need to be addressed: https://blog.melnib.one/2012/06/12/ssh-gateway-shenanigans/
The workaround I use for my own stuff is to have a single jump-host that listens on the public IPv4 address and from there connect to the others. I can still just ssh username@namedhost (which could be username@www.websitehostedonthevm.tld, though I usually give short aliases in .ssh/config) without extra command-line options with the on-time config of adding a host entry in .ssh/config listing the required jump host and internal IP address. Connecting this way (rather than alternatives like manual multi-hop) means all my private keys stay local rather than needing to be on the jump host, without needing to muck around with a key agent.
I even do this despite having a small range of routable IPv4s pointing at home, so I don't really need to most of the time. And as an obscurity measure the jump/bastion host can only be contacted by certain external hosts too, though this does still leave my laptop as a potential single point of security failure (and of course adds latency) and one or any bot trying to get in needs to jump through a few hoops to do so.
169 comments
> We cannot issue an IPv4 address to each machine without blowing out the cost of the subscription. We cannot use IPv6-only as that means some of the internet cannot reach the VM over the web. That means we have to share IPv4 addresses between VMs.
Give a user a option for use IPv6 only, and if the user need legacy IP add it as a additional cost and move on.
Trying to keep v4 at the same cost level as v6 is not a thing we can solve. If it was we wouldn't need v6.
IPv6 does not work on the only ISP in my neighborhood that provides gigabit links. I will not build a product I cannot use.
Even when IPv6 is rolled out, it is only tested for consumer links by Happy Eyeballs. Links between DCs are entirely IPv4 even when dual stacked. We just discovered 20 of our machines in an LAX DC have broken IPv6 (because we tried to use Tailscale to move data to them, which defaults to happy eyeballs). Apparently the upstream switch configuration has been broken for months for hundreds of machines and we are the first to notice.
I am a big believer in: first make it work. On the internet today, you first make it work with IPv4. Then you have the luxury of playing with IPv6.
> IPv6 does not work on the only ISP in my neighborhood that provides gigabit links. I will not build a product I cannot use.
Cool.
Somebody else will, and will likely have a better price (due to the abundance of ipv6 addresses) and you’ll go out of business.
> because we tried to use Tailscale to move data to them, which defaults to happy eyeballs
Not gonna lie, to me that reads like “because we don’t know how to use ipv6”
It's similar to "open source is the most secure because it has the most eyeballs on it", but in reality security bugs will exist for years with no one noticing because people vastly overestimate how any developers will actually spend their time analyzing any given open source software.
Sure, bugs are more likely to be caught in open source and it's more likely someone will take your market share with a more efficient and competitively priced product, but you're overblowing the likelihood of both by a large margin.
> "if you don't do the thing in the most efficient way possible, someone else will steal your lunch"
Well you’re leaving behind both a serious pain point for your users AND you’re leaving in the open a clearly more compute- and money-efficient way to achieve the objective on the table.
It’s literally giving your eventual competitors (because there will be competitors, eventually) a competitive advantage.
Then sure, the market is very wide but… just look at stackoverflow vs chatgpt. As soon as a better alternative came on the market, stackoverflow died to irrelevance within months.
(PS: I use exe.dev quite a lot whenever I want to have a project and basic scripting doesn't work and I want to have a full environment, really thanks for having this product I really appreciate it as someone who has been using it since day one and have recommended/talked about your service in well regards to people :>)
The reason we put so much effort into exposing these publicly is for sharing with a heterogeneous team without imposing a client agent requirement. The web interface should be easy to make public, easy to share with friends with a Google Docs-style link, and ssh should be easy to share with teammates.
That said, nothing wrong with installing tunneling software on the VM, I do it!
Before someone mentions tunnels: Last time I tried to set up a tunnel Happy Eyeballs didn't work for me at all; almost everything went through the tunnel anyway and I had to deal with non-residential IP space issues and way too much traffic.
Discussions about IPv6 quickly end with "we have enough v4 space and there are no services that require v6 anyway". As long as the extra cruft for v4 support remains free or even supported, large ISPs won't care. We're at the point where people need to deal with things like peer to peer connectivity with two sides behind CGNAT which require dedicated effort to even work.
I know it sucks if none of the ISPs in your area support IPv6 and you're left with suboptimal solutions like tunnels from HE, but I think it's only reasonable all this extra cost or effort becomes visible at some point. Half the world is on v6, legacy v4-only connections are becoming the minority now.
It is also available for one of my phone contracts but not tried enabling it yet.
One simple way to check if your ISP have some kind of IPv6 netowork is to see if CDN domains given by YouTube and Facebook have AAAA records.
We shouldn't have to ask for ISPs to add IPv6 support but here we are.
>legacy IP
lol
So far it feels like only LDAP really makes use of it, at least with the tech I interact with
1. Client side: ProxyJump, by far the easiest
2. Server side: use ForceCommand, either from within sshd_config or .ssh/authorized_keys, based on username or group, and forward the connection that way. I wrote a blogpost about this back in 2012 and I assume this still mostly works, but it probably has some escaping issues that need to be addressed: https://blog.melnib.one/2012/06/12/ssh-gateway-shenanigans/
I even do this despite having a small range of routable IPv4s pointing at home, so I don't really need to most of the time. And as an obscurity measure the jump/bastion host can only be contacted by certain external hosts too, though this does still leave my laptop as a potential single point of security failure (and of course adds latency) and one or any bot trying to get in needs to jump through a few hoops to do so.
Not exactly what i built in for, but it'll do the job here too, and able to connect to private addresses on the server side.