Add HTTPS and basic authentication support to elasticsearch #664
Comments
|
I second this. I have legal requirements to protect private patient information in transit and at rest. Native HTTPS, or SSL support would help solve a good piece of this problem. |
|
+1 - an encrypted transport is required at my client site. |
|
Related to this would be locking down ES node to node communication over SSL as well. |
|
Yep - I'd like both an HTTPS entry point and SSL inter-node comms. |
|
I agree +1 for me too. |
|
+1000! |
|
I don't think there's a real need for supporting this in ElasticSearch itself? Most installations where this is neccessary could easily use a reverse proxy such as Nginx? |
|
I think data is very sensitive and shouldn't rely on validation from third parties. The data is in ES, ES should do the auth since it knows it's own data structure the most. |
|
The simplicity of native encrypted transport is highly attractive. The complexity and fragility of wrapping all data in transit through an SSL proxy would be a barrier. |
|
SSL at least ... I mean it's an http transport layer not some abstract peice of code. |
|
Well since it seems people want this but dons't sound like its something you want to build in directly ... maybe a plugin would be good? I looked at Nginx and while it looks supper light weight and pretty clean it would add another virtual hop and, when you already have the extra weight of SSL it would be nice to have the ability to access ES directly with the least amount of extra overhead. I wonder if http://www.openssl.org/ would be helpful in integrating. |
|
i guess a simple tutorial on how to set up nginx would suffice. nginx is not that hard to setup. |
|
I was wondering if anyone was working on this issue, if not I would like to take a shot. |
|
I don't believe so (and I think Shay has a lot of other things going on smiles). I'd contribute, but I'm a .NET programmer by trade, and don't have the skills in Java to make an effective contribution on this issue. If you're going to take a shot, note, that they really have to go hand-in-hand; HTTPS is nice, but without some kind of authentication, you don't gain much for it. Additionally, you'll have to come up with a way to specify the credentials (username/password) of the users that can access the cluster. |
|
I don't have any use-case for HTTPS (i use Nginx as SSL offloader to great effect), but needed HTTP Basic. You can find my plugin here: https://github.com/Asquera/elasticsearch-http-basic My two cents on SSL: Supporting SSL does not only mean implementing all the nuts and bolts (certificate management, etc.), it also has to be efficient and safe. There are other projects (stunnel, nginx, your favourite hardware load balancer) that are much better at doing all that. If you want your elasticsearch to speak ssl externally without configuring nginx - bind it to a local port and put stunnel in front of it. This is a common and tested solution. |
|
I voted for this ticket because I need ES nodes to be able to internally On Wed, Nov 23, 2011 at 3:55 AM, Florian Gilcher <
|
|
@skade: That isn't always feasible; especially when you are on a cloud infrastructure. As an example, Azure, which I've been able to get it to run on, won't let me install any of those. It might be common and tested, but it's not always applicable. |
|
@skade Your suggestion for using @BenHall I'd be interested in lending a hand on the issue as well I would see it as touching two core components, the |
|
@prb: That's interesting -- could you elaborate why an Nginx-based proxy (for example) isn't enough? (I don't understand the nginx isn't going to deal with cluster communications part. You can restrict the in-cluster communication based on IPs.) |
|
Elasticsearch uses the network for both external communications (e.g., over HTTP via the |
|
@prb: You can disable |
|
+1 This is a major deficiency from a government security requirements perspective, and may be a blocker for our project moving from Solr to ElasticSearch, since we have our Solr shards locked down via tomcat. |
|
@asanderson you can deploy elasticsearch as a war file within a wen container if you want using the wares plugin (check the transport-wares repo). Questions in the mailing list. |
|
@kimchy Excellent! Other than performance, are there any other disadvantages to deploying via war? |
|
@asanderson not really, same API, quite simply wrapping as well. If you are up to it, would love to get async support to the servlet based on the servlet 3.0 async feature :) |
|
@kimchy good to know. One of these days, I'd love to contribute, but right now I don't have the bandwidth. Sorry, but the best I can do is to continue to evangelize ElasticSearch. ;-) FWIW, I've spent the last 5+ years replacing expensive COTS products with Solr, and I must say that ElasticSearch out-of-the-box seems to address all of Solr's shortcomings, IMHO. Now, I get to spend the next year or so replacing Solr with ElasticSearch. ;-) |
|
We just released Jetty plugin for elasticsearch. It is a drop-in HTTP transport replacement that exposes full power of embedded Jetty including support for SSL, logging, authentication and access control. It is similar to transport-wares plugin that Shay mentioned above, but instead of running elasticsearch inside a web server, it embeds Jetty web server into elasticsearch. See https://github.com/sonian/elasticsearch-jetty for more information. |
|
Outstanding! Great job! It just keeps getting better and better. ;-) |
|
So I can set up a reverse proxy or use the Jetty plugin to secure access to elasticsearch, but now I can no longer use the Java API... |
|
Or a user running inside an authenticated environment can connect to ES on port 9300 un-authenticated and do what they want. |
|
Assuming you set up firewall rules to only allow specific sets of ip addresses to connect to the ES ports, how hard would it be for someone to spoof one of those ip addresses? As in, an attacker gets in, notices that you use logstash, and then spoofs the ip address of a server that can connect to ES in order to delete logs from that day, or maybe figure out what logs relate to the attack, and only delete those logs. As I think about it, I really don't like the idea of using the firewall to limit what servers can connect to the ES cluster. You'd have to update the rules on every ES server every time you added a node. Or open it up more than is safe... I'd still like to see a detailed explanation of why ES doesn't include user authentication, or ssl authentication between nodes. Relying on a web server like nginx to authenticate means you have to run the web server, not really something I'd like to do when I'm trying to minimize attack surfaces. Relying on a third party plugin means I'm out of luck if that third party stops supporting it. Plus, well, every SQL database I know about offers user authentication. Why should ES do the same? What aspect of ES's (or NoSQL DBs in general) infrastructure eliminates the need for it? |
|
We have ES installed on Windows and in our case, we do not have the option of using Linux/nginx whatsoever. Nginx support for Windows is terrible (connection/worker limits) so that is off the plate. The few plugins I have looked at which add some type of auth/ssl have issues mentioned by others in this thread. It would be ideal for those of us that are unable to run nginx and not willing to use the plugins, for whatever reason, if ES offered native support for auth/ssl as part of the core package. |
|
@blakeja You can use stunnel it works on Windows also. But I must admit it's a B I T C H to setup. I mean stunnel itself is fairly straight forward. But getting ES to play nice with stunnel requires a bit of thinking and knowing very well how ES communicates between all the nodes. Plus MOST the solutions proposed above don't really scale (management wise) well in large clusters anyways. But good news the security stack has been announced: http://www.elasticsearch.com/products/shield/ :) |
|
Shield has been released - closing |
|
Is Shield released in the open source project? |
|
No, but there is an upcoming project which seems to do similar things to Shield. https://github.com/salyh/elastic-defender not sure what state of development it's in though... |
|
No actual code in that project... As for Shield, I really wish they had released it as part of ES. I get that they need to be making money, but the lack of authentication is a glaring hole in the feature set. Nearly every other database product has authentication, ES at least needs built in users per index, or something like that. Maybe, user A has access to index logstash-, user B has access to index applicationA? At least IMO. |
|
Yeah of course, it makes sense. That project is scheduled for release On 10/02/15 19:54, David Reagan wrote:
|
|
Most "nosql" solutions do not have security measures built in. Hadoop did Considering no third party tools have emerged yet for an open source Ivan
|
|
There are a lot of folks out there who assume every piece of software has some measures of security built in and works even in "hostile" environments, like cloud, Amazon etc. The challenge is to keep the search engine code in ES as simple as possible without getting overloaded by "security concerns". Note there is no "security" software that can meet 100% expectations of all users. Shield has also some weakness in setup, which is no surprise because it is still up to the admin skills to set up a "safe enough" environment. See also MongoDB security flaw which exposed thousands of databases to the public internet because admins were not skilled enough: http://www.mongodb.com/blog/post/mongodb-security-best-practices Jörg |
|
To preface this, the ELK stack is open source, and free. I deeply appreciate the work the various contributers have put into it. I'm glad the ES company exists and is making money so they can pay many of the contributers. Heck, I've contacted sales asking about how much support would cost my workplace in the hope we could afford it. (Though, I never got a reply...) I say this because my comments below could come across as somewhat demanding or entitled. I tried to word things as well as I could, but I'm not sure how well I did. If I were a java developer, and I had the time, I'd be trying to create a pull request to add the security support I want. As it is, I'm a sysadmin, and what developement work I do is usually in PHP. So... So far, the only reason I've heard for not adding security from the start, is that it adds extra work to the ES teams lives. It's not like they think it isn't needed/wanted. If that was the case, Shield would not exist. If it's more that just the extra work, I'd love to read a post that goes into details. If it is just the work, then I'd love a detailed explaination of how that's a valid excuse. From my perspective, it's just plain common sense to add, at least, basic auth to the http api. Not having that, just doesn't make any sense to me. Why using third party plugins or a proxy server isn't a good idea has already been addressed in this thread. But, to summarize my thoughts: It complicates deployment, and makes keeping things secure harder. You have to figure out if third party plugins are secure, if they stop being developed, you have to replace them. Adding a proxy server is yet another thing using resources, and yet another thing to configure and keep up to date. Basically, on an ideal ES node, the only thing other than basic OS/Logging/monitoring stuff running should be ES. |
@jerrac - Have you already looked at service providers like https://www.found.no/ who've been offering the security layer for ES for quite a long time now? Do they not meet your needs? |
|
@pulkitsinghal We need the data to be stored in our data center, and our budget is really really tight. So, a third party host isn't something we can consider. |
|
Gosh I'll sound like an arse now ... when you say:
Aren't budget constraints and time constraints the same thing? Isn't it fair for them to hold-off like you do? As for your on-premise piece, you should email Found, they may be willing to do an on-premise install or something for you depending on the price. I'll try to think of more constructive approaches as well and reach out as ideas come. You cam IM me in gitter.im too! |
I recently put a lot of work into closing a security hole on a service we're trying to sunset. It certainly would have been easier to just leave it alone, but it would have left users at risk. That isn't something that is acceptable. So, I am willing to believe that the ES team has reasons for why they didn't implement any security. I just want to know those reasons. 'Cause I can't think of any good ones... |
|
I also want elasticsearch.com to make money and stay in business, and think support and plugins like marvel make great sense to offer for a fee, but having to pay for basic security just seems wrong and broken. And when you hit things like http://www.elasticsearch.org/blog/scripting-security/ you start to wonder. Yes you shouldn't run ES "on the internet" but if someone is "in your network" and scanning that advice doesn't matter. (More common then you might think.) Maybe splitting Shield into a Free vs Pay might help?
Pay could add
The other issue is you can't buy Shield without a support subscription right now. Those of us writing/using open source software can't afford that. I guess we are all waiting for elastic-defender :) |
|
@jerrac in the beginning, ES started just as a toolbox with kimchy as the sole developer, it was not product ready. Read kimchy's comments from late 2010 at http://elasticsearch-users.115913.n3.nabble.com/Document-level-security-and-Connectors-td1818688.html then you understand that security feature are orthogonal to search features and could have been easily added by external software add-ons (as it is now, you can still protect ES today without Shield). It was a matter of priority for completing the core features first. |
|
Heya fellows, let me try and share my thoughts around this: I have always claimed that "just adding basic auth" to ES is not enough, I deeply feel that when it comes to security, you need to have "rounded" features that are defined by what you are trying to achieve, and then how its implemented. Specifically, basic auth as a generic feature in ES itself (and not implemented through a proxy, which is quite simple, check our nginx blog about how to get it configured quickly), means having the ability to authenticate. But that spans more than just HTTP, its also for things like transport client, and potentially node to node authentication too. Sometimes "just" http is good enough, not as a generic function in core ES (where it wouldn't be enough) , but provided by a proxy. The full scale of security was implemented in Shield, its was easier to just take the whole set and implement it in a single package. Note, we went back to ES open source and made sure Shield can be implemented as a plugin to add all the security features, its not a closed source redistribution of ES. This is super important to me personally and generally in work we do, I think that the ability to add it as an extension to open source is very important. In the future, as the usage of ES increases and expands more, and as our usage grows, I definitely can see a case where some "rounded" security features end up being contributed back to the open source. Its a whole different set of work in terms of engineering (compared to building it in a single package). The authentication example in this case definitely falls into this bucket, but then if it happens, its scope will, and must be, more than "just" http basic auth. Just a note of the amount of effort we put into open source, trying to give it some color, as I saw its mentioned, today we have close to 80 developers working at the company, almost all of them focused on projects like Lucene (we have 8 Lucene committers and a lot of work goes into Lucene every day), Elasticsearch, Kibana, Logstash, language clients (perl, python, .NET, ruby, ...), Hadoop and more. |
|
@kimchy So, you want to do security the right way, if you're going to do it at all. And the "right" way involves a LOT of work. I can understand that. I've always tried to do things "right" the first time, even if that takes more initial work. I guess where I disagree with you is that I view security as a feature just as important as every other feature in ES. Even if it doesn't directly relate to what ES is supposed to do. To (hopefully) exagerate, if my data isn't secure and my users are having their identity stolen, or my servers are getting profiled for future hacking, or a disgruntled employee is wiping my logs, then I don't really care how well search works. Here's an imaginary situation, I have no idea how viable this attack would actually be... An attacker owns a server in the same dmz as ES, and are able to see that ES exists (nmap scan). They then add a node to the cluster that copies all the data out of it, and sends it to their servers. Or they just wipe it. Setting some kind of basic auth between nodes, and on the http api, could help prevent (or at least reduce the damage done) that from happening. Since they'd then have to actually own an ES node, or a server that stores the auth info, to do anything. Compare that to mysql. They know mysql exists, but they can only access what databases they have credentials for. Or they'd have to own the entire mysql server. With ES, it's just, "oh, hey, I can connect to the ES api, let's delete everything." Some form of auth wouldn't fix all the security issues, but it sure would help. Anyway, you've answered my questions about why auth isn't part of ES by default. If nothing I've said makes a good point to you, I'm willing to just leave it here. (Side note: Heh, I was just imagining how painful it would be if I had to build a database server for every single mysql based app I manage... If they were based on ES, I'd have to have 20+ ES clusters running, or some kind of complicated custom layer between the ES api and the apps...) |
|
Hey everybody, I've been following this thread for quite some time; and I can fully understand "both sides" of the equation. I don't really have any facts to add, but I think it's really really awesome that you, @kimchy, joined the discussion as the project lead -- even though you have a big company to run and I bet lots of other tasks. Besides that, I really like the constructive tone in the whole thread... For me that just resembles how great of a community / product ElasticSearch is. Thanks everybody 👍 Sebastian |
|
the plugin that just do that(ssl on both http and transport) and nothing more: |
|
Nice job! |
|
There is also https://github.com/floragunncom/search-guard-ssl/ Still hoping Elastic starts including basic security in the free version! Maybe they will announce that at ElasticON 2017! :) |
|
+1 |
|
@chrisdlangton Have a look at Elastic Cloud, which runs on AWS, but comes with X-Pack features enabled by default: https://www.elastic.co/cloud |
This would help with those that are hosting elasticsearch and need security when making calls from another machine which cannot be used with the TCP transport.
HTTPS support should allow for configuration of the SSL certificate from a path, along with any other easily discoverable certificate sources.
The text was updated successfully, but these errors were encountered: