Elastic enables customers to cost-effectively store and search data for deeper insights with no constraints

Elastic announced the beta of searchable snapshots, a new capability that makes it possible to cost-effectively store and search more data to drive critical business decisions, enhance revenue, and reduce costs.

Searchable snapshots provide a tiered approach to searching across data that is saved in different classes of storage, addressing concerns including operational complexity, reduced system resiliency, and query limits.

Elastic customers will be able to minimize costs with low-cost object stores such as Amazon S3, Azure Storage, and Google Cloud Storage. Initially, Elastic is supporting a new lower-cost cold tier of storage, which offloads redundant copies of data to the object stores to drive savings.

With searchable snapshots and the launch of a new cold storage tier, customers will benefit from:

  • Lower storage costs – Data can be moved from more expensive, high-performance storage to far lower-cost data tiers while still remaining searchable, which frees up more performant storage for higher-value data.
  • Streamlined operations – Formalized data tier definitions with built-in data transition rules and integrated index lifecycle management make it easier for customers to get up and running with their data storage policies and manage the full lifecycle of their data automatically.

In a future release, Elastic customers will be able to leverage a frozen tier of storage, where all data can be kept in low-cost object stores. This will unlock the opportunity to drive insights from virtually unlimited lookbacks, with low cost data retention on years of data.

Elastic users will be able to ingest new types of data that they previously may not have considered, driving new innovative projects and use cases.

“Searchable S3 snapshots are going to be a game changer as we use our IT and data collection expertise to address the most urgent human security issues faced by the world’s vulnerable populations,” said Madison Bahmer, CTO, IST Research.

“We’ve typically only kept data in high performance storage for 6 to 12 months at a time; this functionality allows us to search across a much longer period, enabling us to better understand hard-to-reach communities and provide insights about effective communication strategies with community members.”

“As data expands exponentially over time, the cost of storing that data grows to the point where customers are forced to choose between deleting their data or managing increased costs,” said Steve Kearns, vice president, product management, Elastic.

“Searchable snapshots give our customers complete control over optimizing for cost, performance, and depth of data to gain deeper insights, enhance revenue, and drive innovation.”

With database attacks on the rise, how can companies protect themselves?

Misconfigured or unsecured databases exposed on the open web are a fact of life. We hear about some of them because security researchers tell us how they discovered them, pinpointed their owners and alerted them, but many others are found by attackers first.

exposed databases

It used to take months to scan the Internet looking for open systems, but attackers now have access to free and easy-to-use scanning tools that can find them in less than an hour.

“As one honeypot experiment showed, open databases are targeted hundreds of times within a few hours,” Josh Bressers, product security lead at Elastic, told Help Net Security.

“There’s no way to leave unsecured data online without opening the data up to attack. This is why it’s crucial to always enable security and authentication features when setting up databases, so that your organization avoids this risk altogether.”

What do attackers do with exposed databases?

Bressers has been involved in the security of products and projects – especially open-source – for a very long time. In the past two decades, he created the product security division at Progeny Linux Systems and worked as a manager of the Red Hat product security team and headed the security strategy in Red Hat’s Platform Business Unit.

He now manages bug bounties, penetration testing and security vulnerability programs for Elastic’s products, as well as the company’s efforts to improve application security, add new and improve existing security features as needed or requested by customers.

The problem with exposed Elasticsearch (MariaDB, MongoDB, etc.) databases, he says, is that they are often left unsecured by developers by mistake and companies don’t discover the exposure quickly.

“The scanning tools do most of the work, so it’s up to the attacker to decide if the database has any data worth stealing,” he noted, and pointed out that this isn’t hacking, exactly – it’s mining of open services.

Attackers can quickly exfiltrate the accessible data, hold it for ransom, sell it to the highest bidder, modify it or simply delete it all.

“Sometimes there’s no clear advantage or motive. For example, this summer saw a string of cyberattacks called the Meow Bot attacks that have affected at least 25,000 databases so far. The attacker replaced the contents of every afflicted database with the word ‘meow’ but has not been identified or revealed anything behind the purpose of the attack,” he explained.

Advice for organizations that use clustered databases

Open-source database platforms such as Elasticsearch have built-in security to prevent attacks of this nature, but developers often disable those features in haste or due to a lack of understanding that their actions can put customer data at risk, Bressers says.

“The most important thing to keep in mind when trying to secure data is having a clear understanding of what you are securing and what it means to your organization. How sensitive is the data? What level of security needs to be applied? Who should have access?” he explained.

“Sometimes working with a partner who is an expert at running a modern database is a more secure alternative than doing it yourself. Sometimes it’s not. Modern data management is a new problem for many organizations; make sure your people understand the opportunities and challenges. And most importantly, make sure they have the tools and training.”

Secondly, he says, companies should set up external scanning systems that continuously check for exposed databases.

“These may be the same tools used by attackers, but they immediately notify security teams when a developer has mistakenly left sensitive data unlocked. For example, a free scanner is available from Shadowserver.”

Elastic offers information and documentation on how to enable the security features of Elasticsearch databases and prevent exposure, he adds and points out that security is enabled by default in their Elasticsearch Service on Elastic Cloud and cannot be disabled.

Defense in depth

No organization will ever be 100% safe, but steps can be taken to decrease a company’s attack surface. “Defense in depth” is the name of the game, Bressers says, and in this case, it should include the following security layers:

  • Discovery of data exposure (using the previously mentioned external scanning systems)
  • Strong authentication (SSO or usernames/passwords)
  • Prioritization of data access (e.g., HR may only need access to employee information and the accounting department may only need access to budget and tax data)
  • Deployment of monitoring infrastructures and automated solutions that can quickly identify potential problems before they become emergencies, isolate infected databases, and flag to support and IT teams for next steps

He also advises organizations that don’t have the internal expertise to set security configurations and managing a clustered database to hire of service providers that can handle data management and have a strong security portfolio, and to always have a mitigation plan in place and rehearse it with their IT and security teams so that when something does happen, they can execute a swift and intentional response.

Paul Appleby joins Elastic as president, worldwide field operations

Elastic announced the appointment of Paul Appleby as president, worldwide field operations. Appleby was most recently the chief executive officer of Kinetica, and will be responsible for enhancing the customer journey, driving global revenue growth, and developing strategies for addressing the large market opportunity for Elastic.

Appleby will report to Elastic founder and Chief Executive Officer Shay Banon.

Appleby joins Elastic as the company continues to see increasing demand for its enterprise search, observability and security solutions that are built on a single technology stack under a unified pricing model. Elastic’s unified, resource-based pricing enables customers to predictably control costs and fuel rapid adoption across its solutions.

Appleby brings more than 20 years of experience in senior management roles in the enterprise software industry to Elastic. He has a track record of driving significant scale and sustained growth across all channels, building and leading high-performing teams, leading go-to-market operations, and developing new markets.

Prior to Kinetica, Appleby served as president of worldwide sales and marketing of BMC. He also served in senior leadership roles at Salesforce, Siebel Systems, C3 AI, Travelex, and SAP.

“I’m pleased to welcome Paul to Elastic as president of worldwide field operations. He joins us as the demand for our solutions continues to grow, and his more than 20 years of experience in successfully building and leading global teams will help us further scale the company,” said Shay Banon, founder and chief executive officer, Elastic.

“Paul is an industry veteran and an experienced global leader who is uniquely qualified for this key leadership position at Elastic as we build on our free and open heritage and drive value for our customers and partners.”

“Elastic brings customers the speed, scale and simplicity they need to effectively explore and analyze their data using the power of search to drive business insights,” said Paul Appleby, president, worldwide field operations, Elastic.

“I’m excited by Elastic’s solutions and how they help to explore and analyze data differently using the power of search and the boundless ability of organizations who use Elastic to solve their most critical and complex challenges.”

New infosec products of the week: August 21, 2020

Kali Linux 2020.3 released: A new shell and a Bluetooth Arsenal for NetHunter

Offensive Security has released Kali Linux 2020.3, the latest iteration of the popular open source penetration testing platform. Kali’s mobile pentesting platform/app – has been augmented with Bluetooth Arsenal, which combines a set of Bluetooth tools in the app with pre-configured workflows and use cases.

infosec products August 2020

Elastic Security 7.9 delivers a major milestone toward endpoint security integrated into the Elastic Stack

This milestone includes malware prevention on Windows and macOS and advanced detections and deep visibility for all major operating systems. Elastic Security also now offers expanded out-of-the-box protections for cloud security monitoring, as well as a new integration with IBM Security’s SOAR platform, IBM Security Resilient.

infosec products August 2020

BrickStor Security Platform facilitates data security for distributed workforces and hybrid clouds

BrickStor Security Platform is for customers who need to securely protect unstructured data, both on premises and in the cloud, without the complexity or security vulnerabilities inherent to integrating existing legacy storage systems with loosely coupled third-party tools.

infosec products August 2020

NinjaRMM unveils Ninja Data Protection, a backup solution for MSPs and IT professionals

NinjaRMM released Ninja Data Protection, the company’s latest product offering for managed service providers (MSPs) and IT professionals. Ninja Data Protection promises to give NinjaRMM customers more immediate visibility into backup activity, history, and storage usage across their client bases, in addition to more reliable, granular, and streamlined control.

infosec products August 2020

Lacework platform now features Active Host Vulnerability Monitoring and CI/CD integrations

Lacework announced that in the midst of a period of rapid adoption by developers of born-in-the-cloud applications, it will introduce Active Host Vulnerability Monitoring, pre-flight checks, and CI/CD automation workflows to its comprehensive SaaS security offering.

infosec products August 2020

Elastic Security 7.9 delivers a major milestone toward endpoint security integrated into the Elastic Stack

Elastic announced the first major beta milestone for Elastic in delivering comprehensive endpoint security fully integrated into the Elastic Stack, centrally managed under a unified agent and enabled with one click.

Elastic Security 7.9

This milestone includes malware prevention on Windows and macOS and advanced detections and deep visibility for all major operating systems including Windows, macOS, and Linux, all provided under the free distribution tier.

Elastic Security also now offers expanded out-of-the-box protections for cloud security monitoring, as well as a new integration with IBM Security’s security orchestration, automation, and response (SOAR) platform, IBM Security Resilient.

Users also benefit from free access to 200+ prebuilt adversary behavior protections mapped directly to MITRE ATT&CK. New, prebuilt machine learning jobs and threat detection rules enable users to safeguard their cloud infrastructure and applications at scale, helping prevent damage and loss.

New community-driven workflow and UI enhancements unify prevention, detection, and response, strengthening organizations’ ability to operationalize any security use case. Elastic Security UI enhancements include curated and interactive visualizations that enable analysts to efficiently triage and hunt for security threats.

Elastic Security simplifies data ingestion with new integrations for multiple host and cloud data sources, including Microsoft Defender ATP, Windows PowerShell, and G Suite. The curated data onboarding experience supports security operations, DevSecOps, and other use cases.

“Elastic Security helps us perform the threat detection, continuous monitoring, and incident response functions that we need to effectively protect UC Davis. Performing these tasks using a single UI integrates and streamlines all our security operations workflows.

“Since the solution is backed by the security community’s continuing contributions, we gain the capability to handle the latest attacks,” said Jeff Rowe, University of California, Davis, Security Architect.

“The global shift to a remote workforce has accelerated the need for organizations to react fast, implement new controls, and do it all while managing existing budgets and staff.

“Making it easy for organizations to get started with free malware prevention fully integrated into the Elastic Stack helps level the playing field for organizations that are struggling with the typically high cost and complexity of adopting effective endpoint security,” said Mike Nichols, Head of Product, Elastic Security.

“This is our first major beta milestone in delivering comprehensive, integrated endpoint security. At Elastic, we believe that transparency and collaboration with the greater infosec community is fundamental to succeed in stopping threats at scale.”

Elasticsearch security: Understand your options and apply best practices

The ever-escalating popularity of Elasticsearch – the distributed open source search and log analytics engine that has become a staple in enterprise application developers’ tool belts – is well-warranted. Elasticsearch security lapses, however, have been a headline-grabbing thorn in the side of the technology.

Elasticsearch security

The distributed document store too often represents a security blind spot for organizations, inexcusably failing to receive the attention and upkeep that other data storage solutions are normally granted. Data breach incidents involving Elasticsearch have been commonly rooted in this lack of attention, as well as a poor overall understanding of Elasticsearch security requirements.

As an open source solution, Elasticsearch can be downloaded without any subscription or enterprise license required. But in its default configuration, Elasticsearch doesn’t come with enterprise-grade security features. This can add up to a perfect storm from a security perspective: Elasticsearch is tremendously easy to deploy, but just as simple to forget about when it comes to hardening security that properly restricts access and protects data.

In a now-all-too-commonly-seen examples, technology teams expose their development or testing systems to the internet for convenience, and then forget to change to a secure configuration before moving Elasticsearch into production. The result – careless exposure of production Elasticsearch data to anyone who might access it – puts organizations at risk.

Elasticsearch security options

Until recently, the best (and, really, the only) viable option for ensuring Elasticsearch security was using the Elastic Stack extension X-Pack. X-Pack requires purchasing a costly enterprise subscription from Elastic. For that cost, X-Pack does provide valuable enterprise-grade security.

However, there is now another option: the Amazon-initiated Open Distro for Elasticsearch project offers a slate of enterprise-grade security features with open source availability. Among these, Open Distro for Elasticsearch includes encryption of data in-transit – supporting OpenSSL and TLS 1.2. This protects both traffic from external clients and internal traffic among cluster nodes, while offering simplified integration with public key infrastructures and the ability to enable enterprises to satisfy strict regulatory compliance requirements.

Open Distro for Elasticsearch readily integrates into authentication infrastructures as well, allowing enterprises to authenticate users through LDAP/Active Directory, Kerberos, SAML, and other popular protocols.

Open Distro for Elasticsearch also includes role-based access controls (RBACs), featuring granular controls for limiting each user’s access to only those cluster operations, indices, or documents and fields they require. It also enables security incident responses and secures the Elasticsearch cluster in-line with government and industry regulations via audit logs. This logging tracks and records all user actions within the cluster and enables all activity to be monitored.

In comparison, Elastic’s X-Pack similarly features SSL/TLS encryption, authorization and access controls including password protection, RBACs, and IP filtering, and the ability to maintain audit trails. While Elastic has also taken the step of opening its code for X-Pack, the clear caveat remains that the required licensing fees make X-Pack the costlier option for securing Elasticsearch.

5 actions enterprises should take to ensure Elasticsearch security

Whichever solution for achieving Elasticsearch security an enterprise selects, the following best practices should be top-of-mind:

1. Encrypt all data. Utilize TLS to encrypt all traffic within your Elasticsearch cluster, as well as all traffic from data sources connecting to your Elasticsearch cluster.

2. Do not expose your Elasticsearch cluster to the internet without the proper precautions. In cases where such exposure is required, ensure that internet-facing servers use secure configurations and leverage firewalls, least-privilege policies and access controls, proxies, etc.

3. Implement strict access controls. Control access to indices, documents, and more with secure authentication methods and RBACs.

4. Introduce audit logs. Utilize audit logging to track the actions of all users within your Elasticsearch cluster, monitor any suspicious activity, and conduct informed security incident responses.

5. Leverage provider support when necessary. If in need of external expertise and support, enlist a managed Elasticsearch provider capable of mitigating your security risks. Such providers can offer out-of-the-box security features such as encryption, access control, and monitoring and alerts, while ensuring the integrity of your data in accordance with regulatory standards.

Safely realizing the full benefits of Elasticsearch – and there are many of them – requires paying close attention to your data security protections, the same as you would with any database implementation. By selecting a suitable security strategy and adhering to best practices, your organization can get the most out of Elasticsearch while still keeping data fully secure.

How do I select a SIEM solution for my business?

A Security Information and Event Management (SIEM) solution collects and analyzes activity from numerous resources across your IT infrastructure. A SIEM can provide information of critical importance, but how do you find one that fits your organization?

To select an appropriate SIEM solution for your business, you need to think about a variety of factors. We’ve talked to several industry professionals in order to get insight to help you get started.

Jae Lee, Senior Director, Elastic Security

select SIEM solutionSIEM is a mature product category and continues evolving. However, SIEM needs to enable teams to evolve, as SecOps transforms from “traditional” to “adaptive.”

Let’s start with people — traditional skillsets are based on tools (e.g., vulnerability, firewall, IDS/IPS, etc.), but broader skillsets are needed to help practitioners adapt quickly. Manipulating and analyzing data, performing collaborative research, understanding adversaries/tradecraft — SIEM must help augment and develop these skillsets.

Next is process — with improved skills, alerts no longer rule (unless allowed to), and pre-defined, static SOPs / playbooks alone are not enough. Teams now require real-time analysis to hunt — including performing research, reverse-engineering and simulating threats, and more. Context is everything. Hunting and operationalizing effectively requires full visibility — not in a separate tool, but within the SIEM.

Finally, technology. Full visibility isn’t just broad coverage, but fast insights. Also, detections need to work OOTB. Consider endpoint — there, OOTB detections have high accuracy. The same principle should apply in SIEM, without requiring every analyst to be an expert rule author. SIEM isn’t just “technology” — it needs real-world-validated security content.

As SecOps matures, major investments are often required for the care and feeding of a SIEM. You have to stop threats and justify your investment. Give yourself the runway to be confident that once deployed the SIEM can meet your fast-evolving needs, and ask hard questions around scale and flexibility — from detections to integrations, to deployment options, to pricing metrics.

Christopher Meenan, Director, QRadar Product Management and Strategy, IBM Cloud and Cognitive Software

select SIEM solutionThe first thing to think about is what use cases you need to address. Your requirements will look very different depending on whether you need to secure your organization during a cloud transformation, build a unified IT and OT security operations program, or simply address compliance. Your use cases will drive requirements around integrations, use case content, analytics, and deployment methods.

Ask the vendors how they can help address your requirements. Understand which integrations and use case content are included, versus which require a separate license or custom development. Understand what analytics are available and how those analytics are used to detect known and unknown threats. Ask what frameworks, such as MITRE ATT&CK, are natively supported.

If you’re like most companies, your team is understaffed – which means you need usable products that help shorten the learning curve for new analysts and make your experienced team members more efficient. Ask how each solution measurably increases efficiency during the detection, investigation and response processes. Also ask about SaaS deployments and MSSP partnerships if to reduce on-going management requirements.

Most importantly, don’t be shy. Ask for a proof of concept to make sure the tools you’re considering will work for you.

Stephen Moore, Chief Security Strategist, Exabeam

select SIEM solutionThe most seasoned and well-resourced security teams can be easily overwhelmed by the volume of organizational alerts they receive in a day and that complexity – coupled with the inherent difficulties of detecting credential-based attacks – means many SOC analysts now experience several pains that traditional SIEMs can’t solve, including alert fatigue, a lack of skilled analysts and lengthy investigation times.

Many organizations are now migrating their SIEM to the cloud, which allows analysts to harness greater compute power, sift through, interpret and operationalize SIEM data. Now more of their time is spent finding bad things versus platform and server support. But to choose the right SIEM for ‘the business’ you need to consult with it. You need to align its capabilities to the goals, concerns and expectations of the business – which will undoubtedly have changed over the last few months. Above all else, this requires taking the time to ask the questions.

Then, make choices based on known adversary behavior and breach outcomes – focusing specifically on credentials – ensuring your platform is adversary adaptable and object centered. Ask, will it improve your time to answer (TTA) questions, such as ‘which account or asset is associated with this alert?’ or ‘what happened before, during, and after?’

Finally, any solution needs to help your SOC analysts focus on the right things. Key to this is automation – both in the form of incident timelines that display the full scope, acting as the storyboard of the incident, as well as an automated incident response capability for when action must be taken to return the environment to normal. Providing automation of the necessary investigation steps is the most important thing an incident responder can have so they may take action faster and most importantly minimize the risk of an incomplete response.

Wade Woolwine, Principal Security Researcher, Rapid7

select SIEM solutionWhile the term SIEM has “security” as the very first word, event and log management isn’t just for security teams.

When organizations look to invest in a SIEM or replace an existing SIEM, they should consider use cases across security, IT/cloud, engineering, physical security, and any other group who may benefit from a centralized aggregation of logs. Once the stakeholders have been identified, documenting the specific logs, their sources, and any use cases will ensure the organization has a master list of needs against which to evaluate vendors.

Organizations should also recognize that the use cases will change over time and new use cases will be implemented against the SIEM, especially within the security team. For this reason, organizations should also consider the following as hard requirements to support future growth:

  • Support for adding and categorizing custom event sources by your own team
  • Support for cloud based event sources
  • Field searching level with advanced cross-data-type search functionality and regular expression support
  • Saved searches with alerting
  • Saved searches with dynamic dashboard reporting
  • Ability to integrate threat feeds
  • Support for automation platform integration
  • API support
  • Multi-day training included with purchase

Jesper Zerlang, CEO, LogPoint

select SIEM solutionAs the complexity of enterprise infrastructures is increasing, a key component of a Modern SIEM solution is the ability to capture data from everywhere. This includes data on-premises, in the cloud, and from software, including enterprise applications like SAP. In today’s complex threat landscape, a SIEM that fully integrates UEBA and allows enterprises to relevantly enhance security analytics instantly is an absolute necessity.

The efficiency of your SIEM solution is entirely dependent on the data you feed into it. If the license model of a SIEM solution relies on the volume of data ingested or the number of transactions, the cost will be ever-increasing due to the overall growth in data volumes. As a consequence, you may select to skip SIEM coverage for certain parts of your infrastructure to cut costs, and that can prove fatal.

Choose a SIEM with a license model that that support the full digitalization of your business and allows you to fully predict the future cost. This will ensure that your business needs are aligned by your technology choices. And last but not least: Select a SIEM solution that has documented short time-to-value and complete your SIEM project on time. SIEM deployments, whether initial implementation or a replacement, are generally considered complicated and time-consuming. But they certainly don’t have to be.

Elastic Enterprise Search: Giving users the tools to bring search experiences to market quickly

Elastic announced the launch of Elastic Enterprise Search on Elastic Cloud. Elastic Enterprise Search is a suite of search products that dramatically simplifies the process of creating enterprise-grade search experiences for both customer- and employee-facing search applications.

The new offering enables users to deploy both Elastic Workplace Search and App Search with ease and flexibility across platforms and global regions.

Elastic Enterprise Search can be deployed on cloud platforms including AWS, Google Cloud, and Microsoft Azure, with deployment options anywhere Elastic Cloud is available — including 35+ global regions — providing the flexibility and scale needed to ensure speed and manage data sovereignty concerns.

In addition, users can download and run the solution on-premises, from anywhere, and existing Elastic Cloud customers can add Elastic Enterprise Search simply by creating a new deployment in their cloud console.

With Elastic Enterprise Search on Elastic Cloud, customers benefit from an expertly hosted solution, complete with support from the people who built the product. In addition, customers can scale their deployments with ease, enabling full control of costs with a unique, resource-based pricing model that eliminates the confusion created with document-, user-, or query-based pricing that many customers experience.

“We saw a nearly 10x increase in site traffic as the global Coronavirus pandemic compelled the public to stay home and leverage online delivery services like HappyFresh,” said Fajar Budiprasetyo, CTO, HappyFresh.

“Moving our Elastic Enterprise Search deployment to Elastic Cloud allowed us to better serve our customers with faster response times, enhanced overall performance, and ensured the reliability of our services.

“Choosing a cloud region close to our customers increased our flexibility and scalability which enabled our search function to handle the drastic increase in site traffic.”

“Enterprise Search on Elastic Cloud enables our customers to create end-to-end, rich experiences for any use case — from customer-facing websites to employee-focused productivity tools,” said Matt Riley, VP of Product, Enterprise Search, Elastic.

“With this launch, Elastic App Search and Workplace Search are now available as a single cloud-based solution, making it easy to consume and scale — no other vendor offers such a flexible approach to deployment.”

Elastic enhances its solutions to help customers derive insights from data

Elastic, the company behind Elasticsearch and the Elastic Stack, announced new capabilities across its Elastic Enterprise Search, Observability, and Security solutions, helping customers consume Elastic services more intuitively, onboard more data, and power deeper analysis and insights from that data.

Additionally, Elastic users benefit from new, enhanced capabilities in Elastic Cloud, which is now available in 35 regions and offers easier purchasing options across all major public cloud providers.

Elastic Stack

Delivering enhanced navigation, dashboard drilldowns, and visualizations with:

  • A new side navigation in Kibana organized around use cases that groups the apps that matter to users under solutions (Observability and Security) built around users’ needs.
  • More powerful dashboards with new drilldown capabilities and easier authoring in Kibana, delivering a streamlined process for adding and updating dashboard visualizations and metrics.

Enterprise Search

Equipping admins with powerful tools to manage modern search experiences with:

  • Enterprise authentication mechanisms seamlessly integrated with document-level authorization, bringing the combined power of both SAML-based and document-level authentication for a frictionless, secure, personalized, and scalable search experience.
  • Configurable documents view in App Search to keep tabs on ingested data for more dynamic interactions with that data by searching, sorting, and filtering the content as it’s ingested, directly inside the App Search console without having to switch between screens.

Observability

Providing expanded visibility, monitoring, and intelligent service maps with:

  • Complete visibility into Google Cloud operations with an expanded Stackdriver integration that supports metrics collection from the Google Cloud operations suite, providing easy monitoring in the Elastic Stack to optimize and streamline cloud operations.
  • Certificate validity monitoring in Elastic Uptime to automatically track certificate validity and expiration dates on monitored hosts / services — users can view all the certificates detected from their deployed monitors on a single page in the Uptime app, along with relevant information like issuing authority and expiration date.
  • Machine-learning powered service maps that provide health insights via status indicators in Elastic APM that automatically pull data and color service nodes red, yellow, or green based on the severity of the anomaly score tied to the service performance.

Security

Enabling faster response, driving action, and eliminating blind spots with:

  • New Jira Service Desk, Jira Software, and Jira Core integration with Elastic case management, enabling faster response, streamlining workflows, and reducing context switching to enhance analyst productivity.
  • Expanded user interface flexibility that gives security practitioners quicker access to the data they need and new ways to take immediate action with interactive aggregation charts that speed decision making and enable rapid successive hypothesis iterations.
  • New Filebeat modules that easily ingest network and endpoint data to eliminate blindspots and broaden the data immediately available to security teams.

Elastic Cloud

Simplifying operations, expanding deployment options, and streamlining billing for Elastic Cloud with:

  • Support for managing Elastic Cloud workloads using an API or command line interface, as well as the introduction of a dedicated coordinating node that combines ingest and query coordination into a single node, helping to scale with high ingest and query loads.
  • More deployment options across regions and cloud service providers, with service now available in Finland, London, Netherlands, Sao Paulo, Singapore, South Carolina, Taiwan, and Tokyo.
  • AWS GovCloud in beta and FedRAMP Moderate ‘In-Process’ certification, providing increased support for public sector workloads.

“Elastic’s continued delivery of powerful observability solutions has helped us to keep innovating internally, streamlining our own digital transformation focused on IT operations and business performance,” said Oscar Narváez, Tools & Analytic Monitoring Manager, Entel Chile & Perú.

“As a leading telecommunications and technology operator, it’s imperative that we promptly identify and mitigate production and service issues. By automating machine learning analysis with Elastic, our time to resolution was reduced by 80 percent – what used to take hours, now only takes minutes.

“The Elastic Stack is part of the backbone of our digital transformation initiative, providing invaluable visibility across Entel as a whole, and now we’re looking to continue our journey with Elastic, leveraging Elastic Cloud to centralize and scale our multinational operation.”

“The Elastic Stack provides us with valuable player action insights and a real-time view of our infrastructure. This allows us to immediately identify hardware or software problems before they interfere with the free-roam virtual reality experience we provide all around the world,” said Billy Arrigo, Data and Insights Analyst, Zero Latency VR.

“Leveraging the Elastic Cloud and the Elastic Observability solution, we monitor the performance of our system through custom-built Kibana Dashboards and investigate issues that get reported by various beats hosted on our fleet of over 2,000 backpack PCs and other system infrastructure devices. This enables us to ensure a safe and immersive experience for our players, that is unlike anything they have ever done before.”

Elastic launches new alerting framework to bring native alerting workflows to Elastic Stack users

Elastic, the company behind Elasticsearch and the Elastic Stack, announced the launch of a new alerting framework delivered across the Elastic Stack to provide first-class experiences with tailored interfaces that allow users to create powerful alerts in the normal flow of their daily tasks.

The new alerting framework is delivered via Kibana across the Elastic Stack and available within the SIEM, Uptime, APM, and Metrics applications. From monitoring application transactions to tracking brute force login attempts, users are enabled with embedded alerting functionality and easily configured integrations with email platforms, and providers including PagerDuty, ServiceNow, and Slack.

Embedding native alerting within the Elastic Stack delivers on the company’s vision for creating a single, intuitive user experience with integrated workflows that are tailored to a user’s context and use case, and includes predefined detection and action mechanisms.

“The new Elastic Stack integration with PagerDuty enables our joint customers to trigger customized actions on the PagerDuty platform to proactively respond to operations issues,” said Steve Gross, senior director of strategic ecosystem development, PagerDuty.

“Whether you’re monitoring server health metrics or suspicious login attempts, the enhanced alerting framework in the Elastic Stack enables DevOps engineers and security practitioners to sleep easier.”

“Alerting is a critical capability for anyone with time series data, but it’s especially critical for Observability and Security,” said Steve Kearns, vice president, product management, Elastic.

“That’s why we designed our new alerting framework from the ground up to make it easy to build alerting UIs anywhere in Kibana, allowing us to bring intuitive workflows to where the operations and security practitioners need them. With integrations into key third-party systems, from PagerDuty to Slack, it’s never been easier to keep an eye on data from a distance.”

The new alerting framework is being introduced as a beta in the 7.7 release of Kibana and is available immediately on the Elasticsearch Service on Elastic Cloud, or for download.

Elastic Stack 7.7.0: Major updates for Enterprise Search, Observability, and Security

Elastic, the company behind Elasticsearch and the Elastic Stack, announced major updates across the Elastic solution portfolio with dozens of advances to bring efficiency, flexibility, and integrated workflows to teams of every size and across every use case.

Elastic Stack 7.7.0

These innovations build on a unified platform powered by the Elastic Stack to make data actionable in real-time and at scale for enterprise search, logging, APM, metrics, security, business analytics, and more. Enhancements across the Elastic Stack include:

Elastic Stack 7.7.0

Alerting – Introduces a new alerting framework to bring native alerting workflows to users of Enterprise Search, Observability, and Security solutions within the Elastic Stack.

  • Integrates alerting features across the Elastic Stack via Kibana and available within the APM, Metrics, Uptime and SIEM applications, to bring alerting workflows directly to the user, tailored to their unique context and use case.
  • Introduces a full alert management UI and powerful new alert interfaces within Kibana.
  • Incorporates the ability to trigger predefined actions with third-party integrations with communication and workflow platforms including PagerDuty, ServiceNow, and Slack.

Asynchronous search – Allows long running queries to execute in the background, opening the door to new use cases that trade off cost and latency while searching massive amounts of data.

  • Provides the flexibility to balance the speed of search execution, the amount of data that can be searched, and the cost of the hardware to support any use case.
  • Enables teams to manage potentially long-running queries in the background, letting teams track progress and retrieve partial results as they become available.

Elastic Enterprise Search

Workplace Search – Brings a relevant, personalized, and modern unified search experience to organizations of all sizes through the general availability of Workplace Search.

  • Creates a centralized source of truth for internal teams that seamlessly connects to the world’s most widely adopted workplace applications to simplify finding the content they need.
  • Delivers pre-built integrations with the world’s most widely adopted workplace applications including Confluence, Dropbox, GitHub, G Suite by Google Cloud, Jira, Microsoft 365 (formerly Office 365), OneDrive, Salesforce, ServiceNow, SharePoint Online, Zendesk, and more.
  • Provides an easily implemented enterprise search solution with out-of-the-box connectivity to SaaS and cloud-based data sources, indexing content from a modern toolchain.
  • Brings the power of modern search experiences to more platforms and more users with open Workplace Search APIs.

Elastic Observability

Service maps – Provides a graphical view of the dependencies between the services powering an application.

  • Presents real-time view of live data and system dependencies to speed the troubleshooting of issues in today’s distributed and cloud-native environments.
  • Offers an aggregate view of how services interact, along with key summary information about each component, allowing teams to toggle between a 50,000-foot view and a granular view with ease.

Expanded integrations – Adds new, out-of-the-box integrations to collect logs and metrics from many common data sources across the infrastructure ecosystem and simplifies instrumentation across all layers of the technology stack.

  • Ensures teams can quickly gather the context they need from a system to investigate and debug new and complex problems within their infrastructure.
  • Key integrations include:
    • AWS Lambda, Virtual Private Cloud, Amazon Aurora, DynamoDB
    • Azure Database accounts, Kubernetes, and container metrics
    • Google Cloud Platform Pub/Sub and Load Balancing
    • IBM MQ
    • Istio
    • MQTT
    • Pivotal Cloud Foundry
    • Prometheus
    • Redis Enterprise

Elastic Security

Case management – Introduces case management features built into Elastic Security, along with direct integration into ServiceNow ITSM.

  • Provides security operations teams more control over detection and response workflows allowing analysts to open, update, tag, comment on, close, and integrate cases with external systems.
  • Integrates case management with ServiceNow ITSM, allowing analysts to forward information from Elastic SIEM to the ServiceNow platform for cross-org ticket tracking and remediation.

Security notifications – Reduces mean time to respond with direct integrations into existing security operation workflows.

  • Leverages the new alerting framework in the Elastic Stack to surface key anomalies and threats with instant notifications from Elastic SIEM to third-party systems, including email, Slack, and PagerDuty

“The new Elastic Stack integration with PagerDuty enables our joint customers to trigger customized actions on the PagerDuty platform to proactively respond to operations issues,” said Steve Gross, senior director of strategic ecosystem development, PagerDuty.

“Whether you’re monitoring server health metrics or suspicious login attempts, the enhanced alerting framework in the Elastic Stack enables DevOps engineers and security practitioners to sleep easier.”

“This release highlights the power of building our Enterprise Search, Observability, and Security solutions on top of the Elastic Stack. It allows us to introduce core features like alerting, which benefit all of our users, and also build the tailored user experiences that IT, operations, and security teams need to improve visibility, work more efficiently, and scale as their needs evolve,” said Steve Kearns, vice president, product management at Elastic.

Elastic Security 7.6.0: Delivering visibility and threat protection through a unified interface

Elastic, creators of Elasticsearch, announced the release of Elastic Security 7.6.0, which builds on the strengths of Elastic Endpoint Security and Elastic SIEM to deliver unparalleled visibility and threat protection through a unified interface.

This release automates the centralized detection of threats in the SIEM app and enhances endpoint detection capabilities on Windows hosts. Access to new data sources and improvements across the Elastic SIEM app further empower security practitioners to accelerate detection and response.

Approach zero dwell time

Elastic Security 7.6 introduces a new SIEM detection engine to automate threat detection, minimizing mean time to detect (MTTD) and freeing up your security team for security tasks requiring human intuition and skill.

With Elasticsearch at its core, Elastic SIEM already accelerates security investigation time from hours to minutes. This new automated detection capability further reduces dwell time by surfacing threats that would otherwise be missed.

Elastic is also releasing an initial set of nearly 100 out-of-the-box rules aligned with the ATT&CK knowledge base to surface signs of threats often missed by other tools.

Created and maintained by the security experts at Elastic, the rules automatically detect tools, tactics, and procedures indicative of threat activity, and will be continually updated to address new threats.

Risk and severity scores associated with signals generated by the detection engine enable analysts to triage issues rapidly and then turn their attention to the highest-value work.

“Elastic has helped our security team focus on what matters by equipping us with the tools we need to efficiently search millions of logs while reducing the number of alerts to a volume that our security team can manage,” said Maxim Verreault, Security Manager at Skytech Communications.

“With the release of 7.6, out-of-the-box signal detection rules in Elastic SIEM enable us to automate analysis across our observability data and detect and respond to threats the moment they happen.

“Elastic Security 7.6 also provides a great way for the community to connect, as we, the security folks, will be able to share custom signal detection rules so that everyone can benefit from them and detect new emerging threats.”

Rules operate on Elastic Common Schema (ECS)-compliant data gathered from Windows, macOS, and Linux systems, as well as network information from other sources. Security teams have the option to create or customize rules, but should never need to rewrite them for new ECS-compliant data sources added to their environment.

Built-in Elastic SIEM threat detection rules are developed and maintained by the security experts at Elastic, and complement both the machine learning-driven anomaly detection jobs of the SIEM app and host-based protections of Elastic Endpoint Security.

Achieve unprecedented visibility into Windows endpoints

Elastic Security 7.6 delivers unprecedented levels of visibility and protection to Windows systems, which are a major attack target due to their ubiquity and lenient user permissions model. The release deepens visibility into Windows activity and resiliently collects and enriches data from locations otherwise vulnerable to the evasion techniques of advanced threats.

New out-of-the-box detections leverage this data to detect attempts to capture keyboard inputs, load malicious code into other processes, and more. Practitioners can pair events generated by these detection rules with automated responses (e.g., kill a process) to achieve layered prevention.

Combining this visibility and protection with the existing prevention, detection, and response capabilities for macOS and Linux systems provides Elastic Endpoint Security users with complete protection across their entire environment.

Reduce MTTD by quickly seeing what matters most

The new Elastic SIEM app Overview page and broader workflow enhancements enable security practitioners to hunt for and investigate threats fast.

Users can jump right into an investigation by opening a timeline, viewing the latest detection signals, and reviewing alerts from external sources like Elastic Endpoint Security, Palo Alto Networks, Suricata, Zeek, and others. Wherever you are in the SIEM app, you’re never more than a click away from its integrated threat detection and anomaly detection capabilities.

Analysts can also increase operational awareness with new event, alert, and signal histograms, and explore the new and enhanced visualizations in the Hosts and Network views for more specialized analysis.

Keep an eye on your applications

Elastic SIEM also now provides curated visibility into HTTP data, giving you the power to view Elastic APM data from directly within the SIEM app. Multiple out-of-the-box Beats modules provide access to additional ECS-compliant HTTP data, allowing you to easily inspect and visualize all web transactions in one unified view.

Monitor your cloud data

Ingesting data into the Elastic Stack for centralized visualization and analysis is now even simpler. 7.6 introduces support for AWS CloudTrail data and enhances support for Google Cloud Platform, providing essential visibility into the modern attack surface.

Data in the CEF format, whether from the cloud or elsewhere, is also easier than ever to visualize thanks to the updated CEF module for Filebeat.

Elastic App Search: Complete search solution with relevance tuning and analytics built in

Elastic, the company behind Elasticsearch and the Elastic Stack, announced the general availability of Elastic App Search on Elasticsearch Service.

Elastic App Search

Elastic App Search is a ready-to-use, fully complete search solution with user-friendly relevance tuning and analytics built in. And starting today, users can deploy instances with the click of a button right from the Elasticsearch Service dashboard. Now all the tooling needed for a relevant search experience is available with the operational flexibility and scale of Elastic Cloud.

This milestone also unlocks a whole new choice of geolocation options for users: from São Paulo to Singapore and California to Germany, App Search can be hosted everywhere you find our Elasticsearch Service.

Elastic didn’t just make getting started on App Search easier — they’ve also simplified pricing by switching to the same resource-based pricing model that Elasticsearch Service uses. Users only pay for the resources they consume, without worrying about artificial constraints around the number of users, documents, or operations made. It’s a whole new approach to pricing search that’s transparent and fair.

As with all Elastic products, Elastic is committed to making App Search accessible to everyone who wants to run a trial, build a prototype, or go into production. To get new users started, Elastic is introducing a new tier that includes a free 2GB instance to get up and running even faster.

“Elastic App Search is the best way for any developer to add powerful search capabilities to their applications,” said Matt Riley, Product Lead for Elastic Search Solutions.

App Search lets users of almost any skill level easily add a powerful and customizable search experience to any application, website, or mobile app using a refined set of APIs and management tools, including:

  • Fast data ingestion with focused APIs and comprehensive clients
  • A powerful, pre-tuned search engine built on Elasticsearch
  • Comprehensive relevance, tuning, and curation controls from an easy-to-use interface
  • Actionable user analytics to close the loop on the search experience cycle

Elastic Cloud on Kubernetes 1.0 is now available

Elastic Cloud on Kubernetes (ECK) is moving out of beta and into general availability.

Elastic Cloud on Kubernetes

As Elastic announced with the alpha release of ECK back in May 2019, the vision for ECK is to provide an official way to orchestrate Elasticsearch on Kubernetes and provide a SaaS-like experience for Elastic products on Kubernetes.

Kubernetes has continued to grow in popularity and has become the standard for orchestrating container workloads, and Elastic has seen a growing number of users deploying the Elastic Stack on Kubernetes. Elastic has already taken a number of steps to support container workloads, such as releasing official Docker images for Elasticsearch and Kibana, joining the CNCF, and launching Elastic Helm charts. Bringing ECK into general availability is the exciting next step on this journey.

The initial alpha release of ECK built on Elastic’s years of operational knowledge gained from creating Elasticsearch and Elastic Cloud Enterprise and running the Elasticsearch Service. The community reception to the first alpha release (and the three early access releases that followed) has been extremely positive, and with the general availability of ECK Elastic offers users a production-ready solution to deploy and streamline the operation of the Elastic Stack on Kubernetes.

Elastic Cloud on Kubernetes: Day 2 operations simplified

When it comes to deploying software, day 1 is easy; day 2 is more challenging. Built on the Kubernetes Operator pattern, ECK simplifies many day 2 operations — such as scaling, upgrades, and configuration management — when managing one or more deployments of the Elastic Stack on Kubernetes. This reduced operational burden lets users focus on their business requirements and reduces time to value from the Elastic Stack.

Notable features include:

  • Deploy and manage multiple Elasticsearch clusters, including Kibana
  • Seamless upgrades to new versions of the Elastic Stack
  • Simple scaling that allows you to grow with your use cases
  • Default security on every cluster
  • As the creators of Elasticsearch and the rest of the Elastic Stack, Elastic wants ECK to be the best solution for users looking to orchestrate Elasticsearch on Kubernetes. Many users have validated this during the alpha/beta cycles.

“As an early adopter of both Kubernetes and Elastic, we’ve been excited about testing Elastic Cloud on Kubernetes (ECK) as it will allow us to streamline our processes for building and operating Elasticsearch on Kubernetes,” said Michael Lorant, Principal Systems Engineer at Nine, Australia’s largest locally owned media company. “With the release of ECK 1.0 GA, we are looking forward to getting the best features of the Elastic Stack including the infrastructure UI that provides detailed visibility of our Kubernetes environment. We are excited to explore further usage of ECK and Elastic for Kubernetes as it aligns with our strategy of complete application and cluster observability.”

Curated solutions and exclusive features

ECK gives users the complete Elastic experience on Kubernetes, including features and capabilities that you can only get from Elastic — such as APM, Logs, Metrics, SIEM, Canvas, Lens, machine learning, and index lifecycle management. All clusters deployed via ECK include these capabilities. Support for advanced topologies through features like dedicated master and machine learning nodes and hot-warm-cold deployments lets users optimize their deployments further for observability and security use cases.

Elastic has released the core ECK functionality under the free-forever Basic tier to make these exclusive features and capabilities available to all users, no matter where they deploy Elastic products. Users can also access more advanced features through Elastic’s Enterprise Subscription.

Getting started

ECK is built for flexibility and runs on a variety of Kubernetes platforms, including Google Kubernetes Engine, Red Hat OpenShift, Azure Kubernetes Service, Amazon Kubernetes Service, and vanilla Kubernetes.

It’s also super simple to get started. With a one-line command, you can deploy ECK into your Kubernetes environment and start creating clusters in a few minutes. For instructions and more details, be sure to check out the ECK quickstart page.

Use the Elastic Stack to monitor Kubernetes

The Elastic and Kubernetes story extends well beyond just running the Elastic Stack on Kubernetes. The Elastic Stack can also be used to provide comprehensive observability and security capabilities for Kubernetes and its ecosystem:

Elastic Stack 7.5.0 enhances enterprise search, observability, and security

Elastic, creators of Elasticsearch, released Elastic Stack 7.5.0, the latest version of the all-in-one datastore, search engine, and analytics platform.

Elastic Stack 7.5.0

Along with the introduction of Kibana Lens, a fast and intuitive way to craft visualizations, this release offers significant enhancements to Elastic’s Observability, Security, and Enterprise Search solutions.

Elastic security

It is an exciting time for folks using the Elastic Stack to protect their organizations. Since the last release, Elastic has joined forces with Endgame, a leading endpoint security company, and announced the end of per-endpoint pricing for the EPP/EDR space. Unlimited endpoints are now included with Elastic’s Enterprise subscriptions, so users no longer need to choose which machines deserve protection.

For servers, if you’re already collecting security and operational data, why not protect them at the same time? And it isn’t just about servers; the security events from all of your infrastructure, including desktops and laptops, should be available to security analysts. Elastic SIEM 7.5 includes endpoint security data and alerts directly in the SIEM app.

This release also continues the push to detect threats using machine learning, from identifying unusual patterns in DNS activity that could indicate DNS tunneling or command and control behavior, to unusual logins over RDP or using the runas command, and more. The SIEM app itself is expanding to include a number of new visualizations and widgets that make threat hunting easier, from interactive visualizations of host activity to a new TLS view that surfaces unusual certificates and simplifies hunting based on TLS fingerprints, such as JA3 hash.

Kibana Lens: An entirely new data visualization experience

Kibana is, and has always been, the best way to visualize data stored in Elasticsearch and to navigate the Elastic Stack. With 7.5, Elastic is introducing Kibana Lens — an entirely new way to craft visualizations. Lens is designed to work the way users think, letting users rapidly go from raw data to meaningful visualization without needing any previous technical experience or knowledge of Elasticsearch.

It starts with a new drag-and-drop experience, along with the ability to easily switch between chart types and different index patterns. As the user adds fields to the chart, Lens provides smart suggestions to show other views of the data. Combined with the speed of Elasticsearch, Lens makes it faster and easier than ever to visualize, explore, and understand data.

Index time enrichment

Way back in Elasticsearch 5.0, Elastic first introduced the Ingest Pipeline — a way to process and enrich documents at indexing time. By building this directly into Elasticsearch, configuration via API is simple, scaling out is easy, and performance is quite fast. Over the years, Elastic has seen wide adoption of this feature and now relies on it for processing and enrichment in nearly all of its modules — the many data sources that Elastic natively supports. Whether it’s parsing a log line with grok or dissect or adding location data to an IP address, ingest pipelines are increasingly the workhorse doing the ingest-time processing in the Elastic Stack.

With the 7.5 release, Elastic is delivering one of the most requested features: lookup-based enrichment. The new Enrich processor provides an efficient way to query an Elasticsearch index and add the results to a document at indexing time. This allows users to do things like identify web services or vendors based on known IP addresses, add postal codes based on user coordinates, or look up host information ingested from a configuration management database and add the relevant metadata to a document right at indexing time.

Elastic Enterprise Search

Elastic Enterprise Search aims to connect people and teams to the content that matters most to them. For organizations with a significant Microsoft product footprint, Elastic Enterprise Search now provides one-click integrations with SharePoint Online, Office 365, and OneDrive, making it easier than ever to unify and search across content platforms.

To top it off, Enterprise Search also includes a brand new ServiceNow connector, allowing users to centralize all business operation information in one place. With these new sources now available alongside the ones already included — Salesforce, Google Drive, Atlassian JIRA, Confluence, Dropbox, and more — teams can now focus on the task at hand.

Elastic observability

Elastic believes that to truly understand your applications and infrastructure, you need to be able to see, or observe, each layer. Elastic Observability brings together the Elastic Logs, Metrics, APM, and Uptime products to give a view across an organization. Version 7.5 of the Elastic Stack brings a significant expansion of the Elastic Metrics story and adds several key integrations between APM, logging, and security data for organizations adopting observability initiatives.

Elastic’s metrics story has gained steam in recent releases with the addition of Metrics Explorer, a purpose-built user interface for real-time metrics analytics. Elastic has also made it easier to get started with metrics using turnkey data integrations for the most important infrastructure and service metrics, including Kubernetes, Prometheus, and AWS.

In 7.5, Elastic builds on that momentum by introducing turnkey monitoring of Microsoft Azure metrics and logs as part of Elastic’s partnership with Microsoft. Finally, Elastic has also added initial support for viewing endpoint security data directly in the Elastic Metrics and Logs apps. These advances help Elastic Stack users set up monitoring of critical services more quickly and enable them to combine metrics with important events, such as audit logs from endpoint devices, more efficiently.