Visualizing AWS Logging Data in Azure Monitor – Part 1

Posted on July 5, 2019 by mattfeltonma

Hi folks!

2019 is more than halfway over and it feels like it has happened in a flash. It’s been an awesome year with tons of change and even more learning. I started the year neck deep in AWS and began transitioning into Azure back in April when I joined on with Microsoft. Having the opportunity to explore both clouds and learn the capabilities of each offering has been an amazing experience that I’m incredibly thankful for. As I’ve tried to do for the past 8 years, I’m going to share some of those learning with you. Today we’re going to explore one of the capabilities that differentiates Azure from its competition.

One of the key takeaways I’ve had from my experiences with AWS and Microsoft is enterprises have become multicloud. Workloads are quickly being spread out among public and private clouds. While the business benefits greatly from a multicloud approach where workloads can go to the most appropriate environment where the cost, risks, and time tables best suit it, it presents a major challenge to the technical orchestration behind the scenes. With different APIs (application programmatic interface), varying levels of compliance, great and not so great capabilities around monitoring and alerting, and a major industry gap in multicloud skills sets, it can become quite a headache to successfully execute this approach.

One area Microsoft Azure differentiates itself is its ability to easy the challenge of monitoring and alerting in a multicloud environment. Azure Monitor is one of the key products behind this capability. With this post I’m going to demonstrate Azure Monitor’s capabilities in this realm by walking you through a pattern of delivering, visualizing, and analyzing log data collected from AWS. The pattern I’ll be demonstrating is reusable for most any cloud (and potentially on-premises) offering. Now sit back, put your geek hat on, and let’s dive in.

First I want to briefly talk about what Azure Monitor is? Azure Monitor is a solution which brings together a collection of tools that can be used to collect and analyze the large abundance of telemetry available today. This telemetry could be metrics in regards to a virtual machine’s performance or audit logs for Azure Active Directory. The product team has put together the excellent diagram below which explains the architecture of the solution.

Credit – https://docs.microsoft.com/en-us/azure/azure-monitor/overview

As you can see from the inputs on the left, Azure Monitor is capable of collecting and analyzing data from a variety of sources. You’ll find plenty of documentation the product team has made publicly available on the five gray items, so I’m going to instead focus on custom sources.

For those of you who have been playing in the AWS pool, you can think of Azure Monitor as something similar (but much more robust) to CloudWatch Metrics and CloudWatch Logs. I know, I know, you’re thinking I’ve drank the Microsft Kool-Aid.

While I do love to reminisce about cold glasses of Kool-Aid on hot summers in the 1980s, I’ll opt to instead demonstrate it in action and let you decide for yourself. To do this I’ll be leveraging the new API Microsoft introduced. The Azure Monitor HTTP Data Collector API was introduced a few months back and provides the capability of delivering log data to Azure where it can be analyzed by Azure Monitor.

With Azure Monitor logs are stored in an Azure resource called a Log Analytics Workspace. For you AWS folk, you can think of a Log Analytics Workspace as something similar to CloudWatch Log Groups where the data stored in a logical boundary where the data shares a retention and authorization boundary. Logs are sent to the API in JSON format and are placed in the Log Analytics Workspace you specify. A high level diagram of the flow can be seen below.

Credit – https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api

So now that you have a high level understanding of what Azure Monitor is, what it can do, and how the new API works, let’s talk about the demonstration.

If you’ve used AWS you’re very familiar with the capabilities CloudWatch Metrics Dashboards and the basic query language available to analyze CloudWatch Logs. To perform more complex queries and to create deeper visualizations, third-party solutions are often used such as ElasticSearch and Kibana. While these solutions work, they can be complex to implement and can create more operational overhead.

When a peer informed me about the new API a few weeks back, I was excited to try it out. I had just started to use Azure Monitor to put together some dashboards for my personal Office 365 and Azure subscriptions and was loving the power and simplicity of the analytics component of the solution. The new API opened up some neat opportunities to pipe logging data from AWS into Azure to create a single dashboard I could reference for both clouds. This became my use case and demonstration of the pattern of delivering logs from a third party to Azure Monitor with some simple Python code.

The logs I chose to deliver to the API were logs containing information surrounding the usage of AWS access ids and keys. I had previously put together some code to pull this data and write it to an S3 bucket.

Let’s take a look at the design of the solution. I had a few goals I wanted to make sure to hit if possible. My first goal was to keep the code simple. That mean limiting the usage of third-party modules and avoid over complicating the implementation.

My second goal was to limit the usage of static credentials. If I ran the code in Azure, I’d need to setup an AWS IAM User and provision an access id and secret key. While I’m aware of the workaround to use SAML authentication, I’m not a fan because in my personal opinion, it’s using SAML in such a way you are trying to hammer in a square peg in a round hole. Sure you can do it, but you really shouldn’t unless you’re out of options. Additionally, the solution requires some fairly sensitive permissions in AWS such as IAM:ListAccessKeys so the risk of the credentials being compromised could be significant. Given the risks and constraints of authentication methods to the AWS API, I opted to run my code as a Lambda and follow AWS best practices and assign the Lambda an IAM role.

On the Azure side, the Azure Monitor API for log delivery requires authentication using the Workspace ID and Workspace key. Ideally these would be encrypted and stored in AWS Secrets Manager or as a secure parameter in Parameter Store, but I decided to go the easy route and store them as environment variables for the Lambda and to encrypt them with AWS KMS. This cut back on the code and made the CloudFormation templates easier to put together.

With the decisions made the resulting design is pictured above.

I’m going to end the post here and save the dive into implementation and code for the next post. In the meantime, take a read through the Azure Monitor documentation and familiarize yourself with the basics. I’ve also put the whole solution up on Github if you’d like to follow along for next post.

See you next post!

Passing the AZ-300

Posted on May 8, 2019 by mattfeltonma

Hello all!

Over the past year I’ve been buried in Amazon Web Services (AWS), learning the platform, and working through the certification paths. As part of my new role at Microsoft, I’ve been given the opportunity to pursue the Microsoft Certified: Azure Solutions Architect Expert. In the world of multi-cloud who doesn’t want to learn multiple platforms? 🙂

The Microsoft Certified: Azure Solutions Architect Expert certification is part of Microsoft’s new set of certifications. If you’re already familiar with the AWS Certification track, the new Microsoft track is very similar in that it has three paths. These paths are Developer, Administrator, and Architect. Each path consists of two exams, again similar to AWS’s structure of Associate and Professional.

Even though the paths are similar the focus and structure of the first tier of exams for the Microsoft exams differ greatly from the AWS Associate exams. The AWS exams are primarily multiple choice while the Microsoft first level of exams consists of multiple choice, drag and drop, fill in the blank, case studies, and emulated labs. Another difference between the two is the AWS exams focus greatly on how the products work and when and where to use each product. The Microsoft first level exams focus on those topics too, but additionally test your ability to implement the technologies.

When I started studying for the AZ-300 – Microsoft Azure Architect Technologies two weeks I had a difficult time finding good study materials because the exam is so new and has changed a few times since Microsoft released it last year. Google searches brought up a lot of illegitimate study materials (brain dumps) but not much in the way of helpful materials beyond the official Azure documentation. After passing the exam this week, I wanted to give back to the community and provide some tips, links, and the study guide I put together to help prepare for the exam.

To prepare for an exam I have a standard routine.

I first start with referencing the official exam requirements.
From there, I take one or two on-demand training classes. I watch each lesson in a module at 1.2x speed (1x always seems to slow which I think is largely due to living in Boston where we tend to talk very quickly). I then go back through each module at 1.5x to 2.0x taking notes on paper. I then type up the notes and organize them into topics.
Once I’m done with the training I’ll usually dive deep into the official documentation on the subjects I’m weak on or that I find interesting.
During the entirety of the learning process I will build out labs to get a feel for implementation and operation of the products.
I wrap it up by adding the additional learnings from the public documentation and labs into my digital notes. I then pull out the key concepts from the digital notes and write up flash cards to study.
Practice makes perfect and for that I will leverage legitimate practice exams (braindumps make the entire exercise a pointless waste of time and degrade the value of the certification) like those offered from MeasureUp.

Yes, I’m a bit nuts about my studying process but I can assure you it works and you will really learn the content and not just memorize it.

From a baseline perspective, my experience with Microsoft’s cloud services were primarily in Azure Active Directory and Azure Information Protection. For Azure I had built some virtual networks with virtual machines in the past, but nothing more than that. I have a pretty solid foundation in AWS and cloud architectural patterns which definitely came in handy since the base offerings of each of the cloud providers are fairly similar.

For on-demand training A Cloud Guru has always been my go to. Unfortunately, their Azure training options aren’t as robust as the AWS offerings, but Nick Coyler’s AZ-300 course is solid. It CANNOT be your sole source of material but as with most training from the site, it will give you the 10,000 ft view. Once I finished with A Cloud Guru, I moved on to Udemy. Scott Duffy’s AZ-300 course does not have close to the detail of Nick’s course, but provides a lot more hands-on activities that will get you working with the platform via the GUI and the CLI. Add both courses together and you’ll cover a good chunk of the exam.

The courses themselves are not sufficient to pass the exam. They will give you the framework, but docs.microsoft.com is your best friend. There is the risk you can dive more deep into the product than you need to, but reference back to the exam outline to keep yourself honest. Hell, worst case scenario is you learn more than you need to learn. 🙂 Gregor Suttie put together a wonderful course outline with links to the official documentation that will help you target key areas of the public documentation.

Perhaps most importantly, you need to lab. Then lab again. Lab once more, and then another time. Run through the Quickstarts and Tutorials on docs.microsoft.com. Get your hands dirty with the CLI, PowerShell, and the Portal. You don’t have to be an expert, but you’ll want to understand the basics and the general syntax of both the CLI and PowerShell. You will have fully interactive labs where you’ll need to implement the products given a set of requirements.

Finally, I’ve added the study guides I put together to my github. I make no guarantees that the data is up to date or even that there aren’t mistakes in some of the content. Use it as an artifact to supplement your studies as you prepare your own study guide.

Summing it up, don’t just look at the exam as a piece of virtual paper. Look at it as an opportunity to learn and grow your skill set. Take the time to not just memorize, but understand and apply what you learn. Be thankful you work an industry where things change and provides you with the opportunity to learn something new and exercise that big brain of yours.

I wish you the best of luck in your studies and if you have additional materials or a website you’ve found helpful, please comment below.

Thanks!

Capturing and Visualizing Office 365 Security Logs – Part 2

Posted on January 31, 2019 by mattfeltonma

Hello again my fellow geeks.

Welcome to part two of my series on visualizing Office 365 security logs. In my last post I walked through the process of getting the sign-in and security logs and provided a link to some Lambda’s I put together to automate pulling them down from Microsoft Graph. Recall that the Lambda stores the files in raw format (with a small bit of transformation on the time stamps) into Amazon S3 (Simple Storage Service). For this demonstration I modified the parameters for the Lambda to download the 30 days of the sign-in logs and to store them in an S3 bucket I use for blog demos.

When the logs are pulled from Microsoft Graph they come down in JSON (JavaScript Object Notation) format. Love JSON or hate it is the common standard for exchanging information these days. The schema for the JSON representation of the sign-in logs is fairly complex and very nested because there is a ton of great information in there. Thankfully Microsoft has done a wonderful job of documenting the schema. Now that we have the logs and the schema we can start working with the data.

When I first started this effort I had put together a Python function which transformed the files into a CSV using pipe delimiters. As soon as I finished the function I wondered if there was an alternative way to handle it. In comes Amazon Athena to the rescue with its Openx-JsonSerDe library. After reading through a few blogs (great AWS blog here), StackOverflow posts, and the official AWS documentation I was ready to put something together myself. After some trial and error I put together a working DDL (Data Definition Language) statement for the data structure. I’ve made the DDLs available on Github.

Once I had the schema defined, I created the table in Athena. The official AWS documentation does a fine job explaining the few clicks that are provided to create a table, so I won’t re-create that here. The DDLs I’ve provided you above will make it a quick and painless process for you.

Let’s review what we’ve done so far. We’ve setup a reoccurring job that is pulling the sign-in and audit logs via the API and is dumping all that juicy data into cheap object storage which we can further enforce lifecycle policies for. We’ve then defined the schema for the data and have made it available via standard SQL queries. All without provisioning a server and for pennies on the dollar. Not to shabby!

At this point you can use your analytics tool of choice whether it be QuickSight, Tableau, PowerBi, or the many other tools that have flooded the market over the past few years. Since I don’t make any revenue from these blog posts, I like to go the cheap and easy route of using Amazon QuickSight.

After completing the initial setup of QuickSight I was ready to go. The next step was to create a new data set. For that I clicked the Manage Data button and selected New Data Set.

On the Create a Data Set screen I selected the Athena option and created a name for the data source.

From there I selected the database in Athena which for me was named azuread. The tables within the database are then populated and I chose the tbl_signin_demo which points to the test S3 bucket I mentioned previously.

Due to the complexity of the data structure I opted to use a custom SQL query. There is no reason why you couldn’t create the table I’m about to create in Athena and then connect to that table instead to make it more consumable for a wider array of users. It’s really up to you and I honestly don’t know what the appropriate “big data” way of doing it is. Either way, those of you with real SQL skills may want to look away from this query lest you experience a Raiders of The Lost Ark moment.

You were warned.

SELECT records.id, records.createddatetime, records.userprincipalname, records.userDisplayName, records.userid, records.appid, records.appdisplayname, records.ipaddress, records.clientappused, records.mfadetail.authdetail AS mfadetail_authdetail, records.mfadetail.authmethod AS mfadetail_authmethod, records.correlationid, records.conditionalaccessstatus, records.appliedconditionalaccesspolicy.displayname AS cap_displayname, array_join(records.appliedconditionalaccesspolicy.enforcedgrantcontrols,' ') AS cap_enforcedgrantcontrols, array_join(records.appliedconditionalaccesspolicy.enforcedsessioncontrols,' ') AS cap_enforcedsessioncontrols, records.appliedconditionalaccesspolicy.id AS cap_id, records.appliedconditionalaccesspolicy.result AS cap_result, records.originalrequestid, records.isinteractive, records.tokenissuername, records.tokenissuertype, records.devicedetail.browser AS device_browser, records.devicedetail.deviceid AS device_id, records.devicedetail.iscompliant AS device_iscompliant, records.devicedetail.ismanaged AS device_ismanaged, records.devicedetail.operatingsystem AS device_os, records.devicedetail.trusttype AS device_trusttype,records.location.city AS location_city, records.location.countryorregion AS location_countryorregion, records.location.geocoordinates.altitude, records.location.geocoordinates.latitude, records.location.geocoordinates.longitude,records.location.state AS location_state, records.riskdetail, records.risklevelaggregated, records.risklevelduringsignin, records.riskstate, records.riskeventtypes, records.resourcedisplayname, records.resourceid, records.authenticationmethodsused, records.status.additionaldetails, records.status.errorcode, records.status.failurereason  FROM "azuread"."tbl_signin_demo" CROSS JOIN (UNNEST(value) as t(records))

This query will de-nest the data and give you a detailed (possibly extremely large depending on how much data you are storing) parsed table. I was now ready to create some data visualizations.

The first visual I made was a geospatial visual using the location data included in the logs filtered to failed logins. Not surprisingly our friends in China have shown a real interest in my and my wife’s Office 365 accounts.

Next up I was interested in seeing if there were any patterns in the frequency of the failed logins. For that I created a simple line chart showing the number of failed logins per user account in my tenant. Interestingly enough the new year meant back to work for more than just you and me.

Like I mentioned earlier Microsoft provides a ton of great detail in the sign-in logs. Beyond just location, they also provide reasons for login failures. I next created a stacked bar chat to show the different reasons for failed logs by user. I found the blocked sign-ins by malicious IPs interesting. It’s nice to know that is being tracked and taken care of.

Failed logins are great, but the other thing I was interested in is successful logins and user behavior. For this I created a vertical stacked bar chart that displayed the successful logins by user by device operating system (yet more great data captured in the logs). You can tell from the bar on the right my wife is a fan of her Mac!

As I gather more data I plan on creating some more visuals, but this was great to start. The geo-spatial one is my favorite. If you have access to a larger data set with a diverse set of users your data should prove fascinating. Definitely share any graphs or interesting data points you end up putting together if you opt to do some of this analysis yourself. I’d love some new ideas!

That will wrap up this series. As you’ve seen the modern tool sets available to you now can do some amazing things for cheap without forcing you to maintain the infrastructure behind it. Vendors are also doing a wonderful job providing a metric ton of data in their logs. If you take the initiative to understand the product and the data, you can glean some powerful information that has both security and business value. Even better, you can create some simple visuals to communicate that data to a wide variety of audiences making it that much more valuable.

Have a great weekend!

Journey Of The Geek

The chronicles of a Bostonian tech geek navigating through life and technology

Tag Archives: Microsoft

Visualizing AWS Logging Data in Azure Monitor – Part 1

Passing the AZ-300

Capturing and Visualizing Office 365 Security Logs – Part 2