If you made it to this second paragraph, I’m sure you’re thinking to yourself, “Armen, since you’re running this site in AWS, why not use Amazon Redshift or Athena to make your life easy?” You’re absolutely right in that it would be easier, especially given that there are documented solutions for both: S3 to RedShift and CloudFront to Athena However, for those of you that know me well, you know that I often (sometimes to a fault) pick the path least traveled. In this case, especially given that the industry is continuing to drive toward public cloud as a commodity, I wanted to choose the solution that worked best for me.

For this case, especially given my experience with the Kusto Query Language (KQL) over the past couple of years, I wanted the ability to run queries easily without conjuring SQL statements like I would if I used either of the aforementioned AWS solutions. With my target destination set, I put together a rough set of steps that would get me to a point where I could query the data.

• Give Azure least privilege access to the S3 bucket where the logs are located
• Copy the logs to a useful destination in Azure where they could be indexed
• Map the data appropriately so that it can be referenced through KQL statements
• Run KQL against the data to return useful reports

In the past, I struggled to find the correct combination of permissions to read contents from an S3 bucket. I’ll admit that I spent some time trying to figure out the unique combination of GetObject and ListBucket permissions but without success so I scoured the Azure documentation, and discovered the one for the Data Factory S3 Connector, which described what I needed well enough to be able to set up a user and group with the right permission.

- PolicyName: S3ReadAccess
  PolicyDocument:
    Version: 2012-10-17
    Statement:
      - Effect: Allow
        Action:
        - s3:ListBucket
        - s3:GetBucketLocation
        - s3:GetObject
        - s3:GetObjectVersion
        Resource:
        - !Sub 'arn:aws:s3:::${S3BucketName}'
        - !Sub 'arn:aws:s3:::${S3BucketName}/*'

Once the CloudFormation was run in the account where the bucket containing the logs was located, I had a set of credentials that I could use from Azure to access the log data. The next step was to build a landing for the data from the bucket in Azure. I spent a number of days on this problem as this was my first experience building a data pipeline into Azure and I wasn’t sure what I was getting myself into.

My first attempt was to copy the data from S3 to a Storage Account in Azure and then try to figure out how to make it accessible in Data Explorer. This was an epic failure for two reasons: the first being that I wasn’t being very cost conscious in duplicating the data from one source to another, and second is that the transfer lasted over three and a half hours because the Integration Runtime I selected was the smallest size (at least that’s what I believe to be the issue).

The second attempt was to copy the data from S3 to a Data Explorer cluster directly without the intermediate Storage Account step. This attempt was completely iterative as I tested different configuration variations each time until I figured out one that worked well for me.

The first step was to create a basic Data Explorer cluster, and then a database inside the cluster. I effectively followed the steps in the documented QuickStart.

With the database created, the next step was to create the target table in the cluster, and define a schema for the ingested data. Initially, I spent time thinking about mapping each column to an appropriate data explorer scalar type but after evaluating the columns, I noticed that most of them were strings, and went along with it except for a few exceptions. The final schema I used was the following:

"Schema": {"Name":"redacted","OrderedColumns":[
{"Name":"date","Type":"System.String","CslType":"string"},
{"Name":"time","Type":"System.String","CslType":"string"},
{"Name":"edge-location","Type":"System.String","CslType":"string"},
{"Name":"server-bytes","Type":"System.Int32","CslType":"int"},
{"Name":"client-ip","Type":"System.String","CslType":"string"},
{"Name":"client-method","Type":"System.String","CslType":"string"},
{"Name":"edge-fqdn","Type":"System.String","CslType":"string"},
{"Name":"request-uri","Type":"System.String","CslType":"string"},
{"Name":"server-status","Type":"System.Int32","CslType":"int"},
{"Name":"client-referer","Type":"System.String","CslType":"string"},
{"Name":"client-useragent","Type":"System.String","CslType":"string"},
{"Name":"request-uriquery","Type":"System.String","CslType":"string"},
{"Name":"client-cookie","Type":"System.String","CslType":"string"},
{"Name":"edge-result","Type":"System.String","CslType":"string"},
{"Name":"edge-requestid","Type":"System.String","CslType":"string"},
{"Name":"server-fqdn","Type":"System.String","CslType":"string"},
{"Name":"client-protocol","Type":"System.String","CslType":"string"},
{"Name":"client-bytes","Type":"System.String","CslType":"string"},
{"Name":"server-timetaken","Type":"System.Data.SqlTypes.SqlDecimal","CslType":"decimal"},{"Name":"client-forwardedfor","Type":"System.String","CslType":"string"},
{"Name":"client-tls","Type":"System.String","CslType":"string"},
{"Name":"client-tlscipher","Type":"System.String","CslType":"string"},
{"Name":"edge-response","Type":"System.String","CslType":"string"},
{"Name":"client-protocolversion","Type":"System.String","CslType":"string"},
{"Name":"field-encryption","Type":"System.String","CslType":"string"},
{"Name":"field-encryptionfields","Type":"System.String","CslType":"string"},
{"Name":"client-port","Type":"System.String","CslType":"string"},
{"Name":"server-timetofirstbyte","Type":"System.String","CslType":"string"},
{"Name":"edge-errordetail","Type":"System.String","CslType":"string"},
{"Name":"server-contenttype","Type":"System.String","CslType":"string"},
{"Name":"server-contentlength","Type":"System.String","CslType":"string"},
{"Name":"server-rangestart","Type":"System.String","CslType":"string"},
{"Name":"server-rangeend","Type":"System.String","CslType":"string"}]},

Data Explorer has a nifty function to export the create table script. For those that are curious and desire to test out the solution, here it is:

.create table tablename (['date']: string, ['time']: string, ['edge-location']: string, 
['server-bytes']: int, ['client-ip']: string, ['client-method']: string, ['edge-fqdn']: string, 
['request-uri']: string, ['server-status']: int, ['client-referer']: string, 
['client-useragent']: string, ['request-uriquery']: string, ['client-cookie']: string, 
['edge-result']: string, ['edge-requestid']: string, ['server-fqdn']: string, 
['client-protocol']: string, ['client-bytes']: string, ['server-timetaken']: decimal, 
['client-forwardedfor']: string, ['client-tls']: string, ['client-tlscipher']: string, 
['edge-response']: string, ['client-protocolversion']: string, ['field-encryption']: string, 
['field-encryptionfields']: string, ['client-port']: string, ['server-timetofirstbyte']: string, 
['edge-errordetail']: string, ['server-contenttype']: string, ['server-contentlength']: string, 
['server-rangestart']: string, ['server-rangeend']: string)

Each of the columns above mapped directly to the 33 fields (included the new seven that were added in late December of 2019). I figured I would add it to the table schema so that if I wanted to ingest the data from the newer logs in the future, I would be able to.

With the database and the target table created, the next step was to build a pipeline in Data Factory to copy the data from S3 directly to the database in Data Explorer. A Data Factory (ADF) pipeline requires the configuration of Linked Services for each service that you intend for the pipeline to interact with. In my case, I needed to create one for S3, and one for Data Explorer.

The configuration for the S3 Linked Service was not intuitive. In fact, I spent time trying to figure out the correct combination of parameters before I landed on a configuration that worked properly. I landed on the following configuration for my target S3 bucket:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
  {                                                                                                                                                                             
    "etag": "redacted",                                                                                                                             
    "id":  "redacted",                                                                                                                                                                               
    "name": "s3",                                                                                                                                                               
    "properties": {                                                                                                                                                             
      "accessKeyId":  "redacted",                                                                                                                                    
      "additionalProperties": null,                                                                                                                                             
      "annotations": [],                                                                                                                                                        
      "authenticationType": "AccessKey",                                                                                                                                        
      "connectVia": null,                                                                                                                                                       
      "description": null,                                                                                                                                                      
      "encryptedCredential":  "redacted",                                                
      "parameters": null,                                                                                                                                                       
      "secretAccessKey": null,                                                                                                                                                  
      "serviceUrl": "https://s3.amazonaws.com",                                                                                                                                 
      "sessionToken": null,                                                                                                                                                     
      "type": "AmazonS3"                                                                                                                                                        
    },                                                                                                                                                                          
    "resourceGroup":  "redacted",                                                                                                                             
    "type": "Microsoft.DataFactory/factories/linkedservices"                                                                                                                    
  },

The aspect of this configuration that I spent the most time on was with the serviceUrl (line 13). I had assumed I could use the FQDN of my target bucket but because of the way the dataset is configured in ADF, there is no way to specific the root path of the bucket resulting in a data retrieval failure. For this definition, all I needed to specify was the accessKeyId, the secretAccessKey (which is then encrypted), and ensure that the serviceUrl is set to https://s3.amazonaws.com.

The next step was to create a Linked Service for the target Data Explorer instance. The authentication options for the Data Explorer Linked Service are Service Principal, System Managed, and User Managed Identity. In this case, I created an Application SP without any API permissions, generated a client secret, and passed it into the Linked Service definition.

{                                                                                                                                                                               
  "etag": "redacted",                                                                                                                               
  "id": "redacted",                                                                                                                                                                       
  "name": "dataexplorer",                                                                                                                                                       
  "properties": {                                                                                                                                                               
    "additionalProperties": null,                                                                                                                                               
    "annotations": [],                                                                                                                                                          
    "connectVia": null,                                                                                                                                                         
    "credential": null,                                                                                                                                                         
    "database": "primary",                                                                                                                                                      
    "description": null,                                                                                                                                                        
    "endpoint": "redacted",                                                                                                                   
    "parameters": null,                                                                                                                                                         
    "servicePrincipalId": "redacted",                                                                                                               
    "servicePrincipalKey": null,                                                                                                                                                
    "tenant": "redacted",                                                                                                                           
    "type": "AzureDataExplorer"                                                                                                                                                 
  },                                                                                                                                                                            
  "resourceGroup": "redacted",                                                                                                                               
  "type": "Microsoft.DataFactory/factories/linkedservices"           
}                                                                                                                                 

The JSON above doesn’t provide enough visual context, whereas the screenshot below is more clear.

ADF also has a feature to manage credentials for a Linked Service through the Authentication reference method that includes two radio buttons: one for inline (which is what I’m using) and the other for Credential - Eventually, I’ll migrate the inline to Credential so that I can manage the lifecycle a bit better.

Once both Linked Services were created, I could move on to defining the source and destination datasets. A Dataset in ADF is an object referring to and mapping that data that will be sourced or targeted. For my use case, I had two dataset objects, one for the S3 bucket, and one for the Data Explorer table that would contain the ingested the data.

Configuring the source dataset required a bit of exploration as the logs in my bucket are several years old, and I was curious how much the data varied over time. Comparing my data to the schema for the logs provided by Amazon, I noticed that the older data didn’t match. In fact, the schema for the older log files were missing seven fields. After a quick search, I discovered an Amazon blog announcing the release of the new fields as of December 2019. I decided that it wasn’t worth for me to ingest the new fields, so I defined the schema for the source dataset to the original 26 values.

Each of the log file starts with the following two lines:

#Version: 1.0                                                                                                                                                                  
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields

Ideally, I wanted to use the headers defined for each column specified in the second line, but because the line was prepend with #Fields:, it presented a problem for the schema parser (at least to my novice eyes). I decided to set the source dataset to skip the first two lines of each file, and to manually define the schema. In addition, I had to specify that the data was compressed using GZip, using the optimal compression level. The JSON definition for the source dataset is:

  "name": "redacted",         
  "properties": {                      
    "additionalProperties": null,      
    "annotations": null,               
    "columnDelimiter": "\t",           
    "compressionCodec": "gzip",        
    "compressionLevel": "Optimal",     
    "description": null,               
    "encodingName": null,              
    "escapeChar": "\\",                
    "firstRowAsHeader": false,         
    "folder": null,                    
    "linkedServiceName": {             
      "parameters": null,              
      "referenceName": "s3",           
      "type": "LinkedServiceReference" 
    },                                 
    "location": {                      
      "additionalProperties": null,    
      "bucketName": "redacted",
      "fileName": null,                
      "folderPath": null,              
      "type": "AmazonS3Location",      
      "version": null                  
    },                                 
    "nullValue": null,                 
    "parameters": null,                
    "quoteChar": "\"",                 
    "rowDelimiter": null,
    "structure": null,
    "type": "DelimitedText"
	

A few notes:

• The linkedServiceName includes the LinkedService that I defined earlier
• The bucketName in the location block is the simple value of the bucket without being prepended with https:// or s3://, or appended with the traditional s3 FQDN values.
• The column delimiter is specified as a tab character
• The type is defined as DelimitedText

The schema block immediately following the rowDelimiter key started as specified below:

    "schema": [         
      {                 
        "type": "String"
      },                

The String definition was defined a total of 26 times.

Once the source dataset was complete, I could move to the destination dataset. As for the source, this dataset also included a reference to the appropriate LinkedService (in this case dataexplorer).

  "name": "redacted",   
  "properties": {                     
    "additionalProperties": null,     
    "annotations": null,              
    "description": null,              
    "folder": null,                   
    "linkedServiceName": {            
      "parameters": null,             
      "referenceName": "dataexplorer",
      "type": "LinkedServiceReference"
    },                                
    "parameters": null
	

The last piece of this dataset block was the following definition at the end of the JSON:

"structure": null,              
"table": "redacted",    
"type": "AzureDataExplorerTable"

Compared to the source dataset, the type for this dataset is defined as AzureDataExplorerTable. The only other parameter that’s defined in this block is the one for the target table in the database.

With the prerequisites for the pipeline created and configured, the next step was to build the pipeline and define the step inside of it to copy the data. For the first dozen times that I stepped through this process, I used the various wizards and found them to be more complicated (and frustrating) compared to defining each of the components individually. With all of the components defined, it was straight forward to build the pipeline.

A Data Factory Pipeline is simply a grouping of activities. For this case, my pipeline only has one copy activity which uses the previously defined source dataset as the input, and the destination dataset as the output. The JSON to define the source reference is as shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
        "source": {
          "additionalColumns": null,
          "additionalProperties": null,
          "disableMetricsCollection": null,
          "formatSettings": {
            "additionalProperties": null,
            "compressionProperties": null,
            "skipLineCount": "2",
            "type": "DelimitedTextReadSettings"
          },
          "maxConcurrentConnections": null,
          "sourceRetryCount": null,
          "sourceRetryWait": null,
          "storeSettings": {
            "additionalProperties": null,
            "deleteFilesAfterCompletion": null,
            "disableMetricsCollection": null,
            "enablePartitionDiscovery": false,
            "fileListPath": null,
            "maxConcurrentConnections": null,
            "modifiedDatetimeEnd": {
              "type": "Expression",
              "value": "@startOfDay(utcnow())"
            },
            "modifiedDatetimeStart": {
              "type": "Expression",
              "value": "@startOfDay(getPastTime(1, 'Day'))\n"
            },
            "partitionRootPath": null,
            "prefix": null,
            "recursive": true,
            "type": "AmazonS3ReadSettings",
            "wildcardFileName": "",
            "wildcardFolderPath": null
          },
          "type": "DelimitedTextSource"
        }

The most important components in this block are that I used a * for the wildcardFileName key in line 33, and that I defined a function for the start (line 23) and end times (line 27) for the file modified timestamps. This would enable me to run the pipeline once a day without changing any parameters programmatically, and ingest only the logs from the day before without risk of duplication.

The next section of the copy activity was the sink (or target) configuration. Again, as for the source, this was as simple as selected the destination dataset. The JSON for the sink is as below:

        "sink": {
          "additionalProperties": null,
          "disableMetricsCollection": null,
          "flushImmediately": null,
          "ingestionMappingAsJson": null,
          "ingestionMappingName": null,
          "maxConcurrentConnections": null,
          "sinkRetryCount": null,
          "sinkRetryWait": null,
          "type": "AzureDataExplorerSink",
          "writeBatchSize": null,
          "writeBatchTimeout": null
        }

As you can see, other than the type being auto-populated, the pipeline is simply going to use the destination dataset defined in the output. Once the sink was defined, the next configuration section for the copy activity was to define the mapping between the source and the destination. For this definition, as I was configuring the pipeline using the UI, I clicked on the Import Schemas button and crossed my fingers.

After waiting for about a minute, I was greeted with an update to the UI that presented the schema mapping exactly as I intended.

This was it! 🥳 I left the remaining settings as their default, and manually triggered the pipeline. I waited patiently (for almost three and a half hours) until the pipeline was complete…

I really couldn’t believe that after working on building this out for a couple of weeks that the pipeline succeeded, and that the data was now available for me to query in Data Explorer using KQL.

For example, if I wanted to see the latest ten records sorted by timestamp of non-bots (at least from their user agent strings) of request to html content, I could make the following query:

tablename
| extend timestamp = todatetime(strcat(['date'], 'T', ['time']))
| where ['client-useragent'] !contains 'bot'
| where ['request-uri'] contains 'html'
| sort by timestamp desc
| limit 10
| project timestamp, ['edge-location'],['request-uri']

The result from the query is presented to me in 73 ms.

The best part about this solution is that it doesn’t require any additional JavaScript to be imported into the client’s browser. The data logged is from the HTTP requests sent to CloudFront which is providing the CDN ahead of the S3 bucket where this blog is being hosted.

If you made it this far, thank you! I really enjoyed putting this solution together. I learned a ton through the process about services in Azure that I never spent time with before. As I continue building queries and better reporting from the data, I’ll follow up with another post comparing the quality of the data to that which I typically see from Google Analytics.

Given that mutt is the industry standard for an open-source terminal based e-mail client, I knew that I wanted to use it as my point of user interaction with my e-mail. My second requirement was a unified inbox, parallel to my interaction in K-9 Mail preventing the need to navigate from inbox to inbox for new e-mail. After researching solutions, the one that was frequently returned in my search results was Notmuch. Notmuch is an incredibly fast, full text searchable system for e-mail. Its frontend is an API and a collection of CLI tools which absolutely exceeds my requirements. Once I decided that I was going to use notmuch, I looked for an integration with mutt, and discovered that it wasn’t as clean as I hoped. I researched further and discovered NeoMutt! The integration between the two was built in 2016 and includes virtual folders, which solves my unified inbox requirement. The last requirement was the ability to sync each complete e-mail account locally, do what I need to do with it, and then “push” the changes back up when I’m done. This makes e-mail management an activity that happens a couple of times a day, versus real-time (which is an incredible distraction).

I spent a considerable amount of time configuring each of the tools to work together, and generally, my workflow is as easy as the one I use on my mobile with K-9 Mail. On a regulas basis, I pull e-mail down from each of the accounts on my mobile (when on the move), review all of the new mail in a single unified inbox, and then mark for follow-up (and archive), archive, or delete, achieving “Inbox Zero”.

The issue I had with the solution was how Google implemented IMAP for their e-mail product. Through the support of various IMAP extensions, they were able to dynamically build IMAP folders from their special labels as well as labels that a user creates through the UI. For example, an excerpt of the system labels:

The challenge with this implementation is that any and all mail delivered to a given address is stored in All Mail unless explicitly expunged (a.k.a. deleted). Research by the offlineimap team determined that to delete an e-mail completely from Gmail, it needs to be “moved” to Trash and deleted from all other folders.

Were you able to figure the problem out?

If not, I’ll help.

• Gmail only uses one folder, and adds labels to each piece of correspondence indicating its folder
• Standard IMAP protocol only understands folders

Because of these two conditions, offlineimap pulls the same message for every folder that it exists in and saves it as a seperate file.

Note that the same piece of correspondence has two records; one in inbox, and one in archive. If I run a diff on both files, the output is the following:

The only difference is that the entry in the ‘All Mail’ folder has an \Inbox keyword, compared to the record in the Inbox folder having no labels at all. This now became a relatively simple search and replace problem that could be solved on the command line with one of my favorite tools, sed. I put together a list of steps that the script needed to take.

# Steps required to properly delete e-mail via IMAP (not JMAP) from GMail
# supported accounts, which also includes accounts hosted through Google
# Workspace
# 
# Assumptions, using offlineimap v8.0.0 with imaplib2 v3.0.6 and notmuch v0.37
# E-mails to be deleted are tagged with 'trash'
# 1. Find e-mail files that are not currently in trash and are tagged trash
# 2. Delete any records of each that are in folders other than archive
# 3. Remove all X-Keywords from records that are in archive
# 4. Remove MAILDIR hash from filename
# 5. Move the result to the corresponding trash folder in the correct account

Once I had the steps down, it became a development effort. A couple of loops, variables, and a lovely set of Shell Parameter Expansion techniques later, (kudos to @hsanson with their comment in a neomutt thread discussing a similar issue that they had solved), I built out the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Tidy to Trash
for i in $(notmuch search --output=messages not path:/trash/ and tag:trash)
  do 
    for j in $(notmuch search --output=files tag:trash and $i)
      do 
        ACCOUNT=$(echo $j | awk -F/ '{print $5}')
        DIR=$(echo $j | awk -F/ '{print $6}')
        if [ $DIR != 'archive' ]
        then 
          /usr/bin/systemd-cat echo "Remove $j in $DIR"
          rm $j
        elif [ $DIR == 'archive' ]
        then
          /usr/bin/systemd-cat echo "Remove X-Keywords from $j"
          sed -in 's/X-Keywords.*/X-Keywords:/g' $j
          FILE=${j##*/}
          FILE_NOUID=${FILE/,U=[0-9]*:/:}
          FILE_NOUID=${FILE_NOUID/,U=0-9]*/}
          /usr/bin/systemd-cat mv -uv $j ~/mail/$ACCOUNT/trash/cur/$FILE_NOUID
        fi
    done
done

I had used simpler expansions in the past on other scripts, but the set that was built out in lines 16 through 18 used regex to solve an issue that would be much harder otherwise. Per the implementation by offlineimap3, the maildir name is hashed using MD5 and appended to the filename of each piece of mail. During my testing, when I was moving files locally, and syncing the changes with the remote, I found duplicate mails and other inconsistencies. I realized after reviewing this piece of code that the mismatch of the hash to the changed folder was causing the issue. By removing the hash as part of the tidy operation, this ensured consistency between the local and remote mail repositories.

The entire script is in a gist for those that are interested and I’ll continue to update it if I encounter edge cases that need to be dealt with. I reused the trash loop to create an archive loop so that I can use a different notmuch tag (‘archive’) to handle moving mails from inbox to archive when in my unified inbox virtual folder.

I use three macros in my neomutt configuration for when I’m in a virtual folder.

macro index,pager \ca "<modify-labels>+archive\n<next-entry>" "+archive"
macro index,pager \cd "<modify-labels>+trash\n<next-entry>" "+trash"
macro index,pager \cu "<modify-labels>-trash\n" "-trash"

I won’t comment on how many hours I spent researching and eventually building this solution; it was a lot. I learned a lot about IMAP, and in general, how my mail tools could be configured to work together. I give a lot of credit to each of the development teams that maintain the tools that give me the opportunity to manage my e-mail in a way that’s convenient for me.

As a daily Linux user, I spend as much time as I can interacting with communities, services, and systems through a terminal window. The only graphical UI that I use is a browser because Links has a tough time rendering images¹. My daily setup for interacting with e-mail is through a combination of OfflineIMAP, NeoMutt, [msmtp][5, and notmuch. By orchestrating the configuration of these tools, I’m able to effectively manage e-mail through a single interface, completely through the keyboard in a terminal window. The challenge of using WorkMail was that while it satisfied my requirement (as I rarely interacted with e-mail through their UI), my family members dreaded the experience (in comparison to the other large e-mail providers). This led me to find a solution that would satisfy my requirement along with their’s.

Well, Armen, why didn’t you just use O365 as a replacement? Because Microsoft is deprecating personalized e-mail addresses by November 30, 2023.

It was actually my first choice and I was really disappointed when I read the announcement. I considered standing up an Azure Active Directory instance, and building out O365 against it but decided that it would be complete overkill for the family’s needs. I wanted to keep the management overhead as low as possible. In addition, if I ever wanted to apply family controls to the boys’ Microsoft accounts, I wouldn’t be able to as you cannot associate an e-mail created through AAD to a personal Microsoft account. At this point, setting up a Google Workspace was my last option.

Once the decision was made, setting up the Workspace was straight forward and updating the domain MX records were seamless. The concern I had was how to migrate 6K e-mails from one service to another by using my existing toolset. The alternatives were to use Google’s Data Migration Service or keep the WorkMail account live for a period of time while Google imported e-mail over time via… POP3.

Enter OfflineIMAP.

By design, OfflineIMAP adheres to the IMAP standard which is enabled by default for both WorkMail and Workspace. My assumption would be that I could sync the contents of my WorkMail account locally one last time with OfflineIMAP after MX was updated and then use it again after a small configuration change to push the contents to my Workspace account. Without any customization, OfflineIMAP creates a Maildir structure for each account that’s configured during a synchronization using the exact name of the folders in the service. This presents a bit of an issue when your intent is to migrate contents from one directory structure to another.

When I first adopted my e-mail toolset, I spent a considerable time leveraging the nametrans and folderfilter options of OfflineIMAP to bring consistency when interacting with my e-mail (especially across accounts).

[Repository kaleshian@local]
nametrans = lambda folder:  re.sub('archive', '[Gmail]/All Mail',
	re.sub('drafts', '[Gmail]/Drafts',
	re.sub('flagged', '[Gmail]/Starred',
	re.sub('trash', '[Gmail]/Trash',
	re.sub('spam', '[Gmail]/Spam',
	re.sub('inbox', 'INBOX', folder))))))

[Repository kaleshian@remote]
nametrans = lambda folder:  re.sub('\[Gmail\]\/All Mail', 'archive',
	re.sub('\[Gmail\]\/Drafts', 'drafts',
	re.sub('\[Gmail\]\/Starred', 'flagged',
	re.sub('\[Gmail\]\/Trash', 'trash',
	re.sub('\[Gmail\]\/Spam', 'spam',
	re.sub('INBOX', 'inbox', folder))))))

folderfilter = lambda folder: folder in ['[Gmail]/All Mail',
	'[Gmail]/Drafts',
	'[Gmail]/Starred',
	'[Gmail]/Trash',
	'[Gmail]/Spam',
	'INBOX']

There are two blocks above for the account definition in my OfflineIMAP configuration. The first is for the local definition, and the second is for how the service will interact with the IMAP directory structure remotely.

In this case, the local nametrans (or name transformation) is exactly as defined in the remote block, except that it’s reversed. The intent of each definition is to take the remote server structure, and convert it to a user defined standard locally. For my case, I wanted the standard e-mail containers (e.g., inbox, drafts, flagged, trash, and spam) to all the look the same regardless of what account I was reviewing mail in.

The remote for WorkMail had a very different structure, but because I had ensured that locally, all of the accounts had matching directory structures, I was able to point the localfolder for the new remote target to the same directory structure, and let OfflineIMAP sync the contents to the Workspace account.

Overall, I was really pleased with the process and how well OfflineIMAP continues to work for my needs.

An Availability Set is a “logical grouping of VMs that allows Azure to understand how your application is built to provide for redundancy and availability.” When you create an AS in a given region, you’re effectively stating to Azure that the VMs inside of that grouping require orchestration as it relates to how they’re affected by a fault to the underlying compute, network, or storage (at the hypervisor level) or by an update to the hypervisor itself. When either of these events occur, it’s Azure’s responsibility based on the association of the VM to an Availability Set that determines how the VM is handled compared to its workload peer VMs. To handle each of these cases (faults and updates), Azure provides the ability to define how many fault and update domains an Availability Set can manage for a given region. A fault domain is a grouping of virtual machines that share a common power source and network switch. In addition, VMs are also aligned with disk fault domains which ensures that the managed disks associated to those VMs are located (effectively) on unique storage.

It’s this association that determines whether a given region can have an availability set with two or three fault domains when it’s instantiated.

As a Bash shell enthusiast, I figured I could whip up a script to dynamically generate the list by brute forcing the creation of an availability set against every region that my Azure account has access to with the fault domain variable set to 3, and so I did.

1
2
3
4
5
6
7
8
9
for i in $(az account list-locations -o tsv --query [].name); do 
if az group create -o none -l $i -n rg-as-$i 2> /dev/null; 
then 
if az vm availability-set create -o none -n as-$i -g rg-as-$i -l $i --platform-fault-domain-count 3 --platform-update-domain-count 3 2>/dev/null; 
then echo "$i,supported"; 
else echo "$i,unsupported"; 
fi;
fi;
done;

The script is relatively straight forward. Assuming that az cli is configured correctly, the loop starts with listing all of the regions that the account has access to, and then for each one, creates a resource group, and if successful, attemps to create an availability set in that region with the fault domain variable set to 3. If successful, it prints the region and that it’s supported. If not, unsupported.

At this time, only 16 regions supported three fault domains. They are:

• eastus
• eastus2
• southcentralus
• westus2
• westus3
• northeurope
• swedencentral
• westeurope
• centralus
• centralindia
• japaneast
• canadacentral
• francecentral
• brazilsouth
• northcentralus
• westus

I’m curious how this list will change over time. I’m glad that my customer had this requirement because it provided an opportunity for me to learn more about Availability Sets and how their implementation can vary between regions.

DNS over TLS was introduced as part of RFC 7858 in 2016

DNS over HTTP was introduced as part of RFC 8484 in 2018.

Both of these specifications were designed for the purpose of “mitigat[ing] both passive surveillance RFC 7258 and active attacks that attempt to divert DNS traffic to rogue servers (see Section 2.5.1 of RFC 7626).” While their implementations are slightly different, the method of their mitigation is similar. Cloudflare has an overview of the two implementations in their learning center that is worth a read if you’re interested in the technical detail.

At a high level, an upstream resolver listening for DNS queries over TLS is simply communicating with the caller using the standard name resolution protocol but wrapped by a negotiated TLS connection encrypting the content of the query and the response. DNS over HTTP sends a DNS query using the standard HTTP format and the response is delivered as contents with a corresponding content type. This method provides the added protection of blending in with standard web queries as the traffic is delivered over the same ports. The challenge with incorporating DoH versus DoT is that it requires a proxy to forward standard DNS queries from the trusted network to the upstream resolver.

If I chose the easy route to implement DoT, I would be able to configure resolved (part of systemd)9 to forward queries over DoT through the addition of a single key value pair configuration setting DNSOverTLS. As long as the upstream resolver supports DoT, then this value can be set to true (it takes a Boolean, or the value opportunistic). With resolved configured properly, and running on the RPi, the final step is to configure the upstream DNS server to point to the local resolved listener. In the screenshot below, I configured pihole to use the DoH proxy which is listening on port 5053.

Given that my requirement was to implement DoH for my local network, I set out to find a DoH proxy that I could easily implement on the RPi. I had considered configuring Unbound a number of times over the years, but the overhead never seemed worth it. Installing and configuring BIND also seemed like overkill especially considering pihole was using Dnsmasq under the covers. I decided to re-read the pihole documentation to see if the community had shared any other alternatives.

In the official documentation, under the guides section, I discovered cloudflared. Apparently, the amazing folks at cloudflare wrote an open source’d tunnel client to support their customers connecting to their resources over a private connection. A small feature of this client is to listen for standard DNS requests, and proxy them to a DoH resolver. 🎆

I followed the documentation provided by the pihole folks to the last period and in a few minutes, I completed my requirement of sending DNS queries completely encrypted to my resolver of choice. pihole continues to perform as exactly as designed. I discovered from digging through the cloudflared repository that the daemon has a metrics endpoint which I may investigate in the future.

Thanks to the folks at pihole and at cloudflare for contributing to the community!

With the requirement from the customer that I would not be able to augment their dataset, I had to find a way to perform an exact comparison of a value set for a tag against the dataset.

For a problem like this, I start by reviewing the publicy available documentation and this case was no exception. The Azure Policy definition documentation was comprehensive and specifically stated that outside of a dozen functions, all of the Azure Resource Manager template functions could be used within a policy definition. The template functions were nicely grouped by category and I reviewed them to see if there was one that would provide the comparison I needed. There were only two that raised my interest, contains and startsWith, both of which had inherant flaws in their implementation for what I needed them to do. contains was case-sensitive when comparing strings, but because of its lack of precision (by design), it could produce false positives (or negatives) during tag evaluation. startsWith had a similar flaw but was also case-insensitive. At first glance, none of the functions listed in the strings section would satisfy my requirement.

I spent time thinking about the problem over the next two days comparing this problem to ones I solved in the past. Years ago, when I was at RSA, spending time designing and implementing PKI for customers, I quickly learned the two most common formats for an X.509 certificate; DER, which was binary encoded ASN.1 and PEM, which was significantly more human readable. I lost count of how many times I sent a DER encoded certificate to output on the terminal and its bell ringing incessantly until it reached the end of the file. It was during this time that I learned the power of base64 encoding.

For those unfamiliar, “base64 is a group of binary-to-text encoding schemes that represent binary data (more specifically, a sequence of 8-bit bytes) in sequences of 24 bits that can be represented by four 6-bit Base64 digits”¹. In layman’s terms, providing a means to encode binary data in a human readable form. While our requirement for this use case is to compare string data, the nuance of character case needs to be maintained. In reviewing the the base64 alphabet, we observe that case can be preserved.

We can easily create a policy that would determine a precise match for a tag. Using the policy provided in the Microsoft example as reference, one could perform a comparison against a set of tags in the policy definition parameters using the base64 function.

1
2
3
4
5
6
7
8
9
10
11
12
      "if": {
        "allOf": [
          {
            "field": "type",
            "equals": "Microsoft.Resources/subscriptions/resourceGroups"
          },
          {
            "field": "[concat('tags[', parameters('tagName'), ']')]",
            "notEquals": "[base64(parameters('tagValue'))]"
          }
        ]
      },

Once the function was discovered, and we tested it successfully, we were quite pleased at how well it worked. The lesson learned here is that experiences from the past can be leveraged to solve problems in the present.

Azure Monitor is “a comprehensive solution for collecting, analyzing, and acting on telemetry”¹ from within Azure (and on-premise environments). By default, the solution provides you access to the platform activity log which provides insight into the activity against an Azure subscription. However, the data is only retained for 90 days which does not adhere to most enterprise data retention standards.

Enter Log Analytics Workspaces (LAW).

At a high level, LAW “is an Azure resource and a container where [log] data is collected, aggregated, and serves as an administrative boundary.”² for analysis of the data. Log retention is typically the first checkbox that’s marked as part of an audit requirement whether an organization has workloads on-premise or in the public cloud. The challenge is how to make the logs you have actionable. One of the immediate benefits of directing logs to LAW is that it’s query engine is built on top of Azure Data Explorer which is powered by the Kusto Query Language (KQL). In comparison to other query languages (e.g., SQL, GraphQL), Kusto uses a similar syntax pattern that has a low barrier to entry. For example, once you create a workspace and start sending log data to it (after a brief delay for log ingestion), you’ll have a list of tables in the left pane of your workspace as shown below.

Once the tables are visible, stitching together a useful query is straight forward. You can double-click on the table name or start typing it in the query window.

The above short query counts the number of rows in the AzureActivity with a timestamp less than or equal to 24 hours (as defined by the time range selector at the top of the query box). The extent and expanse of KQL is immense. If you’re interested, I highly recommend reviewing the KQL documentation in depth and practicing each of the skills listed in it.

My customer’s requirement was to only send security event data from Azure to their SIEM.

Right. With KQL, this was easy.

After setting up a couple of Windows VMs in my lab environment and configuring them with the Log Analytics Agent, I observed a new table in my LAW named Event and was able to filter on it with examples from the Azure documentation. In addition, I configured Azure Sentinel to receive Security Event data from the instructions in the documentation. Once these events were ingested, I observed the SecurityEvent table in the list of LAW tables. At this point, I knew I had the data I needed.

The next step of the solution was to build automation to poll the LAW with a predefined query to gather only the required events, and send them to Event Hub (where they would be polled by the SIEM).

Enter Logic Apps.

Logic Apps “is a cloud-based platform for creating and running automated workflows that integrate[s] apps, data, services, and systems.” Effectively, a low-code automation solution to perform exactly what my customer needs. I scanned the built-in connectors and found one called Schedule that had a trigger called Recurrence.

Awesome. I had my polling mechanism. I created a logic app, and added the trigger.

The next step was to pull data from my LAW… was there a connector for Azure Monitor?

Yes!

I added the connector, selected the Run query and list results trigger…

… and configured it using the query I had honed within my LAW.

The only nuance was that the time window of the query had to match (or exceed) the interval frequency of the recurrence trigger to ensure that events wouldn’t be missed.

The next few steps required a bit of analysis (read: trial and error). The reason for the churn was because I didn’t quite understand how the records would be fed to my Logic App from the connector. I had assumed that it would be returned as a single blob that needed to be iterated on. I discovered that each row from the query required parsing and had to accommodate that step in my workflow. I discovered the For each action and configured it to take the value from the previous step and pass it to a step that would parse the JSON string.

Configuring the Parse JSON action was… painful. For Parse JSON to be able to understand the incoming JSON object, it effectively needs to know the object structure, the keys, and what types the values will be (e.g., integer, string, etc.) The easy way is to follow the instruction in the action box and to “Use sample schema to generate schema” but we all know that a single example isn’t going to be comprehensive of the data.

I had to find another way.

I tried to export the schema of the source table from the LAW, but it didn’t produce an output that could be automated. I tried using the az cli, piping the output through jq, but that didn’t produce an output that could be directly transferred to the action. I was stuck, and frustrated, but was running out of time, so I conceded and massaged the most reasonable output from the above three options by hand, and produced the following two schemas:

• event.json
• securityevent.json

At the publication time of this post, these two schemas were valid. I was able to create two separate logic apps with these two schemas, and appropriate adjustments to the log query (to respect the table of data that the schema was applicable to).

Once the Parse JSON action was configured, I discovered that I needed an additional step to aggregate the events into a single object before passing it to Event Hub. This action was the responsibility of the Compose action which only required a single argument of Body from the previous For Each.

The last step was to send the completed message to Event Hub which required the Send event action in the Event Hubs category of connectors.

With the last step of the workflow configured, I was able to save the design and execute the workflow successfully. Through this process, I interacted with components of Azure that I would never expect (and was pleasantly surprised to). It reinforced my perspective that without a use case, it’s quite difficult to learn about how public cloud services can be configured (read: cobbled together) to achieve a desired outcome.

Happy solutioning!

Jekyll is an awesome, content management system that requires very little infrastructure. The entire project with extras takes up less than 150MB on disk. My workflow is to develop and test new content on my workstation and checking changes with git when appropriate. Once I’m ready to publish, I push my local repository changes upstream to a CodeCommit repository which kicks off a CodeBuild project. (For more information on how this is configured, check out the post Magic of CloudWatch Events and CodeBuild) When I first started, the complete build and publication to S3 was taking on average 90 seconds which was great. Since then, due to the addition of a couple of extra Jekyll plugins, the time now averages 225 seconds. That’s a 250% increase! I knew that I had to wrangle this issue before it became worse.

The first step in any performance evaluation is to benchmark each step. Thankfully, CodeBuild tracks the start, and end times for each step of the build so I didn’t have to look far. Once I looked at the data, it was pretty obvious where the problem was.

It was time to dig into the build phase of the project to understand why it was taking so long. Enter the CloudWatch Log integration with CodeBuild. For each build in a project, CodeBuild creates (if configured properly) a specific log stream for each build. The contents of the log stream includes any stdout/stderr output during every build phase. As soon as I reviewed the log stream for the last build, I knew why the build phase was taking so long.

In my buildspec.yaml for the project, because I’m using the default AWS container for the build, I am required to install my runtime (in this case ruby), and its respective dependencies. This process also includes bundler, all of the Jekyll gems, and their dependents. Have you figured it out yet? For each build, the container was having to reach out to the Internet, pull all of the gems, and install them before it could even start processing my treasured blog posts. If I could find a way to cache them, I could improve the build time considerably!

Enter CodeBuild’s Cache functionality.

There are two options: local caching, and S3 Local caching wasn’t viable for me as the builds are too infrequent for it to make sense, so I went down the path of using a specific S3 bucket for this cache.

Since I use CloudFormation to manage all of my resources, I updated the template with a couple of changes. The first was to add the S3 bucket:

CodeBuildCacheS3:
  Type: AWS::S3::Bucket
  Properties:
    BucketName:
      !Sub '${AWS::AccountId}-codebuild-cache'

Next was to add the appropriate permissions to get and put objects into the bucket by the role assumed by CodeBuild during execution:

Effect: Allow
Action:
  - s3:PutObject
  - s3:GetObject
Resource: !Sub
  - '${CodeBuildCacheS3Arn}/*'
  - { CodeBuildCacheS3Arn: !GetAtt CodeBuildCacheS3.Arn }

The last piece was adding the relevant cache block to the CodeBuild project:

Cache:
  Type: S3
  Location: !Ref CodeBuildCacheS3

Once I updated the stack, I was eager to re-run the same build to observe the changes by adding the cache…

I did, and didn’t notice any difference at all. I went back to look at the logs and found an entry that indicated a cache miss.

[Container] Unable to download cache: NoSuchKey: The specified key does not exist.

I continued reading through and discovered the following entries in the post build phase:

[Container] Entering phase POST_BUILD
[Container] Uploading S3 cache...
[Container] Phase complete: POST_BUILD State: SUCCEEDED

Awesome! A process within the post build phase was able to upload the contents I had specified to be cached to S3. The file had a unique name using the standard format for Universally Unique Identifiers (UUID). After downloading the file to my local machine, I observed that it was a simple unencrypted tarball.

After unpacking it, I discovered a flat structure (no directories) containing files with numbers for names and a file labeled codebuild.json.

codebuild.json is a properly formatted JSON object with the following header:

{"version":"1.0","content":{"files":[

The header is then followed by JSON objects indicating the path of the file with its proper name. Based on my interpretation and comparison, each JSON object maps to the file in the package starting from the first file number in the package.

Once the cache was populated, each subsequent build and publish was averaging 60 seconds. I was really pleased with how easily I was able to optimize this build process. While this build is small, the impact caching has on large scale builds is significant. For those of you that are using CodeBuild, and are struggling with long build times, I highly recommend looking into using the cache option.

Azure was different. In some areas, it was better, and in some ways, it wasn’t. The most significant difference for me (and one that I continue to struggle with) is that the cornerstone of Azure is Azure Active Directory (AAD). For as long as I can remember, Active Directory was my nemesis. It’s probably the reason I spent more time administrating Linux hosts. In AWS, IAM provides a concise set of controls that are responsible for authentication and authorization of principles against the platform. AAD, in comparison, is responsible for many more features. While AAD “is Microsoft’s cloud-based identity and access management service”¹, it is capable of a lot more, inherently by its design. For the professional without AD experience, discovering the robust integration between Azure and AAD was overwhelming.

Weeks turned into months, and I continued to learn about each of the core services in Azure. I compared them to the services I knew well in AWS and practiced configuring them similarly. I continued using Microsoft’s Learning site for official guidance, and leaned on peers that I was working with to fill in the solution architecture gaps. The challenge with learning individual services is that you treat them as standalone entities. In the real world, they are rarely configured as individuals and part of a larger solution.

Enter the AZ-303 and the AZ-304, the two gates to obtaining the Azure Solutions Architect Expert certificate.

Both exams are designed to measure your solution architecture skills in unique ways. The AZ-303 focuses on implementation of solutions, versus the AZ-304, which focuses on the design of solutions. For those of you that are reading this who are veterans in the industry, you understand the nuance between these two definitions.

In February, I registered for both exams, two weeks apart giving myself the time to prepare. I thought that reading through the public documentation, performing the exercises on learn, and experimenting in my own subscription would be enough to achieve a passing grade. I was wrong. I failed both exams.

When I reviewed the score report, I was expecting the areas for improvement to be with Identity (and Security) correlating it with my weakness in AAD. While there were questions that were AAD specific, it wasn’t the reason I didn’t pass.

For the AZ-303, I needed to improve my data platform knowledge.

For the AZ-304, my monitoring and business continuity knowledge.

Earlier this month, I took both exams again, and passed.

For the AZ-303, an area of improvement, data platforms, became my strength.

For the AZ-304, similarly to the AZ-303, monitoring turned into a strength, closely followed by business continuity.

While I will admit that failing the exams on the first try gave me a baseline of where I should focus my study, the most significant impact on my scores for the second attempt came from real world experience. Over the past few months, I spent the majority of my time working with customers in defining their strategic design and implementation plans for observability and business continuity in Azure. Their real world requirements reinforced the book knowledge that I had gained in my preparation for the exams.

Through this process, I learned and was reminded of why I love this industry. While the technology may change, the experience of actually using them in the real world is where there is value. (The sentiment is absolutely applicable to any industry) As always, I will continue to build my personal trove of experiences to draw on for reference.

From the top, Azure manages access to resources through two planes: control and data. The control plane provides the subscriber to instantiate a resource like a virtual machine, a storage account, a key vault, or an instance of a managed HSM. Once the resource is created, operations against the resource are performed through its respective data plane. The Azure control plane and data plane article is a great read if you’re interested in more detail on this topic.

In the case of Key Vault and Managed HSM, they are managed through two different data planes:

• https://{vault-name}.vault.azure.net for Key Vault
• https://{hsm-name}.managedhsm.azure.net for Managed HSM

This means that when you’re interacting with either of these services through the Azure Portal, or through any of the programmatic interfaces, the respective REST API calls to each service are being made against one of these two endpoints. To add complexity to this topic, while the endpoints are different, the API accessible to the caller is the same: Key Vault REST API

I imagine the efficiency in mapping the existing Key Vault REST API to the underlying Managed HSM API was significantly higher by the engineering team during implementation. It also provides them the ability to extend the capabilities of the Managed HSM to more than key objects in the future.

Now, how does authorization fit into this?

Up to this point, we identified that the only difference between the Key Vault and the Managed HSM is the data plane REST API endpoint. How does Microsoft map the caller’s request to the authorization model? Through the Resource Provider operation list of course! This list identifies (in theory - remember this post) all of actions a caller can be evaluated for by the RBAC model against a given resource.

For example, there is no differences in the createKey REST API call for key vault or a managed HSM; they are the same. This aligns with the Microsoft.KeyVault/vaults/keys/create/action permission in the list of actions for the Key Vault API as there is no corresponding action for a Managed HSM target. By design, Microsoft is using the same API, simply with a different resource.

When KrisTurk86 mentioned his findings, and after spending time performing this research, it made sense why the Key Vault Adminstrator role I suggested wouldn’t provide the right access. He discovered that the Managed HSM Crypto User and Managed HSM Crypto Officer roles were the two default roles that contained the permitted actions against the Managed HSM. When reviewing the matrix of the default Managed HSM roles, it helped reinforce the nuance of the RBAC for these two resources that are incredibly similar.

Combining the REST API and Azure RBAC references is incredibly helpful in determining least privilege on the platform. As you build out your environment, it’s easy to use the built in roles to get the job done (as happened to me in the previous post) but to define least privilege and improve your security posture requires a focussed effort.