[AZURE 70-534 Cheat Sheet and Exam Notes Part-4]Design advanced applications

Series Index 

1. Azure ARM , networking and GLOBAL Infrastructure (2017-10-01)
2. Securing Resources and Azure Security (2017-10-01)
3. Design an application storage and data access strategy (2017-10-01) 
4. Design advanced applications (2017-10-01)
5. Design Azure Web and Mobile Apps (2017-10-01)
6. Design a management, monitoring, and business continuity strategy (2017-10-01)
7. Architect an Azure Compute infrastructure (2017-10-01)

High Performance Computing (HPC)

1. HPC Option in Azure 
1. HPC Pack (On Premise)
2. HPC Pack Hybrid 
3. HPC Pack IAAS 
4. Azure Batch 
2. Compute Intensive VM Series
1. A9, A10, A11 are HPC compute VM 
1.  Support Netwok Intensive application 
2. Support Windows and Linux both
2.  A8, A9 instances support RDMA
1. allow Remote Direct Memory Access 
2. VM are connected by low latency high throughput network 
3. Very much suitable for Message Passing Application  
3. Scaling in not possible in between Non Compute and Compute Optimized Instance and visa Versa
3. HPC not the same as big data:
   • Big data analytics is usually bounded by data volumes and so network IO.
   • HPC usually CPU-bounded.
4. HPC good for financial modelling, media encoding, video and image rendering, smaller compter-aided engineering models, etc.
5. HPC instances are A8/9 (network optimised – high-bandwidth RDMA network 32Gbps within cloud service as well as 10Gbps Ethernet to other services) and A10/11 (compute intensive).
6. Both 8/16 cores, 56/112GB RAM, 382GiB disk.
7. Microsoft HPC Pack 2012 R2 SP1 on Windows Server (on-premises, in Azure or hybrid) – Message Passing Interface (MPI) used (over RDMA network).

Create Long Running Application

Azure Batch

1. It is fully managed HPC in cloud 
2. Auto scale based on compute demand 
3. Light Weight REST Interface 
4. can be used by any cloud application to run schedule job 
5. Azure media services are built on top of it 
6. Support User management, Quota , Budget, Analytics, and Client Plug-In
7. Provide Batch SDK that can be used to build Job Execution Pipeline 
8. Key Concepts 
1. Creates a separate batch account
2. Create a Task Virtual Machine 
3. Create Pool of TVMs 
1. Pool Can be resized as required 
2. Different pool can be based on different user profile 
4. Work Item - How jobs run in TVM pool 
5. Task - Steps to complete job
6. File - Input Data for Task
1. Uploaded to Azure Storage
2. Azure Batch transfer to TVM for processing 
7. Batch App- manage, run and monitor repetitive batch jobs 
8. Batch Application - Name of repeatable batch job 
9. Application Image - a Zip file containing application executable and support file 
10. Cloud Assembally - Zip File that includes 
1. Method to break jobs in  to task 
2. Method to invoke application executable 

Web Jobs 

1. Supported by Web App and Process on same machine good for short term asynchronous jobs 

Worker Roles 

1. It is managed execution environment for long running task that do not required direct input 
2. Azure handles most of infrastructure and task runs in asynchronous mode
3. Used an alternative of HCP for for compute intensive tasks
4. Worker roles can be deployed either via console or visual studio 
5. Run on PAAS VM (Rather IAAS VM)
6. Scale as per demand.
7. Limited RDP is supported diagnostics and trouble shooting 
8. Worker Role Vs WebJobs 
1. Worker Role is processed on separate VM
2. Web jobs are processed on same machine where WebApps are deployed so not suitable for compute intensive jobs
3. Worker role can scale independent of Web Roles

Azure Function 

1. Azure Functions is a serverless compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure.
2. Azure Functions is a solution for easily running small pieces of code, or "functions," in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it.
3. Functions can make development even more productive, and you can use your development language of choice, such as C#, F#, Node.js, Python, or PHP. 
4. Pay only for the time your code runs and trust Azure to scale as needed. Azure Functions lets you develop serverless applications on Microsoft Azure.

Azure Scheduler

1. Azure Scheduler allows you to declaratively describe actions to run in the cloud. It then schedules and runs those actions automatically. Scheduler does this by using the Azure portal, code, REST API, or Azure PowerShell.
2. For extremely busy retail periods, like the annual Black Friday period, it's feasible that the amount of data could more than double on a particular day, forcing the job to exceed a 24 hour run time. When scheduler starts the next job, the previous job will not be complete and scheduler may pick it up again 

Integrate Azure services in a solution

Design connectivity for hybrid applications

Azure Provide flowing option for connectivity and discussed in networking section of this series  

1. Express Route 
2. Service Bus relay 
3. BizTalk Hybrid Connection Manager 
4. VPN

Azure BizTalk Hybrid Connection (ABHC)

1. Hybrid Connections is a feature in Azure BizTalk Services which allows you to connect Azure Websites and Azure Mobile Services to on-premise resources “as if they are on the same private network”.  
2. To do this you create a hybrid connection and then pair it with your cloud resource.  On premise you would install a software agent which will open up a connection to the cloud and you can configure the hybrid connection to be able to then connect to an on premise resource such as a database.
3. Your application in the cloud would then be able to connect to the database as if it’s in the same data centre except that the connection string will be modified so that the connection is routed via the BizTalk Hybrid Connection. Read More Here ...
4. Alternative to Service Bus Relay  
5. Multiple app services can share a connection 
6. Unlike Service Bus Relay Support multiple framework like don.net and node 
7. Only Work with static TCP port and od not support dynamic TCP like Passive FTP
8. Do not Support UDP
9. Security
1. Shared Access Signature is used for Azure App and On Premise Hybrid connection manager 
2. Application level keys for On Premise Hybrid connection manager 
3. key can be rolled and managed independently  
4. Needed Outbound TCP or HTTP on Premises Network to Internet 
5. Inbound firewall connection configuration not required because it works with outbound connection only 
6. Key Port are 9352,5671,80,443
10. ABHC- SQL Server Limitations
1. It is not advised to used SQL express with ABHC in production 
2. Multi subnet Failover not supported 
3. ApplicationINtent =read Only not supported 
4. SQL Authentication must be enabled for ABHC connection

Service Bus Relay 

1. Azure Service Bus Relay is a cloud based service which allows you to host an endpoint in the cloud, Azure will use a port forwarding technology to forward messages received from a client to the listening service implementation which is usually hosted on premise. 
2. Service Bus Relay is intended to allow you to bridge from any application hosted in the cloud of on another data center to bridge through to your on-premise services.
3. An Extension of WCF to make available on the cloud and internet 
4. it is a Alternative of BizTalk Azure 
5. It is Hosted on premises but but listening of new session are delegated to Service Bus Hosted in Azure 
6. Shared Access Signature is used for authentication 
7. Only Support WCF based Services and relay Binding 

Service Bus relay VS BizTalk Hybrid Connection

1. Service Bus Relay only supports a WCF based listening service which uses the WCF relay bindings where as Hybrid Connections uses its on premise listening agent allowing it to connect to a target via any port which can include a WCF or REST service bus also many other applications such as a database
2. Service Bus Relay supports ACS security where as Hybrid Connections support SAS for the cloud hosted endpoint. (Note according to MSDN service bus relay is expected to support SAS sometime in the “near future”)
3. The Service Bus Relay endpoint only exists when the listening service comes online where as the hybrid connections endpoint is created in the cloud as a configured service before the agent is listening
4. Hybrid Connections provides an agent which can be installed on an on-premise server which can then connect to on premise assets. Service Bus Relay does not provide any software to install and you need to configure your own listening component or configure a WCF Service to use a Relay binding.
5. Service Bus Relay can only support clients which can talk web services and are able to retrieve an access token from ACS, Hybrid Connections can support interop with non .net technologies
6. Hybrid Connections can only be consumed by a resource hosted in the cloud such as an Azure Website where as Service Bus Relay can be consumed from a resource hosted outside of Azure such as from a partners data centre
7. Hybrid Connections requires the BizTalk Services feature in Azure which also gives you some more integration features but also includes a higher cost. Service Bus Relay is a smaller feature within Azure and is currently lower cost.

Read More Details Here...

Consideration for Storage Selection 

1. Storage Performance Option 
1. TBD 
2. Storage Concurrency
1.  Last Write Wins 
2. Optimistic
3. Pessimistic 

Azure Messaging Services 

1. Azure Queue 
1. Simple REST Based interface 
2. Reliable Persistent messaging 
3. Use case 
1. App needs to store 80 GB plus message with max 7 days lifespan 
2. Need to track progress of message inside queue  
3. Need server side logs of all transaction against queue 
2. Service Bus Queue 
1. it is like ESB 
2. Support Queuing 
3. Support Publish Subscribe 
4. Web service Remoting 
5. Support Integration Patterns 
6. Use case 
1. When you need pub/sub  mechanism 
2. Ordered Delivery 
3. Duplication detection 
4. Message Stream Implementation 
5. TTL more than 7 Days 
6. Message > 64 KB to 256 KB
7. role based model and access 
8.  Queue size is less then 80 GB
9. Want to use AMPQ message broker 
10. Guaranteed non duplicate ordered delivery is required 
11. required message batching 
12. Need to Publish and consume message in batches 
13. Need Full WCF stack Integration 
7. Azure Service Bus Queue Vs Azure Queue 
1. TBD
8. Queue Design Patters 
1. Competing Consumer 
2. Priority Queue 
3. Queue based Load Leveling 
4. Ref https://docs.microsoft.com/en-us/azure/architecture/patterns/
3. SQL Server Selection Consideations
1. https://docs.microsoft.com/en-us/azure/sql-database/sql-database-paas-vs-sql-server-iaas

Machine Learning (ML)

1. Data is loaded to machine learning using multiple ways 
2. Data can directly be published from application running in Azure 
3. ML Studio is IDE to interact with data
4. Result are exposed via API 
5. it can run Script in R and Python 
6. Provide rebuild algorithm to get started 
7. Market place has several machine learning solution available 
8. Predictive analysis in cloud – as a service, no VMs etc. to manage.
9. Take existing data, analyse by running predictive models and predict future outcomes/trends.
10. Deploy in minutes; drag and drop machine learning algorithms (built-in); use data in Azure; add custom scripts; Marketplace of vendors providing custom solutions.
11. Terminology:
    • Classification (group data).
    • Regression (predict a value).
    • Ranking (order items by criteria).
    • Clustering (take a set of data, e.g. by date range).
12. Get raw data (unstructured or losely structured) -> data cleaning -> build machine learning model -> predict results.
13. Data science says that you need relevant, connected, accurate and enough data in order to be able to use it. The age of the data is only a factor for some uses, while machine learning can often still make use of data if some elements are missing.

Big Data (HD Insights )

1. provide two clustors 
1. Hadoop
2. Apache Spark 
3. HDase 
4. Apache Storm
2. Microsoft Implementation of Hadoop – create clusters in minutes (Windows or Linux); pay per use (no need to leave running); use blob storage as storage layer and Excel to visualise the data.
3. Hadoop uses divide and conquer approach to solving big data problems (chunking): processes the data, then combines it again – using HDFS and MapReduce components.
4. Provision cluster, take large data set (e.g. search engine queries) on master node, distributed to processing nodes (Map). Reduce collects results and collates.
5. Hybrid Hadoop – e.g. for organisations that offer analytics services – burst to cloud…
6. Either site-to-site VPN on-premises to Azure, or ExpressRoute.
7. Supports Storm and HBase clusters natively – can install other software via custom script.
8. Connectors in WebApp (Standard and Premium) – connect to other services (e.g. Azure HDInsight).

Azure Search 

1. Custom Tuning is supported 
2. Search Keys are used to access services 
3. Primary and Secondary admin key is support 
4. Also support search query key for read only access 

Azure Media Services 

1. Interface 
1. Based in REST APIs
2. Storage 
1. Media are stored in BLOB container called Asset 
2. BLOB container is set of blobs is actually Boundry point for access control 
3. Number of Blobs per account is unlimited but max size is 500 TB per account
3. Delivery 
1. Support Storage , Trans-coding and DRM of media 
2. Only Blob can be used for on demand Video but for live streaming table cubes can be used. 
3. Content Server is responsible for pulling resource from storage and delivery to client 
4. Streaming 
1. Support direct encoding Smooth Streaming format 
2. Also support first encoding to MP4 than encoding Smooth Streaming format 
3. Media is stored in multiple bit rate format are converted in real time as requested by client 
4. Supported Streaming formats are Smooth Streaming, HLS and  DASH/CSF
5. Content Protection 
1. Delivery Using AES -128 or Play Ready DRM 
2. Key and Licences are managed by azure 
3. Media encrypted on the fly reducing storage and chances of key leaks 
4. Can use AD Authentication 
6. Media Indexing 
1. MAVIS Windows Azure Service can can be access via REST API
1. Perform speech recognition on video content 
2. Build vocabulary
3. Generate caption and keyboard 
4. content is index and searchable
7. Azure Media Played 
1.   free solution for azure media services playback 
2. Unified Java script interface support only Azure Media services 
3. Uses HTML by default 
4. Single Unified UI 
8. Monitoring Plan 
1. Three plans are supported , NONE, Verbose , Minimal 
2. Minimal provide aggregated data 
9. Supported Encryption 
1. None 
2. Common Encryption or Play Ready DRM
3. Envelop Encryption for HTTP live Streaming (HIL)
10. Access policy 
1. is used to permission and duration of access 
11. Locators 
1. Locators Provides entry point for accessing files an asset 
2. Locators  can have different start type and connection times using same permission and duration settings
3. There are two type of Locators 
1. On Demand Origin locators are used for streaming 
2. SAS URL are locators used to upload or download media files for streaming 
12. Job And Task 
1. Job are used to process audio and video 
2. Jobs are bind to media and Each media requires individual job to do any processing 
3. Job is combination of Task and task can be chained 
13. Channels (Streaming End Points)
1. Media service account by default provides five channels 
2. Each channel can have running Programs, Max three currently running program at any time per account.
3. program is an administrative tool to control publishing and storage of live stream segment 
4. Programs are times events on channels and managed by channels 
5. Archive Window Length - how long recorded content will be saved 
14. Streaming End Point 
1. Delivers content directly to client player, application or CDN
2. Support live stream or Video on demand 
3. Scaling in 200MPBS increments
4. default 2 Streaming end point per Media service account  
15. How to setup
1. In management portal, create new Media Service with name, storage account and region.
2. Start the Media Service.
3. Scale up streaming units (1 unit=200Mbps).
4. Upload a video file (from local or from Azure storage) – will be stored in storage account without encryption.
5. Publish the file.
6. Configure the encoding options, then video is uploaded into portal (can encode multiple times for different formats with different names).
7. View the media content (copy link into browser).

Azure Content Delivery Network 

1. Only Public BLOB with anonymous access  are cached at CDN
2. CND URL format http;//<ID>.vo.msecnd.net/<path>
3. Suggested to use only for Static content are supported with Azure CDN
4. Cloud Service must deliver content via port HTTP:80  for CDN caching with AZURE
5. If CND need to be used with HTTPS then
1. BYO certificate not supported , CDN only Certificated are supported.
2. Must use CDN domain name not custom domain name 
3. Even if HTTP is on HTTP is still supported.
6. Caching public objects from a storage account at point of presence (POP) for faster access close to users (and to scale when a lot of traffic hits).
7. Content served from local edge location. If content not there (first serve), it fetches information from the origin and caches locally.
8. Drastic reduction in traffic on original content (so faster access and more scalable!)
   Use a CDN for lower latency, higher throughput, improved performance!
9. POP locations separate to Azure regions – not full-fledged DCs.
10. CDN origin can be Azure Storage, Apps, Cloud Services or Media Services (including live streaming) – or a custom origin on any web server.
11. CDN Edge is a cache – not a permanent store.
12. Anycast protocol is used to route user to closest endpoint.
13. Create a CDN endpoint: http://cdnname.azureedge.net/
14. Change website code to point to the CDN. Route dynamic content to origin, static to CDN.
15. Can set a custom domain too (e.g. cdn.domain.com) – avoid browser warnings about content from other domains.
16. Can also enable HTTPS – need to upload the SSL certificate.
17. Default cache is 72 hours – cache control header can be used to control (any value >300s). Use to ensure not serving stale content.
18. Use CDN to cache images, scripts, CSS from Azure Cloud Service but have to provide using HTTP on port 80.
19. Pricing based on bandwidth (between edge and origin) and requests.

Azure App Service, 

1. App Service is fully managed deployment platform support multiple languages and platfroms 
2. App Service Environment (ASE)?
1. An App Service Environment is a Premium service plan option of Azure App Service that provides a fully isolated and dedicated environment for securely running Azure App Service apps at high scale, including Web Apps, Mobile Apps, and API Apps.
2. App Service Environments are ideal for application workloads requiring: 
3. Very high scale 
4. Isolation and secure network access 
5. Customers can create multiple App Service Environments within a single Azure region, as well as across multiple Azure regions. This makes App Service Environments ideal for horizontally scaling state-less application tiers in support of high RPS workloads.
6. https://docs.microsoft.com/en-us/azure/app-service/environment/app-service-app-service-environment-intro

API Management, 

TBD..........................

Azure Cache,

TBD..........................

Service Bus, 

TBD..........................

Event Hubs,

TBD..........................

Stream Analytics, and IoT Hub; 

TBD..........................