NAGESH D N

Python Vaccine API Implementation

2021-05-07T00:00:00+05:30

Currently Govt has open sourced covid vaccine availability API’s in India as given here, Vaccine slots are limited and difficult to know when those slots gets opened. Here is simple bot created to monitor the vaccine slots and notify interested users when those slots are available.

#!/usr/bin/env python3
import json,urllib3,datetime,time,logging,telegram,configparser

config = configparser.ConfigParser()
config.read_file(open('api.conf'))
weburl = config.get('cowin', 'url')
TOKEN = config.get('telegram', 'token')
CHAT_ID = config.get('telegram', 'chat_id')
x=datetime.datetime.now()

day=x.strftime("%d-%m-%Y")
weburl+=day
http = urllib3.PoolManager()
logging.basicConfig(filename='./vaccine.log',level=logging.INFO,format='%(asctime)s %(message)s')
bot = telegram.Bot(token=TOKEN)

while True:
    found=0
    try:
        response = http.request('GET', weburl)
        json_data = json.loads(response.data)
    except Exception as ex:
        logging.info("API is not responding!")
        time.sleep(10)
        continue

    for x in json_data['centers']:
        for y in x['sessions']:
            if y['min_age_limit'] == 18 and y['available_capacity'] > 0:
                #print(y)
                message = '18+ vaccine available in #BBMP: '+ str(x['pincode']) +' '+ str(y['date'])+ ' ' + x['name'] +' Avail slots: '+ str(y['available_capacity'])
                logging.info(message)
                bot.sendMessage(chat_id = CHAT_ID, text =message)
                found=1
                break
            else:
                continue
            break
    if (found == 0):
        time.sleep(4)
    else:
        continue

API is enforcing 100 calls/5 mins rate limit per IP, which translates to 1 call/3 seconds,hence after every iteration, the bot sleeps for 4 seconds Incase slots are open ( found=1), need to validate and alert without this 4 seconds delay, hence utilizes double break points and skips the 4 seconds delay with found=1 flag

Example snapshot of the telegram alerts is as follows:

Systemd timers for cron activities

2020-12-21T00:00:00+05:30

Cron utility in Linux has been helpful to schedule tasks to run at a scheduled time. A few of the disadvantages of cron are

If Instance/server is not available/down at the cron schedule, cron doesn’t run those tasks until the next event is triggered
If the cron task need to run other tasks after running the main task at the scheduled time, there are only hacks to create this dependency
If the cron task needs to run within a given time interval to avoid running at the same time
If a cron task needs to run after an event, such as 3 minutes after the system restarted. Cron doesn’t support this functionality
To check the status of the last cron task to see if it ran/failed with logs

Most of the above issues require an alternative to cron. the above issues are fixed using system timer in Linux as follows:

Example Systemd Timer unit file

# test.timer
[Unit] 
Description=Runs the test.service 5 minutes after boot-up & at/around 6 PM every weekday.
[Timer] 
OnCalendar=Mon..Fri 18:00
## supports multiple time formats
Persistent=True
##If the machine is powered down and missed Oncalendar event, Persistent=true triggers if time has elapsed.
OnBootSec=5 m 
## Event-Based, triggers the event 5 minutes after the machine is rebooted
RandomizedDelaySec=1800
## Randomly add up to 30 mins to the schedule. This is to ensure the task runs between 6 PM to 6.30 PM.

Unit=test.service 
## service/task which needs to be run when the event/time occurs
[Install] 
WantedBy=basic.target

Example Systemd service unit file # test.service [Unit] Description= test.service that needs to be run

[Service]
ExecStartPre=<some executable>
ExecStart=<some executable>
ExecStartPost=<some executable>
## ExectStart runs only if ExecStartPre is successful and ExecStartPost runs only if ExecStart is successful
[Install] 
WantedBy=basic.target

To list all the system timers,last/next schedule

ubuntu@ip-11-223-3-40:/tmp$ systemctl list-timers --all
NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES                     
Tue 2020-12-22 00:00:00 IST 1h 19min left Mon 2020-12-21 16:06:50 IST 6h ago       logrotate.timer              logrotate.service             
Tue 2020-12-22 00:00:00 IST 1h 19min left Mon 2020-12-21 16:09:06 IST 6h ago       man-db.timer                 man-db.service                
Tue 2020-12-22 01:02:49 IST 2h 22min left Mon 2020-12-21 15:04:47 IST 7h ago       motd-news.timer              motd-news.service             
Tue 2020-12-22 03:01:02 IST 4h 20min left Mon 2020-12-21 15:05:43 IST 7h ago       certbot.timer                certbot.service               
Tue 2020-12-22 04:58:50 IST 6h left       Mon 2020-12-21 14:54:08 IST 7h ago       apt-daily.timer              apt-daily.service             

To Check the syntax of calendar event

ubuntu@ip-11-223-3-40:~/tmp$ systemd-analyze calendar "Mon..Fri 06:01"
  Original form: Mon..Fri 06:01             
Normalized form: Mon..Fri *-*-* 06:01:00    
    Next elapse: Tue 2020-12-22 06:01:00 IST
       (in UTC): Tue 2020-12-22 00:31:00 UTC
       From now: 7h left    

Performance Optimization of http endpoints

2020-04-01T00:00:00+05:30

http endpoints running on a small instances which needs to be optimized for peformance. can be done using 2 ways

“Throwing resource to the problem” - easier approach,increase size of the backend instance, this increases cost.
“Optimize Protocols” - to improve/reduce the traffic reaching small instance, similar or marginal cost increase.

Some opportunities to optimize protocols in nginx:

http/2 instead of http/1.1

http/2 provides header compression,multiplexing and server push features, since http request is foundational, any small improvement will have huge impact on performance and costs.

enabling http/2 protocol in nginx can be done as
```
 listen 443 ssl http2;
```
TLS1.3 and TLS1.2

Transport Layer Security(TLS) is successor of Secure Sockets Layer(SSL), TLS 1.2 requires 2 round trips for handshake, TLS 1.3 completes handshake with 1 round trip, this reduces encryption latency by 50% and thus improves performance.
TLS 1.3 also offers 0-RTT (zero Round trip time) Resumption, which helps clients which are previously connected to server with zero-handshake and thus improves performance, this feature is vulnerable to Replay attacks. Not all clients supports TLS1.3, hence better to enable both TLS 1.3 and TLS 1.2

enabling TLS V1.3 and 0-RTT in nginx can be done as
```
 ssl_protocols TLSv1.3 TLSv1.2;
 ssl_early_data on;
```
http/3 and http/2

http/3 is evovled version of http/2 and upcoming standard for http, http/3 is based on QUIC (Quick UDP Internet Connections) which offers much improved handshake than TCP+TLS 1.3 combined and other performance improvements.

support to enable this protocol in nginx is limited , and expected to be added soon

S3 website public access prevention

2020-02-20T00:00:00+05:30

S3 public websites have limitation to access only using http and not https, this can be addressed using cloudfront(http/https) as frontend of website and S3 public website as backend(http). However both frontend and backend can be accessed separately and can be considered as duplicates.

To ensure access to S3 public website only from cloudfront, cloudfront can pass origin header as Referer which S3 would whitelist using bucket policy as given below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::swbucketname/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": "secretfors3"
                }
            }
        }
    ]
}

this effectively restricts all public access limited to cloudfront,which caches frequent data and improves speed.

Serverless Enterprise Logging issues

2020-02-15T00:00:00+05:30

While developing and deploying Function as a service(Faas) applications leveraging AWS Lambda ,API Gateways etc. Serverless frameworks promised to keep development and deployments simple and enabling public cloud of choice.

Serverless Enterprise/Pro Edition which offers monitoring,alerts etc to serverless applications resulted in huge logging for each function invocations e.g.

SERVERLESS_ENTERPRISE 
{
    "type": "transaction",
    "origin": "sls-agent",
    "payload": {
        "duration": 576.2143135070801,
        "endTime": "2020-02-15T05:18:46.646536Z",
        "logs": {},
        "operationName": "s-transaction-function",
        "schemaType": "s-span",
        "schemaVersion": "0.0",
        "spanContext": {
            "spanId": "1c28085b-efd5-4591-865a-e38ef0bda0e7",
            "traceId": "0874e9f5-ed95-4e3c-aa37-d16a54f3602c",
            "xTraceId": "Root=1-5e477f35-53f776209c7518c6e40ea991;Parent=5412c6a632e94e9d;Sampled=0"
        },
        .... tens of such lines for each function invocation

There are no documented ways to disable serverless enterprise logging and reporting, A simple Programming mistake which triggered hundreds of function calls to Lambda resulted in >80 GB Cloud watch logs, and more than 2GB data transfer out to serverless.com, which exceeded AWS budgets by many folds.

Steps to Control serverless logging and bill shocks in AWS :

Simple step to disable serverless enterprise logging is - logout from serverless using cli sls logout and comment app and org labels in servererless.yml
Disable unwanted logging/warnings generated by functions, Serverless Enterprise framework with Python3.8 caused multiple warnings, migrating it back to python3.6 resolved it and saved logging.
Implement Guard Rails for number of AWS lambda invocations using ReservedConcurrency to reasonable limits (default account concurrency limit which allows upto 1000 lamnba functions to run in parallel)
Implement Rate limiter for AWS API gateway end points to be within reasonable/expected limits.

Stop Cloud9 IDE immediately to save costs

2020-02-07T00:00:00+05:30

AWS cloud9 IDE simplifed access and development in Cloud by using cloud native IDE and associated terminal for CLI access. To reduce wastage of unused EC2 instances which are the backend of Cloud9, Cloud9 IDE detects presence of user based on activity and shutdowns EC2 instances using predetermined interval.

The minimum time to configure inactivity is 30 minutes, which means when user is not using cloud9 upto 30 mins, Cloud9 shutsdown EC2 instance.

To enable immediate shutdown of cloud9 environment on determinstic use cases, user can accomplish this by following process.

1.write lambda function which stops the requested EC2 instance ID. simple code for this is..

import boto3
region = 'us-east-1'
ec2 = boto3.client('ec2', region_name=region)
def stopEC2(event, context):
    print("Received Instance ID:"+event['queryStringParameters']['instance_id'])
    if(
            event['httpMethod'] == 'GET' and
            event['path'] == "/someconfidentialpath/ec2stop" and
            event['queryStringParameters']['instance_id'] != ''
            ):
        instance_id=event['queryStringParameters']['instance_id']
        ec2.stop_instances(InstanceIds=[instance_id])
        print('stopped your instances: ' + instance_id)

Deploy the above lambda function with IAM role which has permission to Stop EC2 instances.
Configure AWS API Gateway which proxies the request to above given lambda function.
Configure a http bookmark for API Gateway with querystring parameters as instance ID. e.g: https://api.nageshdn.com/someconfidentialpath/ec2stop?instance_id=i-012345678923

Whenever user is leaving the desk, user just need to click the above http bookmark, which will trigger lambda function to shutdown EC2 instances and saves costs upto 30 mins every time.

Rolling back github private repo

2019-04-29T00:00:00+05:30

As detailed in the earlier post,hosted this website in github private repo with github pages until today.

Last week,Received following notification about github pages not supported in private-free accounts

Hence rolling back github private repo with github pages to public repo to continue to host this blog.

Github Private repo for pages

2019-01-13T00:00:00+05:30

This website uses github as repository/editor for markdown and uses AWS for distribution using S3/Cloudfront. Github repo has to be public/open if it is not subscribed to paid plans. The other alternative to have private repo is bitbucket or AWS codecommit, which are free, but not as easy/popular compared to github.

Earlier this month github announced unlimited free private repository with collobarators limited upto 3. Hence moved this repository to private, However this breaks github pages features, which works only if it is public or pro account. Since this site is using github pages only as failover (when AWS S3 is down).

Github Private repo for hosting pages

2019-01-13T00:00:00+05:30

This website uses github as repository/editor for markdown and AWS for distribution using S3/Cloudfront. Github repo has to be public/open if it is not subscribed to paid plans. The other alternative to have private repo is bitbucket or AWS codecommit.

Earlier this month github announced unlimited free private repository with collobarators limited upto 3 users. Hence moved this repository to private, However this breaks github pages features, which works only if it is public or pro account. Since this site is using github pages only as failover (when AWS S3 is down).

Need to explore alternatives for backup/github pages for private repo.

S3 Failover to Github

2018-12-20T00:00:00+05:30

AWS S3 storage and S3 websites are highest available services in terms of uptime, as they are backed by AWS SLA of 99.99% availability in a given year. AWS health status also leverages S3, however there are chances still S3 can go down, here is report, where AWS S3 was down for few hours and AWS also had no access to update the health status.

With this context, is there a way to automatically failover when highly reliably AWS S3 fails and still run your services? since this site relies on github as a repo/editor, few months ago, github also started to support https on custom domain names , Hence github pages is right alternative for AWS S3 for static sites. So the change required is to add github pages as failover whenever underlying S3 fails or configuration/deployment failure to S3.

github pages are automatically available at username.github.io , need to update the Route 53 DNS record from existing simple/Alias record to Failover(Primary)/CNAME. github does not supports cname at apex domain(example.com), Hence need to resolve this using Alias record which redirects example.com to www.example.com ( setting redirect at S3 bucket).

Route 53 health check is configured to check the status of website using index.html page over port 80 at S3 public endpoint,this would be free of charge as it is using AWS endpoint, default health check would check every 30 seconds from more than 10 s3 regions, this healthchecks would generate around 1,000,000 get request against the S3 bucket, made changes to reduce this to minimum required 3 regions to reduce this flooding of the requests.

Created second record as Failover(Secondary)/CNAME record with target as username.github.io without any healthcheck,to simulate the failover,changed S3 index.html object to be unavailable, here is how DNS health check finds it and DNS failover works when this occurs.

DNS healthcheck Notification over email
DNS response on failure (points to username.github.io)

On failover to secondary(github pages), website worked with fully functionalty including support for https for both apex and subdomain as required. Main advantage with this change is automatic standby/failover and cost optimizations if the primary website using AWS becomes costly to manage (CDN/S3 charges), can easily make Secondary as Primary by just modifying an object in S3/Route 53.